JP7588679B2 - Audio signal processing device, program, and audio signal processing method - Google Patents

Description

The present invention relates to real-time, noise reduction, sliding window, neural networks, and data streams.

The following documents relate to the present invention:

Santiago Pascual, Antonio Bonafonte, and Joan Serra, "Segan: Speech enhancement generative adversarial network," arXiv preprint arXiv:1703.09452, 2017. Francois G. German, Qifeng Chen, and Vladlen Koltun, "Speech denoising with deep feature losses," arXiv preprint arXiv:1806.10522, 2018. Craig Macartney and Tillman Weyde, "Improved speech enhancement with the wave-u-net," arXiv preprint arXiv:1811.11307, 2018. Szu-Wei Fu, Chien-Feng Liao, Yu Tsao, and Shou-De Lin, "Metricgan: Generative adversarial networks based black-box metric scores optimization for speech enhancement," in International Conference on Machine Learning. PMLR, 2019, pp. 2031-2041. Ruilin Xu, Rundi Wu, Yuko Ishiwaka, Carl Vondrick, and Changxi Zheng, "Listening to sounds of silence for speech" denoising," in Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, Eds. 2020, vol. 33, pp. 9633-9648, Curran Associates, Inc. Alexandre Defossez, Gabriel Synnaeve, and Yossi Adi, "Real time speech enhancement in the waveform domain," arXiv preprint arXiv:2006.12847, 2020. Xiang Hao, Xiangdong Su, Radu Horaud, and Xiaofei Li, "Fullsubnet: A full-band and sub-band fusion model for real-time single-channel speech enhancement," in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 6633-6637. Tyler Vuong, Yangyang Xia, and Richard M. Stern, "A modulation-domain loss for neural-network-based realtime speech enhancement," 2021. Qiquan Zhang, Aaron Nicholson, Mingjiang Wang, Kuldip K Paliwal, and Chenxu Wang, "Deepmmse: A deep learning approach to mmse-based noise power spectral density estimation," IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 1404-1415, 2020. Hyeong-Seok Choi, Sungjin Park, Jie Hwan Lee, Hoon Heo, Dongsuk Jeon, and Kyogu Lee, "Real-time denoising and dereverberation with tiny recurrent u-net," arXiv preprint arXiv:2102.03207, 2021. Sepp Hochreiter and Jurgen Schmidhuber, "Long shortterm memory," Neural computation, vol. 9, no. 8, pp. 1735-1780, 1997. Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T. Freeman, and Michael Rubinstein, “Looking to listen at the cocktail party: A speaker-independent audio-visual model. for speech separation," ACM Transactions on Graphics, vol. 37, no. 4, pp. 1-11, July 2018. Y. Wang, A. Narayanan, and D. Wang, "On training targets for supervised speech separation," IEEE/ACM Transactions on Audio, Speech, and Language. Processing, vol. 22, no. 12, pp. 1849-1858, 2014. Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter, “Audio set: An ontology and human-labeled dataset for audio events," in Proc. IEEE ICASSP 2017, New Orleans, LA, 2017. Joachim Thiemann, Nobutaka Ito, and Emmanuel Vincent, “The diverse environments multi-channel acoustic noise database (demand): A database of multichannel environmental noise recordings, "in 21st International Congress on Acoustics, Montreal, Canada, June 2013, Acoustical Society of America, The dataset itsself is archived on Zenodo, with DOI 10.5281/zenodo. 1227120. Cassia Valentini-Botinhao, Xin Wang, Shinji Takaki, and Junichi Yamagishi, “Investigating rnn-based speech enhancement methods for noise-robust text-to-speech," in 9th ISCA Speech Synthesis Workshop, 2016, pp. 146-152. Cees H Taal, Richard C Hendriks, Richard Heusdens, and Jesper Jensen, “An algorithm for intelligence prediction of time-frequency weighted noise speech," IEEE Transactions on Audio, Speech, and Language Processing, vol. 19, no. 7, pp. 2125-2136, 2011. ITU-T Recommendation, "Perceptual evaluation of speech quality (pesq): An objective method for end-toend speech Quality assessment of narrow-band telephone networks and speech codes," Rec. ITU-T P. 862, 2001. Yi Hu and Philippos C. Loizou, "Evaluation of objective quality measurements for speech enhancement," IEEE Transactions on Audio, Speech, and Language Processing, vol. 16, no. 1, pp. 229-238, 2008. Mark A. Clements Schuyler R. Quackenbush, Thomas P. Barnwell, Objective Measures Of Speech Quality, Prentice Hall, Englewood Cliffs, NJ, 1988.

These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter.

13 illustrates a schematic of dynamic versus fixed magnitude sliding windows. 1 shows a schematic diagram of data padding. 1 shows a schematic comparison of real-time performance. 1 illustrates a schematic diagram of a network structure. 1 illustrates a schematic representation of denoised audio. An example of the hardware configuration of a computer 1200 functioning as the data processing device 100 is illustrated in schematic form.

1. Introduction Real-time speech denoising is a highly demanding audio processing task - perhaps the most demanding ever - as our world remains darkened by the coronavirus disease (COVID) pandemic and online meetings have become the "new normal" of our daily social life. Especially recently, all prior art real-time denoising techniques are based on neural networks [1, 2, 3, 4, 5, 6, 7, 8]. They explore novel network structures to achieve reasonable denoising quality while retaining the simplicity of the network to reduce the processing time.

Unlike offline audio denoising, in a real-time setting the audio signal is provided in a streaming manner. The network must process signal samples as they arrive and generate output samples with as short a delay as possible. As a result, a common strategy in nearly all real-time techniques is to use a fixed-length sliding window buffer.

This seems like a natural choice: the network waits until the sliding window buffer is filled with incoming audio samples and then denoises the data in that buffer. The resulting signal data is later fed to a real-time audio player. In this way, the network expects an input signal of fixed length L. As long as the processing time of the network is always less than L, the overall delay from the arrival of a signal sample to the play-out time of the corresponding denoised sample is bounded, greater than L and less than 2L (see the analysis in Section 2.1).

However, in practice, it is difficult, if not impossible, to choose a buffer length L that ensures real-time performance. A large L will cause a long delay; a small L may result in a network processing time longer than L, resulting in choppy audio playback. This is because in a real computing environment, there are always other background processes (e.g., 4K video playback and gaming). The shorter L is, the more susceptible the denoising network is to the CPU occupancy of other processes. In short, the sliding window strategy, although fundamental for real-time denoising, remains elusive to careful investigation.

We propose a different sliding window strategy, namely dynamic sliding window. In our approach, the input buffer length is not fixed. The network takes the currently buffered data regardless of its length and starts processing it without waiting. While the network is running, newly received data is accumulated in the buffer and is ready to be processed when the network finishes the current round of denoising. This sliding window strategy is conceptually simple, but is more robust to the CPU occupancy of other processes, thereby resulting in shorter and more stable delays. To show its advantages, we publicly analyze the audio playback delays caused by our approach and a commonly used fixed-size sliding window. To the best of our knowledge, this is the first time that network delays under different sliding window strategies are investigated.

Many existing real-time denoising networks (e.g., [6, 7, 8]) cannot easily integrate dynamic sliding windows (see the discussion in Section 2.2). Therefore, we propose a lightweight denoising network adapted for the real-time setting: accepting streaming signals and processing them using dynamic sliding windows. By carefully padding and reusing data across the sliding window, our network does not suffer from loss in denoising quality compared to the offline non-streaming case.

We have performed extensive experiments comparing our proposed model with several prior art real-time denoising methods. The results show that our proposed model obtains comparable denoising quality in terms of all quality metrics, but with the lowest playback delay among all compared methods. Most importantly, compared to prior real-time denoising methods [6], our model is more robust in maintaining real-time performance in real-world scenarios where other background tasks such as Zoom meetings, 4K video editing, and video games may preempt CPU cycles.

2. Methodology We start by analyzing the delay in audio playback when a fixed-size sliding window is used, which is compared to the delay when using our proposed dynamic sliding window (Section 2.1). Motivated by our analysis, we then propose a lightweight denoising network that exploits the dynamic sliding window for faster and more robust denoising (Section 2.2).

Figure 1 shows a schematic of dynamic vs. fixed size sliding windows. Different windows are indicated by different hatching. Each window is processed by the network individually. The grey areas indicate periods where CPU cycles are occupied by other processes, thus increasing the network processing time.

2.1 Sliding Windows for Data Streams Fixed-Size Sliding Windows In almost all existing network-based denoising models, the incoming audio signal is treated as contiguous non-overlapping windows [X ₁ , X ₂ , ...] in time. Each window hosts a constant length L of audio samples (i.e., X _i ∈ R ^L ) that are filled in a streaming manner. The network F takes the latest unprocessed window X _i and outputs the denoised result F(X _i ), and then waits until the next window X _{i + 1} is filled (see Fig. 1(a)). Let t _k denote the processing time of the network for window X _k . Note that although the window size is fixed, in practice t _k varies over time due to the CPU occupancy of other background processes. Our analysis shows that the delay (or lag) d _i from the moment X _i is received to the moment the network outputs F(X _i ) can be expressed as follows:

The derivation of equation (1) is non-trivial. Due to limited space here, we skip the details of the derivation, but we have made it publicly available ^online.1

Our analysis (1) reveals that in the ideal case where t _i < L is always satisfied, the denoising network would run smoothly in real time without any accumulated delay; the playback delay would be upper bounded by 2L. However, in reality, the execution of the network is often affected by other background computational processes, and its processing time t _i can be longer than L. At the same time, audio samples of length t _i arrive and accumulate in the buffer. To process this amount of data, the network must:
times, which in turn can lead to an accumulation of delay in the audio playback (hence the summation term in (1)).

Dynamic Sliding Window. We propose to dynamically adjust the size of the sliding window. As soon as the network finishes processing a data window X _i , the newly received data in the buffer has a length t _i , which may or may not be greater than L. Regardless of the length of the buffer, we denoise the available buffered data without waiting. Figure 1b illustrates the process. With this strategy, the delay d _i for playing a window X _i is:

When i=1, the delay of the first window is _d1 = _L0 + _t1 , where _L0 is the initial window size to begin the denoising process at the beginning. We again refer the reader to our online literature for the derivation of equation (2).

The analysis shows that d _i depends only on the network processing time of two successive windows X _k−1 and X _k . In contrast to a fixed-size sliding window (shown in (1)), there is no accumulating delay. Therefore, our approach is more robust to variations in computing power. This is a notable advantage, since in a real computing environment, the computing power of denoising varies constantly (see Section 3.3).

2.2 Network Structure and Data Padding Although the dynamic sliding window strategy is independent of a specific network structure, many existing real-time denoising networks [6, 7, 8, 9, 10] cannot be easily adapted to take advantage of it. Some of them require a given sliding window size before the network runs [6]. Others have focused on reducing the inference cost of the network, but it is still unclear how they handle the incoming data stream [7, 8, 9, 10].

Proposed network structure. We propose a lightweight denoising network that uses a dynamic sliding window. Our network builds on the denoising components of [5] (and is therefore much simpler than the denoising model therein). The input to our network is a spectrogram s _x obtained by applying STFT to a data window X _i . The spectrogram s _x is first processed by a 2D convolutional layer with kernel size (5,5) and dilation (1,1) in the time-frequency domain. The resulting feature map is fed into a hidden unidirectional LSTM [11] of size 400. Finally, three fully connected layers of size (400, 600, 512) are applied for each time bin. Similar to other speech enhancement models [5, 12, 13], our network outputs a complex-valued mask c of the same dimension as s _x . Finally, the denoised audio signal is subjected to an inverse STFT to obtain a
is obtained by applying
denotes the Hadamard product.

During training, we optimize the subsequent loss function:
In the formula
shows the ground truth spectrogram of clean audio. When computing the STFT, we set the number of FFT bins to 510, the Hanning window size to 400, and the hop length to 128.

Figure 2 shows a schematic illustration of data padding: two consecutive windows of size 8 and 6 are processed by two 1D convolutions with kernel size 3. In this case, we pad two future elements for each window (22, 23 for window X _i ) and reuse two elements of the first convolution result from the previous window (15, 16 taken from processing window X _i ).

Data padding. Our network (and many others) has convolutional layers, which require padding to process boundary data. To handle streaming data in a sliding window manner, this means that we need to buffer enough “future” data when processing the data window (e.g., blocks 16 and 17 for window X _i−1 in FIG. 2). Waiting for the arrival of padding data introduces an additional delay (48 ms in our implementation). However, we can reuse the result of the convolution with the padding data of the next sliding window (see the diagram in FIG. 2). Apart from saving some computational cost, reusing the padding data is important for the denoising quality of the network. It ensures that our network maintains the same denoising quality as if it were suddenly taking in the entire signal. Perhaps surprisingly, such a guarantee is still lacking in existing real-time denoising networks (see experiments in Section 3.2).

3. Experiments Our experiments are two-fold: we evaluate the denoising quality of our network by comparing it with prior art real-time denoising models (Section 3.2). We then demonstrate the performance advantage of a dynamic sliding window over a traditional fixed-size sliding window approach (Section 3.3).

3.1 Experimental Setup Datasets. We perform experiments on two publicly available datasets. The first one, provided by Xu et al. [5], has clean audio selected from AVSPEECH [12] and noise from AudioSet [14] and DEMAND [15]. We call this dataset the ADD dataset. In addition, we also validate the benchmark of Valentini [16], which contains audio clips from 28 speakers; each clip has corresponding clean and noise versions.

Evaluation metric. To evaluate the quality of noise reduction, we use the following widely used objective metrics: (i) STOI: Short-Time Objective Intelligibility [17]; (ii) PESQ: Perceptual evaluation of speech quality (we use the narrowband version) [18]; (iii) CSIG: MOS predictor of code distortion [19]; (iv) CBAK: MOS predictor of background noise aggressiveness [19]; (v) COVL: MOS predictor of overall quality [19]; (vi) SSNR: Segmental Signal-to-Noise Ratio [20].

We use two metrics to evaluate the real-time performance of a network: the average network processing time (D _N ) and the maximum audio playout delay (D _A ).
_D.N.
where M is the total number of examples in the sliding window and t _i is the processing time of window X _i . When an audio sample arrives, D _A measures the maximum delay between the arrival time of the audio sample and its playout time (after cleanup). Unless otherwise stated, the denoising network is run on a CPU (3.60 GHz Inter 8-Core i7-9700K) and the metrics are measured in milliseconds.

[Table 1]
Denoising quality of the AAD dataset in both offline (top) and real-time (bottom) settings. All scores are the average result of the input audio at SNRs [-10, -7, -3, 0, 3, 7, 10]. The runners-up are underlined.

3.2. Real-time Audio Denoising Quality We compared our denoising model with several recently proposed real-time denoising networks, including Demucs48 [6], FullSub [7], and RNN-Mod [8]. We train these models on the same aforementioned dataset to evaluate the denoising quality in both real-time and offline settings. In the offline setting, the audio signal is provided immediately, so no sliding window is required. By comparing the denoising quality of the two settings from the same network, we hope to understand to what extent the sliding window strategy affects the denoising quality.

For Demucs48, we use the sliding window implementation provided. FullSub and RNN-Mod do not provide a streaming implementation, and we found that their denoising quality becomes unstable when a dynamic sliding window is added. Therefore, we adopt for them a fixed size sliding window of size 80 ms with 16 ms padding at the past and future ends. We choose this sliding window setting because our experiments have shown that it provides the best possible real-time denoising quality while maintaining acceptable latency.

Table 1 summarizes the results of the evaluation on the ADD dataset. Our model has the best or comparable real-time denoising quality for all quality metrics. It is also worth noting that our model was the only one that ensured the same real-time denoising quality as its offline counterpart thanks to the data padding strategy. All other models suffer from a degradation in quality when switching from the offline to the real-time setting. Furthermore, the real-time denoising quality results for the Valentini benchmark are reported in Table 2.

3.3. Real-time performance Controlled experiments. First, we measured the D _N and D _A of different network models (see Table 3). These models take input data of different lengths, and we also measure the network execution time to process a 200 ms signal at once, without splitting it into multiple windows. All measurements were done without extensive background processes. The results show that in a dedicated computing environment, our network is as fast as the prior art models.

[Table 2]
Valentini noise reduction quality.

[Table 3]
Timing Comparison: In addition to D _N and D _A , we also report the cost of network inference on 200 ms of audio (S _N ), where the numbers include the mean and standard deviation of the timing.

Next, we conduct controlled experiments to understand the real-time performance of dynamic sliding windows and fixed-size sliding windows in the presence of CPU resource variations. To that end, we create two denoising models: both use the same denoising network trained on the AAD dataset (Section 2.2). The first one uses a dynamic sliding window (referred to as D-model), while the second one uses a fixed-size sliding window (referred to as F-model). To make a fair comparison, we set the initial window size L ₀ of the D-model to be the same as the fixed window size of the F-model (L ₀ =16 ms). We use the two models to denoise the same set of audio, each of which has a duration of 5 seconds. To improve the computational power variations, after 2 seconds we intentionally delay the network processing by a factor s, where s is randomly selected from [1, 7]. This simulates CPU occupancy by background processes in a controlled way as the inventors ensure that the same length of delay is added to both the D and F models.

FIG. 3 shows D _N and D _A measured as the audio stream arrives over time. At the beginning, both can run smoothly in real time (with playback lag N _A <100 ms). After 2 seconds, the computational power starts to fluctuate, preventing some sliding windows from being processed in time. As a result, the playback lag D _A of the F model accumulates and the output audio playback becomes choppy. In contrast, D _A of the D model remains stable because it can dynamically increase the window size to catch up with the delay. This experiment confirms the inventor's theoretical analysis in equations (1) and (2).

Figure 3 shows a schematic comparison of real-time performance between dynamic and fixed sliding window approaches. We artificially randomly increase the network processing time by 1-7 times after 2 seconds to simulate variations in CPU power. The curves are averaged over 100 trials with a fixed random seed.

Real-world experiments. We next examine the real-time performance of our model in real-world scenarios, where background processes can preempt CPU cycles. Here, we denoise the same set of audio while different software runs in the background, including 4K video playback, Zoom meetings, iMovie video editing, and the video game Apex. To run the Apex game, we use a Windows 10 PC with an 8-core Intel CPU (3.60GHz i7-9700K) and GPU (NVIDIA GeForce RTX 2070 SUPER); other software tests are done on a MacBook Pro (2.3GHz Inter Quad-Core i5). We choose these software because they require more CPU cycles.

[Table 4]
Demucs 48 and our real-time noise reduction lag when other background software is running. Each cell shows D _A (D _N ) with both the mean and standard deviation. Numbers were measured using 20 seconds of audio.

We measure D _N and D _A using our model and Demucs 48, respectively, running these software individually. We compare with Demucs 48, as it has its own streaming implementation and provides comparable denoising quality to ours. The results are reported in Table 4. When background processes become quite computationally intensive (e.g., iMovie and Apex), the playback lag of Demucs 48 increases dramatically, whereas the lag of our model remains mild and stable. This is clear evidence of the performance advantage of our model and dynamic sliding window strategy.

4. Conclusion We have proposed a dynamic sliding window strategy for real-time speech denoising. Through careful analysis and experiments, we have demonstrated its advantages over the widely used fixed-size sliding window strategy. Our denoising network achieves real-time denoising quality comparable to SOTA while retaining a short lag by utilizing a dynamic sliding window. Notably, it enables our model to perform robustly in real-world scenarios in the presence of other background tasks.

6 shows an example of a hardware configuration of a computer 1200 functioning as a data processing device 100, a CEP device 200, a batch processing device 300, or a selection device 400. A program installed on the computer 1200 can cause the computer 1200 to function as one or more "units" of the device according to the present embodiment, or to perform operations associated with the device or to execute one or more "units" thereof according to the present embodiment, and/or to perform a process or execute a step of a process according to the present embodiment. Such a program can be executed by causing the CPU 1212 to cause the computer 1200 to perform specific operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

The computer 1200 according to this embodiment includes a CPU 1212, a RAM 1214, and a graphics controller 1216, which are connected to each other via a host controller 1210. The computer 1200 also includes input/output units such as a communication interface 1222, a storage device 1224, a DVD drive, and an IC card drive, which are connected to the host controller 1210 via an input/output controller 1220. The DVD drive may be a DVD-ROM drive, a DVD-RAM drive, or the like. The storage device 1224 may be a hard disk drive, a solid state drive, or the like. The computer 1200 also includes legacy input/output units such as a ROM 1230 and a keyboard, which are connected to the input/output controller 1220 via an input/output chip 1240.

The CPU 1212 operates according to the programs stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphics controller 1216 acquires image data issued by the CPU 1212, which is in the RAM 1214 or a frame buffer provided in the graphics controller 1216 itself, and causes the image data to be displayed on the display device 1218.

The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive reads programs or data from a DVD-ROM or the like and provides the programs or data to the storage device 1224. The IC card drive reads programs and data from an IC card and/or writes programs and data to the IC card.

ROM 1230 stores therein a boot program and the like executed by computer 1200 during operation, and/or programs that depend on the hardware of computer 1200. I/O chip 1240 can also connect various I/O units to I/O controller 1220 via USB ports, parallel ports, serial ports, keyboard ports, mouse ports, and the like.

The program is provided by a computer-readable storage medium such as a DVD-ROM or an IC card. The program is read from the computer-readable storage medium, installed in storage device 1224, RAM 1214, or ROM 1230, which are also examples of computer-readable storage media, and executed by CPU 1212. Information processing written in the program is read by computer 1200, resulting in cooperation between the program and the various types of hardware resources described above. The apparatus or method can be configured by implementing calculations or processing of information in accordance with the use of computer 1200.

For example, when communication is taking place between computer 1200 and an external device, CPU 1212 can execute a communication program loaded into RAM 1214 and instruct communication interface 1222 to perform communication processing based on the processing written in the communication program. Under the control of CPU 1212, communication interface 1222 reads and transmits transmission data stored in a transmission buffer area provided in RAM 1214, storage device 1224, a DVD-ROM, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes received data received from the network to a reception buffer area or the like provided in the recording medium.

The CPU 1212 may also cause all or a necessary portion of a file or database stored in an external recording medium such as the storage device 1224, a DVD drive (DVD-ROM), an IC card, etc. to be read into the RAM 1214, and perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write the processed data back to the external recording medium.

Various types of information, such as various types of programs, data, tables, databases, etc., are stored in the recording medium and processed. The CPU 1212 can perform various types of processing on the data read from the RAM 1214 to write the results back to the RAM 1214, which processing is described throughout this disclosure and specified by the sequential instructions of the program, and includes various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search/replacement, etc. The CPU 1212 may also search for information in a file, database, etc. in the recording medium. For example, when multiple entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 1212 can search for an entry whose attribute value of the first attribute meets a specified condition from the multiple entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that meets the specified condition.

The above-described programs or software modules may be stored on a computer-readable storage medium on the computer 1200 or in the vicinity of the computer 1200. In addition, a recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the programs to the computer 1200 via the network.

The blocks of the flowcharts and block diagrams of the present embodiment may represent stages of a process in which an operation is performed or a "unit" of an apparatus is responsible for performing an operation. Particular stages and "units" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and/or a processor provided with computer-readable instructions stored on a computer-readable storage medium. Dedicated circuitry may include digital and/or analog hardware circuitry, and may include integrated circuits (ICs) and/or discrete circuits. For example, programmable circuitry may include reconfigurable hardware circuitry, such as logical AND, OR, XOR, NAND, NOR, and other logic operations, flip-flops, registers, and memory elements, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), and the like.

A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by a suitable device, and as a result, a computer-readable storage medium having instructions stored thereon includes a product that includes instructions that can be executed to create a means for performing the operations specified in the flowchart or block diagram. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable storage media may include floppy (registered trademark) disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), Blu-ray (registered trademark) disks, memory sticks, integrated circuit cards, and the like.

The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk (registered trademark), JAVA (registered trademark), C++, etc., and conventional procedural programming languages such as the "C" programming language or similar programming languages.

The computer-readable instructions may be provided to a processor or programmable circuit of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, either locally or over a local area network (LAN), a wide area network (WAN) such as the Internet, so that the processor or programmable circuit of the general-purpose computer, special-purpose computer, or other programmable data processing apparatus executes the computer-readable instructions to generate means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

Although the present invention has been described with reference to the above embodiments, the technical scope of the present invention is not limited to the above embodiments. It is clear to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the claims that embodiments with such modifications or improvements can also be included in the technical scope of the present invention.

The order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, embodiments, and drawings is not specifically stated as "before" or "prior to," and it should be noted that they may be realized in any order, unless the output of a previous process is used in a later process. Even if the operational flow in the claims, embodiments, and drawings is described using "first" or "next" for convenience, it does not mean that it is essential to perform the process in this order.

Claims (3)

an acquisition unit configured to acquire an audio signal in a streaming manner; and a denoising unit configured to denoise the audio signal by using a dynamic sliding window with an input buffer of a non-fixed length ,
11. An audio signal processing apparatus, wherein the noise removal unit is configured to buffer a portion of the audio signal in the dynamic sliding window having a first size, then start removing noise from the portion of the audio signal and buffering a next portion of the audio signal, and then stop the buffering when the noise removal is completed . A program for causing a computer to function as the audio signal processing device according to claim 1 . acquiring an audio signal in a streaming manner; and denoising the audio signal acquired in the acquiring step by using a dynamic sliding window in which an input buffer length is not fixed,
11. A method for processing an audio signal, wherein the denoising step comprises buffering a portion of the audio signal in the dynamic sliding window having a first size, then commencing denoising the portion of the audio signal and buffering a next portion of the audio signal, and then stopping the buffering when the denoising is completed .