JP7775899B2 - Time window generation device, method and program - Google Patents

Description

The present invention relates to technology for processing audio signals such as voice.

A method using short-time Fourier transform is used to analyze acoustic signals in real time in terms of time frequency. In this method, a time window is used to extract a signal of a certain length and treat it as a periodic signal. The frequency resolution and dynamic range of the time window are determined according to its shape. Here, there is a trade-off between frequency resolution and dynamic range. Therefore, improving the separation performance of adjacent frequency components can result in the overlooking of small components.

Since there is no time window that is effective for all signals, it is necessary to use a time window that suits the situation appropriately.

In conventional acoustic signal processing systems, a specific window function was used in a fixed manner, or multiple pre-prepared window functions were used by switching between them (see, for example, non-patent document 1).

Yuma Koizumi, "Sound source enhancement and phase control based on deep learning," Journal of the Acoustical Society of Japan, Vol. 75, No. 3, pp. 156-163, 2019

However, until now, there has been no technology to generate appropriate time windows.

The present invention aims to provide a time window generation device, method, and program for generating an appropriate time window.

A time window generation device according to one aspect of the present invention includes a signal clipping unit that generates a clipped signal by clipping a sound signal to a predetermined length, an analysis window generation unit that generates an analysis window using the clipped signal and an analysis window model parameter, a synthesis window generation unit that generates a synthesis window using at least the clipped signal or the analysis window, a frequency domain conversion unit that converts the clipped signal into a frequency domain using the analysis window to generate a frequency domain signal, a signal processing unit that performs predetermined processing on the frequency domain signal to generate a processed frequency domain signal, a time domain conversion unit that converts the processed frequency domain signal into a time domain using the synthesis window to generate a time domain signal, and a learning unit that at least learns the analysis window model parameter using the time domain signal and ground truth data corresponding to the time domain signal. The predetermined processing is a speech signal enhancement process. The ground truth data is a noise-free original sound signal.

Appropriate time windows can be generated.

FIG. 1 is a diagram illustrating an example of the functional configuration of a time window generating device. FIG. 2 is a diagram showing an example of a processing procedure of the time window generation method. FIG. 3 is a diagram illustrating an example of the functional configuration of a computer.

The following describes in detail an embodiment of the present invention. Note that components with the same functions in the drawings are given the same numbers, and duplicate explanations will be omitted.

[Time window generation device and method]
As shown in FIG. 1, the time window generating device includes, for example, a signal extracting unit 1, an analysis window generating unit 2, a synthesis window generating unit 3, a frequency domain transforming unit 4, a signal processing unit 5, a time domain transforming unit 6, and a learning unit 7.

The time window generation method is realized, for example, by each component of the time window generation device performing the processing of steps S1 to S7 described below and shown in Figure 2.

Here, a time window refers to at least one of an analysis window and a synthesis window.

The following describes each component of the time window generation device.

<Signal extraction section 1>
A sound signal is input to the signal extractor 1 .

The signal extraction unit 1 generates an extracted signal by extracting the sound signal to a predetermined length (step S1).

The generated cut-out signal is output to the analysis window generation unit 2 and the synthesis window generation unit 3.

<Analysis window generation unit 2>
The extracted signal is input to the analysis window generator 2. The analysis window generator 2 also receives analysis window model parameters generated by the learning unit 7, which will be described later.

The analysis window generation unit 2 generates an analysis window using the extracted signal and an analysis window model determined by the analysis window model parameters (step S2). The generated analysis window is output to the frequency domain transformation unit 4.

The analysis window model parameters are, for example, analysis window model parameters generated by the learning unit 7, which will be described later. If this is the first processing by the analysis window generation unit 2 and there are no analysis window model parameters generated by the learning unit 7, the analysis window generation unit 2 uses predetermined analysis window model parameters. In this case, the analysis window generation unit 2 may generate and output a predetermined analysis window.

<Synthesis window generation unit 3>
The extracted signal is input to the synthesis window generation unit 3. The synthesis window generation unit 3 also receives synthesis window model parameters generated by the learning unit 7, which will be described later.

The synthesis window generation unit 3 generates a synthesis window using the extracted signal and a synthesis window model determined by the synthesis window model parameters (step S3). The generated synthesis window is output to the time domain transformation unit 6.

The synthesis window model parameters are, for example, synthesis window model parameters generated by the learning unit 7 described below. If this is the first processing by the synthesis window generation unit 3 and there are no synthesis window model parameters generated by the learning unit 7, the synthesis window generation unit 3 uses predetermined synthesis window model parameters. In this case, the synthesis window generation unit 3 may generate and output a predetermined synthesis window.

<Frequency domain transform unit 4>
The frequency domain transform unit 4 receives the extracted signal and the analysis window.

The frequency domain transform unit 4 generates a frequency domain signal by transforming the extracted signal into the frequency domain using an analysis window (step S4). The generated frequency domain signal is output to the signal processing unit 5.

The frequency domain transformation unit 4 performs transformation to the frequency domain using techniques such as short-time Fourier transform.

<Signal processing unit 5>
The signal processing unit 5 receives a frequency domain signal.

The signal processing unit 5 performs predetermined processing on the frequency domain signal to generate a processed frequency domain signal (step S5). The generated processed frequency domain signal is output to the time domain transform unit 6.

Examples of specified processing are at least one of processing to emphasize specified signals such as voice (in other words, processing to emphasize voice signals, processing to suppress noise), and classification processing to classify noise, etc.

When the specified processing is speech signal enhancement processing, the signal processing unit 5 estimates a speech enhancement filter, for example, using the frequency domain signal. Then, the signal processing unit 5 multiplies the estimated speech enhancement filter by the frequency domain signal to generate a frequency domain signal in which the speech is enhanced. This frequency domain signal is an example of a processed frequency domain signal.

The predetermined processing may include noise classification processing. In this case, the signal processing unit 5 uses the frequency domain signal to estimate the noise contained in the frequency domain signal. The noise label resulting from this estimation is output to the learning unit 7.

<Time domain transform unit 6>
The time domain transform unit 6 receives the processed frequency domain signal and the synthesis window.

The time domain transform unit 6 generates a time domain signal by transforming the processed frequency domain signal into the time domain using a synthesis window (step S6). The generated time domain signal is output to the learning unit 7. The generated time domain signal may be output from the time window generation device as a result of predetermined processing by the signal processing unit 5.

The frequency domain transformation unit 4 performs transformation to the time domain using techniques such as inverse short-time Fourier transform.

<Study Section 7>
The time domain signal is input to the learning unit 7. The learning unit 7 also receives correct data corresponding to the time domain signal.

The learning unit 7 learns the analysis window parameters and synthesis window parameters using the time domain signal and the correct data corresponding to the time domain signal (step S7).

The analysis window parameters and synthesis window parameters are learned using, for example, gradient descent.

When the specified processing in the signal processing unit 5 is speech signal enhancement processing, the noise-free original sound signal becomes the correct data. In this case, a gradient method or the like is used to learn the analysis window parameters and synthesis window parameters so that the mean squared error, calculated by squaring the difference between the time-domain signal and the original sound signal and averaging it, becomes small.

If the predetermined processing in the signal processing unit 5 includes noise classification processing, the true noise label is input to the learning unit 7 as correct data. Also, if the predetermined processing in the signal processing unit 5 includes noise classification processing, the noise label estimated by the predetermined processing in the signal processing unit 5 is further input to the learning unit 7. In this case, the learning unit 7 may learn the analysis window parameters and synthesis window parameters using the true noise label and the estimated noise label in addition to the time-domain signal and the correct data corresponding to the time-domain signal.

The processing from step S1 to step S7 described above may be repeated as appropriate.

The learning unit 7 learns the analysis window parameters and synthesis window parameters based on the time domain signal, which is a signal that has undergone predetermined processing in the signal processing unit 5 and been converted to the time domain. Therefore, it can be said that the analysis window parameters and synthesis window parameters are learned taking into account the predetermined processing in the signal processing unit 5, which is the processing subsequent to the analysis window generation unit 2 and the synthesis window generation unit 3. In this way, by learning the analysis window parameters and synthesis window parameters taking into account the processing subsequent to the analysis window parameters, it is possible to generate time windows more appropriately than before.

[Variations]
The above describes the embodiments of the present invention, but the specific configuration is not limited to these embodiments, and it goes without saying that even if design changes are made as appropriate within the scope of the present invention, they are still included in the present invention.

The various processes described in the embodiments may not only be performed chronologically in the order described, but may also be performed in parallel or individually depending on the processing capacity of the device performing the processes or as needed.

For example, data exchange between components of the time window generation device may be performed directly or via a memory unit not shown.

The synthesis window can be generated from the analysis window. Therefore, the synthesis window generation unit 3 may generate the synthesis window using the analysis window generated by the analysis window generation unit 2. In other words, the synthesis window generation unit 3 may generate the synthesis window using at least the extracted signal or the analysis window.

In this case, the learning unit 7 does not need to learn the synthesis window model parameters. In other words, the learning unit 7 only needs to learn the analysis window model parameters using the time-domain signal and the ground truth data corresponding to the time-domain signal.

Not only the processing in the analysis window generation unit 2 and the synthesis window generation unit 3, but also certain processing in the signal processing unit 5 may be implemented using deep learning.

That is, the predetermined processing in the signal processing unit 5 may be processing that generates a processed frequency domain signal using a model determined by the frequency domain signal and the model parameters. In this case, the learning unit 7 may further learn the model parameters using at least the time domain signal and the correct answer data corresponding to the time domain signal.

The predetermined processing in the signal processing unit 5 may also include processing for estimating noise labels using a model determined by the frequency domain signal and the noise estimation model parameters. In this case, the learning unit 7 may further learn the noise estimation model parameters using the true noise labels input to the learning unit 7 and the noise labels estimated by the signal processing unit 5, in addition to the time domain signal and the correct answer data corresponding to the time domain signal.

[Programs, recording media]
The processing of each unit of each of the above-mentioned devices may be realized by a computer, in which case the processing content of the functions that each device should have is described by a program. Then, by loading this program into storage unit 1020 of computer 1000 shown in Figure 3 and running it on arithmetic processing unit 1010, input unit 1030, output unit 1040, display unit 1060, etc., various processing functions of each of the above-mentioned devices are realized on the computer.

The program describing this processing content can be recorded on a computer-readable recording medium. A computer-readable recording medium is, for example, a non-transitory recording medium, specifically a magnetic recording device, optical disk, etc.

This program may be distributed, for example, by selling, transferring, or lending portable recording media such as DVDs or CD-ROMs on which the program is recorded. Furthermore, this program may be distributed by storing it in a storage device of a server computer and transferring it from the server computer to other computers via a network.

A computer executing such a program, for example, first stores the program recorded on a portable recording medium or transferred from a server computer in its own non-transitory storage device, auxiliary storage unit 1050. Then, when executing a process, the computer loads the program stored in its own non-transitory storage device, auxiliary storage unit 1050, into storage unit 1020 and executes processing in accordance with the loaded program. Alternatively, as an alternative execution form of this program, the computer may load the program directly from a portable recording medium into storage unit 1020 and execute processing in accordance with the program. Furthermore, each time a program is transferred from a server computer to this computer, the computer may execute processing in accordance with the received program. Alternatively, the above-described processing may be executed using a so-called ASP (Application Service Provider) type service, which realizes processing functions simply by issuing execution instructions and obtaining results, without transferring the program from the server computer to this computer. In this embodiment, the program includes information used for processing by an electronic computer that is equivalent to a program (such as data that is not a direct instruction to a computer but has properties that dictate computer processing).

In this embodiment, the device is configured by executing a predetermined program on a computer, but at least some of the processing may be implemented in hardware. For example, the signal extraction unit 1, analysis window generation unit 2, synthesis window generation unit 3, frequency domain transformation unit 4, signal processing unit 5, time domain transformation unit 6, and learning unit 7 may be configured as processing circuits.

It goes without saying that other modifications are possible as long as they do not deviate from the spirit of this invention.

Claims (4)

a signal cutout unit that cuts out a sound signal to a predetermined length to generate a cutout signal;
an analysis window generation unit that generates an analysis window using the extracted signal and an analysis window model determined by analysis window model parameters;
a synthesis window generator that generates a synthesis window using at least the extracted signal or the analysis window;
a frequency domain transform unit that generates a frequency domain signal by transforming the extracted signal into a frequency domain using the analysis window;
a signal processing unit that performs predetermined processing on the frequency domain signal to generate a processed frequency domain signal;
a time domain transform unit that generates a time domain signal by transforming the processed frequency domain signal into a time domain signal using the synthesis window;
a learning unit that learns at least the analysis window model parameters using the time domain signal and ground truth data corresponding to the time domain signal;
Including,
the predetermined processing is a speech signal enhancement processing,
The correct answer data is a noise-free original sound signal,
the synthesis window generation unit generates a synthesis window using the extracted signal and a synthesis window model determined by a synthesis window model parameter;
the learning unit further learns the synthesis window model parameters using the time-domain signal and ground truth data corresponding to the time-domain signal.
Time window generator. 2. The time window generating device of claim 1 ,
the predetermined processing is processing for generating a processed frequency domain signal using the frequency domain signal and a model determined by model parameters;
the learning unit further learns the model parameters by using at least the time-domain signal and ground truth data corresponding to the time-domain signal.
Time window generator. a signal cutting step in which a signal cutting unit cuts out a sound signal to a predetermined length to generate a cut-out signal;
an analysis window generation step in which an analysis window generation unit generates an analysis window using the extracted signal and an analysis window model determined by analysis window model parameters;
a synthesis window generation step in which a synthesis window generation unit generates a synthesis window using at least the extracted signal or the analysis window;
a frequency domain transform step in which a frequency domain transform unit transforms the extracted signal into a frequency domain using the analysis window to generate a frequency domain signal;
a signal processing step in which a signal processing unit performs predetermined processing on the frequency domain signal to generate a processed frequency domain signal;
a time domain transform step in which a time domain transform unit transforms the processed frequency domain signal into a time domain signal using the synthesis window;
a learning step in which a learning unit learns at least the analysis window model parameters using the time domain signal and ground truth data corresponding to the time domain signal;
Including,
the predetermined processing is a speech signal enhancement processing,
The correct answer data is a noise-free original sound signal,
In the synthesis window generating step, a synthesis window is generated using the extracted signal and a synthesis window model determined by a synthesis window model parameter;
In the learning step, the synthesis window model parameters are further learned using the time-domain signal and ground truth data corresponding to the time-domain signal.
Time window generation method. A program for causing a computer to function as each unit of the time window generating device according to claim 1 or 2 .