JP7772107B2 - Signal processing device, signal processing method, and signal processing program - Google Patents

Description

The present invention relates to a signal processing device, a signal processing method, and a signal processing program.

There is currently technology to remove reverberation components from voice or music signals recorded with a remote microphone. Reverberation is a signal component that arrives at a microphone later than the original signal, for example, due to the original signal being reflected off the walls, floor, ceiling, etc.

Signals containing reverberation can lead to performance degradation in speech recognition, noise reduction using signal processing, sound source separation, etc., so performance degradation can be avoided by removing reverberation from the signal in advance. The removal of reverberation components as described above can also be applied to, for example, improving the performance of hearing aids and automatic music notation creation.

One technique for removing reverberation components is WPE (Weighted Prediction Error). WPE assumes an autoregressive model of reverberation and predicts current reverberation components from past observed signals. WPE then estimates an inverse filter that cancels out the predicted reverberation components, and performs dereverberation using this estimated inverse filter. WPE can remove reverberation, but when the number of sound sources is greater than or equal to the number of microphones, the MINT theorem shows that a causal inverse filter (an inverse filter that uses only past signals) cannot exist.

SwitchingWPE (see Non-Patent Document 1) is a technology that partially solves the above problem. SwitchingWPE is an improved version of WPE, and achieves dereverberation by switching between multiple WPE filters for each time-frequency bin of the signal. Here, the parameter used to select the time-frequency bin to which the WPE filter is applied is called Switch.

Rintaro Ikeshita,et al., "Blind Signal Dereverberation Based on Mixture of Weighted Prediction Error Models", IEEE SIGNAL PROCESSING LETTERS, VOL. 28, 2021, 399.

In SwitchingWPE, the switch is optimized using the weighted power minimization criterion (maximum likelihood criterion), so the optimized switch is not necessarily the optimal switch in terms of other evaluation criteria (e.g., speech recognition rate, signal distortion measure, etc.). For example, a switch optimized by SwitchingWPE is not necessarily the optimal switch for speech recognition. As a result, the speech recognition rate for signals after dereverberation using SwitchingWPE may not be high.

Therefore, the objective of this invention is to solve the above-mentioned problems and achieve improved performance in removing reverberation components in SwitchingWPE according to the purpose.

To solve the above-mentioned problems, the present invention provides a SwitchingWPE having multiple WPE filters that remove reverberation components from an observed signal and a Switch for switching between the multiple WPE filters for each time frequency of the observed signal; a reception unit that receives input of an evaluation criterion for the signal after reverberation components have been removed by the SwitchingWPE; a learning unit that uses a training dataset for removing reverberation components of a signal and trains a model that outputs an estimation result for the Switch such that the signal after reverberation components have been removed by the SwitchingWPE is optimized using the evaluation criterion; a Switch setting unit that sets the Switch estimated by the trained model for the observed signal in the SwitchingWPE; and a filter setting unit that calculates an optimal WPE filter for the set Switch and sets it in the SwitchingWPE, wherein the SwitchingWPE removes the reverberation components of the input signal using the set Switch and the set WPE filter.

According to the present invention, it is possible to improve the performance of reverberation component removal in SwitchingWPE according to the purpose.

FIG. 1 is a diagram for explaining an overview of SwitchingWPE. FIG. 2 is a diagram illustrating an outline of the signal processing device. FIG. 3 is a diagram illustrating an outline of a signal processing device when the evaluation standard is SDR. FIG. 4 is a diagram illustrating an example of the configuration of a signal processing device. FIG. 5 is a diagram illustrating an example of a processing procedure of the signal processing device. FIG. 6 is a diagram showing the evaluation results of the reverberation component removal performance of the signal processing device. FIG. 7 is a diagram showing an example of the configuration of a computer that executes a signal processing program.

Hereinafter, a form (embodiment) for implementing the present invention will be described with reference to the drawings. The present invention is not limited to this embodiment.

[SwitchingWPE]
First, an overview of SwitchingWPE, which is a basic technology used by the signal processing device of this embodiment, will be described using Fig. 1. SwitchingWPE clusters the time frequencies of an observed signal (e.g., a speech signal) and uses Switch to switch between multiple WPE filters for each time frequency, thereby achieving dereverberation of the observed signal. A WPE filter is calculated for each Switch. The Switch and WPE filters of SwitchingWPE are alternately optimized using a weighted power minimization criterion (maximum likelihood criterion).

[overview]
Next, an overview of the signal processing device of this embodiment will be described with reference to Fig. 2. The signal processing device estimates an optimal Switch for dereverberation using SwitchingWPE from an observed signal using a model such as a DNN (Deep Neural Network). Then, the signal processing device dereverberates the observed signal using SwitchingWPE using the estimated Switch.

For example, a signal processing device uses a dereverberation training dataset to train a Switch estimation model that receives an input of an observed signal and outputs an optimal Switch estimation result for that observed signal. Note that this training dataset is a dataset that shows the input signal and a signal obtained by removing the reverberation components from that input signal (the dereverberation correct signal).

Here, before training the above-mentioned Switch estimation model, the signal processing device accepts input of evaluation criteria for the signal after dereverberation by SwitchingWPE (e.g., SDR (Signal-to-Distortion Ratio), Scale invariant SDR, intelligibility such as STOI (Short-Time Objective Intelligibility measure), Cepstral distance, WER (Word Error Rate) in ASR (Automatic Speech Recognition), etc.).

Then, the signal processing device uses the dereverberation training dataset to train a Switch estimation model that estimates a Switch such that the signal (dereverberated signal) after reverberation components have been removed by SwitchingWPE is optimized using the above evaluation criteria.

For example, consider a case where the evaluation criterion for the dereverberated signal is SDR and the Switch estimation model is implemented using a DNN. In this case, the signal processing device optimizes the DNN to maximize the SDR between the dereverberated signal output by SwitchingWPE for an input signal included in the training dataset and the dereverberated target signal for that input signal included in the training dataset (see Figure 3).

Then, the signal processing device inputs the observed signal into the optimized DNN and obtains an estimated Switch optimized for the observed signal. The signal processing device then uses the estimated Switch to dereverberate the observed signal using SwitchingWPE. In this way, the signal processing device can achieve improved performance in removing reverberation components in SwitchingWPE according to the purpose (evaluation criteria).

[Configuration example]
Next, an example of the configuration of the signal processing device 10 will be described with reference to Fig. 4. The signal processing device 10 includes an input/output unit 11, a storage unit 12, and a control unit 13.

The input/output unit 11 is an interface that handles the input and output of various information. For example, the input/output unit 11 accepts inputs such as the observed signal to be dereverberated and evaluation criteria for the dereverberated signal. Furthermore, for example, the input/output unit 11 outputs the dereverberated signal.

The memory unit 12 stores data that the control unit 13 references when performing various processes. For example, the memory unit 12 stores evaluation criteria for dereverberated signals, a training dataset for dereverberation, a switch estimation model (parameters of the switch estimation model) for estimating the optimal switch, etc.

The above Switch estimation model is a model that takes an observed signal as input to SwitchingWPE131 and outputs an estimation result of the optimal Switch in SwitchingWPE131. This Switch estimation model is realized, for example, by a DNN. The Switch estimation model is trained by the learning unit 133.

The control unit 13 is responsible for controlling the entire signal processing device 10. The control unit 13 includes a SwitchingWPE 131, a reception unit 132, a learning unit 133, a Switch setting unit 134, and a filter setting unit 135.

SwitchingWPE131 derives reverberation from the observed signal using Switch and multiple WPE filters. Switch is a parameter for switching between multiple WPE filters for each time frequency of the observed signal. The WPE filters remove the reverberation components of the observed signal.

The reception unit 132 receives input of evaluation criteria for the dereverberated signal. The evaluation criteria include, for example, SDR, Scale invariant SDR, clarity such as STOI, Cepstram distance, and WER in ASR. The evaluation criteria for the dereverberated signal received by the reception unit 132 may be any of the above evaluation criteria, or a combination of multiple evaluation criteria.

The learning unit 133 uses a dereverberation learning dataset, takes the observed signal as input, and learns a Switch estimation model that outputs a Switch estimation result such that the signal from which the reverberation components have been removed by SwitchingWPE 131 is optimized using the evaluation criteria input by the receiving unit 132.

For example, when the evaluation criteria are SDR, Scale invariant SDR, STOI, or Cepstral distance, the learning unit 133 sets the dereverberated signal (correct signal) for the input signal of the training dataset as correct data. Then, using the correct data, the learning unit 133 trains a Switch estimation model that estimates the Switch of SwitchingWPE131 so that the result of evaluating the reverberation signal output by SwitchingWPE131 for the input signal of the training dataset using the above evaluation criteria is optimized.

When the evaluation criterion is ASR, the learning unit 133 uses the correct text (a transcribed sentence of the input speech) as the correct data instead of the above-mentioned correct signal. In this case, the learning unit 133 inputs the dereverberated signal output by SwitchingWPE 131 to the ASR, and trains the Switch estimation model so that the recognition result by ASR matches the correct text as closely as possible (so that the ASR accuracy rate improves). For example, the learning unit 133 trains the Switch estimation model so that the WER of the recognition result by ASR is as small as possible.

The Switch setting unit 134 sets the Switch output by the learned Switch estimation model for the observed signal to SwitchingWPE 131. In addition, the filter setting unit 135 calculates the optimal WPE filter for the Switch set by the Switch setting unit 134 and sets it to SwitchingWPE 131. The method for calculating the optimal WPE filter here may be, for example, the same method as the method for calculating the WPE filter in conventional SwitchingWPE.

Then, SwitchingWPE131 removes the reverberation components of the input observation signal using the Switch set by the Switch setting unit 134 and the WPE filter set by the filter setting unit 135.

By doing this, the signal processing device 10 can achieve improved performance in removing reverberation components according to the purpose for the SwitchingWPE 131.

[Example of processing procedure]
Next, an example of the processing procedure of the signal processing device 10 will be described with reference to Fig. 5. First, the receiving unit 132 of the signal processing device 10 receives an input of an evaluation criterion for a signal after reverberation components have been removed by the SwitchingWPE 131 (S1).

Next, the learning unit 133 uses a dereverberation learning dataset to learn a Switch estimation model that takes a signal as input and outputs a Switch estimation result such that the signal from which the reverberation components have been removed by SwitchingWPE131 is optimized using the evaluation criteria accepted in S1 (S2: Model learning).

After S2, the signal processing device 10 accepts input of an observed signal (S3). Then, the Switch setting unit 134 sets the Switch estimated by the Switch estimation model learned in S2 for the observed signal input in S3 to SwitchingWPE131 (S4: Setting the Switch). Then, the filter setting unit 135 calculates the optimal WPE filter for the set Switch and sets it to SwitchingWPE131 (S5: Setting the WPE filter).

Then, SwitchingWPE131 removes the reverberation components of the input observation signal using the Switch set by the Switch setting unit 134 and the WPE filter set by the filter setting unit 135 (S6).

By doing this, the signal processing device 10 can achieve improved performance in removing reverberation components according to the purpose for the SwitchingWPE 131.

[Evaluation results]
Next, the evaluation results of the reverberation component removal performance of the signal processing device 10 will be described with reference to Fig. 6. Here, the signal processing device 10 was evaluated for its reverberation component removal performance on speech data containing reverberation components that was created by simulation.

The signal processing device 10 used a dereverberation training dataset to train a DNN that estimates the optimized Switch for a signal from which reverberation components have been removed by SwitchingWPE131, using SDR as the evaluation criterion. The audio data to be evaluated was audio data recorded with one microphone. The number of Switches used by SwitchingWPE131 was set to three. The comparison targets were the observed signal (unprocessed), WPE, and SwitchingWPE.

As shown in Figure 6, it was confirmed that the speech data dereverberated by the signal processing device 10 using SwitchingWPE, which uses the Switch estimated by the above-mentioned DNN, had a higher SDR than WPE and SwitchingWPE. It was also confirmed that the speech data dereverberated by the signal processing device 10 had a lower word recognition error rate than WPE and SwitchingWPE.

This confirms that the signal processing device 10 can achieve improved performance in removing reverberation components according to the purpose compared to SwitchingWPE131.

[System configuration, etc.]
Furthermore, the components of each unit shown in the figure are conceptual functional units and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by each device can be realized by a CPU and a program executed by the CPU, or can be realized as hardware using wired logic.

Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information, including the processing procedures, control procedures, specific names, various data, and parameters shown in the above documents and drawings, can be changed as desired unless otherwise specified.

[program]
The signal processing device 10 can be implemented by installing a program (signal processing program) as package software or online software on a desired computer. For example, by executing the program on an information processing device, the information processing device can function as the signal processing device 10. The information processing device referred to here includes mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone Systems), as well as terminals such as PDAs (Personal Digital Assistants).

Figure 7 shows an example of a computer that executes a signal processing program. The computer 1000 has, for example, memory 1010 and a CPU 1020. The computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. Each of these components is connected by a bus 1080.

The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM (Random Access Memory) 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the programs that define the processes executed by the signal processing device 10 described above are implemented as program modules 1093 in which computer-executable code is written. The program modules 1093 are stored, for example, on the hard disk drive 1090. For example, a program module 1093 for executing processes similar to the functional configuration of the signal processing device 10 is stored on the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

In addition, data used in the processing of the above-described embodiment is stored as program data 1094, for example, in memory 1010 or hard disk drive 1090. Then, the CPU 1020 reads the program module 1093 or program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as needed and executes it.

The program module 1093 and program data 1094 may not necessarily be stored on the hard disk drive 1090, but may also be stored on a removable storage medium, for example, and read by the CPU 1020 via the disk drive 1100, etc. Alternatively, the program module 1093 and program data 1094 may be stored on another computer connected via a network (such as a LAN (Local Area Network), WAN (Wide Area Network)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

10 Signal processing device 11 Input/output unit 12 Storage unit 13 Control unit 131 SwitchingWPE
132 Reception unit 133 Learning unit 134 Switch setting unit 135 Filter setting unit

Claims (6)

a SwitchingWPE having a plurality of WPE filters that remove reverberation components of the observed signal and a Switch for switching the plurality of WPE filters for each time frequency of the observed signal;
a receiving unit that receives an input of an evaluation criterion for the signal after the reverberation component has been removed by the SwitchingWPE;
a learning unit that uses a dereverberation learning dataset to learn a model that estimates the Switch such that a signal from which reverberation components have been removed by the SwitchingWPE is optimized using the evaluation criterion; and
A Switch setting unit that sets a Switch estimated by the model after learning to the SwitchingWPE for an observed signal;
A filter setting unit calculates an optimal WPE filter for the set Switch and sets it to the SwitchingWPE,
The SwitchingWPE is
a signal processing device that removes a reverberation component of the observed signal using the set Switch and the set WPE filter. The evaluation criteria are:
2. The signal processing device according to claim 1, wherein the signal processing parameter is at least one of a signal-to-distortion ratio, a scale-invariant signal-to-distortion ratio, intelligibility, a cepstral distance, and a word error rate in automatic speech recognition. the dereverberation training data set includes an input signal and a signal obtained by removing reverberation components from the input signal, which is correct data of the input signal;
The learning unit
2. The signal processing device according to claim 1, wherein the model is trained so that an evaluation result based on the evaluation criterion for the signal after the reverberation components have been removed, using the signal after the reverberation components have been removed output by the SwitchingWPE and the ground truth data, for the input signal, is optimized. The model is
The signal processing device according to claim 1 , wherein the model estimates the switch using a deep neural network (DNN). A signal processing method performed by a signal processing device, comprising:
receiving an input of an evaluation criterion for the signal after the reverberation components have been removed by a SwitchingWPE having a plurality of WPE filters that remove the reverberation components of the observed signal and a Switch for switching the plurality of WPE filters for each time frequency of the observed signal;
a step of training a model that outputs Switch estimation results such that a signal from which reverberation components have been removed by the SwitchingWPE is optimized using the evaluation criterion, using a training dataset for dereverberation;
A step of setting the Switch estimated by the model after learning to the SwitchingWPE for an observed signal;
A process of calculating an optimal WPE filter for the set Switch and setting it as the SwitchingWPE;
and removing the reverberation component of the observation signal using the SwitchingWPE to which the Switch and the WPE filter are set. receiving an input of an evaluation criterion for the signal after the reverberation components have been removed by a SwitchingWPE having a plurality of WPE filters that remove the reverberation components of the observed signal and a Switch for switching the plurality of WPE filters for each time frequency of the observed signal;
a step of training a model that outputs Switch estimation results such that a signal from which reverberation components have been removed by the SwitchingWPE is optimized using the evaluation criterion, using a training dataset for dereverberation;
A step of setting the Switch estimated by the model after learning to the SwitchingWPE for an observed signal;
A process of calculating an optimal WPE filter for the set Switch and setting it as the SwitchingWPE;
and removing the reverberation component of the observed signal using the Switch and the SwitchingWPE in which the WPE filter is set.