JP7619564B2 - Sound collection device, sound collection program, and sound collection method - Google Patents

Description

The present invention relates to a sound collection device, a sound collection program, and a sound collection method, and, for example, to an area sound collection process for collecting sound coming from a sound source present in a target area (hereinafter referred to as "target area sound").

Conventionally, an area sound collection technology using a multi-channel microphone is MUBASE (Multiple Beam-forming Area Sound Enhancement) described in Non-Patent Document 1. MUBASE is a method that emphasizes the area in the front direction by utilizing the ability to extract disturbing sounds from the surroundings by using the difference in the observation signals of two-channel microphones.

Figure 7 shows an example of how MUBASE is used to emphasize and acquire sound components from the front direction (the direction in which the target area sound is located) using observation signals from a microphone array MA equipped with two microphones Ml and Mr.

Here, when the observation signals of the microphones Ml and Mr are respectively expressed as the following formulas (1) and (2) (f is a frequency bin index), the difference of the observation signals can be expressed as the following formula (3). The difference of the observation signals becomes a filter (hereinafter referred to as a "difference filter") that faces null in the front direction, and interference sounds from outside the area are extracted. However, it is known that the estimated interference sounds obtained by the difference filter have weaker power at lower frequencies than the actual interference sounds. Using the estimated interference sounds obtained by the difference filter, the sound from the sound source _yf in the target area in front can be extracted using subtraction expressed by the following formula (4).

In equation (4), the subtraction coefficient _αf is a hyperparameter. In equation (4), the width of the emphasized area changes depending on the value of coefficient _αf . The larger the value of _αf , the narrower the beam becomes.

Kazuhiro Katagiri, Tokuo Yamaguchi, Takashi Yazu, and Yoong Keok Lee, “Multiple beam-forming area sound enhancement (MUBASE) and stereophonic area sound reproduction (SASR) system”, SIGGRAPH Asia 2015 Emerging Technologies, 2015.

In the conventional area sound collection processing using MUBASE, the optimum value of the coefficient _αf varies depending on the position of the sound source within the target area.

For example, assuming that an attempt is made to manually adjust the coefficient _αf in area sound collection processing using the conventional MUBASE, if the coefficient _αf is too large, over-subtraction occurs, and the signal obtained by the sound collection processing (a signal in which the target area sound is emphasized) is distorted, whereas if the coefficient _αf is too small, suppression of interference sounds (non-target area sounds) becomes insufficient, making it difficult to optimally adjust the coefficient _αf .

In view of the above problems, there is a need for a sound collection device, a sound collection program, and a sound collection method that are more robust against environmental changes (e.g., movement of the sound source) related to the sound source within the target area.

The first sound collection device of the present invention is characterized in having a target sound extraction processing means for using a learning model to obtain a target sound enhancement signal that emphasizes the target sound component contained in a first input signal from a first microphone that constitutes a microphone array and a differential signal that is the difference between the first input signal and a second input signal from a second microphone that constitutes the microphone array, and a learning means for obtaining the learning model by learning and processing the first input signal, the differential signal, and data including the target sound signal as teacher data .

The second sound collection program of the present invention is characterized in that it functions as a target sound extraction processing means for using a learning model to obtain a target sound emphasis signal that emphasizes the target sound component contained in a first input signal from a first microphone that constitutes a microphone array and a differential signal that is the difference between the first input signal and a second input signal from a second microphone that constitutes the microphone array, and as a learning means for obtaining the learning model by learning and processing data including the first input signal, the differential signal, and the target sound signal as teacher data .

The third invention is a sound collection method performed by a sound collection device, the sound collection device being equipped with a target sound extraction processing means and a learning means , and the target sound extraction processing means uses a learning model to obtain a target sound enhancement signal that emphasizes the target sound component contained in a first input signal from a first microphone that constitutes a microphone array and a differential signal that is the difference between the first input signal and a second input signal from a second microphone that constitutes the microphone array, and obtains the learning model by learning and processing data including the first input signal, the differential signal, and the target sound signal as teacher data .

The present invention provides a sound collection process that is more robust against environmental changes related to sound sources within a target area.

FIG. 2 is a block diagram showing a functional configuration of a first target area sound extraction unit according to the embodiment. FIG. 11 is a block diagram showing a functional configuration of a second target area sound extraction unit according to the embodiment. 1 is a block diagram showing a functional configuration of a sound collection device according to an embodiment. 1 is a block diagram showing an example of a hardware configuration of a sound collection device according to an embodiment. FIG. 1 is a diagram showing an experimental environment for a sound collection device according to an embodiment. 11A to 11C are diagrams showing experimental results of the sound collection device according to the embodiment. FIG. 1 is a diagram showing a conventional sound collection process using a two-channel microphone array.

(A) Main embodiment Hereinafter, an embodiment of a sound collection device, a program, and a method according to the present invention will be described in detail with reference to the drawings.

(A-1) Configuration of the Embodiment FIG. 3 is a block diagram showing the functional configuration of the sound collection device 100 of this embodiment.

The sound collection device 100 performs target area sound collection processing to collect target area sound from a sound source in the target area using a microphone array MA having two microphones Mr and Ml.

The microphone array MA is placed anywhere in the space in which the target area exists. Note that in this embodiment, for ease of explanation, it is assumed that there is only one target area (target sound source placed in the target area) that is the target of sound pickup by the microphone array MA.

Next, the internal configuration of the sound collection device 100 will be described.

The sound collection device 100 includes a signal input unit 101, a target area sound extraction unit 102, and a signal output unit 103. The detailed processing of each functional block constituting the sound collection device 100 will be described later.

The signal input unit 101 has the function of converting the acoustic signals (analog signals) observed by each microphone into digital signals, and then converting them into a signal format that can be processed by the target area sound extraction unit 102 (in this embodiment, a frequency domain signal). The signal input unit 101 converts the acoustic signals (analog signals) observed by each microphone from analog signals to digital signals, and further converts them from the time domain to the frequency domain (for example, by using a fast Fourier transform, etc.), and supplies them to the target area sound extraction unit 102.

It should be noted that, here, the observation signals (acoustic signals converted into the frequency domain) of the microphones Ml and Mr supplied from the signal input unit 101 to the target area sound extraction unit 102 are represented as _Xr and _Xl , respectively.

The target area sound extraction unit 102 has the function of estimating and extracting the target area sound components from the signal supplied from the signal input unit 101.

The signal output unit 103 converts the signal output from the target area sound extraction unit 102 from the frequency domain to the time domain and outputs the signal in a predetermined format. Note that there are no limitations on the format or method of signal output by the signal output unit 103.

Next, an example of the hardware configuration of the sound collection device 100 will be described.

The sound collection device 100 may be configured entirely from hardware (e.g., a dedicated chip, etc.), or may be configured partially or entirely as software (program). The sound collection device 100 may be configured, for example, by installing a program (including the sound collection program of the embodiment) on a computer having a processor and memory.

Figure 4 is a block diagram showing an example of the hardware configuration of the sound collection device 100.

Figure 4 shows an example of the hardware configuration when the sound collection device 100 is configured using software (computer).

The sound collection device 100 shown in FIG. 4 has, as a hardware component, a computer 400 on which a program (including the sound collection program of the embodiment) is installed. The computer 400 may be a computer dedicated to the sound collection program, or may be configured to be shared with programs of other functions.

The computer 400 shown in FIG. 4 has a processor 401, a primary storage unit 402, and a secondary storage unit 403. The primary storage unit 402 is a storage unit that functions as a working memory (work memory) for the processor 401, and can be, for example, a high-speed memory such as a dynamic random access memory (DRAM). The secondary storage unit 403 is a storage unit that records various data such as an operating system (OS) and program data (including data of the sound collection program according to the embodiment), and can be, for example, a non-volatile memory such as a FLASH (registered trademark) memory, HDD, or SSD. In the computer 400 of this embodiment, when the processor 401 starts up, the OS and programs (including the sound collection program according to the embodiment) recorded in the secondary storage unit 403 are read, deployed on the primary storage unit 402, and executed.

The specific configuration of the computer 400 is not limited to that shown in FIG. 4, and various configurations can be applied. For example, if the primary storage unit 402 is a non-volatile memory (e.g., FLASH (registered trademark) memory, etc.), the secondary storage unit 403 may be excluded.

Next, we will provide an overview of the target area sound extraction process performed by the target area sound extraction unit 102.

The target area sound extraction process described here is designed as a process for extracting target area sound from observation signals of two microphones, similar to the conventional MUBASE. In the conventional MUBASE process, the above formula (4) is applied, but the optimal coefficient _αf varies depending on the arrival angle of the target sound source and the interference sound (non-target area sound), and it may be difficult to set it manually. In addition, in the conventional MUBASE process, if the value of _αf is too large, oversubtraction may occur, and the sound within the target area may be distorted. On the other hand, in the conventional MUBASE process, if the value of _αf is small, the interference sound outside the area may not be suppressed very much.

In this embodiment, the target area sound extraction unit 102 will be described as applying a configuration (hereinafter referred to as "deep area sound collection" or "DMUBASE") that collects target area sound by learning using a deep neural network (DNN) to perform the subtraction calculation represented by the above formula (4). In this embodiment, the target area sound extraction unit 10 can achieve highly accurate area sound collection by using deep area sound collection (DMUBASE), regardless of the arrival angle of the target sound or interfering sound.

Incidentally, in area sound collection processing, it is desirable to be robust against the movement of sound sources within the target area, but in deep area sound collection (DMUBASE), filters are learned in a data-driven manner, so constraints are needed to ensure robustness.

Therefore, it is desirable for deep area sound collection (DMUBASE) to meet the requirements of "being able to suppress interfering sounds outside the area" and "being robust against the movement of the target sound source within the area."

In light of the above, in this embodiment, for example, either the first target area sound extraction unit 102A shown in FIG. 1 or the second target area sound extraction unit 102B shown in FIG. 2 is applied as the model architecture of the target area sound extraction unit 102.

First, we will explain the first target area sound extraction unit 102A shown in Figure 1.

The first target area sound extraction unit 102A has an estimation processing unit 200, a mask processing unit 210, a phase processing unit 220, and a difference extractor 230.

The estimation processing unit 200 estimates the components of the interference sound (non-target area sound) contained in _Xr based on the observation signals _Xl and _Xr of each microphone array, and outputs a signal (hereinafter referred to as a "mask signal") in which a coefficient (filter coefficient) for suppressing the components of the non-target area sound contained in _Xr is set. In the mask signal, a filter coefficient (any value between 0 and 1) for suppressing the components of the interference sound (non-target area sound) contained in _Xr is set for each frequency.

Specifically, the estimation processing unit 200 uses a DNN to perform processing to estimate a mask signal based on the observed signal | _Xr | and |d| = | _Xr - _Xl |, which is the difference between _Xl and _Xr (output of a differential filter). Note that, in this embodiment, in order to perform processing to extract components of the target area sound from the observed signal Xr of the microphone Mr, processing to estimate a mask signal is performed based on the observed signal | _Xr | and the output of a differential filter | _Xr - _Xl |, but processing to estimate a mask signal may be performed based on the observed signal | _Xl | as a reference and the output of the filter | _Xl - _Xr |.

The mask processing unit 210 masks (attenuates; suppresses; filters) the components of the interference sound (non-target area sound) contained in |X _r | based on the mask signal (filter coefficient) supplied from the estimation processing unit 200, and outputs a signal in which the target area sound is emphasized.

The phase processing unit 220 performs processing to impart (synchronize) a phase of _Xr (phase information) to the signal supplied from the mask processing unit 210, and outputs the result. Here, the signal output from the phase processing unit 220 is represented as "y^". Here, y^ is the output signal of the first target area sound extraction unit 102A. In other words, here, y^ can be said to be the result of extracting (emphasizing; estimating) the target area sound.

The difference extractor 230 obtains the difference between y^ (the result of estimating the destination area sound) output from the phase processing unit 220 and the clean destination area sound (hereinafter referred to as "y") which serves as the teacher label (correct label) in machine learning, and feeds this difference back to the estimation processing unit 200 as a loss (loss in machine learning). In other words, the difference extractor 230 is an element which functions only when the estimation processing unit 200 is made to perform learning processing. Therefore, if new learning processing has already not been performed by the estimation processing unit 200, the difference extractor 230 may be excluded from the first destination area sound extraction unit 102A.

As described above, in the first target area sound extraction unit 102A, a mask signal for extracting a sound source (target area sound) within a target area corresponding to a sector-shaped area in front of the microphone array MA (see Figure 7) is estimated by a neural network (estimation processing unit 200) that receives as input the observed signal | _Xr | and |d| in which non-target area sound (interference sound) is dominant, obtained as the output of a differential filter.

The first target area sound extraction unit 102A in this embodiment is compatible with both an operation mode in which the estimation processing unit 200 executes a learning process (hereinafter referred to as the "learning processing mode"), and an operation mode in which the target area sound extraction process (extraction of the mask signal and y^) is performed based on the supplied observation signals Xl and Xr (hereinafter referred to as the "signal processing mode"). Note that the first target area sound extraction unit 102A may be configured not to support the learning processing mode (for example, a configuration in which a learning model has already been acquired or a learning model is acquired from outside, etc.).

When the first target area sound extraction unit 102A operates in the learning processing mode, upon being supplied with a data set (hereinafter referred to as the "teacher data set") including samples of observed signals ( _Xl , _Xr ) as teacher data and clean target area sound y as a teacher label, the first target area sound extraction unit 102A acquires | _Xr | and |d| from the observed signals ( _Xl , _Xr ) of the teacher data set and supplies them to the estimation processing unit 200, and also feeds back the loss (difference) extracted by the difference extractor 230 to the estimation processing unit 200. This allows the estimation processing unit 200 to acquire a learning model that has been learned (deep learning) based on the teacher data set.

Next, an example of the internal configuration of the estimation processing unit 200 will be described with reference to FIG. 1.

Here, the example of Figure 1 will be described as the internal configuration of the estimation processing unit 200, but as the estimation processing unit 200, various machine learning (deep learning) frameworks can be applied as long as learning processing and signal processing based on the above teacher dataset are possible.

In this embodiment, the neural network of the estimation processing unit 200 is described as having the five-layer configuration shown in FIG. 1, but various configurations can be applied as long as learning processing and signal processing based on the above teacher data set are possible.

In the estimation processing unit 200 shown in Fig. 1, "FC layers 211, 212", "FC layers 221, 222", "FC layer 231", "FC layer 241", and "FC layer 251" are arranged in this order from the input layer. In the estimation processing unit 200 shown in Fig. 1, | _xr | and |d| are input to the input FC layers 211 and 212, respectively. In the neural network of the estimation processing unit 200 shown in Fig. 1, only the FC layer 251 has an activation function of sigmoid, and the activation functions of the other FC layers are ReLU (Rectified Linear Unit).

In the neural network of the estimation processing unit 200 shown in Fig. 1, after a nonlinear transformation is performed on each of | _xr | and |d| by the FC layers 211, 212, 221, and 222, the two inputs are combined and input to the third layer, the FC layer 231. Furthermore, in the neural network of the estimation processing unit 200 shown in Fig. 1, the two subsequent FC layers 241 and 251 perform a transformation (inverse transformation) and output a mask signal (time-frequency mask). As described above, the activation function of the FC layer 251 is sigmoid, so that the estimation processing unit 200 can output a coefficient (filter coefficient) represented by a numerical value between 0 and 1 for each frequency.

In the estimation processing unit 200, by configuring a neural network as shown in Fig. 1, a process of extracting a mask signal (filter coefficient) that emphasizes a target area sound whose sound source is in the front direction of the microphone array MA from |d| where the interfering sound is dominant and | _Xr | which is an observation signal (observation sound) can be learned from data, and a subtraction process corresponding to the formula (4) can be learned by the DNN. In other words, in the estimation processing unit 200, by configuring a neural network as shown in Fig. 1, it is possible to perform area sound collection processing that is robust against the movement of a sound source in the target area in the front direction of the microphone array MA. In particular, in the estimation processing unit 200, the FC layer 231, which is an intermediate layer to which two inputs are connected, is configured to be responsible for the above-mentioned subtraction process.

Next, the second target area sound extraction unit 102B shown in FIG. 2 will be described. In FIG. 2, the same or corresponding parts as those in FIG. 1 above are denoted by the same or corresponding reference numerals.

The following describes the differences between the second target area sound extraction unit 102B and the first target area sound extraction unit 102A.

As shown in FIG. 2, the second target area sound extraction unit 102B has an estimation processing unit 300, a phase processing unit 220, and a difference extractor 230.

The estimation processing unit 200 performs processing to estimate a mask signal based on the observed signals | _Xr | and |d| of the microphone array, but the estimation processing unit 300 differs from the estimation processing unit 200 in that it outputs a power spectrum obtained as a result of estimating the target area sound y (the spectrum of a signal in which the components of the target area sound are emphasized; a signal in the frequency domain).

The phase processing section 220 performs processing to add (synchronize) the phase of _Xr (phase information) to the power spectrum supplied from the estimation processing section 300, and outputs the result as a signal y^.

As described above, in the second target area sound extraction unit 102B, the power spectrum of the sound source (target area sound) in the target area corresponding to the sector-shaped area in front of the microphone array MA (see Figure 7) is estimated by a neural network (estimation processing unit 300) that receives as input the observed signal | _Xr | and |d| in which non-target area sound (interference sound) is dominant, obtained as the output of a differential filter.

The second target area sound extraction unit 102B may be configured to support a learning processing mode and a signal processing mode, similar to the first target area sound extraction unit 102A.

Next, an example of the internal configuration of the estimation processing unit 300 will be described with reference to FIG. 2.

Here, the example of Figure 2 will be described as the internal configuration of the estimation processing unit 300, but as the estimation processing unit 300, various machine learning (deep learning) configurations can be applied as long as learning processing and signal processing based on the above teacher data set are possible.

Here, the neural network of the estimation processing unit 300 differs from the estimation processing unit 200 in that the FC layer 251 at the last stage (output layer) of the neural network of the estimation processing unit 300 is replaced with an FC layer 351, as shown in FIG. 2. The FC layer 351 of the estimation processing unit 300 differs from the estimation processing unit 200 in that the activation function is ReLu instead of sigmoid. This allows the FC layer 351 of the estimation processing unit 300 to output a power spectrum.

In the estimation processing unit 300, a mechanism is configured using a neural network as shown in FIG. 2 to output a target area sound whose sound source is in front of the microphone array MA from |d|, where the interfering sound is dominant, and | _Xr |, which is the observed sound, so that the subtraction processing equivalent to equation (4) can be learned from data.

(A-2) Operation of the Embodiment Next, the operation of the sound collection device 100 in this embodiment having the above-mentioned configuration (sound collection method according to the embodiment) will be described.

First, we will explain the processing performed when the target area sound extraction unit 102 of the sound collection device 100 operates in the learning processing mode.

When a teacher data set is supplied to the target area sound extraction unit 102 operating in a learning processing mode, the target area sound extraction unit 102 acquires | _Xr | and |d| from the observed signals ( _Xl , _Xr ) of the teacher data set and inputs them to the neural network to perform learning processing for deep area sound collection (learning the process of extracting target area sound using a neural network).

When the first target area sound extraction unit 102A is applied to the sound collection device 100, the first target area sound extraction unit 102A inputs | _Xr | and |d| to the estimation processing unit 200. At this time, the first target area sound extraction unit 102A extracts the loss between the signal y^ output from the phase processing unit 220 and the teacher label y by the difference extractor 230 and feeds it back to the estimation processing unit 200. The first target area sound extraction unit 102A performs a learning process for deep area sound collection (learning the process of extracting the target area sound by a neural network) by the above-mentioned feedback.

On the other hand, when the second target area sound extraction unit 102B is applied to the sound collection device 100, the second target area sound extraction unit 102B inputs | _Xr | and |d| to the estimation processing unit 300. At this time, the second target area sound extraction unit 102B extracts the loss of the power spectrum output from the estimation processing unit 300 by the difference extractor 230 and feeds it back to the estimation processing unit 300. The second target area sound extraction unit 102B performs a learning process for deep area sound collection (learning the process of extracting the target area sound by a neural network) by the above-mentioned feedback.

Next, we will explain the operation of the target area sound extraction unit 102 of the sound collection device 100 when it operates in signal processing mode.

Assume here that observed signals ( _Xl , _Xr ) are supplied from the microphone array MA (microphones Mr, Ml) to the target area sound extraction unit 102 operating in signal processing mode via the signal input unit 101. The target area sound extraction unit 102 then supplies | _Xr | and |d| to the neural network (the estimation processing unit 200 or the estimation processing unit 300), and as a result, obtains y^ and supplies it to the signal output unit 103. The signal output unit 103 converts y^ from the frequency domain to the time domain and outputs it.

Next, the inventors will explain the experimental results and evaluation results of an experiment (hereinafter referred to as "this experiment") in which they actually constructed a sound collection device 100, performed a process to collect target area sound, and evaluated its quality.

Figure 5 shows the environment of this experiment.

Figure 5 shows an example in which microphones Mr, Ml, the target sound source, and the interfering sound source are all on the same plane. Also, in Figure 5, the position of the midpoint of line L connecting the positions (center positions) of microphones Mr, Ml (center point of microphone array MA) is shown as P1. Furthermore, in Figure 5, the direction of microphone Mr as viewed from P1 is 0°, and the direction of microphone Ml as viewed from P1 is 180°, and the target sound source and the interfering sound source arrive from any angle between 0° and 180° as viewed from P1. Hereinafter, the direction in which the apparent sound source and the interfering sound source exist from P1 will also be referred to as the "arrival angle" or "arrival direction". Also, as shown in Figure 5, the target sound source and the interfering sound source (sound source in a non-target area) are located on a semicircular line at a distance of 1 m from P1.

In this experiment, in both the learning processing mode (during training) and the signal processing mode (during signal processing), the TIMIT corpus (see Reference 1 below) was used as the dry source (signal) of the target sound source, and the TIMIT corpus or the DEMAND (Diverse Environments Multi-channel Acoustic Noise Database) corpus (see Reference 2 below) was used as the dry source (signal) of the interference sound.

Reference 1: JS Garofolo, LF Lamel, WM Fisher, JGFiscus, DS Pallett, NL Dahlgren, V. Zue, “TIMIT acoustic phonetic continuous speech corpus,” Linguistic Data Consotrium, 1992.
Reference 2: J. Thiemann, N. Ito, and E. Vincent, “The diverseenvironments multi-channel acoustic noise database(DEMAND): A database of multichannel environmental noise recordings”, The Journal of the Acoustical Society of America, vol. 133, p. 3591,05, 2013.

In this experiment, the observed signals (acoustic signals) captured by microphones Ml and Mr in a sound field (model environment) like that shown in Figure 5 were obtained by computer simulation, and the obtained observed signals were then input to the sound collection device 100 to evaluate the results.

Specifically, in this experiment, a simulation was performed using PyRoomAcoustics (see Reference 3 below) with a sound field (model environment) as shown in FIG. 5 to obtain an impulse response, and the obtained impulse response was convoluted with the above-mentioned dry sources (the dry sources of the target sound source and the interfering sound source) to obtain the observation signals _Xl and _Xr of the microphones Ml and Mr.

Reference 3: Scheibler, E. Bezzam, I. Dokmani´c, “Pyroomacoustics: A Python package for audio room simulations and array processing algorithms”, Proc. IEEE ICASSP, 2018

In addition, in the simulation of this experiment, the SNR of the observed signals _Xl and _Xr is adjusted to be approximately 0.0 [dB]. In the following, the 3D space of the sound field for this experiment is represented by a three-dimensional coordinate system of (x, y, z).

In the simulation of this experiment, the size of the room that constitutes the sound field of the model environment was set to (x, y, z) [m] = (5, 3, 3), and the coordinates of the 2ch microphones Ml and Mr were set to (x, y, z) [m] = (2.49, 1.5, 1) and (x, y, z) [m] = (2.51, 1.5, 1), respectively. This results in a distance of 2 [cm] between microphones Ml and Mr. In addition, in the simulation of this experiment, the sound absorption coefficient of the room was set to 0.2, and the number of reflections in the room was set to 3.

In this experiment, simulations were performed for the following cases: a first target area sound extraction unit 102A that estimates the target area sound by mask estimation is applied to the target area sound extraction unit 102 of the sound collection device 100 (hereinafter referred to as the "first experimental model of the present invention"), a second target area sound extraction unit 102B that estimates the target area sound by power spectrum estimation is applied (hereinafter referred to as the "second experimental model of the present invention"), and target area sound estimation using conventional MUBASE is applied (hereinafter referred to as the "experimental model of conventional configuration").

Next, we will explain the position of each sound source in the simulation of this experiment.

In this experiment, the position of the target sound source was fixed at 90° during learning, and during testing, the target sound source was moved within the area (within a range of 80° to 90° at a distance of 1 m from P1) to verify whether the sound collection device 100 satisfied the above two requirements. In addition, the interfering sound source was randomly placed in 1 to 3 of a total of 8 positions, namely 0°, 15°, 30°, 45°, 135°, 150°, 165°, and 180°, during both learning and testing. In this experiment, such changes in the positions of the target sound source and the interfering sound source were performed in data processing units (e.g., word units) on the corpus.

Next, we will explain the detailed parameter settings for the simulation of this experiment.

In a simulation using the "conventional MUBASE model," the value of α was set to 200/(f+0.01) in the target area sound extraction unit 102 when estimating non-target area sound (interference sound) using a differential filter, based on the tendency for the power to be weaker at lower frequencies.

In addition, in training the "first experimental model of the present invention" and the "second experimental model of the present invention", the batch size was set to 32, the number of epochs was set to 200, and the mean squared error was used as the loss function. In training the "first experimental model of the present invention" and the "second experimental model of the present invention", Adam (see Reference 4 below) was used as the optimization algorithm, and the learning rate was set to 0.001.

Reference 4: D. Kingma, and J. Ba, “Adam: A method for stochastic optimization”, International Conference on Learning Representations (ICLR), 2015.

In this experiment, an environment was constructed for the first experimental model of the present invention (mask estimation), the second experimental model of the present invention (power spectrum estimation), and the conventional experimental model (MUBASE), and the above-mentioned learning process and signal processing (test process) were performed for each of them. In the signal processing (test process) of this experiment, two indices, SNR (Signal-to-Noise Ratio) and STOI (Short-Time Objective Intelligence), were measured for each of the three experimental models. In addition, in the test process of this experiment, test processes were performed for each experimental model in a pattern in which the position of the target sound source was fixed at 90° (hereinafter referred to as the "target sound source fixed pattern") and a pattern in which the target sound source was moved randomly between 80° and 90° (hereinafter referred to as the "target sound source moving pattern"). Figure 6 shows the results of this experiment.

(A-3) Advantages of the Embodiment According to this embodiment, the following advantages can be obtained.

In a conventional configuration using MUBASE, area sound collection processing was performed by spectral subtraction involving a predetermined coefficient, but the sound collection device 100 of this embodiment performs deep area sound collection (DMUBASE) using a neural network trained with teacher data. In particular, the sound collection device 100 of this embodiment utilizes the fact that in the two-channel microphone array MA, the output d of the differential filter can be taken to obtain interference sounds (non-target area sounds) arriving from directions other than the front, and by inputting the output d of the differential filter (information in which interference sounds are dominant) into the neural network together with the observation signal, an output can be obtained in which the target area sounds in the front direction are emphasized.

As described above, in the area sound collection process, it is desirable to be robust against the movement of the sound source within the target area, but in the deep area sound collection (DMUBASE), the filter is learned in a data-driven manner, so constraints are required to ensure robustness. In addition, in the sound collection device 100 of this embodiment, robustness against environmental changes is improved by using physical information such as the output d of the differential filter as auxiliary information, rather than a simple data-driven (e.g., only the observed signal). As shown in FIG. 6, both of the experimental models that reproduce the configuration of this embodiment (the first and second experimental models of the present invention) exceeded the accuracy of the experimental model (MUBASE) of the conventional configuration in both the target sound source fixed pattern and the target sound source moving pattern. In other words, it was confirmed that the experimental model of the present invention is more robust against the movement of the sound source of the target area sound than the conventional one.

(B) Other Embodiments The present invention is not limited to the above-described embodiments, and modified embodiments such as those exemplified below can also be mentioned.

(B-1) In the above embodiment, the sound collection device 100 has been described as being compatible with both the learning processing mode and the signal processing mode (test mode), but if a learning model is stored in advance, it may be configured to be compatible with only the signal processing mode and to exclude the means (learning means) required for the learning processing mode.

100...sound collection device, 101...signal input section, 102...target area sound extraction section, 103...signal output section, 102A...first target area sound extraction section, 200...estimation processing section, 212, 221, 222, 231, 241, 251, 211,...FC layer, 210...mask processing section, 220...phase processing section, 230...difference extractor, 102B...second target area sound extraction section,...estimation processing section 300, 212, 221, 222, 231, 241, 351, 211...FC layer, 230...difference extractor.

Claims (5)

a target sound extraction processing means for acquiring a target sound emphasis signal in which a component of a target sound included in a first input signal is emphasized from a first input signal from a first microphone constituting a microphone array and a difference signal which is a difference between the first input signal and a second input signal from a second microphone constituting the microphone array, using a learning model;
a learning means for acquiring the learning model by performing learning processing on data including the first input signal, the difference signal, and the target sound signal as teacher data;
A sound collecting device comprising: the learning model outputs a mask coefficient that suppresses components of non-target sounds other than the target sound included in the first input signal from the first input signal and the differential signal;
The sound collection device according to claim 1 , wherein the target sound extraction processing means acquires the target sound emphasis signal by suppressing the non-target sound components from the first input signal using the mask coefficient. The sound collection device according to claim 1, characterized in that the learning model outputs a target sound emphasis signal in which non-target sound components contained in the first input signal are suppressed and the target sound components are emphasized from the first input signal and the difference signal. Computer,
a target sound extraction processing means for acquiring a target sound emphasis signal in which a component of a target sound included in a first input signal is emphasized from a first input signal from a first microphone constituting a microphone array and a difference signal which is a difference between the first input signal and a second input signal from a second microphone constituting the microphone array, using a learning model ;
a learning means for acquiring the learning model by performing learning processing on data including the first input signal, the difference signal, and the target sound signal as teacher data;
A sound recording program characterized by functioning as follows. In the sound collection method performed by the sound collection device,
The sound collection device includes a target sound extraction processing means and a learning means ,
the target sound extraction processing means acquires, using a learning model, a target sound enhancement signal in which a component of the target sound included in a first input signal is enhanced from a first input signal from a first microphone constituting a microphone array and a differential signal which is a difference between the first input signal and a second input signal from a second microphone constituting the microphone array ;
The learning means obtains the learning model by performing a learning process using data including the first input signal, the difference signal, and the target sound signal as teacher data.
A sound collection method comprising:

Images