JP7010900B2 - Sound source localization device and sound source localization method - Google Patents

Description

The present invention relates to a sound source localization device for estimating the position of a sound source and a sound source localization method.

Conventionally, there is known a device that estimates the position of a sound source in a three-dimensional space based on the sound received by a plurality of sound receiving devices. The non-patent document uses a sound source using a machine learning model that includes a deep neural network (hereinafter referred to as "DNN") trained to output discrete sound source positions for inputting audio features. A technique for estimating the position of is disclosed.

Ryu Takeda, and Kazunori Komatani, "Discriminative Multiple Sound Source Localization based on Deep Neural Networks using Independent Location Model", Proceedings of IEEE Workshop on Spoken Language Technology (SLT), pp.603-609, Dec. 16, 2016.

In the above technique, it is necessary to train a machine learning model using learning data in which the position of a sound source is known. Therefore, for example, when the sound source exists at a position where the machine learning model has not learned, the accuracy of sound source localization may decrease. In general, in the sound source localization process, the waveforms of a plurality of voices are compared. Therefore, if the voice contains noise, the accuracy of the sound source localization may decrease due to the influence of the noise superimposed on the waveform. there were.

Therefore, the present invention has been made in view of these points, and an object of the present invention is to provide a sound source localization device and a sound source localization method capable of improving the accuracy of sound source localization.

In the sound source localization device according to the first aspect of the present invention, the first sound received by the first sound receiving device and the second sound receiving device installed at a position different from the first sound receiving device receive sound. A plurality of processing layers included in a machine learning model capable of outputting text information indicating the content of the input voice to each of the acquisition unit for acquiring the second voice and the first voice and the second voice. A unit that is commonly activated in both the propagation control unit for propagating, the post-stage processing layer selected from the plurality of processing layers, and the pre-stage processing layer that is the processing layer immediately before the post-stage processing layer is included. One or more first outputs output from the post-stage processing layer and the pre-stage processing layer based on the first sound, and one or more outputs from the post-stage processing layer and the pre-stage processing layer based on the second sound. An extraction unit that extracts the second output, one or more first acoustic feature quantities based on the one or more first outputs, and one or more second acoustics based on the one or more second outputs. Based on the feature amount detection unit that detects the feature amount, the one or more first acoustic feature amounts, and the one or more second acoustic feature amounts, the sounds included in the first voice and the second voice, respectively. It has a sound source localization unit that estimates the position of a sound source that is a source, and an output unit that outputs information indicating the position of the sound source estimated by the sound source localization unit.

The sound source localization unit receives a time when the first sound receiving device receives a sound corresponding to the first acoustic feature amount and a sound received by the second sound receiving device corresponding to the second acoustic feature amount. The position of the sound source may be estimated based on the relationship between the time and the position of the first sound receiving device and the position of the second sound receiving device.

The extraction unit has a plurality of post-stage first outputs output from the post-stage processing layer due to the propagation of the first voice in the order of the pre-stage processing layer and the post-stage processing layer which are a part of the plurality of processing layers, and the above-mentioned One or more post-stage first outputs and one or more commonly activated from the plurality of post-stage second outputs output from the post-stage processing layer due to the second voice propagating in the order of the pre-stage processing layer and the post-stage processing layer. A plurality of pre-stage first outputs output from the post-stage extraction unit that extracts the post-stage second output, and the pre-stage processing layer that is a factor that activates the one or more rear-stage first outputs and the one or more post-stage second outputs. The output and the pre-stage extraction unit that extracts one or more pre-stage first outputs and one or more pre-stage second outputs that are commonly activated among the plurality of pre-stage second outputs output from the pre-stage processing layer. You may have.

The pre-stage extraction unit has one or more pre-stage first outputs and one or more pre-stage second outputs based on the activated size of the plurality of pre-stage first outputs and the plurality of pre-stage second outputs. The output may be extracted.

The machine learning model may include a recurrent neural network, and the post-processing layer may be any one of an output layer, a recursive layer, a fully connected layer, a pooling layer, and a convolution layer.
The pretreatment layer may be any one of a recursive layer, a fully connected layer, a pooling layer, a convolution layer, and an input layer.

When the last layer, which is the last treatment layer, is selected as the subsequent treatment layer among the plurality of treatment layers, the extraction unit has one or more first layers that are commonly activated in the last treatment layer. When there is no output and the one or more second outputs, the one or more first outputs and the one or more second outputs that are commonly activated in the treatment layer before the rearmost layer may be extracted. ..

The sound source localization device is a part of the first acoustic feature amount based on the mutual correspondence relationship from the one or more first acoustic feature amount and the one or more second acoustic feature amount specified by the feature amount detection unit. And a part of the selection unit for selecting a part of the second acoustic feature amount may be further provided, and the sound source localization part may be divided into the part of the first acoustic feature amount and the part of the second acoustic feature amount. Based on this, the position of the sound source may be estimated.

The sound source localization device may further have an instruction receiving unit that receives an instruction to select a processing layer to be used as the subsequent processing layer among the plurality of processing layers, and the extraction unit may have the instruction receiving unit. The processed layer indicated by the received instruction may be used as the post-processed layer.

The extraction unit selects one of the plurality of processing layers as the post-processing layer, extracts the one or more first outputs and the one or more second outputs, and then selects the pre-processing layer. The processed layer may be selected as the subsequent processing layer to extract another one or more first outputs and one or more second outputs.

The sound source localization method according to the second aspect of the present invention includes a step of acquiring the first sound received by the first sound receiving device and a second sound receiving device installed at a position different from the first sound receiving device. A plurality of processes included in a machine learning model capable of outputting text information indicating the content of the input voice to each of the step of acquiring the second voice received by the device and the first voice and the second voice. A unit that is commonly activated in both the treatment layer of the step of propagating the layer, the post-stage treatment layer selected from the plurality of treatment layers, and the pre-stage treatment layer that is the treatment layer immediately before the post-stage treatment layer is included. One or more first outputs output from the post-stage processing layer and the pre-stage processing layer based on the first sound, and one or more outputs from the post-stage processing layer and the pre-stage processing layer based on the second sound. A step of extracting a second output, detecting one or more first acoustic features based on the one or more first outputs, and one or more second acoustic features based on the one or more second outputs. It is a source of sound included in each of the first voice and the second voice based on the step of detecting the amount and the one or more first acoustic feature quantities and the one or more second acoustic feature quantities. It has a step of estimating the position of the sound source and a step of outputting information indicating the estimated position of the sound source.

In the extraction step, a plurality of post-stage first outputs and a plurality of post-stage first outputs output from the post-stage processing layer due to the propagation of the first voice in the order of the pre-stage processing layer and the post-stage processing layer which are a part of the plurality of processing layers. One or more post-stage first outputs and one or more commonly activated from a plurality of post-stage second outputs output from the post-stage processing layer due to the propagation of the second sound in the order of the pre-stage processing layer and the post-stage processing layer. A plurality of pre-stage first outputs output from the post-stage extraction step for extracting the post-stage second output, and the pre-stage processing layer that has become a factor for activating the one or more post-stage first outputs and the one or more post-stage second outputs. A pre-stage extraction step for extracting one or more pre-stage first outputs and one or more pre-stage second outputs that are commonly activated among the one output and the plurality of pre-stage second outputs output from the pre-stage processing layer. May have.

After executing the pre-stage extraction step, the post-stage extraction step is performed by using the one or more pre-stage first outputs and the one or more pre-stage second outputs as the plurality of post-stage first outputs and the plurality of post-stage second outputs. You may do it.
The latter-stage extraction step and the first-stage extraction step may be executed for each of the plurality of processing layers.

According to the present invention, there is an effect that the accuracy of sound source localization can be improved.

It is a figure for demonstrating the outline of a sound source localization system. It is a figure which shows an example of the structure of a machine learning model. It is a figure which shows the structure of the sound source localization apparatus. It is a figure for demonstrating the extraction process performed by an extraction part. It is a figure for demonstrating the extraction process performed by an extraction part. It is a figure for demonstrating the extraction process performed by an extraction part. It is a figure for demonstrating the extraction process performed by an extraction part. It is a flowchart which shows the flow of the process performed by a sound source localization device. It is a flowchart which shows the flow of the process performed by the extraction unit.

[Overview of sound source localization system S]
FIG. 1 is a diagram for explaining an outline of the sound source localization system S. The sound source localization system S is a system that estimates the position of the sound source in the sound source localization, that is, the three-dimensional space. The sound source localization system S has a plurality of sound receiving devices 1 and a sound source localization device 2.

The sound receiving device 1 is, for example, a microphone. The sound receiving device 1 is a device that converts the received sound into an electric signal. In the example shown in FIG. 1, the sound source localization system S has a first sound receiving device 1a and a second sound receiving device 1b installed at a position different from the first sound receiving device 1a as a plurality of sound receiving devices 1. Has.

The sound source localization device 2 is, for example, a PC (Personal Computer). The sound source localization device 2 estimates the position of a sound source that is a source of a plurality of sounds based on a plurality of sounds received by each of the plurality of sound receiving devices 1 installed at different positions, thereby performing sound source localization. It is a device that executes processing. The sound source localization device 2 may execute the sound source localization process based on the plurality of sounds received by each of the plurality of sound receiving devices 1 stored in advance, or the plurality of electrically connected sound receiving devices 1 may be executed. Sound source localization processing may be executed based on a plurality of sounds received in real time by each. Information (for example, position and orientation) regarding a plurality of sound receiving devices 1 is preset in the sound source localization device 2.

By the way, in recent years, AI (Artificial Intelligence) has been installed in various devices such as smartphones, smart speakers, and home appliances, and various machine learning models for voice recognition that recognize the contents spoken by humans have been used in each AI. There is. Devices equipped with AI are becoming widespread, and along with this, the accuracy of AI technology including voice recognition is improved, and the resistance to noise in voice recognition is increasing. Therefore, the inventor of the present application has found that a machine learning model for speech recognition can be diverted to a sound source localization technique.

Specifically, the sound source localization device 2 executes the sound source localization process by using the machine learning model M for voice recognition. The machine learning model M is a model trained to output text information indicating the content of the input voice. The content of the voice indicated by the text information is, for example, the content spoken by a person.

FIG. 2 is a diagram showing an example of the configuration of the machine learning model M. The machine learning model M includes a recurrent neural network (hereinafter referred to as "RNN (Recurrent Neural Network)"). The RNN included in the machine learning model M is a block of long-term short-term memory (hereinafter referred to as “LSTM (Long Short Term Memory)”) or a gated recurrent unit (hereinafter referred to as “GRU (Gated Recurrent Unit)”). May include. The machine learning model M may further include a convolutional neural network (hereinafter referred to as "CNN (Convolutional Neural Network)"). Further, the machine learning model M may further include a hidden Markov model (hereinafter referred to as "HMM (Hidden Markov Model)") and a mixed Gaussian model (hereinafter referred to as "GMM (Gaussian Mixture Model)").

As an example shown in FIG. 2, the machine learning model M has a first convolution layer (hereinafter referred to as “input layer M1”), a first pooling layer M2, a second convolution layer M3, a second pooling layer M4, and a third convolution layer. Layer M5, 4th convolutional layer M6, 1st fully connected layer M7, 2nd fully connected layer M8, 1st recursive layer M9, 2nd recursive layer M10 and CTC (Connectionist Temporal Classification) loss (loss of connectionist temporal classification) Hereinafter, it has an “output layer M11”).

The machine learning model M may further have a gate layer having a function of deleting or adding information. The machine learning model M may have a forgetting gate layer, an input gate layer, and an output gate layer, for example, when the RNN includes a block of LSTM. Further, the machine learning model M may have an update gate layer and a reset gate layer, for example, when the RNN includes a block of GRU. In the present specification, of the two adjacent processing layers, the upstream processing layer when the input voice propagates is referred to as a pre-stage processing layer, and the downstream processing layer is referred to as a post-stage processing layer.

The processing layers that can be the subsequent processing layers are an output layer M11, a recursive layer (second recursive layer M10, first recursive layer M9), a connecting layer (second fully connected layer M8, first fully connected layer M7), and a convolution layer (convolution layer M7). It is any one of a fourth convolution layer M6, a third convolution layer M5, a second convolution layer M3) and a pooling layer (second pooling layer M4, first pooling layer M2). Further, the processing layers that can be the pre-stage processing layer are a recursive layer (second recursive layer M10, first recursive layer M9), a connecting layer (second fully connected layer M8, first fully connected layer M7), and a convolutional layer (fourth). It is any one of a convolution layer M6, a third convolution layer M5, a second convolution layer M3), a pooling layer (second pooling layer M4, first pooling layer M2), and an input layer M1. When the machine learning model M has a gate layer, the processing layer that can be the pre-stage processing layer may be a gate layer. The sound source localization device 2 inputs the acquired first voice and the second voice into the machine learning model M, and forward-propagates each processing layer from the input layer to the output layer, that is, causes the first voice to be inferred. The text information corresponding to and the text information for the second voice are output.

In the example shown in FIG. 1, first, the sound source localization device 2 acquires the first sound received by the first sound receiving device 1a and the second sound received by the second sound receiving device 1b (FIG. 1). (1)). The first voice and the second voice include a target sound (for example, a human voice) emitted from a sound source targeted for sound source localization, and noise emitted from a sound source other than the target sound source. The sound source localization device 2 inputs the acquired first voice and the second voice into the machine learning model M, and propagates a plurality of processing layers included in the machine learning model M ((2) in FIG. 1).

The sound source localization device 2 uses the calculation result in each processing layer that has come to recognize the content of the voice indicated by the text information output by the machine learning model M, that is, the feature amount with a high degree of abstraction by deep learning. The acoustic feature amount common to the first voice and the second voice is detected ((3) in FIG. 1). Here, the sound source localization device 2 detects the common acoustic features in the reverse order of the propagation order. By doing so, the sound source localization device 2 can detect an acoustic feature amount based on a feature amount having a high degree of abstraction.

The sound source localization device 2 is derived from, for example, an acoustic feature amount derived from the target sound included in the first voice and a target sound contained in the second voice by detecting a common acoustic feature amount. It is detected that there is a corresponding relationship with the acoustic features. The correspondence relationship is a relationship in which the target sound included in the first voice indicated by the acoustic feature amount of the first voice and the target sound included in the second voice indicated by the acoustic feature amount of the second voice match or approximate.

The sound source localization device 2 is based on the detected acoustic feature amount corresponding to the first voice and the detected acoustic feature amount corresponding to the second voice, and the sound included in each of the first voice and the second voice (target sound). ) Is estimated ((4) in FIG. 1). Then, the sound source localization device 2 outputs information indicating the estimated position of the sound source ((5) in FIG. 1).

By doing so, the sound source localization system S uses the acoustic feature amount derived from the target sound contained in each of the first voice and the second voice, so that the noise contained in the first voice and the second voice is used. The position of the sound source, which is the source of the target sound, can be estimated without being affected by. As a result, the sound source localization system S can improve the accuracy of the sound source localization.

In the above, the target sound included in each of the first voice and the second voice has been described as a human voice, but the present invention is not limited to this, and the sound corresponds to the content output by the machine learning model M to the text information. All you need is. For example, when the content output by the machine learning model M is a musical note of a staff notation, the target sound may be the sound of a musical instrument.
Hereinafter, the details of the sound source localization device 2 will be described.

[Structure of sound source localization device 2]
FIG. 3 is a diagram showing the configuration of the sound source localization device 2. The sound source localization device 2 has an operation unit 21, a storage unit 22, and a control unit 23.

The operation unit 21 is an input device that accepts user operations. The storage unit 22 is a storage medium such as a ROM (Read Only Memory), a RAM (Random Access Memory), and a hard disk. The storage unit 22 stores various programs executed by the control unit 23.

The control unit 23 is, for example, a CPU (Central Processing Unit). The control unit 23 controls the function related to the sound source localization device 2 by executing the program stored in the storage unit 22. By executing the program, the control unit 23 functions as an acquisition unit 231, a propagation control unit 232, an extraction unit 233, an instruction reception unit 236, a feature amount detection unit 237, a selection unit 238, a sound source localization unit 239, and an output unit 240. do.

The acquisition unit 231 acquires the first sound received by the first sound receiving device 1a and the second sound received by the second sound receiving device 1b. For example, the first voice and the second voice are stored in the storage unit 22, and the user performs an operation to execute the sound source localization process for the first voice and the second voice in the sound source localization device 2. , The acquisition unit 231 acquires the first voice and the second voice stored in the storage unit 22. The acquisition unit 231 inputs the acquired first voice and the second voice to the propagation control unit 232.

The acquisition unit 231 may input the first voice and the second voice converted into a format that can be input to the machine learning model M to the propagation control unit 232. In this case, the acquisition unit 231 may function as a conversion unit that converts voice into a predetermined format. For example, when the format that can be input to the machine learning model M is an image format, the acquisition unit 231 converts the first image obtained by converting the first sound into an image and the second image obtained by converting the second sound into an image. The first voice and the second voice may be input to the propagation control unit 232. The acquisition unit 231 may, for example, convert the acquired voice into an image displaying cepstrum, which is the result of Fourier transforming the spectrum of the voice.

The propagation control unit 232 propagates propagation processing by propagating a plurality of processing layers included in the machine learning model M capable of outputting text information indicating the content of the input voice to each of the first voice and the second voice. To execute. In the example shown in FIG. 2, the propagation control unit 232 propagates each of the processing layers from the input layer M1 to the output layer M11 included in the machine learning model M to each of the first voice and the second voice in order.

The extraction unit 233 includes a unit that is commonly activated in both the processing layer of the post-stage processing layer selected from the plurality of processing layers and the processing layer of the pre-stage processing layer immediately before the post-stage processing layer. By extracting one or more first outputs output from the post-stage processing layer and the pre-stage processing layer based on the above, and one or more second outputs output from the post-stage processing layer and the pre-stage processing layer based on the second sound. Execute the extraction process. Specifically, the extraction unit 233 includes one or more first outputs output from the post-stage processing layer and the pre-stage processing layer based on the first frame included in the first voice, and the second frame included in the second voice. Therefore, one or more second outputs output from the post-stage processing layer and the pre-stage processing layer are extracted based on the second frame corresponding to the first frame.

In the corresponding first frame and the second frame, the first text information output by the machine learning model M by inputting the first voice to the machine learning model M and the second voice are input to the machine learning model M. This is a frame in which the machine learning model M recognizes a part (for example, a character) common to the second text information output by the machine learning model M. For example, when "good morning" is included in the first text information and the second text information as the content of the voice, the corresponding first frame and the second frame are each character (for example, the first character "o". ) Is a frame included in each of the first voice and the second voice that led to the recognition of the machine learning model M.

The details of the extraction process performed by the extraction unit 233 will be described later, but the extraction unit 233 is a part of the subsequent first output and the second output which are a part of the first output commonly activated in the subsequent processing layer. The post-stage extraction unit 234 that extracts a certain post-stage second output, and the pre-stage first output that is a part of the first output that is commonly activated in the pre-stage processing layer and the front-stage second output that is a part of the second output. It has a pre-stage extraction unit 235 for extraction.

The first output and the second output extracted by the extraction unit 233 are information indicating the activated unit among the plurality of units included in the processing layer. The definition of "activation" may be, for example, when the product of the output value of the unit or the output value of the unit and the weight of the connection of the unit exceeds a predetermined threshold value, or a predetermined number or a predetermined number in descending order of output. It may be included in a predetermined ratio. Further, in the processing layer other than the fully connected layer (for example, a convolutional layer), for example, it may be included in a predetermined number or a predetermined ratio in descending order of output for each channel. The channel is the output calculated by convolution for each filter. The definition of "common activation" may be when both the first output and the second output are activated, and the product of the first output and the second output exceeds a predetermined threshold value. It may be a case, or it may be a case where it is included in a predetermined number or a predetermined ratio in descending order.

It is preferable that the extraction unit 233 selects the rearmost treatment layer, which is the last treatment layer, as the post-treatment layer among the plurality of treatment layers. However, there may be no first and second outputs that are commonly activated in the rearmost layer. Therefore, when the last layer is selected as the post-treatment layer among the plurality of treatment layers, the extraction unit 233 has one or more first outputs and one or more second outputs that are commonly activated in the rearmost layers. In the absence of, one or more first outputs and one or more second outputs that are commonly activated in the treatment layer prior to the rearmost layer may be extracted.

For example, when the extraction unit 233 selects the output layer M11, which is the rearmost layer, as the post-processing layer, there is no one or more first outputs and one or more second outputs that are commonly activated in the output layer M11. And. In this case, the extraction unit 233 repeatedly searches for one or more first outputs and one or more second outputs that are commonly activated for each processing layer before the output layer M11. The extraction unit 233 is, for example, when there is one or more first outputs and one or more second outputs that are commonly activated in the second recursive layer M10 which is the processing layer immediately before the output layer M11, the second recursive. The layer M10 is selected as the post-processing layer. Then, the extraction unit 233 extracts one or more first outputs and one or more second outputs that are commonly activated in the second recursive layer M10 selected as the post-processing layer.

Further, in the extraction unit 233, for example, when the machine learning model M is a DNN-HMM hybrid system or a DNN-GMM-HMM tandem system, the output layer unit corresponds to the HMM state, so that the first voice is input. The states common to the plurality of states estimated to be maximum likelihood by inputting and the plurality of states estimated to be maximum likelihood by inputting the second voice may be extracted as the first output and the second output. .. By applying the processing only to the DNN portion in this way, the extraction unit 233 has acoustic features in each of the extraction units 233 even when the machine learning model M is a DNN-HMM hybrid system, a DNN-GMM-HMM tandem system, or the like. Amounts can be associated.

The extraction unit 233 may select the processing layer specified by the user as the subsequent processing layer. Specifically, first, the instruction receiving unit 236 receives an instruction to select a processing layer to be used as the subsequent processing layer from the plurality of processing layers via the operation unit 21. Then, the extraction unit 233 uses the processing layer indicated by the instruction received by the instruction reception unit 236 as the subsequent processing layer. In the example shown in FIG. 2, the extraction unit 233 uses the second recursive layer M10 indicated by the instruction received by the instruction receiving unit 236 as the subsequent processing layer when the user selects the second recursive layer M10. The extraction unit 233 inputs the extracted first output and the second output to the feature amount detection unit 237.

The feature amount detection unit 237 detects one or more first acoustic feature amounts based on one or more first outputs, and detects one or more second acoustic feature amounts based on one or more second outputs. Specifically, the feature amount detection unit 237 first searches for a corresponding acoustic feature amount based on one or more first outputs and one or more second outputs. Then, based on the search result, the feature amount detection unit 237 has one or more first acoustic feature amounts based on one or more first outputs having a corresponding relationship, and one or more first acoustic features based on one or more second outputs. 2 Acoustic features are detected.

The feature amount detection unit 237 is, for example, one or more first acoustics based on one or more first outputs derived from the target sound included in the first sound among one or more first outputs and one or more second outputs. A second acoustic feature based on a feature and one or more second outputs derived from a target sound included in the second sound, and one or more first acoustic features corresponding to each of the one or more first acoustic features. 2 Sound features are detected. In this way, the feature amount detection unit 237 detects one or more first acoustic feature amounts and one or more second acoustic feature amounts in a corresponding relationship, so that the first sound receiving device becomes the first acoustic feature amount. It is possible to specify the time when the corresponding sound is received and the time when the second sound receiving device receives the sound corresponding to the second acoustic feature amount. Further, the feature amount detection unit 237 can prevent the accuracy of sound source localization from being lowered due to the noise contained in the first voice and the second voice by detecting the acoustic feature amount derived from the target sound. can. The feature amount detection unit 237 inputs the detected first acoustic feature amount and the second acoustic feature amount to the selection unit 238.

The selection unit 238 is a part of the first acoustic feature amount and a part based on the mutual correspondence relationship from one or more first acoustic feature amount and one or more second acoustic feature amount detected by the feature amount detection unit 237. Select the second acoustic feature amount of. Specifically, the selection unit 238 removes the erroneously detected correspondence and selects one or more first acoustic features and one or more second acoustic features based on the removed correspondence. The erroneous detection of the correspondence relationship is, for example, a state in which there is a contradiction in the correspondence relationship between the first acoustic feature amount and the second acoustic feature amount. False detection of correspondence is, for example, when the distance to the sound source calculated to estimate the position of the sound source, that is, the value indicating the depth of the sound source becomes a negative number and does not fall within the normal range, or three or more. In the case of estimating the sound source position based on the sound of the above, the value indicating the depth of the same sound source differs remarkably depending on the combination of the sounds.

Further, the false detection of the correspondence relationship may be in a state where a contradiction occurs in the time series of the acoustic features. For example, the first acoustic feature amount A1 which is one of one or more first acoustic feature quantities corresponds to the second acoustic feature amount B1 which is one of one or more second acoustic feature quantities. The first acoustic feature amount A2, which is one of the first acoustic feature amounts of one or more first acoustic feature amounts and is later than the first acoustic feature amount A1 in chronological order, is one or more first acoustic features. It is assumed that it corresponds to the second acoustic feature amount B2, which is one of the feature amounts. In this case, the erroneous detection of the correspondence is when the second acoustic feature amount B1 is ahead of the second acoustic feature amount B2 in time series, or the second acoustic feature amount B1 is from the second acoustic feature amount B2. Even after the time series, the time interval between the second acoustic features B1 and B2 is significantly different from the time interval between the first acoustic features A1 and A2. The selection unit 238 removes the erroneously detected correspondence by, for example, narrowing down based on the RANSAC (Random Sampling Consensus) method or the least squares median (LMedS: Least Median of Square) method.

The sound source localization unit 239 estimates the position of the sound source, which is the source of the sound contained in each of the first voice and the second voice, based on one or more first acoustic features and one or more second acoustic features. do. The sounds included in the first voice and the second voice are the sounds corresponding to the contents of the voice output by the machine learning model M in the text information, and are the sounds included in both the first voice and the second voice.

Specifically, first, the sound source localization unit 239 is the depth of the sound source which is the source of the first sound and the second sound based on one or more first acoustic features and one or more second acoustic features. To estimate. Then, the sound source localization unit 239 estimates the position of the sound source in the three-dimensional space based on the estimated depth of the sound source.

The sound source localization unit 239 has a time when the first sound receiving device receives a sound corresponding to the first acoustic feature amount and a time when the second sound receiving device receives a sound corresponding to the second acoustic feature amount. Based on the relationship and the relationship between the position of the first sound receiving device and the position of the second sound receiving device, the position of the sound source that is the source of the sound contained in each of the first sound and the second sound is estimated. May be good. The sound source localization unit 239 is, for example, the time when the first sound receiving device receives the sound corresponding to the first acoustic feature amount and the time when the second sound receiving device receives the sound corresponding to the second acoustic feature amount. The position of the sound source may be estimated based on the bicurved surface calculated based on the difference between the sound source and the relative positions of the first sound receiving device and the second sound receiving device.

Further, when the sound source localization unit 239 has three sound receiving devices as a plurality of sound receiving devices, the difference in the time when each of the three sound receiving devices receives the sound corresponding to the acoustic feature amount and the three sound receiving devices. The position of the sound source may be estimated based on the intersection line (hyperboloid) of the two hyperboloids calculated based on the relative position of the sound receiving device. Further, in the sound source localization unit 239, when there are four sound receiving devices as a plurality of sound receiving devices, the difference in the time when each of the four sound receiving devices receives the sound corresponding to the acoustic feature amount and the four sound receiving devices. The position of the sound source may be estimated based on the intersection of the three hyperboloids calculated based on the relative position of the sound receiving device.

The sound source localization unit 239 estimates the position of the sound source based on a part of the first acoustic feature amount and a part of the second acoustic feature amount based on the correspondence after the selection unit 238 removes the false detection. May be good. The sound source localization unit 239 may estimate the position of the sound source by using a known technique.

The output unit 240 outputs information indicating the position of the sound source estimated by the sound source localization unit 239.

[Extraction process]
Subsequently, the extraction process performed by the extraction unit 233 will be described. As described above, the extraction unit 233 has a rear stage extraction unit 234 and a front stage extraction unit 235. The post-stage extraction unit 234 has a plurality of post-stage first outputs and second voices output from the post-stage processing layer because the first voice propagates in the order of the pre-stage processing layer and the post-stage processing layer which are a part of the plurality of processing layers. From the plurality of posterior second outputs output from the posterior processing layer due to the propagation of Is extracted. Specifically, the post-stage extraction unit 234 has a correspondence relationship with the first frame in the plurality of post-stage first outputs and the second sound output from the post-stage processing layer based on the first frame included in the first sound. From the plurality of post-stage second outputs output from the post-stage processing layer based on the second frame, one or more post-stage first outputs and one or more post-stage second outputs that are commonly activated are extracted.

The pre-stage extraction unit 235 is output from a plurality of pre-stage first outputs output from the pre-stage processing layer that has become a factor for activating one or more post-stage first outputs and one or more post-stage second outputs, and from the pre-stage processing layer. Of the plurality of pre-stage second outputs, one or more pre-stage first outputs and one or more pre-stage second outputs that are commonly activated are extracted.

4 to 7 are diagrams for explaining the extraction process performed by the extraction unit 233. 4 to 6 show a state of propagation from the pre-processed layer to the post-processed layer. In FIGS. 4 to 7, the connecting line connecting the units shown by the solid line indicates that the unit to be connected was activated, and the connecting line shown by the broken line indicates that the unit to be connected was not activated. In addition, the thickness of the line indicating the binding line indicates the magnitude of activation of the unit to be bound.

In the example shown in FIG. 4, the treatment layer M20 is a post-stage treatment layer, and the treatment layer M19 is a front-stage treatment layer. FIG. 4A is a state before extraction, and FIG. 4B is a state after extraction. In the first voice, units U1 and U2 are activated in the processing layer M20, and units U1 and U2 are activated in the processing layer M19. In the second voice, the unit U2 is activated in the processing layer M20, and the unit U2 is activated in the processing layer M19.

In this case, the post-stage extraction unit 234 is used in the units U1, U2, and the second voice, which are the rear-stage first outputs output from the processing layer M20, which is the post-stage processing layer, based on the first frame included in the first voice. The unit U2, which is the second output in the subsequent stage, output from the processing layer M20 based on the second frame corresponding to the first frame is compared. Then, the rear stage extraction unit 234 extracts the unit U2 of the rear stage first output and the unit U2 of the rear stage second output which are commonly activated.

Subsequently, the front-stage extraction unit 235 has the units U1, U2, and the rear-stage first output, which are the front-stage first outputs, output from the front-stage processing layer M19, which is a factor that activates the unit U2 of the rear-stage first output. The unit U2 which is the second output of the previous stage output from the processing layer M19 which became a factor for activating the unit U2 of two outputs is compared. Then, the pre-stage extraction unit 235 extracts the unit U2 of the pre-stage first output and the unit U2 of the pre-stage second output that are commonly activated.

The pre-stage extraction unit 235 extracts one or more pre-stage first outputs and one or more pre-stage second outputs based on the activated size of the plurality of pre-stage first outputs and the plurality of pre-stage second outputs. You may. Even if the pre-stage extraction unit 235 extracts, for example, one or more pre-stage first outputs and one or more pre-stage second outputs that are most activated among the plurality of pre-stage first outputs and the plurality of pre-stage second outputs. good.

When the extraction unit 233 extracts the output from the processing layer M20 to the processing layer M19, the extraction unit 233 extracts the output for the next processing layer. Specifically, the extraction unit 233 repeats the process of extracting the first output and the second output that are commonly activated for each processing layer in the reverse order of the order propagated by the propagation control unit 232. conduct. More specifically, the extraction unit 233 selects one of the plurality of processing layers as the post-processing layer, extracts one or more first outputs and one or more second outputs, and then serves as the pre-processing layer. The selected processing layer is selected as the subsequent processing layer, and another one or more first outputs and one or more second outputs are extracted. By doing so, the extraction unit 233 can improve the accuracy of comparison with respect to the first voice and the second voice.

The extraction unit 233 has one or more first outputs output from the post-stage processing layer and the pre-stage processing layer based on each of the plurality of first frames included in the first voice, and the second frame included in the second voice. One or more second outputs output from the post-stage processing layer and the pre-stage processing layer may be extracted based on the second frame corresponding to one frame.

FIG. 5 shows a state in which the first frame n and the first frame n-1 in the first voice propagate from the pre-stage processing layer to the post-stage processing layer. FIG. 6 shows a state in which the second frame m and the second frame m-1 in the second voice propagate from the pre-stage processing layer to the post-stage processing layer. The first frame n and the second frame m are the first frame and the second frame having a corresponding relationship, and the first frame n-1 and the second frame m-1 are the first frame and the second frame having a corresponding relationship. This is the frame immediately before each frame. In the case of FIGS. 5 and 6, the processing layer M18 of the first frame n and the second frame m is a post-stage processing layer, and the processing layers M18 and the first frame of the first frame n-1 and the second frame m-1. The processing layer M17 of n and the second frame m is a pre-stage processing layer. 5 (a) and 6 (a) are states before extraction, and FIGS. 5 (b) and 6 (b) are states after extraction.

In the example shown in FIG. 5, the units U11 and U12 are activated in the processing layer M18 of the first frame n in the first voice. In the example shown in FIG. 6, the unit U12 is activated in the processing layer M18 of the second frame m in the second voice.

In this case, the post-stage extraction unit 234 is the unit U11, U12, which is the second-stage first output output from the processing layer M18, which is the post-stage processing layer in the first frame n of the first voice, and the first frame in the second voice. The unit U12 which is the second output in the subsequent stage output from the processing layer M18 in the second frame m corresponding to n is compared. Then, the rear stage extraction unit 234 extracts the unit U12 of the rear stage first output and the unit U12 of the rear stage second output which are commonly activated.

Subsequently, the pre-stage extraction unit 235 is output from the pre-stage first stage processing layer M18 of the first-stage processing layer, which is the pre-stage processing layer that has become a factor for activating the rear-stage first output unit U12 in the first stage stage n. Units U11 and U12 that are one output, and the rear stage in the second frame m The front stage that was output from the processing layer M18 of the second frame m-1 that was the factor that activated the unit U12 of the second output. The unit U12, which is the second output, is compared. Then, the subsequent stage extraction unit 234 extracts the unit U12 of the first output of the previous stage and the unit U12 of the second output of the previous stage which are commonly activated.

Further, the front-stage extraction unit 235 is a front-stage first output output from the processing layer M17 of the first-stage processing layer n, which is a factor that activates the unit U12 of the rear-stage first output in the first frame n. It is the front second output output from the processing layer M17 of the second frame m, which is the front processing layer that has become a factor for activating the unit U12 of the second output of the second frame in the units U11, U12, and the second frame m. Compare units U12. Then, the pre-stage extraction unit 235 extracts the unit U12 of the pre-stage first output and the unit U12 of the pre-stage second output that are commonly activated.

When the gate layer is included in the post-processing layer from which the post-stage extraction unit 234 has extracted one or more rear-stage first outputs and one or more post-stage second outputs, the front-stage extraction unit 235 is commonly activated in the gate layer. The first output of the previous stage and the second output of the previous stage may be extracted.

In the example shown in FIG. 7, the unit U22 is the first output and the second output extracted by the subsequent extraction unit 234, the gate G2 is an output gate layer, and the cell C2 is a memory cell. FIG. 7A is a state before extraction, and FIG. 7B is a state after extraction. In the first voice, the gate G2 and the cell C2 are activated. In the second voice, cell C2 is activated.

In this case, the front-stage extraction unit 235 uses the gate G2 and the cell C2, which are factors that activate the unit U22 of the rear-stage first output, and the cell C2, which is a factor that activates the unit U22 of the rear-stage second output. Compare. Then, the pre-stage extraction unit 235 extracts the cell C2 of the first output of the pre-stage and the cell C2 of the second output of the pre-stage that are commonly activated.

It is preferable that the extraction unit 233 repeats the above-mentioned extraction process up to the input layer. However, the extraction unit 233 may end the extraction process at an intermediate process layer (for example, a pooling layer or a convolution layer) without performing the extraction process up to the first process layer. As described above, the extraction unit 233 can extract the feature amount having a high degree of abstraction by performing the extraction process in the order opposite to the order propagated by the propagation control unit 232.

[Processing of sound source localization device 2]
Subsequently, the flow of processing performed by the sound source localization device 2 will be described. FIG. 8 is a flowchart showing the flow of processing performed by the sound source localization device 2. This flowchart starts, for example, when the first voice and the second voice stored in the storage unit 22 are acquired (S1). The acquisition unit 231 inputs the acquired first voice and the second voice to the propagation control unit 232.

The propagation control unit 232 transfers a plurality of processing layers from the input layer M1 to the output layer M11 included in the machine learning model M to each of the first voice and the second voice input from the acquisition unit 231 from the input layer M1. Propagate in order (S2). The extraction unit 233 performs a process of extracting one or more first outputs and one or more second outputs that are commonly activated in both the treatment layers of the post-stage treatment layer and the front-stage treatment layer (S3).

FIG. 9 is a flowchart showing the flow of processing performed by the extraction unit 233. The extraction unit 233 determines whether or not the instruction receiving unit 236 has received an instruction to select a processing layer to be used as the subsequent processing layer from the plurality of processing layers via the operation unit 21 (S31).

When the extraction unit 233 determines that the instruction reception unit 236 has received the instruction (YES in S31), the extraction unit 233 uses the processing layer indicated by the instruction received by the instruction reception unit 236 as the subsequent processing layer (S32). For example, when the extraction unit 233 determines that the instruction receiving unit 236 has received the instruction indicating the second recursive layer M10, the extraction unit 233 selects the second recursive layer M10 indicated by the instruction received by the instruction receiving unit 236 as the subsequent processing layer. .. On the other hand, when the extraction unit 233 determines that the instruction reception unit 236 has not received the instruction (NO in S31), the extraction unit 233 has one or more first layers that are commonly activated in the rearmost layer (for example, the output layer M11). It is determined whether or not there is one output and one or more second outputs (S33).

When the extraction unit 233 determines that there is one or more first outputs and one or more second outputs that are commonly activated in the second recursive layer M10 (YES in S33), the output that is the last layer. The layer M11 is used as a post-processing layer (S34). On the other hand, when the extraction unit 233 determines that there is no one or more first outputs and one or more second outputs commonly activated in the second recursive layer M10 (NO in S33), the second recursive layer. For each treatment layer prior to M10, one or more first outputs and one or more second outputs that are commonly activated are repeatedly searched. Then, the extraction unit 233 uses a processing layer having one or more first outputs and one or more second outputs (for example, the first recursive layer M9) that are commonly activated as the subsequent processing layer (S35). .. The extraction unit 233 is one or more firsts output from the post-processing layer and the pre-processing layer based on the first voice, which are commonly activated in both the selected post-processing layer and the pre-processing layer. Based on the output and the second voice, one or more second outputs output from the post-stage processing layer and the pre-stage processing layer are extracted.

Specifically, first, the post-stage extraction unit 234 is commonly active from a plurality of post-stage first outputs output from the selected post-stage processing layer and a plurality of post-stage second outputs output from the selected post-stage processing layer. One or more posterior first outputs and one or more posterior second outputs are extracted (S36). Then, the front-stage extraction unit 235 has a plurality of front-stage first outputs output from the front-stage processing layer that has become a factor for activating one or more rear-stage first outputs and one or more rear-stage second outputs extracted by the rear-stage extraction unit 234. Of the outputs and the plurality of pre-stage second outputs output from the pre-stage processing layer, one or more pre-stage first outputs and one or more pre-stage second outputs that are commonly activated are extracted (S37).

Subsequently, the extraction unit 233 determines whether or not there is another processing layer before the pre-stage processing layer (S38). When the extraction unit 233 determines that there is another processing layer (for example, the second fully connected layer M8) before the previous stage processing layer (for example, the first recursive layer M9) (YES in S38), the first The recursive layer M9 is used as a post-processing layer (S39), and the processing is returned to S46. On the other hand, when the extraction unit 233 determines that there is no other processing layer before the pre-stage processing layer (for example, the input layer M1) (NO in S38), the extracted one or more first outputs and one or more are extracted. The second output is input to the feature amount detection unit 237, and the extraction process is completed.

Returning to FIG. 8, the feature amount detection unit 237 searches for the corresponding acoustic feature amount based on one or more first outputs and one or more second outputs, and based on the search result, the correspondence relationship is established. One or more first acoustic features based on one or more first outputs and one or more second acoustic features based on one or more second outputs are detected (S4). Subsequently, the selection unit 238 determines whether or not there is a corresponding relationship erroneously detected between the first acoustic feature amount and the second acoustic feature amount detected by the feature amount detection unit 237 (S5). The selection unit 238 narrows down based on, for example, the RANSAC method.

When the selection unit 238 determines that the first acoustic feature amount and the second acoustic feature amount have a erroneously detected correspondence relationship (YES in S5), the first acoustic feature amount and the first acoustic feature amount having the erroneously detected correspondence relationship are present. 2 The acoustic features are removed (S6), and a part of the first acoustic features and a part of the second acoustic features based on the correspondence after the removal are selected.

The sound source localization unit 239 has a correspondence relationship in which the selection unit 238 determines that there is no erroneously detected correspondence between the first acoustic feature amount and the second acoustic feature amount (NO in S5) or erroneously detected. After removing the first acoustic feature amount and the second acoustic feature amount, the sound included in each of the first sound and the second sound based on one or more first acoustic feature amounts and one or more second acoustic feature amounts. The position of the sound source, which is the source of the above, is estimated (S7). Then, the output unit 240 outputs information indicating the position of the sound source estimated by the sound source localization unit 239 (S8).

[Effects in this embodiment]
As described above, the sound source localization system S propagates a plurality of processing layers included in the machine learning model M to each of the acquired first voice and the second voice. The sound source localization system S produces one or more first outputs and one or more second outputs that are commonly activated in both the processing layers of the post-stage processing layer and the front-stage processing layer in the reverse order of the propagation order. Each processing layer is extracted, and one or more first acoustic features and one or more second acoustic features that are in a corresponding relationship are detected, respectively. Then, the sound source localization system S is a sound source that is a source of sound included in each of the first voice and the second voice based on the detected one or more first acoustic features and one or more second acoustic features. The position is estimated, and information indicating the estimated position of the sound source is output.

By doing so, the sound source localization system S uses the acoustic feature amount derived from the target sound contained in each of the first voice and the second voice, so that the noise contained in the first voice and the second voice is used. The position of the sound source, which is the source of the target sound, can be estimated without being affected by. As a result, the sound source localization system S can improve the accuracy of the sound source localization.

The sound source localization system S uses, for example, multiple listening point sounds received by a plurality of sound receiving devices 1, and obtains information indicating the direction of a specific sound source to obtain the sound emitted from the specific sound source. You can distinguish it from other voices. Further, the sound source localization device 2 can remove noise by suppressing the sound in a direction different from that of the sound source to be heard, for example.

Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes can be made within the scope of the gist thereof. be. For example, all or part of the device can be functionally or physically distributed / integrated in any unit. Also included in the embodiments of the present invention are new embodiments resulting from any combination of the plurality of embodiments. The effect of the new embodiment produced by the combination has the effect of the original embodiment together.

1 Sound receiving device 2 Sound source localization device 21 Operation unit 22 Storage unit 23 Control unit 231 Acquisition unit 232 Propagation control unit 233 Extraction unit 234 Later stage extraction unit 235 Front stage extraction unit 236 Instruction reception unit 237 Feature amount detection unit 238 Selection unit 239 Sound source localization Part 240 Output part

Claims (14)

An acquisition unit that acquires the first sound received by the first sound receiving device and the second sound received by the second sound receiving device installed at a position different from that of the first sound receiving device.
A propagation control unit that propagates a plurality of processing layers included in a machine learning model capable of outputting text information indicating the content of the input voice to each of the first voice and the second voice.
Based on the first voice, which comprises a unit that is commonly activated in both the post-treatment layer selected from the plurality of treatment layers and the pre-treatment layer which is the treatment layer immediately before the post-treatment layer. One or more first outputs output from the post-stage processing layer and the pre-stage processing layer, and one or more second outputs output from the post-stage processing layer and the pre-stage processing layer based on the second voice are extracted. Extractor and
A feature amount detection unit that detects one or more first acoustic features based on the one or more first outputs and detects one or more second acoustic features based on the one or more second outputs.
A sound source that estimates the position of a sound source that is a source of sound contained in each of the first voice and the second voice based on the one or more first acoustic features and the one or more second acoustic features. Localization part and
An output unit that outputs information indicating the position of the sound source estimated by the sound source localization unit, and an output unit.
Sound source localization device with. The sound source localization unit receives a time when the first sound receiving device receives a sound corresponding to the first acoustic feature amount and a sound received by the second sound receiving device corresponding to the second acoustic feature amount. The position of the sound source is estimated based on the relationship between the time and the position of the first sound receiving device and the position of the second sound receiving device.
The sound source localization device according to claim 1. The extraction unit
The plurality of post-stage first outputs and the second voice output from the post-stage processing layer due to the propagation of the first voice in the order of the pre-stage processing layer and the post-stage processing layer which are a part of the plurality of processing layers are the front stages. One or more post-stage first outputs and one or more post-stage second outputs that are commonly activated from the plurality of post-stage second outputs output from the post-stage treatment layer by propagating in the order of the treatment layer and the post-stage treatment layer. The post-stage extraction unit to be extracted and
A plurality of pre-stage first outputs output from the pre-stage processing layer, and a plurality of outputs from the pre-stage processing layer, which are factors that activate the one or more post-stage first outputs and the one or more post-stage second outputs. Of the first-stage second outputs of the above, one or more pre-stage first outputs and one or more pre-stage second outputs that are commonly activated, and a pre-stage extraction unit.
Have,
The sound source localization device according to claim 1 or 2. The pre-stage extraction unit has one or more pre-stage first outputs and one or more pre-stage second outputs based on the activated size of the plurality of pre-stage first outputs and the plurality of pre-stage second outputs. Extract the output,
The sound source localization device according to claim 3. The machine learning model includes a recurrent neural network.
The post-treatment layer is any one of an output layer, a recursive layer, a fully connected layer, a pooling layer, and a convolutional layer.
The sound source localization device according to claim 3 or 4. The pretreatment layer is any one of a recursive layer, a fully connected layer, a pooling layer, a convolution layer, and an input layer.
The sound source localization device according to any one of claims 3 to 5. When the last layer, which is the last treatment layer, is selected as the subsequent treatment layer among the plurality of treatment layers, the extraction unit has one or more first layers that are commonly activated in the last treatment layer. When there is no output and the one or more second outputs, the one or more first outputs and the one or more second outputs that are commonly activated in the processing layer before the rearmost layer are extracted.
The sound source localization device according to any one of claims 1 to 6. From the one or more first acoustic features and the one or more second acoustic features specified by the feature detection unit, a part of the first acoustic features and a part of the second acoustic features based on the mutual correspondence relationship. It also has a selection unit for selecting acoustic features.
The sound source localization unit estimates the position of the sound source based on the first acoustic feature amount of the part and the second acoustic feature amount of the part.
The sound source localization device according to any one of claims 1 to 7. Further having an instruction receiving unit for receiving an instruction to select a processing layer to be used as the subsequent processing layer among the plurality of processing layers.
The extraction unit uses the processing layer indicated by the instruction received by the instruction receiving unit as the post-stage processing layer.
The sound source localization device according to any one of claims 1 to 8. The extraction unit selects one of the plurality of processing layers as the post-processing layer, extracts the one or more first outputs and the one or more second outputs, and then selects the pre-processing layer. The processed layer is selected as the subsequent processing layer, and another one or more first outputs and one or more second outputs are extracted.
The sound source localization device according to any one of claims 1 to 9. The step of acquiring the first sound received by the first sound receiving device, and
A step of acquiring a second sound received by a second sound receiving device installed at a position different from that of the first sound receiving device, and
A step of propagating a plurality of processing layers included in a machine learning model capable of outputting text information indicating the content of the input voice to each of the first voice and the second voice.
Based on the first voice, which comprises a unit that is commonly activated in both the post-treatment layer selected from the plurality of treatment layers and the pre-treatment layer which is the treatment layer immediately before the post-treatment layer. One or more first outputs output from the post-stage processing layer and the pre-stage processing layer, and one or more second outputs output from the post-stage processing layer and the pre-stage processing layer are extracted based on the second voice. Steps and
A step of detecting one or more first acoustic features based on the one or more first outputs and detecting one or more second acoustic features based on the one or more second outputs.
A step of estimating the position of a sound source that is a source of sound included in each of the first voice and the second voice based on the one or more first acoustic features and the one or more second acoustic features. When,
A step to output information indicating the estimated position of the sound source, and
Sound source localization method having. The extraction step is
The plurality of post-stage first outputs and the second voice output from the post-stage processing layer due to the propagation of the first voice in the order of the pre-stage processing layer and the post-stage processing layer which are a part of the plurality of processing layers are the front stages. One or more post-stage first outputs and one or more post-stage second outputs that are commonly activated from the plurality of post-stage second outputs output from the post-stage treatment layer by propagating in the order of the treatment layer and the post-stage treatment layer. The post-extraction step to extract and
A plurality of pre-stage first outputs output from the pre-stage processing layer, and a plurality of outputs from the pre-stage processing layer, which are factors that activate the one or more post-stage first outputs and the one or more post-stage second outputs. Of the first-stage second outputs of the above, one or more previous-stage first outputs and one or more previous-stage second outputs that are commonly activated are extracted.
The sound source localization method according to claim 11. After executing the pre-stage extraction step, the post-stage extraction step is performed by using the one or more pre-stage first outputs and the one or more pre-stage second outputs as the plurality of post-stage first outputs and the plurality of post-stage second outputs. Execute,
The sound source localization method according to claim 12. The latter-stage extraction step and the first-stage extraction step are executed for each of the plurality of processing layers.
The sound source localization method according to claim 12 or 13.

Images