JP7721089B2 - Sound processing device, sound processing method and program - Google Patents

Description

The present invention relates to an audio processing device, an audio processing method, and a program.

Microphone array processing is a fundamental technology in acoustic signal processing. Microphone array processing uses multi-channel acoustic signals collected using a microphone array, and examples include sound source localization and sound source separation. Sound source localization is a method for estimating the direction of a sound source from multi-channel acoustic signals. Sound source separation is a method for extracting components arriving from individual sound sources from multi-channel acoustic signals. Sound source localization and sound source separation are useful for identifying individual sounds, such as when speech is spoken in a noisy environment or when multiple sound sources are present. Sound source localization and sound source separation are applied to a variety of applications, including robot audition, smart speakers, teleconferencing systems, and meeting minutes creation.

Microphone array processing uses an acoustic transfer function that indicates the transfer characteristics of sound from a sound source to a sound receiving point. The sound source transfer function may be calculated geometrically using a mathematical model assuming a free sound field, or may be measured in advance in a laboratory using sound sources installed in multiple directions. However, such an acoustic transfer function differs from that measured in the acoustic environment in which microphone array processing is actually used (sometimes referred to as the "usage environment" in this application). This can result in a decrease in the performance of sound source localization or sound source separation. It may be possible to measure the acoustic transfer function in advance in the usage environment to ensure the performance of sound source localization or sound source separation. The acoustic transfer function changes from the time of measurement as the acoustic environment changes, but re-measuring the acoustic transfer function requires a lot of time and effort. Therefore, re-measuring the acoustic transfer function in the usage environment is not practical.

Kazuhiro Nakadai, Masayuki Takigahira, Kumasuke Kawai, and Hiroshi Nakajima, "Improvement of Sound Source Localization and Separation by Continuous Online Adaptation of Transfer Functions," Japanese Society for Artificial Intelligence Second Type Research Meeting Materials, AI Challenge Research Group SIG-Challenge-058-07, <https://doi.org/10.11517/jsaisigtwo.2021.Challenge-058_07>, published December 16, 2021.

Non-Patent Document 1 describes a method for estimating and sequentially updating an acoustic transfer function using acoustic signals acquired by a microphone array. This method makes it possible to acquire an acoustic transfer function online using any sound source, without the need to set up equipment for remeasurement. However, acoustic signals acquired in the usage environment are not necessarily suitable for acquiring an acoustic transfer function. For example, significant noise may be temporarily mixed into the acoustic signal. As a result, it may not be possible to stably estimate the acoustic transfer function.

This embodiment has been made in consideration of the above points, and aims to provide an acoustic processing device, an acoustic processing method, and a program that can stably estimate an acoustic transfer function in the usage environment.

(1) The present application has been made to solve the above-mentioned problems, and one aspect of this embodiment is an audio processing device comprising: a memory unit that stores a first acoustic transfer function for each sound source direction; a sound source direction estimation unit that calculates a spatial spectrum for each sound source direction for each frame based on the transform coefficients in the frequency domain of the acoustic signal for each channel and the first acoustic transfer function, and estimates the sound source direction where the spatial spectrum is maximum as the estimated sound source direction; a representative estimated sound source direction determination unit that determines a representative estimated sound source direction that is a representative value of the estimated sound source directions based on the frequency distribution of the estimated sound source directions in an observation period consisting of a plurality of frames; an outlier removal unit that removes transform coefficients of frames where the estimated sound source direction falls outside a predetermined tolerance range from the representative estimated sound source direction; an acoustic transfer function estimation unit that estimates a representative value of the acoustic transfer function from the sound source to the sound collection unit for the acoustic signal during the observation period as a second acoustic transfer function based on the transform coefficients of the acoustic signal in the remaining frames; and an acoustic transfer function update unit that updates the first acoustic transfer function for the representative estimated sound source direction using the second acoustic transfer function.

(2) Another aspect of the present embodiment is the sound processing device of (1), wherein the representative estimated sound source direction determiner may determine the estimated sound source direction with a maximum frequency as the representative estimated sound source direction.

(3) Another aspect of this embodiment is the sound processing device of (1), wherein the acoustic transfer function update unit may update the weighted average of the first acoustic transfer function and the second acoustic transfer function to a new first acoustic transfer function.

(4) Another aspect of this embodiment is the sound processing device of (1), wherein the acoustic transfer function update unit determines the reliability of the representative estimated sound source direction based on the frequency with which the estimated sound source direction during the observation period falls within the tolerance range, and may increase the ratio of the second acoustic transfer function to the first acoustic transfer function as the reliability increases.

(5) Another aspect of this embodiment is the sound processing device of (1), in which the allowable range is equal to the representative estimated sound source direction and does not need to include directions different from the representative estimated sound source direction.

(6) Another aspect of this embodiment is the sound processing device of (1), wherein the sound source direction estimation unit may calculate the spatial spectrum by multiplying an input vector including the transformation coefficient for each channel by a pseudo-inverse matrix of an acoustic transfer function vector including the first acoustic transfer function for each channel.

(7) Another aspect of this embodiment may be a program for causing a computer to function as the sound processing device of (1).

(8) Another aspect of this embodiment is an acoustic processing method in an acoustic processing device including a storage unit that stores a first acoustic transfer function for each sound source direction, wherein the acoustic processing device executes the following acoustic processing steps: a sound source direction estimation step for calculating a spatial spectrum for each sound source direction for each frame based on the transform coefficients in the frequency domain of the acoustic signal for each channel and the first acoustic transfer function, and estimating the sound source direction for which the spatial spectrum is maximum as the estimated sound source direction; a representative sound source direction determination step for determining a representative estimated sound source direction that is a representative value of the estimated sound source directions based on the frequency distribution of the estimated sound source directions in an observation period consisting of a plurality of frames; an outlier removal step for removing transform coefficients of frames in which the estimated sound source direction falls outside a predetermined tolerance range from the representative estimated sound source direction; an acoustic transfer function estimation step for estimating a representative value of the acoustic transfer function from the sound source to the sound collection unit for the acoustic signal in the observation period as a second acoustic transfer function based on the transform coefficients of the acoustic signal in the remaining frames; and an acoustic transfer function updating step for updating the first acoustic transfer function for the representative estimated sound source direction using the second acoustic transfer function.

According to this embodiment, it is possible to stably estimate an acoustic transfer function in a real acoustic environment.
According to the above-described configurations (1), (7), and (8), the second acoustic transfer function is calculated based on the conversion coefficient of the acoustic signal that gives an estimated sound source direction within a predetermined range from the representative estimated sound source direction, and the calculated second acoustic transfer function can be used to update the first acoustic transfer function in association with the representative estimated sound source direction. A representative value of the acoustic transfer function obtained based on the acoustic signal that statistically gives the representative estimated sound source direction or an estimated sound source direction approximate thereto is updated as the second acoustic transfer function in association with the representative estimated sound source direction, so that a first acoustic transfer function with a stable correspondence relationship with the sound source direction can be obtained.

According to the configuration (2) described above, the estimated sound source direction with the highest frequency within the observation period is determined as the representative estimated sound source direction, so the most likely estimated sound source direction is simply determined as the representative estimated sound source direction.

According to the configuration (3) described above, when the observation period is changed, the first acoustic transfer function is not completely replaced by the second acoustic transfer function through updating, and some of its components remain. This avoids sudden fluctuations in the first acoustic transfer function, thereby ensuring system stability.

The configuration (4) described above allows the first acoustic transfer function to be updated using the second acoustic transfer function, with emphasis being placed on acoustic signals that provide more reliable estimated sound source directions. This improves the reliability of the updated first acoustic transfer function.

The above-described configuration (5) makes it possible to simply determine whether or not to exclude the conversion coefficient of the acoustic signal that gives the estimated sound source direction, depending on whether the estimated sound source direction is equal to the representative estimated sound source direction.

The configuration (6) described above makes it possible to estimate the sound source direction based on the spatial spectrum calculated by a simple matrix operation. Since it does not require many computational resources, it can be implemented economically.

1 is a schematic block diagram illustrating an example of the configuration of a sound processing system according to an embodiment of the present invention. 10 is a data flowchart showing an example of acoustic transfer function adaptation processing according to the present embodiment. FIG. 2 is a diagram illustrating an example of the configuration of a sound collection unit. FIG. 2 is an explanatory diagram illustrating an acoustic transfer function. FIG. 1 illustrates a laboratory. FIG. 10 is a diagram illustrating an example of a success rate for each observation period. FIG. 10 is a diagram illustrating an example of success rates for each type of first acoustic transfer function. FIG. 10 is a schematic block diagram showing an example of the configuration of a sound processing system according to a first modified example of the present embodiment. FIG. 10 is a schematic block diagram showing an example of the configuration of a sound processing system according to a second modified example of the present embodiment.

(First embodiment)
A first embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a schematic block diagram showing an example of the configuration of a sound processing system S1 according to this embodiment.
The sound processing system S1 includes a sound processing device 10 and a sound collection unit 20.

The sound processing device 10 stores a first acoustic transfer function, which indicates the transfer characteristics of sound from a sound source, for each sound source direction in advance. In this application, the acoustic transfer function stored in advance or temporarily in the sound processing device 10 is referred to as the "first acoustic transfer function," and the acoustic transfer function estimated based on the acoustic signal acquired from the sound collection unit 20 is referred to as the "second acoustic transfer function," thereby distinguishing between the two.

The sound processing device 10 acquires multi-channel acoustic signals from the sound collection unit 20. The sound processing device 10 calculates a conversion coefficient in the frequency domain of the acoustic signal for each channel for each frame, and calculates a spatial spectrum for each sound source direction based on the calculated conversion coefficient and the first acoustic transfer function. The sound processing device 10 estimates the sound source direction where the spatial spectrum has a local maximum as the estimated sound source direction (sound source localization). The sound processing device 10 accumulates the estimated sound source direction for each frame and the acoustic signals used to estimate the estimated sound source direction, and generates a frequency distribution (histogram) of the estimated sound source direction over an observation period consisting of multiple frames. The sound processing device 10 determines a representative value of the estimated sound source direction as the representative estimated sound source direction based on the frequency distribution of the estimated sound source direction.

The sound processing device 10 identifies frames of the acoustic signal that provide an estimated sound source direction that is outside a predetermined tolerance range from the representative estimated sound source direction, and removes the transformation coefficients of the acoustic signal of those frames as outliers. The sound processing device 10 calculates an acoustic transfer function from the sound source to the sound collection unit 20 based on the transformation coefficients that remain. The sound processing device 10 estimates a representative value of the acoustic transfer function during the observation period as a second acoustic transfer function. The sound processing device 10 updates the first acoustic transfer function for the determined representative estimated sound source direction using the estimated second acoustic transfer function.

The sound processing device 10 may perform microphone array processing using the updated first acoustic transfer function. Microphone array processing may include processing such as sound source separation in addition to sound source localization. Sound source separation includes processing of extracting sound components from individual sound sources as sound source components from the multi-channel acoustic signals acquired from the sound collection unit 20 based on the estimated sound source direction.

The sound processing device 10 may use one or both of the sound source direction estimated by performing sound source localization and the sound source signal indicating the sound source components extracted by performing sound source separation for other processing within the device itself, or may output them to another device (not shown, sometimes referred to as an "output destination device" in this application). As another process, the sound processing device 10 may, for example, estimate the presence of an object in the estimated sound source direction. The sound processing device 10 may perform speech recognition processing on the sound source components or sound source signal from a specific sound source direction (speaker), obtain spoken text indicating the content of the utterance, or estimate the speaker.

The sound collection unit 20 has multiple microphones 20-1 to 20-M and functions as a microphone array. The number of microphones, M, is an integer greater than or equal to two. The number of microphones, M, corresponds to the number of channels. Each microphone is placed at a different position and has an actuator that collects sound waves arriving at that microphone. The actuator converts the arriving sound waves into an acoustic signal. The converted acoustic signal is output to the sound processing device 10 wirelessly or via a wired connection. Each microphone corresponds to a channel of the acoustic signal.

The arrangement of the multiple microphones may be fixed or variable. The positions of the multiple microphones need only be different from one another. The sound collection unit 20 illustrated in FIG. 3 is configured as an eight-channel circular microphone array. In FIG. 3, individual microphones are indicated by black circles. The eight microphones are arranged at equal intervals around the circumference. The eight microphones are arranged on the side of a housing that forms a body of revolution that is rotationally symmetrical about a rotation axis parallel to the vertical direction, and their relative positions are fixed. The sound collection unit 20 includes an output interface (not shown). The output interface aggregates the eight-channel acoustic signals recorded by the individual microphones and outputs them in parallel via a wired connection to the sound processing device 10. However, the number and arrangement of microphones are not limited to this. The number M of microphones may be two to seven or less, or nine or more. The positions of the individual microphones may be arranged in a straight line, as illustrated in FIG. 4.

The sound processing device 10 may be configured as a general-purpose information and communication device such as a PC (Personal Computer) or a multi-function mobile phone, or may be configured as a dedicated device such as a measuring instrument or a monitoring device.
In the following description, the sound collection unit 20 is configured separately from the sound processing device 10 as an example, but may be separate from the sound processing device 10 .

Next, an example of the functional configuration of the sound processing device 10 according to this embodiment will be described.
The sound processing device 10 includes an input/output unit 110 , a control unit 120 , and a storage unit 150 .
The input/output unit 110 is connected wirelessly or wired to other devices to input and output various types of data. The input/output unit 110 outputs M-channel acoustic signals from the sound collection unit 20 to the control unit 120 as input data. When output data is input from the control unit 120, the input/output unit 110 can output the input output data to an output destination device (not shown). The output data may include information obtained by microphone array processing. Such information includes, for example, estimated sound source direction information indicating an estimated sound source direction, a sound source signal indicating sound source components arriving from a sound source, and the like. The input/output unit 110 may be, for example, an input/output interface, a communication interface, or a combination thereof.

The control unit 120 executes processes to realize the functions of the sound processing device 10, processes to control those functions, and the like. The control unit 120 may be configured using dedicated components as a whole or for each individual function, or may be configured as a computer system including a processor such as a CPU (Central Processing Unit) and various storage media. The processor reads a predetermined program stored in advance in the storage medium and executes the processes instructed by the various commands written in the read program to realize the functions of the control unit 120. The functional configuration of the control unit 120 will be described later.

The memory unit 150 includes a storage medium that temporarily or permanently stores various types of data. The memory unit 150 stores various types of data (including parameters, etc.) used by the control unit 120, and various types of data acquired by the control unit 120 or other functional units (including input data input from the outside, intermediate data during processing, and generated data generated as a result of processing). The memory unit 150 stores an acoustic transfer function set. The acoustic transfer function set includes a first acoustic transfer function for each microphone (channel) for each frequency for each sound source direction. In this application, the sound source direction associated with the first acoustic transfer function in the acoustic transfer function set may be referred to as the "target direction."

As the initial value H _T of the first acoustic transfer function, a pre-measured acoustic transfer function may be used, or an acoustic transfer function pre-calculated using a predetermined geometric model may be used. Examples of the geometric model that may be used include a plane wave model that assumes the propagation of a plane wave in a free sound field, and a spherical wave model that assumes the propagation of a spherical wave from a sound source located a predetermined distance from the sound pickup unit 20. The initial acoustic transfer function set includes, as elements, initial values H _T (θ ₁ ) to H _T (θ _N ) of the first acoustic transfer function for each sound source direction for each channel and frequency. The initial value H _T (θ ₁ ) etc. can be calculated using a geometric model assuming the arrival of a sound wave from a sound source direction θ ₁ etc. N indicates the number of predetermined sound source directions. The distance between adjacent sound source directions directly affects the accuracy of the sound source direction estimated by sound source localization. The greater the number of sound source directions, the more accurate the sound source direction is expected to be, but the amount of calculation required to calculate the spatial spectrum in sound source localization increases.

Next, an example of the functional configuration of the control unit 120 according to this embodiment will be described. The control unit 120 is configured to include a frequency analysis unit 122, a sound source direction estimation unit 124, a representative estimated sound source direction determination unit 126, an outlier removal unit 128, a representative acoustic signal determination unit 130, an acoustic transfer function estimation unit 132, and an acoustic transfer function update unit 134. Unless otherwise specified, the processes of the sound source direction estimation unit 124, the representative estimated sound source direction determination unit 126, the outlier removal unit 128, the representative acoustic signal determination unit 130, the acoustic transfer function estimation unit 132, and the acoustic transfer function update unit 134 may be executed independently for each frequency.

The frequency analysis unit 122 receives M-channel acoustic signals from the sound collection unit 20 via the input/output unit 110. The acquired M-channel acoustic signals each represent a time series (waveform) of amplitude at each sample time in the time domain. The frequency analysis unit 122 performs frequency analysis on the time-domain acoustic signal for each channel, frame by frame over a predetermined period (e.g., 20 ms-100 ms), and converts it into transform coefficients for each frequency in the frequency domain. For each channel, the set of transform coefficients between frequencies represents a frequency spectrum. For frequency analysis, the frequency analysis unit 122 can use techniques such as the short-time Fourier transform (STFT) and the discrete Fourier transform (DFT). The frequency analysis unit 122 outputs input signal information indicating the transform coefficients obtained by the transform to the sound source direction estimation unit 124 and the acoustic transfer function estimation unit 132.

The sound source direction estimation unit 124 refers to the acoustic transfer function set stored in the storage unit 150 and calculates a spatial spectrum S _sp (θ) for each frequency using the conversion coefficients of each channel indicated in the input signal information input from the frequency analysis unit 122. The spatial spectrum S _sp (θ) can be considered as an index indicating the degree of possibility that a sound source exists for each target direction θ relative to the position of the sound collection unit 20. The sound source direction estimation unit 124 can perform calculations using an acoustic transfer function set consisting of the first acoustic transfer function H _E , the target direction θ, and the input vector X. As exemplified in Equation (1), the sound source direction estimation unit 124 can estimate the direction in which the spatial spectrum is maximum as the estimated sound source direction φ. A specific example of a method for calculating the spatial spectrum S _sp (θ) will be described later. The sound source direction estimation unit 124 outputs estimated sound source direction information indicating the estimated sound source direction to the representative estimated sound source direction determination unit 126 and the outlier removal unit 128. Furthermore, the sound source direction estimation unit 124 outputs the input signal information to the outlier removal unit 128 in association with the estimated sound source direction information.

The representative estimated sound source direction determiner 126 receives estimated sound source direction information from the sound source direction estimator 124. The representative estimated sound source direction determiner 126 generates a frequency distribution of the estimated sound source direction φ indicated in the estimated sound source direction information for each preset observation period. The observation period is a period including multiple frames for which a representative estimated sound source direction is determined at one time. More specifically, the representative estimated sound source direction determiner 126 identifies the estimated sound source direction indicated in the estimated sound source direction information for each frame included in the observation period, and counts the number of frames for each estimated sound source direction by adding one to the number of frames for the identified estimated sound source direction (increment). The representative estimated sound source direction determiner 126 can obtain the number of frames (frequency) for each estimated sound source direction counted during the observation period as a frequency distribution (histogram) of the estimated sound source direction.

The representative estimated sound source direction determination unit 126 determines the estimated sound source direction (mode) with the largest number of frames in the identified section as the representative estimated sound source direction θ'. The representative estimated sound source direction determination unit 126 outputs representative estimated sound source direction information indicating the determined representative estimated sound source direction θ' to the outlier removal unit 128.

The outlier removal unit 128 receives input of estimated sound source direction information and input signal information from the sound source direction estimation unit 124, and inputs representative estimated sound source direction information from the representative estimated sound source direction determination unit 126. The outlier removal unit 128 determines, for example, whether the estimated sound source direction is equal to the representative estimated sound source direction. When the estimated sound source direction is within a predetermined range from the representative estimated sound source direction, the outlier removal unit 128 adopts input signal information corresponding to the estimated sound source direction information indicating that estimated sound source direction. When the estimated sound source direction differs from the representative estimated sound source direction, the outlier removal unit 128 removes the input signal information corresponding to the estimated sound source direction information indicating that estimated sound source direction as an outlier and discards it. In this way, the outlier removal unit 128 functions as a mode filter. The outlier removal unit 128 outputs the adopted input signal information to the representative sound signal determination unit 130.

For each observation period, the representative acoustic signal determination unit 130 determines the representative value (e.g., average value) between frames of the transformation coefficients for each channel indicated in the input signal information input from the outlier removal unit 128 as the representative transformation coefficient for each frequency. The representative acoustic signal determination unit 130 outputs representative input signal information indicating the determined representative transformation coefficient to the acoustic transfer function estimation unit 132.

The acoustic transfer function estimation unit 132 receives representative input signal information from the representative acoustic signal determination unit 130. For each frequency, the acoustic transfer function estimation unit 132 estimates the acoustic transfer function from the sound source to the microphone corresponding to that channel as a second acoustic transfer function H' based on the representative conversion coefficient for each channel indicated in the representative input signal information. The second acoustic transfer function H' corresponds to the representative value of the acoustic transfer function estimated during the observation period. When estimating the second acoustic transfer function H', the acoustic transfer function estimation unit 132, for example, normalizes the amplitude and phase of the representative conversion coefficient for each channel between channels.

The acoustic transfer function estimator 132 can calculate the second acoustic transfer function H′ according to, for example, Equation (2). In the example of Equation (2), the representative input vector X is divided by its norm |X| to normalize the amplitude of the representative transform coefficient. For example, the square root of the sum of squares can be used as the norm. The representative input vector X is a vector having, as elements, representative transform coefficients X _m for each channel _m at a certain frequency. The normalized amplitude is a real value between 0 and 1. The phase of the representative transform coefficient is normalized by multiplying it by the complex conjugate of the quotient obtained by dividing the sum Σ _m X _m of the representative transform coefficients X m across channels by its absolute value |Σ _m X _m |. Phase normalization causes the average value of the phase across channels, weighted by the amplitude of the representative transform coefficient of each channel, to be 0. In this embodiment, the acoustic transfer function may be a value in which the amplitude and phase are relativized across channels, and need not necessarily be an absolute value. The acoustic transfer function estimating unit 132 outputs second acoustic transfer function information indicating the estimated second acoustic transfer function H′ to the acoustic transfer function updating unit 134 .

The acoustic transfer function updating unit 134 receives second acoustic transfer function information from the acoustic transfer function estimating unit 132 and receives representative estimated sound source direction information from the representative estimated sound source direction determining unit 126. For each frequency, the acoustic transfer function updating unit 134 identifies the second acoustic transfer function H' for each channel indicated by the input second acoustic transfer function information as the second acoustic transfer function corresponding to the representative estimated sound source direction indicated in the representative estimated sound source direction information. The acoustic transfer function updating unit 134 uses the identified second acoustic transfer function H' to update the first acoustic transfer function H _E corresponding to the representative estimated sound source direction from the acoustic transfer function set stored in the storage unit 150.

The acoustic transfer function update unit 134 calculates a new updated first acoustic transfer function H _E (θ') by using, for example, exponential smoothing, a weighted average of the second acoustic transfer function H' at that time and the first acoustic transfer function H _E (θ') related to the representative estimated sound source direction θ' to be updated. In the example of equation (3), the weighting coefficient β by which the second acoustic transfer function H' is multiplied is a predetermined positive real value having a maximum value of 1. The first acoustic transfer function H _E (θ') before updating is multiplied by the weighting coefficient (1-β). The weighting coefficients β and (1-β) correspond to the ratios to the second acoustic transfer function H' and the first acoustic transfer function _{H E} (θ'), respectively. Therefore, the larger the weighting coefficient β, the more importance is placed on the second acoustic transfer function H' as the first acoustic transfer function H E (θ'). When the weighting coefficient β is 1, the second acoustic transfer function H' is replaced by the first acoustic transfer function H _E for each frame. That is, the larger the weighting coefficient β, the more the first acoustic transfer function H E (θ') includes the influence of the presence or absence of sound from the sound source included in the second acoustic transfer function H _' , temporary changes in the acoustic environment, erroneous estimation of the sound source direction, etc. The smaller the weighting coefficient β, the more the temporary fluctuations in the second acoustic transfer function H' are smoothed.

The acoustic transfer function update unit 134 stores the new first acoustic transfer function H _E (θ') in the storage unit 150 in place of the original, unupdated first acoustic transfer function H _E (θ') in association with the estimated sound source direction θ'.

The acoustic transfer function update unit 134 may set the weighting coefficient β for the second acoustic transfer function H' so that the higher the reliability of the representative estimated sound source direction θ', the larger the weighting coefficient β. The acoustic transfer function update unit 134 may use the ratio of the number of frames in which the estimated sound source direction φ falls within a predetermined range from the representative estimated sound source direction θ' to the total number of frames as the reliability of the representative estimated sound source direction θ'. More specifically, the acoustic transfer function update unit 134 may set the weighting coefficient β to L/2K, where L is the number of frames in the observation period in which the estimated sound source direction φ becomes the representative estimated sound source direction θ', and K is the total number of frames in the observation period. When L = K, the weighting coefficient β has a maximum value of 0.5.

The representative estimated sound source direction determination unit 126 may identify an estimated sound source direction in which the number of counted frames is greater than a preset lower limit of the number of frames as a significant estimated sound source direction, and may determine a section consisting of multiple significant adjacent sound source directions. The representative estimated sound source direction determination unit 126 may also determine, as the representative estimated sound source direction, a representative value of the estimated sound source direction in which the number of frames is maximum among the identified sections. This eliminates particularly isolated estimated sound source directions and prevents them from being selected as the representative estimated sound source direction.

When the estimated sound source direction is within a predetermined tolerance range (e.g., ±3 to 5°) from the representative estimated sound source direction, the outlier removal unit 128 adopts input signal information corresponding to the estimated sound source direction information indicating that estimated sound source direction. When the estimated sound source direction is outside the predetermined range from the representative estimated sound source direction, the outlier removal unit 128 may remove and discard the input signal information corresponding to the estimated sound source direction information indicating that estimated sound source direction as an outlier. As a result, the conversion coefficient for the acoustic signal that provides an estimated sound source direction that is close to the representative estimated sound source direction is also used to estimate the second acoustic transfer function. A tolerance range of 0° corresponds to the mode filter described above.

Note that the number of sound sources from which sound is radiated at one time is not necessarily limited to one, and may be two or more, or sound may not be radiated from a sound source temporarily or continuously. Therefore, only when one direction in which the spatial spectrum S _sp (θ) is maximized and is greater than a predetermined spatial spectrum threshold is detected, the sound source direction estimating unit 124 may output estimated sound source direction information indicating the detected direction as an estimated sound source direction θ' to the representative estimated sound source direction determining unit 126 and the outlier removing unit 128. As described above, the acoustic transfer function updating unit 134 can update the first acoustic transfer function H _E (θ') associated with one estimated sound source direction θ' notified by the sound source direction estimating unit 124 in the estimated sound source direction information, by using the second acoustic transfer function H'. In this case, when determining the weighting coefficient β, the acoustic transfer function updating unit 134 may determine the weighting coefficient β by setting the number of frames in which the number of estimated sound source directions in the observation period is one as L and the number of frames in which the one estimated sound source direction φ is within the representative estimated sound source direction θ′ (or within an acceptable range) as K.

In other words, the sound source direction estimating unit 124 does not output estimated sound source direction information to the representative estimated sound source direction determining unit 126 and the outlier removing unit 128 when two or more directions in which the spatial spectrum _{S sp} (θ) is maximized and greater than a predetermined spatial spectrum threshold are detected, or when no direction in which the spatial spectrum S sp (θ) is maximized and greater than a predetermined spatial spectrum threshold is detected. Therefore, when determining the second acoustic transfer function, input signal information acquired when the number of estimated sound source directions is two or more or when no estimated sound source direction is detected is not used. On the other hand, when two or more estimated sound source directions are detected, sounds arriving from multiple sound sources are superimposed at the microphone, so the ratio of the conversion coefficients between the channels does not provide the ratio of the acoustic transfer functions for the sound source direction associated with a specific sound source. When a sound source direction is not detected, no significant sound arrives at the microphone from the sound source. Therefore, when one sound source is detected, sound source localization, estimation of acoustic transfer functions, and updating are limited, thereby preventing deterioration in the accuracy of estimating acoustic transfer functions.

In two-dimensional space, the sound source direction may be defined by an azimuth angle from a representative point (e.g., the center of gravity) of the sound collection unit 20. In this case, the arrangement of sound source directions associated with each first acoustic transfer function constituting the acoustic transfer function set may be, for example, a one-dimensional array distributed on a circumference parallel to a horizontal plane centered on the position of the sound collection unit 20. In three-dimensional space, the sound source direction may be defined by a pair of an azimuth angle and an elevation angle. The arrangement of sound source directions may be a two-dimensional array distributed on a spherical surface centered on the position of the sound collection unit 20. The acoustic transfer function set may be configured to include a first acoustic transfer function for each sound source position. In this case, the arrangement of sound source positions is a three-dimensional distribution in three-dimensional space. The sound source position is expressed in three-dimensional coordinates based on the position of the sound collection unit 20 and corresponds to a combination of the sound source direction and the distance from the reference position. However, although this embodiment mainly describes a case where the distribution of sound source positions is a one-dimensional array, it is also applicable to two-dimensional or three-dimensional arrays.

When the acoustic transfer function set includes a first acoustic transfer function for each sound source position, the sound source direction estimation unit 124 can estimate the sound source position as the information to be estimated. The sound source direction estimation unit 124 calculates a spatial spectrum for each sound source position instead of the sound source direction, and identifies the sound source position where the spatial spectrum is maximized (or at its maximum). The acoustic transfer function update unit 134 sets the identified sound source position as the estimated sound source position, and updates the first acoustic transfer function associated with the estimated sound source position using the second acoustic transfer function estimated by the sound source direction estimation unit 124 using the above-described method.

(Example of sound source localization)
Next, an example of sound source localization will be described. The sound source direction estimator 124 can use, for example, beamforming in sound source localization. The sound source direction estimator 124 can calculate, as an estimated sound source direction, the sound source direction θ that gives the maximum value of the spatial spectrum S _sp (θ) exemplified in equation (4). The spatial spectrum S _sp (θ) is obtained by multiplying the input vector X by the pseudo-inverse matrix H(θ) ⁺ of the acoustic transfer function vector H(θ). The acoustic transfer function vector H(θ) is a vector [H ₁ (θ), H ₂ (θ), ..., H _M (θ)] ^T having, as elements of each column, a transfer function from the sound source to the microphone for each channel.

The sound source direction estimation unit 124 may use a method other than beamforming for sound source localization. For example, the sound source direction estimation unit 124 may use a method such as the MUSIC (Multiple Signal Classification) method or a delay and sum method.

(Acoustic Transfer Function Adaptation Processing)
Next, the acoustic transfer function adaptation process according to this embodiment will be described. Fig. 2 is a data flowchart showing an example of the acoustic transfer function adaptation process according to this embodiment. In the following description, an example will be taken in which the outlier remover functions as a mode filter.
(Step S<b>102 ) The frequency analysis unit 112 receives the M-channel acoustic signal x from the sound collection unit 20 .
(Step S104) The frequency analysis unit 112 performs frequency analysis on the acoustic signal for each channel for each frame and converts it into an input vector X (input signal information) indicating a transformation coefficient in the frequency domain. A set of input vectors [X ₁ , X ₂ , ..., X _K ] for each frame in the observation period forms an input signal group Z.

(Step S106) The sound source direction estimator 124 refers to the acoustic transfer function set and calculates, for each frame, the sound source direction at which the spatial spectrum S _sp (θ) is maximized for each frequency, as the estimated sound source direction φ, using the conversion coefficients indicated in the input vector X. A set of estimated sound source directions [φ ₁ , φ ₂ , ..., φ _K ] for each frame in the observation period forms a localization direction group Φ.
(Step S108) The representative estimated sound source direction determiner 126 counts the frequency distribution indicating the frequency (number of frames) of each estimated sound source direction for each observation period, and determines the most frequently occurring estimated sound source direction as the representative estimated sound source direction θ′.

(Step S110) The outlier removal unit 128 removes, as an outlier, an input vector X" of a frame that gives an estimated sound source direction φ different from the representative estimated sound source direction θ', from among the input vectors X for each frame. A set of input vectors [X ₁ ', X ₂ ', ..., X _L '] that remain without being removed forms an outlier-removed input signal group Z'.

(Step S112) The representative acoustic signal determination unit 130 generates a representative input vector <X> that indicates, as a representative transformation coefficient, the representative value of the transformation coefficients for each channel indicated in the input vector X' of the frame in which the outliers were not removed and which remains in the observation section.
(Step S114) The acoustic transfer function estimation unit 132 normalizes the representative conversion coefficient for each channel indicated in the representative input signal information for each frequency between channels, and estimates the acoustic transfer function from the sound source to the microphone corresponding to that channel as the second acoustic transfer function H'.
(Step S116) The acoustic transfer function update unit 134 updates the weighted average value (1-β)H _E +βH' of the first acoustic transfer function H _E and the second acoustic transfer function H' associated with the representative estimated sound source direction θ' for each frequency as a new first acoustic transfer function H _E.

Note that the process in Figure 2 may be repeated for each observation period, or may be repeated at a cycle shorter than the observation period, for example, every frame.

(Evaluation experiment)
Next, an evaluation experiment conducted to evaluate the effectiveness of the above-described embodiment will be described. The evaluation experiment was conducted in a laboratory with an acoustic environment similar to that of a typical office environment. The laboratory was approximately rectangular in shape. The laboratory dimensions were 7.0 m in width (x direction), 4.0 m in length (y direction), and 3.0 m in height (z direction) (see FIG. 5 ). Tables were installed in the center and on the periphery of the laboratory, with multiple chairs placed around the tables. A circular microphone array (see FIG. 3 ) was installed as the sound collection unit 20 on the central table, and the height of the table from the floor was 0.9 m. A notebook personal computer and other items were placed on the peripheral table.

Prior to the evaluation experiment, a sound source signal was acquired. A male voice selected from the Corpus of Spontaneous Japanese (CSJ) was used as the sound source. The loudspeaker was moved slowly clockwise along a circle at a distance of 0.78 m from the center of the microphone array, and sound was emitted according to the sound source signal. The height of the center of the loudspeaker from the floor was set to 1.0 m. Under these conditions, eight-channel acoustic signals representing the sound coming from the sound source and picked up by the microphone array were acquired for 20 minutes. The sampling frequency was set to 16 kHz. During this time, the loudspeaker made three revolutions around the circle.
As the initial value H _T of the first acoustic transfer function, an acoustic transfer function calculated based on the positional relationship between the speaker and the microphone array installed in the target direction was set in advance, assuming a free sound field model.

The frequency analysis unit 122 performed STFT for frequency analysis. In STFT, the frame length and shift width were set to 512 points and 256 points, respectively. The Hann window was used as the window function. Frames with an average sound pressure of -24 dB or higher were considered valid frames. The acoustic signals in valid frames were adopted, and other frames were discarded as silent intervals. Each time the time equivalent to the observation period elapsed, a new observation period was set. In other words, the shift width of the observation period was set to a period equivalent to that observation period.

The evaluation experiment consisted of two verification items. In the first verification, the relationship between the length of the mode filter (equivalent to the observation period) and sound source localization performance was investigated. In the first verification, the first acoustic transfer function was updated by performing the process illustrated in Figure 2 using acoustic signals acquired in advance for each of several observation periods. The observation period was set to 11 different intervals of 60 frames, ranging from 60 frames (0.96 seconds) to 600 frames (9.6 seconds).

The directional resolution of the sound source direction for the first acoustic transfer function was set to 5°. The directional resolution corresponds to the spacing between the sound source directions associated with each first acoustic transfer function. The maximum and minimum frequencies of the frequency band to be processed were set to 300 Hz and 6000 Hz, respectively. Sound source localization was then performed using an acoustic transfer function set including the first acoustic transfer function obtained by updating, and the estimated sound source direction was compared with the target sound source direction. The target sound source direction corresponds to the known, correct sound source direction. The delay-and-sum method was used as the sound source localization method.

The success rate was calculated as an evaluation index for sound source localization performance. The success rate corresponds to the ratio of the number of successful frames to the number of valid frames. A successful frame corresponds to a frame in which sound source localization was successful. In this test, frames in which the estimated sound source direction was within a specified range (e.g., 5°) from the target sound source direction were counted as successful frames. Therefore, a higher success rate indicates better sound source localization performance. In this test, the number of valid frames was 4,322.

Figure 6 illustrates the success rate for each observation period. Figure 6 shows that the success rate reached its highest at 90.42% when the observation period was 120 frames (equivalent to 1.92 seconds). Considering that the speakers were moved in this test, this indicates that the performance of sound source localization does not degrade even if the speakers are assumed to be stationary for approximately 1.92 seconds. Overall, the shorter the observation period, the higher the success rate. This is thought to be due to the fact that the longer the observation period, the higher the frequency of outliers due to changes in the target sound source direction associated with movement, and the lower the update frequency of the first acoustic transfer function. On the other hand, a short observation period reduces the statistical reliability of the distribution of estimated sound source directions used to determine the representative estimated sound source direction. This actually contributes to a lower success rate. The fact that the success rate reaches its highest when the observation period is 120 frames can be seen as an indication of the increase and decrease that occurs depending on the observation period.

In the first verification, the relationship between reliability and the type of first acoustic transfer function was investigated. A success rate was determined for each of the following three types of first acoustic transfer function: (1) This embodiment: The weighted sum of the first acoustic transfer function and the second acoustic transfer function is updated to a new first acoustic transfer function; (2) Existing method (method described in Non-Patent Document 1): The first acoustic transfer function is updated for each frame; (3) No update: The first acoustic transfer function is not updated. However, the observation period was set to 120 frames.

Figure 7 is a diagram illustrating the success rate for each type of first acoustic transfer function. Figure 7 shows the success rate in the order of no update, existing method, and this embodiment. The success rate increases in the order of existing method, no update, and this embodiment. The success rate was 85.59% without updating, 80.63% with the existing method, and 90.42% with this embodiment. This result supports the effectiveness of this embodiment. One reason why the success rate with the existing method is lower than the success rate without updating is thought to be that, because the first acoustic transfer function is updated uniformly for each frame, outliers occur and cases in which an estimated sound source direction with low reliability is not rejected.

(Modification)
Next, a modification of this embodiment will be described. The following description will mainly focus on the differences from the above-described embodiment, and unless otherwise specified, the same reference numerals as in the above-described embodiment will be used to refer to the description therein. FIG. 8 is a schematic block diagram showing an example configuration of a sound processing system S1 according to a first modification of this embodiment. The sound processing system S1 according to this modification is applied to sound source separation. In the sound processing system S1, the control unit 120 of the sound processing device 10 includes a frequency analysis unit 122, a sound source direction estimation unit 124, a representative estimated sound source direction determination unit 126, an outlier removal unit 128, a representative sound signal determination unit 130, an acoustic transfer function estimation unit 132, and an acoustic transfer function update unit 134, as well as a sound source separation unit 136 and a sound source signal generation unit 138.

The frequency analysis unit 122 outputs the input signal information to the sound source direction estimation unit 124 , the acoustic transfer function estimation unit 132 , and also to the sound source separation unit 136 .
The sound source direction estimating unit 124 outputs the estimated sound source direction information to the representative estimated sound source direction determining unit 126 and the outlier removing unit 128 as well as to the sound source separating unit 136 .
The acoustic transfer function estimating unit 132 outputs the second acoustic transfer function information to the acoustic transfer function updating unit 134 as well as to the sound source separating unit 136 .

The sound source separation unit 136 receives input signal information from the frequency analysis unit 122 and estimated sound source direction information from the sound source direction estimation unit 124. The sound source separation unit 136 extracts sound source components arriving from the estimated sound source direction from the conversion coefficients for each channel indicated in the input signal information. The sound source separation unit 136, for example, refers to an acoustic transfer function set stored in the storage unit 150 and calculates a separation matrix W(H _E , θ') from a first acoustic transfer function H _E related to the estimated sound source direction θ'. As exemplified in Equation (5), the sound source separation unit 136 multiplies the input vector X by the separation matrix W(H _E , θ') to calculate an output vector Y (separated sound source) indicating, for each frequency, an output value estimated as a sound source component arriving from a sound source present in the estimated sound source direction θ'. The input vector X is a vector including, as elements, conversion coefficients for each channel indicated in the input signal information. When multiple estimated sound source directions are detected, the sound source separation unit 136 can determine an output value for each sound source (estimated sound source direction). The sound source separation unit 136 outputs output signal information indicating the output value determined for each frequency for each sound source to the sound source signal generation unit 138.

The sound source signal generation unit 138 converts the output values for each frequency indicated in the output signal information input from the sound source separation unit 136 for each sound source into a time series of amplitude for each sample time in the time domain. When converting the output values for each frequency in the frequency domain into a time series of amplitude, the sound source signal generation unit 138 can use the inverse process of frequency analysis, for example, an inverse discrete Fourier transform. The sound source signal generation unit 138 can generate a sound source signal by linking the time series of amplitude obtained for each frame for each sound source between frames. The sound source signal generation unit 138 may output the generated sound source signal to an output destination device via the input/output unit 110, or may store it in the storage unit 150.

The sound source direction estimation unit 124 may detect a plurality of directions in which the spatial spectrum S _sp (θ) is maximized and exceeds a predetermined spatial spectrum threshold. In such a case, the sound source direction estimation unit 124 may output estimated sound source direction information indicating each of the plurality of sound source directions as an estimated sound source direction to the sound source separation unit 136. This is because in such a case, it is estimated that a plurality of significant sound sources exist. The sound source localization in the sound source direction estimation unit 124 and the sound source separation in the sound source separation unit 136 may be performed simultaneously with the update of the first acoustic voltage function in the acoustic transfer function update unit 134, but they do not necessarily have to be synchronized. In other words, when two or more sound sources are detected, the estimated sound source direction information is not output to the representative estimated sound source direction determiner 126 and the outlier remover 128, and even if the representative estimated sound source direction information is not input from the representative estimated sound source direction determiner 126 to the acoustic transfer function update unit 134, the sound source separation in the sound source separation unit 136 is permitted.

The sound source separation unit 136 can apply, for example, the beamforming method described above as a sound source separation technique. In this case, the sound source separation unit 136 may employ, as a separation matrix, a pseudo-inverse matrix H ⁺ (θ′) of an acoustic transfer function vector H(θ′) associated with an estimated sound source direction θ′ estimated using the beamforming method. Other sound source separation techniques that can be used include, for example, the GHDSS (Geometric-contrained High-order Decorrelation-based Source Separation) method. The GHDSS method includes a process of adaptively calculating a separation matrix W so as to minimize a cost function J(W). The cost function J(W) is a weighted sum of separation sharpness J _SS (W) and geometric constraint J _GC (W). The separation sharpness J _SS (W) is an index value indicating the degree to which a component of a certain sound source is mixed into the sound source component Y of another sound source. The geometric constraint degree J _GC (W) is an index value indicating the degree of error between the output sound source signal and the original sound source signal emitted from the sound source.

Next, a second modified example of this embodiment will be described. The following description will mainly focus on the differences from the above-described embodiment and modified example, and unless otherwise specified, the same reference numerals as in the above-described embodiment will be used and the description thereof will be incorporated. The sound processing system S1 according to this modified example forms part of a robot system (not shown). Figure 9 is a schematic block diagram showing an example configuration of the sound processing system S1 according to this modified example. One or both of the sound processing device 10 and the sound collection unit 20 that form the sound processing system S1 may be built into the robot's housing.

In the sound processing device 10, the control unit 120 includes a frequency analysis unit 122, a sound source direction estimation unit 124, a representative estimated sound source direction determination unit 126, an outlier removal unit 128, a representative sound signal determination unit 130, an acoustic transfer function estimation unit 132, an acoustic transfer function update unit 134, a sound source separation unit 136, and a sound source signal generation unit 138, as well as an operation control unit 140. That is, in the sound processing system S1, the sound source direction estimation unit 124, the sound source separation unit 136, and the sound source signal generation unit 138 may function as a robot audition functional block that realizes robot audition.

The control unit 120 may further include a speech recognition processing unit (not shown). The speech recognition processing unit may identify the type of sound source by performing known speech recognition processing on sound source components related to each sound source (sound source identification). The speech recognition processing unit may identify a human speaker as the type of sound source. The sound source direction estimation unit 124 may notify another device of estimated sound source direction information indicating the estimated sound source direction for the identified type of sound source, or may output a sound source signal converted from output signal information for the identified type of sound source to another device.
The sound source direction estimation unit 124 is capable of estimating the sound source position as described above, and outputs estimated sound source direction information indicating the estimated sound source position to the representative estimated sound source direction determination unit 126, the outlier removal unit 128, the sound source separation unit 136, and also to the operation control unit 140.

The operation control unit 140 receives estimated sound source direction information from the sound source direction estimation unit 124 and output signal information indicating sound source components from the sound source separation unit 136. The operation control unit 140 controls the operation of the operating mechanism 40 using one or both of the estimated sound source position and sound source components. The operation control unit 140 may, for example, perform self-location estimation and environmental map creation based on the estimated sound source position and sound source components (SLAM: Simultaneous Localization and Mapping). The operation control unit 140 can estimate the presence of an object (including a person) that is the sound source at the estimated sound source position by performing sound source identification. The operation control unit 140 may determine the existence probability of an object that is the sound source using a predetermined density function model so that the probability increases the closer to the estimated sound source position. The operation control unit 140 can, for example, create an environmental map by superimposing the spatial distribution of the existence probability of each object between objects. In the path planning, the movement control unit 140 may determine a travel path that does not pass through areas where the probability of an object's existence is higher than a predetermined probability. The travel path is represented by a target position for each time. The movement control unit 140 may determine the estimated direction of a predetermined type of sound source as the target direction relative to the front of the robot. The movement control unit 140 outputs a control signal to the movement mechanism 40 that indicates either or both of the target position and target direction at that time.

The movement mechanism 40 is built into the robot's housing and controls the robot's movement based on control signals input from the movement control unit 140. The movement mechanism 40 is equipped with a motor (not shown) that serves as a power source and an encoder (not shown) that detects the position and direction of the movement mechanism 40. The motor moves the robot so that it approaches the target position or direction specified by the control signal. The encoder sequentially outputs movement information to the movement control unit 140, indicating the position and direction detected at that time as the movement state.

As described above, the sound processing device 10 according to this embodiment includes: a storage unit 150 that stores a first acoustic transfer function for each sound source direction; a sound source direction estimation unit 124 that calculates a spatial spectrum for each sound source direction for each frame based on the transform coefficients in the frequency domain of the acoustic signal for each channel and the first acoustic transfer function, and estimates the sound source direction for which the spatial spectrum is maximum as the estimated sound source direction; a representative estimated sound source direction determination unit 126 that determines a representative estimated sound source direction that is a representative value of the estimated sound source directions based on the frequency distribution of the estimated sound source directions in an observation period consisting of a plurality of frames; an outlier removal unit 128 that removes transformation coefficients of frames whose estimated sound source directions fall outside a predetermined tolerance range from the representative estimated sound source direction; an acoustic transfer function estimation unit 132 that estimates, as the second acoustic transfer function, a representative value of the acoustic transfer function from the sound source to the sound collection unit for the acoustic signal in the observation period based on the transformation coefficients of the acoustic signal in the remaining frames; and an acoustic transfer function update unit 134 that updates the first acoustic transfer function for the representative estimated sound source direction using the second acoustic transfer function.
With this configuration, the second acoustic transfer function is calculated based on the conversion coefficient of the acoustic signal that gives an estimated sound source direction within a predetermined range from the representative estimated sound source direction, and the first acoustic transfer function can be updated using the calculated second acoustic transfer function in association with the representative estimated sound source direction. A representative value of the acoustic transfer function obtained based on the acoustic signal that statistically gives the representative estimated sound source direction or an estimated sound source direction that is approximate thereto is updated as the second acoustic transfer function in association with the representative estimated sound source direction, thereby obtaining a first acoustic transfer function with a stable correspondence with the sound source direction. By using such a first acoustic transfer function, it is possible to improve the reliability of sound source localization, sound source separation, and other microphone array processing using any acoustic signal online.

Furthermore, the sound source direction estimating unit 124 may determine the estimated sound source direction φ having a local maximum (for example, the largest) frequency as the representative estimated sound source direction θ′.
According to this configuration, the estimated sound source direction with the maximum frequency within the observation period is determined as the representative estimated sound source direction, so that the most likely estimated sound source direction is simply determined as the representative estimated sound source direction.

Furthermore, the acoustic transfer function update unit 134 may update the first acoustic transfer function H _E to a new weighted average value of the first acoustic transfer function and the second acoustic transfer function (for example, βH′+(1−β)H _E ).
With this configuration, when the observation period is changed, the first acoustic transfer function is not completely replaced by the second acoustic transfer function through updating, but some of its components remain. This prevents abrupt fluctuations in the first acoustic transfer function, thereby improving system stability.

In addition, the acoustic transfer function update unit 134 may determine the reliability (e.g., L/K, where K is the number of frames in the observation period) of the representative estimated sound source direction based on the frequency (e.g., the number of frames L) at which the estimated sound source direction falls within an acceptable range during the observation period, and may increase the ratio β of the second acoustic transfer function to the first acoustic transfer function as the reliability increases (e.g., L/2K).
According to this configuration, the first acoustic transfer function can be updated using the second acoustic transfer function, with emphasis being placed on acoustic signals that provide more reliable estimated sound source directions, thereby improving the reliability of the updated first acoustic transfer function.

Furthermore, the allowable range of the estimated sound source direction is equal to the representative estimated sound source direction and does not need to include directions different from the representative estimated sound source direction.
According to this configuration, it is possible to simply determine whether or not to exclude the transform coefficient of the acoustic signal that gives the estimated sound source direction, depending on whether or not the estimated sound source direction is equal to the representative estimated sound source direction.

Alternatively, the sound source direction estimator 124 may calculate the spatial spectrum by multiplying an input vector X including a conversion coefficient for each channel by a pseudo-inverse matrix H ⁺ of an acoustic transfer function vector including a first acoustic transfer function for each channel.
This configuration allows the sound source direction to be estimated based on the spatial spectrum calculated by a simple matrix operation, which does not require many computational resources and can be implemented economically.

One embodiment of the present invention has been described in detail above with reference to the drawings, but the specific configuration is not limited to that described above, and various design modifications can be made without departing from the spirit of the present invention.

S1...sound processing system, 10...sound processing device, 20...sound collection unit, 40...operation mechanism, 110...input/output unit, 120...control unit, 122...frequency analysis unit, 124...sound source direction estimation unit, 126...representative estimated sound source direction determination unit, 128...outlier removal unit, 130...representative sound signal determination unit, 132...acoustic transfer function estimation unit, 134...acoustic transfer function update unit, 136...sound source separation unit, 138...sound source signal generation unit, 140...operation control unit, 150...storage unit

Claims (8)

a storage unit that stores the first acoustic transfer function for each sound source direction;
For each frame, a spatial spectrum is calculated for each sound source direction based on a transform coefficient in the frequency domain of the acoustic signal for each channel and the first acoustic transfer function;
a sound source direction estimation unit that estimates the sound source direction in which the spatial spectrum is maximized as an estimated sound source direction;
a representative estimated sound source direction determining unit that determines a representative estimated sound source direction that is a representative value of the estimated sound source directions based on a frequency distribution of the estimated sound source directions in an observation period consisting of a plurality of frames;
an outlier removal unit that removes transform coefficients of frames whose estimated sound source directions are outside a predetermined tolerance range from the representative estimated sound source direction;
an acoustic transfer function estimation unit that estimates, as a second acoustic transfer function, a representative value of an acoustic transfer function from a sound source to a sound collection unit of the acoustic signal during the observation period based on a transformation coefficient of the acoustic signal of the remaining frames;
an acoustic transfer function update unit that updates a first acoustic transfer function for the representative estimated sound source direction using the second acoustic transfer function;
An acoustic processing device comprising: The sound processing device according to claim 1 , wherein the representative estimated sound source direction determiner determines the estimated sound source direction with a maximum frequency as the representative estimated sound source direction. The sound processing device according to claim 1 , wherein the acoustic transfer function update unit updates the first acoustic transfer function to a new first acoustic transfer function by a weighted average value of the first acoustic transfer function and the second acoustic transfer function. the acoustic transfer function update unit determines a reliability of the representative estimated sound source direction based on a frequency with which the estimated sound source direction falls within the tolerance range during the observation period;
The sound processing device according to claim 3 , wherein the ratio of the second acoustic transfer function to the first acoustic transfer function is increased as the reliability increases. The sound processing device according to claim 1 , wherein the tolerance range is equal to the representative estimated sound source direction and does not include directions different from the representative estimated sound source direction. The sound source direction estimation unit
The sound processing device according to claim 1 , wherein the spatial spectrum is calculated by multiplying an input vector including the conversion coefficients for each channel by a pseudo-inverse matrix of an acoustic transfer function vector including the first acoustic transfer function for each channel. A program for causing a computer to function as the sound processing device described in claim 1. A sound processing method in a sound processing device including a storage unit that stores a first acoustic transfer function for each sound source direction,
The sound processing device comprises:
For each frame, a spatial spectrum is calculated for each sound source direction based on a transform coefficient in the frequency domain of the acoustic signal for each channel and the first acoustic transfer function;
a sound source direction estimating step of estimating the sound source direction in which the spatial spectrum is maximized as an estimated sound source direction;
a representative sound source direction determining step of determining a representative estimated sound source direction that is a representative value of the estimated sound source directions based on a frequency distribution of the estimated sound source directions in an observation period consisting of a plurality of frames;
an outlier removal step of removing transform coefficients of frames in which the estimated sound source direction is outside a predetermined tolerance range from the representative estimated sound source direction;
an acoustic transfer function estimating step of estimating, as a second acoustic transfer function, a representative value of an acoustic transfer function from a sound source to a sound collection unit of the acoustic signal during the observation period based on a transformation coefficient of the acoustic signal of the remaining frames;
an acoustic transfer function updating step of updating a first acoustic transfer function for the representative estimated sound source direction using the second acoustic transfer function;
An acoustic processing method that performs the above.