JP5235724B2 - Utterance front / side direction estimation apparatus, method and program - Google Patents

Description

  The present invention relates to a technique for estimating the utterance direction of a speaker from an audio signal input to a microphone.

  A system for exchanging voice information such as a telephone or a voice conference terminal is generally called a voice communication system. In the video conference system, the video is added to the audio information and presented, so that the situation of the place is easily transmitted, but in the audio communication system, it is difficult to grasp the situation of the other party. One of the information on the other party's situation is utterance direction information. By receiving this information from the other party, it is possible to grasp the direction in which the speaker is speaking and to facilitate communication.

Conventional techniques for estimating such speech direction information are disclosed in Non-Patent Documents 1 and 2 and the like, and a configuration example is shown in FIG. The speech direction estimation apparatus 10 in this configuration example estimates speech direction information as follows.
(i) The voice from the speaker 1 is picked up using M (M is an integer of 2 or more) microphones 11-1,. The collected analog signal is converted into a digital signal vx (t) = [x ₁ (t),..., X _M (t)] ^T by the AD converter 12. Here, t represents an index of discrete time.

(ii) The frequency domain transform unit 13 receives a set (frame) of the digital signals composed of a plurality of samples as an input, and performs frequency domain signal VX (ω, n) = [X ₁ (ω, n) by fast Fourier transform or the like. _{, ···, X M (ω,} n)] to convert to ^T. Here, ω represents a frequency index, and the total number of frequency indexes is Ω. N represents the index of the frame.
(iii) The fixed beamformer design unit 14 sets the fixed beamformer VG (ω, r, θ) = [G ₁ (ω, r, θ),..., G _M (for each speaker position and direction. ω, r, θ)] ^T is designed. G _i (ω, r, θ) is a coefficient to be multiplied by the frequency component x _i (ω, n) of the i-th microphone in order to emphasize / suppress the sound source at the speaker position r and the speech direction θ.

In designing, acoustic propagation characteristics between a sound source and a microphone VH (ω, r, θ) = [H ₁ (ω, r, θ),..., H _{M for} each predetermined speaker position and direction. (ω, r, θ)] ^T is obtained using simulation values or actual measurement values. Here, H _i (ω, r, θ) represents an acoustic propagation characteristic between the sound source at the speaker position r and the speech direction θ and the i-th microphone.
The fixed beamformer VG (ω, r, θ) is designed as a value that satisfies the expressions (1) and (2) representing the relationship with the acoustic propagation characteristics.
VH (ω, r _T , θ _T ) ^H · VG (ω, r _T , θ _T ) = 1 (1)
VH (ω, r _U , θ _U ) ^H · VG (ω, r _T , θ _T ) = 0 (2)
Expressions (1) and (2) emphasize the output power of the speaker position r _T and the speech direction θ _T and suppress the output power of the other speaker positions r _U and the speech direction θ _U. It shows that VG (ω, r, θ) is designed.

(iv) In the product-sum calculation unit 15, the frequency domain signal VX (ω, n) = [X ₁ (ω, n),..., X _M (ω, n)] ^T and the fixed beamformer VG (ω , r, θ) = [G ₁ (ω, r, θ),..., G _M (ω, r, θ)] ^T as an input, for each frequency ω, speaker position r, and speech direction θ. Multiplying the frequency component X _i (ω, n) corresponding to each microphone by the coefficient G _i (ω, r, θ) of the fixed beamformer, and adding the obtained M components, the output Y (ω, n, r, θ) is calculated. This calculation is synonymous with calculating Y (ω, n, r, θ) = VG (ω, r, θ) ^H · VX (ω, n).
(v) The power calculator 16 calculates and outputs power | Y (ω, n, r, θ) | ² from the output Y (ω, n, r, θ) from the product-sum calculator 15.

(vi) The frequency averaging processing unit 17 averages the power | Y (ω, n, r, θ) | ² output from the power calculation unit 16 by frequency, and calculates AY (n, r, θ). obtain. In this calculation, if F ₀ is defined as the frequency index used in the averaging process, and | F ₀ | is defined as the total number of frequency indexes,

Is equivalent to calculating Note that F ₀ satisfies Ω ≧ | F ₀ |.
(vii) The sound source direction selection unit 18 searches for the speaker position r and the speech direction θ where the power AY (n, r, θ) averaged by frequency for each frame is maximum, and the power AY (n , r, θ) is determined as the estimated speech direction θ _out (n).

Hiroshi Nakajima, “Extended Beamforming for Estimating Sound Source Direction”, Proceedings of the Acoustical Society of Japan, September 2005, p.619-620 Hiroshi Nakajima and 8 others, “Sound source directivity estimation using extended beamforming”, Acoustical Society of Japan Proceedings, September 2005, p.621-622

The following two points can be cited as problems of the prior art.
(i) To deal with utterances at any position and to estimate the direction of utterance with high accuracy, a large number of microphones are required and the position of the microphones must be devised. As the output power | Y (ω, n, r, θ) | ² of the fixed beamformer designed for each position and speech direction is different, the speech direction can be estimated with higher accuracy. However, assuming a sound source with strong directivity in front of the mouth, such as a sound wave emitted from the mouth of the speaker, as shown in FIG. Depending on the speaker position / speech direction, there is no difference in the output power of the fixed beamformer, and the speech direction estimation error increases (for example, in the experiment of Non-Patent Document 2, 64 microphones are used). Therefore, in order to reduce the error, a large number of microphones are required, the apparatus becomes large, and it is difficult to use it by attaching it to a portable apparatus such as a telephone or an audio conference terminal.

(ii) High reverberation direction estimation performance cannot be obtained in a reverberant environment where the reverberation time (the time from the direct wave arrival time to the 60 dB attenuation from the direct wave pickup power) is 250 msec or more. Below, since many strong reflected waves are mixed, it is difficult to design the acoustic propagation characteristics VH (ω, r, θ) with high accuracy. Therefore, ambiguity occurs in the output of the fixed beamformer, and the estimation accuracy deteriorates. For example, in an actual environment room that is not subjected to low reverberation processing, reverberation time is generally about 250 to 500 msec, so that accurate estimation is difficult.

  An object of the present invention is to make it possible to appropriately estimate a speech direction even in a reverberant environment where it is not necessary to arrange a large number of microphones so as to surround a speaker and a reverberation time is 250 msec or more. It is to provide a lateral direction estimation apparatus, method, and program.

  The utterance front / side direction estimation apparatus of the present invention collects voice signals uttered by a speaker around a microphone array composed of a plurality of microphones by the plurality of microphones, and between the voice signals collected by each microphone. Generate a correlation matrix that represents the correlation, and estimate the direction of the speaker speaking to the microphone array from the eigenvalues obtained by decomposing the correlation matrix into an eigenvalue matrix and an eigenvector matrix .

  According to the utterance front / side direction estimation apparatus of the present invention, it is not necessary to arrange a large number of microphones so as to surround a speaker, and the utterance direction can be appropriately estimated even in a reverberation environment where the reverberation time is 250 msec or more. Is possible.

The figure which shows the function structural example of the utterance front and side direction estimation apparatus of 1st Embodiment. The figure which shows the example of a processing flow of the utterance front and side direction estimation apparatus of 1st Embodiment. The image figure which shows the positional relationship of a microphone, a speaker, and a speech direction. The image figure about the method of judging front direction and sideways. The figure which shows the propagation characteristic of an audio | voice signal in a time domain. The image figure which shows the correlation with an utterance direction, a base vector, and an eigenvalue / eigenvector. The image figure which shows the relationship between an utterance direction and an eigenvalue. The figure which shows the function structural example of the utterance front and side direction estimation apparatus of 2nd Embodiment. The figure which shows the example of a processing flow of the utterance front and side direction estimation apparatus of 2nd Embodiment. The image figure about the method of determining left-right orientation. The figure which shows notionally the acoustic propagation vector group which comprises a spatial correlation matrix, and the eigenspace of a spatial correlation matrix. The figure which shows the function structural example of the front and side direction determination part 125. FIG. The figure which shows the processing flow of the front and side direction determination part 125. FIG. The figure which shows the function structural example of the front and side direction determination part 145. FIG. The figure which shows the processing flow of the front and side direction determination part 145. The figure which shows the function structural example of the front and side direction determination part 165. FIG. The figure which shows the verification environment and conditions of an effect. The figure which shows the verification result about the relationship between an utterance direction and an eigenvalue. The figure which shows the verification result of left-right direction estimation. The figure which shows the service structural example which incorporated this invention in the audio conference terminal. The figure which shows the function structural example of the speech direction estimation apparatus by a prior art. The image figure which shows the relationship between the microphone and speaker by a prior art.

[First Embodiment]
FIG. 1 shows a functional configuration example of the utterance front / side orientation estimation apparatus 100 of the present invention, and FIG. 2 shows a processing flow example thereof.
The utterance front / sideways estimation apparatus 100 includes a microphone array 101 composed of M (M is an integer of 2 or more) microphones 101-1 to 101 -M, an AD converter 12, a frequency domain converter 13, and a correlation matrix. A calculation unit 102, an eigenvalue decomposition unit 103, a frequency averaging processing unit 104, and a front / side orientation determination unit 105 are provided. Among them, the AD conversion unit 12 and the frequency conversion unit 13 are the same as those used in the speech direction estimation apparatus 10 described in the background art.

  In the prior art, as shown in FIG. 16, it was necessary to arrange a large number of microphones so as to surround the speaker. However, in the present invention, M microphones 101-1 to 101-M can be provided. It is sufficient to arrange them closely. Although the number of microphones constituting such a microphone array 101 is not too large, according to the configuration of the present invention described below, it is possible to estimate the utterance direction with two or more microphones. The arrangement may be two-dimensional or three-dimensional. By adopting a configuration in which a small number of microphones are arranged densely in this way, it can be attached to a portable device such as a telephone or an audio conference terminal, and the direction of speech by surrounding speakers can be estimated It becomes. The speaker speaks around the microphone array 101. FIG. 3 shows an image of a speaker talking around a microphone array 101 composed of seven microphones as viewed from above, and the direction of the arrow is the direction of speech. FIG. 3 (a) shows a state where a speaker is speaking in a front direction at each position, and FIG. 3 (b) shows a state where a speaker is speaking in a horizontal direction.

The AD conversion unit 12 converts the analog voice signals uttered by the speaker 1 collected by the M microphones 101-1 to 101 -M into digital voice signals x ₁ (t),..., X _M (t (S1). Here, t represents an index of discrete time.

The frequency domain transform unit 13 receives as input a set (frame) of the above digital speech signals composed of a plurality of discrete time samples, and performs frequency domain digital speech signals X ₁ (ω, n),. It is converted into X _M (ω, n) and output (S2). Here, n represents a frame index, and ω represents a frequency index. The total number of frequency indexes is Ω.

The correlation matrix calculation unit 102 receives the digital audio signals X ₁ (ω, n),..., X _M (ω, n) in the frequency domain as inputs, and an M × M correlation matrix R representing the correlation between the signals. (ω, k) is sequentially generated and output for each frequency ω by equation (3) (S3).
R (ω, k) = E [VX (ω, n) ・ VX ^H (ω, n)] (3)
Here, VX (ω, n) = [X ₁ (ω, n),..., X _M (ω, n)] ^T
Note that H represents conjugate transposition, and E calculates VX (ω, n) · VX (ω, n) ^H for each frame, and then calculates an expected value for each L frame by averaging processing or the like. It is an operator. That is, the correlation matrix is sequentially output at a rate of once per L frame, and k represents an output index of the correlation matrix. L is preferably an integer greater than or equal to M.

The eigenvalue decomposition unit 103 receives the correlation matrix R (ω, k) as input, and first, M eigenvalues λ ₁ (ω, k),..., Λ _M (ω, k) so as to satisfy Equation (4). ) Eigenvalue matrix Λ (ω, k), which is a diagonal matrix with each square as a diagonal element, and M eigenvectors vv ₁ (ω, n),..., Vv _M (ω, k) Is decomposed into eigenvector matrix V (ω, k) by the eigenvalue decomposition method.
R (ω, k) = V (ω, k) · Λ (ω, k) · V ^H (ω, k) (4)
Where Λ (ω, k) = diag [λ ₁ ² (ω, k),..., Λ _M ² (ω, k)]
λ ₁ (ω, k) ≧ λ ₂ (ω, k) ≧ ・ ・ ・ ≧ λ _M (ω, k)
V (ω, k) = [vv ₁ (ω, k),..., Vv _M (ω, k)] ^T
vv _i (ω, k) = [v _{i, 1} (ω, k),..., v _{i, M} (ω, k)]
Note that diag [•] is an operator having the components in [•] as elements of a diagonal matrix.
Then, the first eigenvalue λ ₁ (ω, k), which is the maximum eigenvalue, is normalized by Expression (5), and the first normalized eigenvalue nλ ₁ (ω, k) is output (S4).

The frequency averaging processing unit 104 calculates an average value for the first normalized eigenvalue nλ ₁ (ω, k) obtained for each frequency ω by the equation (6), and obtains the first averaged eigenvalue aλ ₁ ( k) is output (S5).

F ₁ is an index of frequencies used for averaging, | F ₁ | is a total number of frequency indexes, and F ₁ is appropriately set so as to satisfy Ω ≧ | F ₁ |.

The front / side orientation determination unit 105 compares the first averaged eigenvalue aλ ₁ (k) with a predetermined threshold value thr1, and if aλ ₁ (k) <thr1, the speaker is oriented sideways with respect to the microphone array. If not, it is determined that the user has spoken in the front direction, and the determination result is output (S6). The determination image is illustrated in FIG. Here, thr1 may be arbitrarily set depending on the environment and the position of the speaker. Since the first averaged eigenvalue aλ ₁ (k) is obtained for each frame group k, the determination result is also output for each frame group k.

  The theoretical background that can determine the speech direction with such a configuration will be described. FIG. 5 shows the propagation characteristics of the audio signal in the time domain. Propagation characteristics can be broadly divided into three types: direct wave, initial reverberation, and rear reverberation. In the time zone in which direct wave and initial reverberation are observed, the directivity with respect to the microphone array composed of a plurality of microphones. It is known that waves with In particular, in the initial reverberation time zone (after the arrival of the direct wave, the time from the direct wave pickup power to the attenuation of 10 dB), a strong reflected wave with directionality is mixed, but the power of this reflected wave depends on the direction of speech. Change. Specifically, since the direct wave power increases as the utterance direction is the front direction, the reflected wave power decreases, and the direct wave power decreases as it is in the horizontal direction. The power of will increase. The present invention uses such a property to estimate the speech direction.

This will be described below in accordance with the configuration of the present invention. FIG. 6 shows how the utterance state in front and side affects each eigenvalue λ _i (ω, k) of the correlation matrix R (ω, k). Here, a case where a microphone array is configured by three microphones is illustrated. When facing the front, many direct waves reach the microphone array, and the arrival rate of the reflected waves is relatively low. Therefore, as shown in FIG. Compared to the basis vector group to be expressed, it has a large power. At this time, the first eigenvalue λ ₁ (ω, k) is significantly larger than the second eigenvalue λ ₂ (ω, k) and the third eigenvalue λ ₃ (ω, k). On the other hand, in the case of the horizontal orientation, the direct waves that reach the microphone array are reduced, so that more reflected waves arrive accordingly. For this reason, as shown in FIG. 6B, the power of the basis vectors expressing the direct wave decreases, and the power of the basis vector group expressing the reflected wave increases. At this time, the first eigenvalue λ ₁ (ω, k) is smaller than that in the front direction, and conversely, the second eigenvalue λ ₂ (ω, k) and the third eigenvalue λ ₃ (ω, k) are in the front direction. Larger than the case. FIG. 7 shows an image of the difference that occurs in each eigenvalue between the case of facing forward and the case of facing sideways.

  From the above, it can be seen that the change in speech direction is reflected in the eigenvalue. Then, by using the first eigenvalue that significantly reflects the strength of the direct wave as a determination parameter, it is possible to appropriately determine that the first eigenvalue is front-facing if it is greater than a certain threshold value, and that it is lateral if it is smaller. .

  As described above, according to the utterance front / sideways estimation apparatus of the first embodiment, since a small number of microphones need only be arranged densely, a compact configuration can be achieved without enclosing the speaker with a large number of microphones. It becomes. In addition, since the reverberation is actively used, it is possible to appropriately estimate the utterance direction even in a reverberation environment where the reverberation time is 250 msec or more. In addition, since the eigenvalue decomposition process, which is the core of the process in the present invention, has a small amount of calculation, it is advantageous when it is incorporated in a device having a low CPU specification such as a portable terminal.

[Second Embodiment]
The first embodiment determines only whether the utterance direction is the front direction or the horizontal direction. However, the second embodiment can further determine whether the utterance direction is the left direction or the right direction. is there.
FIG. 8 shows an example of the functional configuration of the utterance front / sideways estimation apparatus 200 of the present invention, and FIG.

  The utterance front / sideways estimation apparatus 200 includes a microphone array 101 composed of M microphones 101-1 to 101-M (M is an integer of 2 or more), an AD converter 12, a frequency domain converter 13, and a correlation matrix. Calculation unit 102, eigenvalue decomposition unit 201, frequency averaging processing unit 104, front / lateral orientation determination unit 105, first eigenvector power calculation unit 202, second frequency averaging processing unit 203, and left / right direction determination unit 204. Among these, the same names and symbols as described in the first embodiment are attached except for the eigenvalue decomposition unit 201, the first eigenvector power calculation unit 202, the second frequency averaging processing unit 203, and the left / right direction determination unit 204. Since these are the same components as those described above, descriptions of functions and processes are omitted.

The eigenvalue decomposition unit 201 performs the same processing as the eigenvalue decomposition unit 103 of the first embodiment, and further outputs a first eigenvector vv ₁ (ω, k) corresponding to the first eigenvalue λ ₁ (ω, k) ( S7).
The first eigenvector power calculation unit 202 receives the first eigenvector vv ₁ (ω, k) as input, and v _1,1 (ω, k),... Constituting the first eigenvector vv ₁ (ω, k),. For M elements of v _{1, M} (ω, k), the power is calculated by the equation (7), respectively, and M power elements pv _1,1 (ω, k),..., pv _{1, M} (ω, k) is output (S8).
pv _{1, i} (ω, k) = | v _{1, i} (ω, k) | (7)

The second frequency averaging processing unit 203 formulas _M power elements pv _1,1 (ω, k),..., Pv _{1, M} (ω, k) generated for each frequency ω, respectively. The average value is calculated according to (8), and M averaged power elements apv _1,1 (k),..., Apv _{1, M} (k) are output (S9).

Here, F ₂ is an index of frequencies used for averaging, | F ₂ | is the total number of frequency indexes, and F ₂ is appropriately set so as to satisfy Ω ≧ | F ₂ |.
In addition, you may implement the process of said S8 and S9 in parallel with the process of S5 and S6.

The left / right direction determination unit 204 receives the determination result of the front / side direction determination unit 105 and M averaged power elements apv _1,1 (k),..., Apv _{1, M} (k) as inputs. -If the determination result in the landscape orientation determination unit 105 is front-facing, it is output as it is as a determination result, and if it is landscape orientation, the result is determined by further deciding whether to speak leftward or rightward Is output (S10). The left-right orientation determination is performed by using two averaged power elements apv _{1, α} (k), apv corresponding to two microphones 101-α and 101-β among the M microphones constituting the microphone array 101. This is done by taking a ratio of _{1, β} (k) and comparing it to a predetermined threshold thr2. FIG. 10 illustrates an example of the determination image in the horizontal direction. In this example, a case where the speech is directed toward the midpoint between the microphone α and the microphone β shown in FIG. 10A is set as apv _{1, α} (k) / apv _{1, β} (k) = thr2, and this direction is set as follows. The reference speech direction in the left-right direction is determined based on the magnitude relationship between apv _{1, α} (k) / apv _{1, β} (k) and thr2. Specifically, since the rate of attenuation of apv _{1, β} (k) is larger than apv _{1, α} (k) as it goes to the left, apv _{1, α} (k) / apv _{1, β} (k )> Thr2 can be determined to be leftward (FIG. 10 (b)), and the more rightward is, the greater the rate at which apv _{1, α} (k) attenuates than apv _{1, β} (k). Therefore, when apv _{1, α} (k) / apv _{1, β} (k) <thr2, it can be determined to be rightward (FIG. 10 (c)). Note that the two microphones have the widest left-right spacing with respect to the speaker's position so that the average power elements apv _{1, α} (k) and apv _{1, β} (k) tend to differ. It is desirable to choose.

  The theoretical background that can determine the left-right orientation with such a configuration will be described. As described in the first embodiment, the power of the basis vector representing the direct wave is significantly reflected in the first eigenvalue. Therefore, the value of each power element corresponding to the M microphones of the first eigenvector corresponding to the first eigenvalue is a measure of how strong the direct wave reaches each of the M microphones. Can think. The power at which direct waves reach each microphone varies depending on the direction of speech. Therefore, as described above, the ratio of the power elements of any two microphones in the reference utterance direction is set as a threshold (thr2), and the threshold and the power elements of the two microphones in a certain utterance direction It is possible to determine whether the utterance is uttered in the left direction or the utterance in the right direction with respect to the reference utterance direction from the magnitude relationship.

  As described above, according to the utterance front / sideways estimation apparatus of the second embodiment, in addition to the effects of the configuration of the first embodiment, it is possible to determine which side is the left or right direction with respect to the sideways direction. It is possible to communicate more smoothly.

[Front / Horizontal Judgment Section]
As a method for determining whether the speaker is facing frontward or sideways, the method using the magnitude of the first averaged eigenvalue has been described in the first embodiment. A method for determining whether the utterance direction of the speaker is the front direction or the side direction will be described in more detail.
FIG. 11 conceptually shows the acoustic propagation vector group constituting the spatial correlation matrix and the eigenspace of the spatial correlation matrix for each of the case where the speaker is front-facing, left-facing, and right-facing. The acoustic propagation vector group constituting the spatial correlation matrix is considered to be a combination of the direct wave, the initial reflected wave, and the rear reverberation. It is assumed that the shape of the eigenspace formed by these eigenvectors can be represented by the shape shown by the dotted line.

  This shape depends on the magnitude of the acoustic propagation wave that is the base of the direct wave because the initial reflected wave is mixed smaller in the front direction than in the case of the left and right direction. That is, in the case of facing the front, the shape has a large deviation in the side length. In addition, since the initial reflected wave is mixed greatly in the case of the horizontal orientation on the left and right, the shape is greatly affected by the initial reflected wave while depending on the acoustic propagation that is the base of the direct wave. That is, the shape of the eigenspace of the spatial correlation matrix in the horizontal direction is a shape in which the side length deviation is smaller than that in the front direction.

  It is also possible to determine whether the speaker is facing front or side by evaluating the shape of this eigenspace. FIG. 12 shows a functional configuration example of the front / side orientation determination unit 125 of the method for evaluating the eigenspace shape, the eigenvalue decomposition unit 123, and the frequency averaging processing unit 124. FIG. 13 shows a processing flow of the front / side orientation determination unit 125. The front / side orientation determination unit 125 includes an eigenspace shape evaluation unit 125a and a determination unit 125b.

The eigenvalue decomposition unit 123 is basically the same as the eigenvalue decomposition unit 103 described above. However, the eigenvalues λ ₁ ,..., Λ _M are normalized by the equation (9), and the first normalized eigenvalue sequence nΛ (ω, k ) = [Nλ ₁ (ω, k),..., Nλ _M (ω, k)].

Here, subscript m of nλ _m (ω, k) is substituted for 1 to M, and by calculating each, the elements constituting the first normalized eigenvalue sequence nΛ (ω, k) are calculated. .
The frequency averaging processing unit 124 calculates an average value for the first normalized eigenvalue sequence nΛ (ω, k) obtained for each frequency ω by the equation (10), and calculates the first averaged eigenvalue sequence aΛ (k ) = [Aλ ₁ (k),..., Aλ _M (k)].

The eigenspace shape evaluation unit 125a includes a first averaged eigenvalue sequence aΛ (k) = [aλ ₁ (k),..., Aλ _M (k) composed of a number of first averaged eigenvalues corresponding to the number M of microphones. ] Is input, and the eigenspace shape evaluation value Sλ shown in Expression (11) is output (S125a).

Here, K ₁ is an integer that satisfies 1 ≦ K ₁ <M, and is a constant set in advance in the eigenspace shape evaluation unit 125a. K ₁ may be given from the outside. The eigenspace shape shown in FIG. 11 can be simulated by Sλ obtained by accumulating the K _first averaged eigenvalues aλ _m (k).

The determination unit 125b determines the eigenspace shape evaluation value Sλ with the threshold value thr1, and determines whether or not the direction of the utterance of the speaker is the front direction (S125b). The threshold value thr1 is set to, for example, a predetermined value for determining the speaker direction based on the eigenspace shape evaluation value Sλ obtained empirically from data with different speaker directions. If the eigenspace shape evaluation value Sλ is larger than the threshold value thr1, it is determined that the front direction has a relatively large side length deviation, and the front flag S _F = 1 is set (S125c). That is, it is determined that the speaker is speaking from the front of the microphone.

On the other hand, if the eigenspace shape evaluation value Sλ is smaller than the threshold value thr1, the eigenspace shape evaluation value Sλ is determined to be a lateral direction with a relatively small side length deviation and is reset to the front flag S _F = 0 (S125d). ). That is, it is determined that the speaker is speaking from the side of the microphone.
FIG. 14 shows a functional configuration example of the front / side orientation determination unit 145 that obtains the shape of the eigenspace by another method. The front / side orientation determination unit 145 is different from the front / side orientation determination unit 125 in that the eigenspace shape evaluation value Sλ is obtained by Expression (12).

Considered constant _{K 2} and threshold thr2 is the same as the front-lateral determination unit 125. A processing flow of the front / side orientation determination unit 145 is shown in FIG. Since the processing flow is basically the same as that of the front / side orientation determination unit 125, a description thereof will be omitted. In this way, the eigenspace shape evaluation value Sλ obtained by multiplying and summing a predetermined number of first averaged eigenvalue sequences may be used.

In addition, it is possible to reduce the total amount of calculation of the utterance front / sideways estimation device by using other numbers (K ₁ , K ₂ ) that accumulate or sum the first averaged eigenvalue sequence. That is, it is useless to obtain the first averaged eigenvalue sequence more than necessary to determine the eigenspace shape. Therefore, K ₁ and K ₂ may be given to the eigenvalue decomposition unit 123 to calculate a necessary number of first normalized eigenvalues. By doing so, the calculation amount in the eigenvalue decomposition unit 123 and the frequency averaging processing unit 124 can be reduced.

Further, a front / side orientation determination unit 165 shown in FIG. 16 is also conceivable. The front / side orientation determination unit 165 includes a norm evaluation unit 165a and a determination unit 165b. The norm evaluation unit 165a evaluates the first averaged eigenvalue sequence using Expression (13), and outputs the norm ^Cθ˜ (k).

Here ^mΛ θ~, the model eigenvalues column was prepared for each ^- a candidate for speaker orientation _{^{θ mΛ θ~ = [mλ 1 θ~}} , ..., mλ M θ~] is. This model eigenvalue m [lambda ^Shita～ is, for example, a value calculated in advance from speaker facing candidate θ data measured every ^~.
The Nomuru value C ^Shita～ from candidates theta ^~ smallest speaker orientation is ^(k), it determines whether front or sideways determining unit 165b. For example, if the norm value is minimum when facing left, it is determined that the side is facing horizontally. The idea of the determination is the same as that of the front / side determination units 125 and 145 described above.

When the configuration of the utterance front / side orientation estimation device of each of the above embodiments is realized by a computer, the processing contents of the functions that each device should have are described by a program. The processing functions are realized on the computer by executing the program on the computer. In this case, at least a part of the processing content may be realized by hardware.
Further, the various processes described above are not only executed in time series according to the description, but may be executed in parallel or individually as required by the processing capability of the apparatus that executes the processes. In addition, it can change suitably in the range which does not deviate from the meaning of this invention.

[Verification of effects]
In the sound collection environment shown in FIG. 17 (a), the effect of the present invention was verified under the conditions shown in FIG. 17 (b). The definition of the utterance direction is as shown in FIG.
FIG. 18 shows the relationship between each utterance direction and the first to seventh averaged eigenvalues aλ _i (k). Here, the second to seventh averaged eigenvalues are obtained by the same method as the first averaged eigenvalue. FIG. 18A shows the case where the reverberation time is 250 msec, and FIG. 18B shows the case where the reverberation time is 400 msec. In any reverberation time, the first averaged eigenvalue varies greatly depending on the speech direction, but the second to seventh averaged eigenvalues have a small difference depending on the speech direction. The first eigenvalue is the largest when it is 0 ° (frontward) and the smallest when it is ± 90 ° (laterally). From this verification result, it can be seen that the configuration of the present invention for determining the front / sideways orientation based on the first averaged eigenvalue shown in the first embodiment is valid and effective.

  FIG. 19 shows a comparison between the utterance direction in the left-right direction estimated by the configuration of the second embodiment and the actual utterance direction. FIG. 19A shows the case where the reverberation time is 250 msec, and FIG. 19B shows the case where the reverberation time is 400 msec. In any reverberation time, a correct answer rate of 75% or more was obtained. From this verification result, it can be seen that the configuration of the present invention for determining the left-right orientation from the two power elements of the first eigenvector shown in the second embodiment is valid and effective.

[Service application example]
FIG. 20 is a structural example of a service in which the present invention is incorporated in an audio conference terminal. Assume that the conference hall A and the conference hall B are connected by a voice terminal through a network. Speech direction information is extracted from a voice signal picked up by a microphone attached to the voice conference terminal, and transmitted to the other party along with the voice information. By presenting the utterance direction information as visual information on the other party side, it is possible to convey the situation of the place that is difficult to convey only with the voice information.

  Also, the speaker orientation estimation technology can be applied to a minutes system that records the state of communication in a meeting using video and audio. The speaker orientation estimation technology can tag the recorded voice and video as to who spoke to whom, which is useful for organizing the minutes. In addition, services performed by detecting face orientation in images, for example, surveillance / crime prevention services used with surveillance cameras and intercoms, and services that determine whether or not attention is paid to advertisements using digital signage (electronic signage) Etc. can also be replaced with an audio signal output.

Claims (9)

Generates a correlation matrix representing a correlation between the audio signal picked up by the plurality of microphones constituting the Ma Ikurohon'arei, from at least the largest eigenvalue of the correlation matrix, speaker is frontally with respect to the microphone array An utterance front / side direction estimation device that estimates whether a utterance or a sideways utterance. De Ijitaru audio signal picked up by the microphone of the M constituting the microphone array (M is an integer of 2 or more), and the frequency domain conversion unit for converting the frequency domain from the time domain in units of frames consisting of a plurality of samples,
For each frequency, a correlation matrix calculator Ru obtain the correlation matrix of M × M representing the correlation between each of said digital audio signal converted into the frequency domain,
By comparing the evaluation value based on at least the maximum eigenvalue of the correlation matrix with a predetermined threshold value , it is determined whether the speaker speaks in front or side with respect to the microphone array and outputs the result. A front / side orientation determination unit,
An utterance front / side orientation estimation device characterized by comprising:   A frequency domain conversion unit for converting a digital audio signal collected by M microphones (M is an integer of 2 or more) constituting a microphone array from a time domain to a frequency domain in units of frames composed of a plurality of samples;
  A correlation matrix calculator that obtains an M × M correlation matrix representing the correlation between the digital audio signals converted into the frequency domain for each frequency;
  An eigenvalue decomposition unit for obtaining a first normalized eigenvalue obtained by normalizing the maximum eigenvalue of the correlation matrix;
  By comparing the first normalized eigenvalue or the first averaged eigenvalue that is the frequency average of the first normalized eigenvalue with a predetermined threshold value, the speaker speaks in front of the microphone array. A front / side determination unit that determines whether the user has spoken in a horizontal direction and outputs the result,
An utterance front / side orientation estimation device characterized by comprising:   A frequency domain conversion unit for converting a digital audio signal collected by M microphones (M is an integer of 2 or more) constituting a microphone array from a time domain to a frequency domain in units of frames composed of a plurality of samples;
  A correlation matrix calculator that obtains an M × M correlation matrix representing the correlation between the digital audio signals converted into the frequency domain for each frequency;
  An eigenvalue decomposition unit for obtaining a first normalized eigenvalue obtained by normalizing the eigenvalue of the correlation matrix;
  A sum or product of a plurality of first normalized eigenvalues including at least a first normalized eigenvalue corresponding to the largest eigenvalue and not including a first normalized eigenvalue corresponding to the smallest eigenvalue, or of the first normalized eigenvalue Among the first averaged eigenvalues that are frequency averages, a plurality of first averaged eigenvalues that include at least the first averaged eigenvalue corresponding to the largest eigenvalue and do not include the first averaged eigenvalue corresponding to the smallest eigenvalue. By comparing the sum or product with a predetermined threshold value, it is determined whether the speaker speaks in front or side to the microphone array, and outputs a result.
An utterance front / side orientation estimation device characterized by comprising: Generates a correlation matrix representing a correlation between the sound signal picked up by the plurality of microphones constituting the Ma Ikurohon'arei, from at least the largest eigenvalue of the correlation matrix, speaker is frontally relative to said microphone array An utterance front / side direction estimation method that estimates whether a utterance or a sideways utterance. De Ijitaru audio signal picked up by the microphone of the M constituting the microphone array (M is an integer of 2 or more), and the frequency domain conversion step of converting the frequency domain from the time domain in units of frames consisting of a plurality of samples,
For each frequency, a correlation matrix calculating step Ru obtain a correlation matrix of M × M representing the correlation between each of said digital audio signal converted into the frequency domain,
By comparing the evaluation value based on at least the maximum eigenvalue of the correlation matrix with a predetermined threshold value , it is determined whether the speaker speaks in front or side with respect to the microphone array and outputs the result. Front / side orientation determination step,
An utterance front / side direction estimation method comprising:   A frequency domain conversion step of converting a digital audio signal collected by M microphones (M is an integer of 2 or more) constituting a microphone array from a time domain to a frequency domain in units of frames composed of a plurality of samples;
  A correlation matrix calculating step for obtaining an M × M correlation matrix representing the correlation between the digital audio signals converted into the frequency domain for each frequency;
  An eigenvalue decomposition step of obtaining a first normalized eigenvalue obtained by normalizing the maximum eigenvalue of the correlation matrix;
  By comparing the first normalized eigenvalue or the first averaged eigenvalue that is the frequency average of the first normalized eigenvalue with a predetermined threshold value, the speaker speaks in front of the microphone array. A front / landscape determination step that determines whether the user has spoken in a horizontal direction and outputs the result,
An utterance front / side direction estimation method comprising:   A frequency domain conversion step of converting a digital audio signal collected by M microphones (M is an integer of 2 or more) constituting a microphone array from a time domain to a frequency domain in units of frames composed of a plurality of samples;
  A correlation matrix calculating step for obtaining an M × M correlation matrix representing the correlation between the digital audio signals converted into the frequency domain for each frequency;
  An eigenvalue decomposition step of obtaining a first normalized eigenvalue obtained by normalizing the eigenvalue of the correlation matrix;
  A sum or product of a plurality of first normalized eigenvalues including at least a first normalized eigenvalue corresponding to the largest eigenvalue and not including a first normalized eigenvalue corresponding to the smallest eigenvalue, or of the first normalized eigenvalue Among the first averaged eigenvalues that are frequency averages, a plurality of first averaged eigenvalues that include at least the first averaged eigenvalue corresponding to the largest eigenvalue and do not include the first averaged eigenvalue corresponding to the smallest eigenvalue. A front-side / right-side determination step for determining whether the speaker has spoken in front or side to the microphone array by comparing the sum or product with a predetermined threshold, and outputting the result; and
An utterance front / side direction estimation method comprising: The program for functioning a computer as an apparatus as described in any one of Claims 1-4 .

Images