JP5034607B2 - Acoustic echo canceller system - Google Patents

Description

  The present invention belongs to an acoustic echo cancellation technology for a telephone conference system or a video conference system having a speaker and a microphone.

  There is a telephone conference system or a video conference system that has a speaker and a microphone on both sides and can talk by voice with a person connected by a network. This system has a problem that sound output from the speaker is mixed into the microphone. Therefore, conventionally, the acoustic output from the speaker (acoustic echo) mixed in the microphone has been removed using the acoustic echo canceller technique. If the acoustic environment of the conference room is unchanged, it is possible to first learn how to transmit the sound in the space (impulse response) only once and use the impulse response to completely remove the acoustic echo. However, when the conference participant moves from the seat, the acoustic path of the acoustic echo changes, so the learned impulse response and the actual impulse response become unmatched, and the acoustic echo cannot be completely removed. In the worst case, the residual echo turns around and the volume gradually increases, causing a phenomenon called howling and making it difficult to talk.

  In view of this, a method has been proposed in which an acoustic response is sequentially learned and the acoustic echo is always completely canceled by following the change in the acoustic path (for example, Non-Patent Document 1).

  In addition, a method of canceling acoustic echoes using a microphone array has been proposed (for example, Patent Document 1). In the prior art, the performance of the echo canceller is not sufficient, so when the near-end speaker and far-end speaker speak at the same time, the voice of the speaker with low volume is completely shut out, and the one-side call state is set. Preventing howling is done. However, there is a problem that it is difficult to talk in a one-side call.

JP-A-2005-136701 Peter Heitkamper, “An Adaptation Control for Acoustic Echo Cancers,” IEEE Signal Processing Letters, Vol. 4, No. 6, 1997/6. R. O. Schmidt, “Multiple Emitter Location and Signal Parameter Estimation,” IEEE Trans. Antennas and Propagation, vol.34, no.3, pp.276-280, 1986. Masato Togami, Akio Amano, Hiroshi Shinjo, Ryota Kamoshida, Junichi Tamamoto, Kou Egawa, “Hearing Function of Human Symbiotic Robot“ EMIEW ”,” 22nd AI Challenge Study Group, pp. 59-64, 2005

  In the conventional method of learning impulse responses sequentially and following changes in the acoustic path, it is possible to learn sequentially when the sound is coming from only the speaker, but the speaker speaks and the speaker in the conference room speaks. In such a case, learning becomes impossible. In the worst case, learning of the impulse response fails, and acoustic echo cannot be removed at all. Therefore, it is necessary to determine whether sound is output from only the speaker or a speaker in the conference room (double talk detection).

  In the present invention, the situation where the sound from the speaker is dominant is detected, and adaptive control of the echo canceller is performed. As a configuration example for this purpose, the arrival direction of a sound source can be estimated by using a microphone array having a plurality of microphone elements. In a more preferred aspect, it is possible to determine a situation in which sound from a speaker is dominant by detecting a phase difference between sounds input to a plurality of microphone elements. The determination can be made by comparison with a threshold value stored in advance. In addition, in a preferable aspect, there is provided an acoustic echo canceller adaptation unit that extracts only a band-divided signal whose sound source arrival direction is the speaker direction and adapts the acoustic echo canceller with the band-divided signal.

  The acoustic echo canceller can cancel the echo by creating a pseudo sound from the speaker and subtracting it from the input voice.

  As a typical example of the configuration of the present invention, a microphone for inputting sound, an AD converter for digitally converting a signal from the microphone, and an information processing for processing a digital signal from the AD converter and suppressing an acoustic echo component A device, an output interface for sending a signal from the information processing device to the network, an input interface for receiving the signal from the network, a DA converter for analog conversion of the signal from the input interface, and a signal from the DA converter The conference system includes a speaker that outputs sound, and the information processing device controls optimization timing of the information processing device based on a state of sound input to the microphone. The AD converter and the DA converter may be integrated.

  Preferably, the information processing apparatus is optimized at the timing when the sound input to the microphone is mainly from the direction of the speaker. The determination can be made by setting an appropriate threshold value, for example.

  More preferably, the information processing apparatus uses an adaptive filter, an acoustic echo canceller adaptation unit that optimizes the adaptive filter, and the adaptive filter to suppress an acoustic echo component that is a mixture of speaker sounds from a digital signal. And an acoustic echo canceling unit.

  More preferably, the microphone is a microphone array having a plurality of microphone elements, the AD converter is a plurality of AD converters for digitally converting a signal for each microphone element, and the information processing device is supplied from a plurality of AD converters. A phase difference calculation unit that calculates a phase difference between sounds input to a plurality of microphone elements based on the signal, and the sound input to the microphone array is output from the speaker from the phase difference output by the phase difference calculation unit. It has a frequency distribution part which determines whether it is a voice of.

  More preferably, the information processing apparatus has a band dividing unit that divides a digital signal into bands, and the band dividing unit divides each digital signal digitally converted for each microphone element, and the phase difference calculation unit divides the digital signal. The phase difference between sounds input to a plurality of microphone elements for each band is calculated, and the frequency distribution unit is a speaker output signal from the phase difference for each band output by the phase difference calculation unit. The acoustic echo canceller adapting unit suppresses the mixing of the speaker sound from the microphone element signal only in the band determined by the frequency distribution unit as the speaker output signal. The acoustic echo canceller is adapted from the signal of each microphone element using the adaptive filter. To remove the code component. In the band dividing unit, for example, a frequency from 20 Hz to 16 kHz is divided every 20 Hz. Thus, by performing control for each frequency band, highly accurate echo cancellation can be performed.

  More preferably, in order to determine whether or not the output signal is a speaker output signal, the frequency distribution unit measures in advance a transfer function of sound transmitted from the speaker to the microphone array, and the sound is output from the speaker from the measured transfer function. When calculating the phase difference for each band of the microphone array, store the phase difference for each band in the external storage device, and store the phase difference between the microphone elements for each band of the band-divided signal. Is equal to or less than a predetermined threshold value, the band division signal is determined to be a speaker output signal.

  More preferably, it has a user interface characterized in that the user designates the number of speakers and the physical position relative to the microphone array in advance, and the sound from the speakers is determined from the number and physical positions of the speakers designated by the user interface. Has an echo phase difference calculation processing unit that calculates a phase difference for each band of the microphone array, and stores the phase difference for each band in an external storage device, and stores the stored phase difference and band When the phase difference between the microphone elements for each band of the divided signal is equal to or less than a predetermined threshold value, the band divided signal is determined to be a speaker output signal.

  More preferably, using a phase difference of each microphone array, a sound source direction histogram is calculated across the band, and a sound source localization unit that estimates the sound source direction from the histogram is provided. The magnitude of the signal estimated to have been calculated is calculated, and the magnitude of the calculated signal is determined to be the loudspeaker output signal or the loudspeaker output signal after the acoustic echo canceller. In comparison with the size of the band division signal determined to be equal to or smaller than the predetermined size, the size of the band division signal is reduced or all are set to zero.

  The echo canceller can be controlled dynamically according to the situation of the conference room.

  Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. The present invention is, for example, a teleconference system using an IP network line, and two (or more) sites connected by a network communicate with each other using a teleconference facility including a microphone array and a speaker. , Enable conversations between speakers at both sites. Hereinafter, both of these sites are referred to as a near end and a far end.

  FIG. 1 is a diagram showing a configuration of hardware according to the present invention arranged at the near end and the far end of a site. A microphone array 1 composed of at least two or more microphone elements, an A / D converter for converting analog sound pressure values inputted from the microphone array into digital data, and a D / A converter 2 for converting digital data into analog data A central processing unit 3 for processing the output of the conversion device 2, for example, a volatile memory 4, a hub 5 for connecting to a network and transmitting / receiving data to / from the far end, and D / A converted analog data The speaker 6 that converts the sound pressure through the speaker, for example, a non-volatile external storage medium 7.

  The multi-channel sound pressure values recorded by the microphone array 1 are sent to the AD / DA converter 2 and converted into multi-channel digital data. The converted digital data is stored in the memory 4 via the CPU 3.

  The far-end audio sent via the hub 5 is sent to the AD / DA converter 2 via the CPU 3 and output from the speaker 6. The far-end voice output from the speaker 6 is mixed with the voice recorded by the microphone array 1 together with the near-end talker voice. Accordingly, the far-end sound output from the speaker is also mixed in the digital sound pressure data stored in the memory 4. The CPU 3 performs echo cancellation processing to suppress the mixed far-end voice from the digital sound pressure data stored in the memory 4 and transmits only the near-end speaker voice to the far-end side via the hub 5. To do. Echo cancellation processing suppresses the far-end sound by using data such as the number of speakers and the position of the speakers, as well as data related to the sound transmission from the speakers to the microphone array stored in advance in the external storage medium 7 To do.

  FIG. 2 is a diagram showing a software configuration of the present invention, and is executed digitally by the CPU 3 except for the A / D converter 8 that converts analog data into digital data. This is the most convenient method if the processing capacity of the CPU is sufficient. Alternatively, a hardware configuration equivalent to this may be used, and digital or analog processing may be used.

  The main functional blocks are a band division unit 9 that converts digital data into band-divided data, a phase difference calculation unit 10 that calculates a phase difference between each microphone channel of the band division signal, and a phase difference calculation unit that calculates From the phase difference, for each band of the band-divided signal, a frequency distribution unit 11 that determines whether acoustic echo is dominant or speaker voice is dominant, an acoustic echo canceller adaptation unit 12 that adapts an adaptive filter for acoustic echo cancellation, It consists of a pseudo echo generation unit 13 that pseudo-generates an acoustic echo transmitted to the microphone array from the speaker reference signal, an acoustic echo cancellation unit 14 that suppresses the acoustic echo from the input signal using an adaptive filter, and the like. Multi-channel analog sound pressure data recorded by the microphone array 1 is converted into multi-channel digital data x (t) by the A / D converter 8. The converted multi-channel digital data is sent to the band dividing unit 9 and converted into multi-channel band divided data x (f: τ). Here, since the microphone array includes a plurality of microphone elements, the A / D conversion unit 8 and the band dividing unit 9 may be arranged in parallel by the number of microphone elements.

  For band division, short-time Fourier transform, wavelet transform, band-pass filter, or the like is used. In the band dividing unit, for example, a frequency from 20 Hz to 20 kHz is divided every 20 Hz. τ is a frame index when performing short-time frequency analysis. The band-divided data is sent to the phase difference calculation unit 10. The phase difference calculator 10 calculates the phase difference for each microphone channel by [Equation 1].

xi (f, τ) is the f-th band division data of i channel. Similarly, xj (f, τ) is the f-th band division data of j channel. Δi, j (f, τ) is a phase difference regarding the f-th band of the i channel and the j channel. The calculated phase difference for each microphone channel is sent to the frequency distribution unit 11. In the frequency distribution unit 11, the phase difference Spi, j (f) of the echo component from the speaker to the microphone array set in advance and the phase difference for each microphone channel are defined as ei, j (f, τ) is calculated, and if the sum of ei, j (f, τ) with respect to index i, j is equal to or less than a preset threshold, it is determined that the f-th band is a band in which echo is dominant, and index i , j is determined to be near-end speech if the sum of ei, j (f, τ) is equal to or greater than a preset threshold value.

The frequency component determined to be an echo is sent to the acoustic echo canceller adaptation unit 12. The acoustic echo canceller adaptation unit 12 stores adaptive filter setting conditions for each of the divided frequency bands. The acoustic echo canceller adaptation unit 12 uses the pseudo echo component Echo i (f, τ) output by the pseudo echo generation unit 13 for the frequency band determined by the frequency distribution unit 11 as an echo, using [Equation 3]. Adapt the adaptive filter hi, τ (f, T).

Echo i (f, τ) is a pseudo echo component of the i-th channel microphone. hi, τ (f, T) is a filter of the T-tap in the f-th band of the adaptive filter of the i-th channel microphone, and is a filter that has been adapted with signals up to τ−1 frames. L is the tap length of the adaptive filter. Adaptation may be performed for each frequency in this way, and when the number of bands determined to be in the speaker direction at a time τ is equal to or greater than a predetermined threshold, all the frequency components at the time τ [Equation 3] You may adapt with The sound source localization for each frequency may be performed based on the MUSIC method or the modified delay sum array method (see Non-Patent Document 3). The pseudo echo generation unit 13 generates a pseudo echo component e ^ (f, τ) defined by [Equation 4].

d (f, τ) is a band division signal of the original signal output to the speaker. Further, the pseudo echo generator 13 updates the echo phase difference DB with [Equation 5] from the pseudo echo.

The phase difference between microphones of the adaptive filter is stored in the echo phase difference DB. The acoustic echo cancellation unit 14 uses the adaptive filter adapted by the acoustic echo canceller adaptation unit 12 to generate and output the audio digital data x ^ i (f, τ) after acoustic echo suppression by [Equation 6].

As described above, in this embodiment, by using a microphone array having a plurality of microphone elements, a band in which the speaker sound is dominant is determined from the phase difference between microphones, and adaptive control is performed only for that band. A filter with high echo suppression performance can be generated. In addition, since the double talk can be detected, it is possible to adapt the acoustic echo canceller when sound is emitted only from the speaker. Therefore, it is possible to always follow fluctuations in the acoustic path, and learning of the impulse response is paused when the speaker in the conference room speaks when sound is emitted from the speaker. Become.

  FIG. 3 is a block diagram of a system that sets the initial value of the echo phase difference DB using GUI information that specifies the number of speakers and speaker positions. From the function block and database of the echo phase difference calculation processing unit 16 that calculates the phase difference of the acoustic echo from the set number and position of the speakers, the speaker number / position setting GUI 15 for designating the number of speakers and the physical position used for the video conference. Thus, it is realized by a CPU and storage means.

  In the speaker number / position setting GUI 15, the number of speakers and the position of the speaker with respect to the microphone array 1 are set. In the speaker number / position setting GUI 15, it is essential to set the speaker direction with respect to the microphone array 1, but the number and position information of the speakers set in the speaker number / position setting GUI are sent to the echo phase difference calculation processing unit 16. Sent. The echo phase difference calculation processing unit 16 estimates the phase difference Spi, j (f) between the microphones of the i-channel and the j-th channel of the acoustic echo based on the assumption of FarField from the positions that span the number of speakers. The estimated echo phase difference is stored in the echo phase difference DB.

  FIG. 4 shows that the adaptive algorithm and sound source direction of the present invention are used to switch between using an echo canceller or a voice switch, so that even in a large-scale conference room exceeding the limit of the echo canceller performance, It is possible to realize a video conference system that does not cause howling. In addition to the voice echo canceling unit 14, a sound source localization unit 17 that estimates the power of the speaker sound, a VoiceSwitch determination unit 18 that determines whether to use the VoiceSwitch from the size of the acoustic echo and the power of the speaker sound, An output signal generation unit 19 that outputs a signal after acoustic echo suppression is provided.

  The A / D conversion unit, the band division unit, the phase difference calculation unit, the frequency distribution unit, the acoustic echo canceller adaptation unit, the pseudo echo generation unit, and the acoustic echo cancellation unit are the same processing as in FIG. The sound source localization unit 17 calculates a histogram of phase differences of frequency components that are not regarded as echo components by the frequency distribution unit 11. The sound source direction is identified from the peak of the calculated phase difference histogram. The number of sound source directions to be identified is determined in advance, or when it is a peak of a histogram and the frequency is equal to or higher than a certain threshold, it is regarded as the sound source direction. For the identified sound source direction, the sum of all powers is defined as near-end speaker power. The sound source localization unit 17 outputs near-end speaker power. The VoiceSwitch determination unit 18 calculates, as the acoustic echo power, the sum of the power after acoustic echo cancellation for the frequency for which the acoustic distribution is determined to be dominant by the frequency distribution unit. If the ratio between the calculated acoustic echo power and near-end speaker power is equal to or greater than a predetermined threshold value, it is determined that the frame is mainly an acoustic echo and there is no speaker, and it is determined that VoiceSwitch is used. Also, if it is below a predetermined threshold, it is determined that there is a speaker in the frame, and it is determined that VoiceSwitch is not used. When it is determined that the VoiceSwitch is used, the output signal generation unit 19 generates and outputs a signal in which all values are set to 0. When it is determined that VoiceSwitch is not used, a signal after acoustic echo cancellation output by the acoustic echo canceller is output. If there are many residual echoes in the signal after acoustic echo cancellation, the VoiceSwitch determination unit 18 determines that the VoiceSwitch is used and does not transmit a signal including the residual echo. If a signal including residual echo is transmitted, the system becomes a closed loop, and howling may occur due to the residual echo. In order to prevent residual echo, it is desirable to use VoiceSwitch to prevent the echo from looping. However, if VoiceSwitch is always used, the near-end speaker and the far-end speaker cannot talk at the same time. Therefore, since the VoiceSwitch determination unit 18 of the present invention uses VoiceSwitch only for frames in which residual echo exists, when the residual echo does not occur, the near-end speaker and the far-end speaker can talk simultaneously. When residual echo occurs, switching to VoiceSwitch will dramatically reduce the possibility of howling. In this embodiment, the acoustic echo power used for determination by the VoiceSwitch determination unit 18 is obtained from the signal after acoustic echo cancellation. However, the acoustic echo power may be calculated from the power of the signal before acoustic echo cancellation.

  Also, the residual echo of all frequencies and the near-end speaker power of all frequencies are compared to determine the use of VoiceSwitch. For each subband including several frequency bins, the residual echo and the near-end speaker power You can switch whether to use VoiceSwitch for each subband. In this case, for each subband, the subband determined to use the VoiceSwitch is output by the output signal generator 19 with the value 0 replaced. For the subbands determined not to use VoiceSwitch, the output signal generation unit 19 outputs a signal after acoustic echo cancellation.

  FIG. 5 shows an overall view of the system when the present invention is applied to a video conference system. This system is a video conference system characterized in that the adaptation of the acoustic echo canceller is controlled on the calculator 101 by using the phase difference calculated by the phase difference calculator 10 and the information on the sound source direction.

  FIG. 5 shows the system configuration of one site. In the video conference system 100, the computer 101 performs acoustic signal processing, image processing, and communication processing. An A / DD / A device 102 is connected to the computer 101, and an audio signal recorded by the microphone array 105 is converted into a digital audio signal by the A / DD / A device 102 and sent to the computer 101. The microphone array 105 has a plurality of microphone elements.

The computer 101 performs acoustic signal processing on the digital audio signal, and the processed audio signal is sent to the network via the hub 103. Here, the computer 101 includes the CPU 3, the memory 4, and the external storage medium 7 shown in FIG.

  The external storage medium 7 may be inside the computer 101 or outside the computer 101. Then, in the CPU 3 in the computer 101, as shown in FIG. 2, a band dividing unit 9, a phase difference calculating unit 10, a frequency distributing unit 11, an acoustic echo canceller adapting unit 12, a pseudo echo generating unit 13, an acoustic echo canceller unit 14, or as shown in FIG. 9 to be described later, has an audio transmission unit 201, an acoustic echo canceller adaptation unit 204, an acoustic echo canceller unit 205, an audio recording unit 203, an audio reception unit 207, an audio reproduction unit 208, As a result, an acoustic echo canceller is realized.

  The image signal of the other base sent to the video conference system 100 via the hub 103 is sent to the image display device 104 and displayed on the screen. The audio signal of the other base sent through the hub 103 is output from the speaker 106.

  The sound received by the microphone array 105 includes an acoustic echo transmitted from the speaker 106 to the microphone array 105 and needs to be removed. As the digital cable 110 and the digital cable 113, a USB cable or the like is used.

  FIG. 6 shows a configuration of a conventional acoustic echo suppression process using an acoustic echo model and an acoustic echo canceller using an adaptive filter when sound is transmitted from a speaker to a microphone element.

  All signals are expressed with z-transform. The received signal d (z) is emitted from the speaker and arrives in a form in which the impulse response H (z) of the room is convoluted with the microphone element. The impulse response H (z) includes a direct sound from the speaker to the microphone and a reflection (acoustic echo) from the wall, floor, ceiling, or the like.

  In addition to the acoustic echo, speaker sound N (z) is mixed in the microphone element. If the microphone element signal X (z) is transmitted as it is, an acoustic echo is included in the transmitted voice, and the signal loops. In the worst case, howling occurs and the call becomes impossible. Therefore, it is necessary to suppress only the acoustic echo from the transmitted voice.

  The adaptive filter W (z) is a filter that has learned the impulse response H (z) of the room adaptively. By applying W (z) to the received signal, a pseudo acoustic echo can be created. The adaptive filter W (z) is adapted using, for example, the NLMS method. In the NLMS method, the adaptive filter is updated such that W (z) = W (z) +2 μX (z) N ′ (z) * / | X (z) | ^ 2.

  In the case of W (z) = H (z), only the speaker voice N (z) can be extracted by subtracting the pseudo acoustic echo from the microphone signal.

  When N (z) = 0, the transmitted voice is 0 in the case of W (z) = H (z). That is, in the above update formula of the NLMS method, W (z) is adaptively changed so that the transmitted voice becomes zero.

  However, when N (z) is not 0, W (z) is adaptively changed so that the transmitted voice becomes 0, and W (z) fails to adapt. Therefore, when N (z) is not 0, it is necessary to perform control so as not to adapt.

  FIG. 7 shows a system to which the present invention is applied, which has a double-talk detector having a function of controlling not to adapt when N (z) is not 0. This double talk detector determines whether N (z) is 0, and applies the adaptive filter only when N (z) is close to 0.

  This system to which the present invention is applied is characterized in that the double talk detection unit 702 received by the microphone array makes a determination using information on the arrival direction of the sound source obtained by the phase difference calculation unit 701.

  If the acoustic echo canceller is updated when N (z) is not 0, there is a risk that adaptation will fail and the filter will diverge, so a double-talk detector is essential to avoid that risk. .

  FIG. 8 shows the flow of an audio stream in a video conference system at two sites and the flow of an audio stream in a conference system at three or more sites. Here, the phase difference calculation unit may be in a server or may exist in a CPU at each site.

  In the case of two sites, the transmission signal after acoustic echo cancellation is transmitted from the video conference system at site A to the video conference system at site B via the network and reproduced at site B. The audio from site B is sent to site A for playback.

  In the case of three or more locations, data is once collected in a server or CPU, redistributed to each location, and reproduced.

  FIG. 9 shows a block configuration of a video conference system when the present invention is applied. The received voice transmitted via the network is received by the voice receiving unit 207. The received received voice is sent to the voice playback unit 208. The voice playback unit 208 plays back the received voice using a speaker.

  The received voice is sent to the acoustic echo canceller unit 205. The audio recording unit 203 records the audio signal of the microphone array. The recorded audio signal is sent to the acoustic echo canceller unit 205.

  The acoustic echo canceller unit 205 generates a pseudo echo from the acoustic echo cancellation filter stored in the acoustic echo cancellation filter DB211 and the received voice, and subtracts the pseudo echo from the audio signal of the microphone array. As a result of the subtraction, the remaining error signal is sent to the acoustic echo canceller adaptation unit 204.

  The acoustic echo canceller adaptation unit 204 adapts the acoustic echo canceller so that the error signal becomes zero. The adapted result is stored in the acoustic echo cancellation filter DB211. The error signal output from the acoustic echo cancellation unit 205 is sent to the audio transmission unit 201.

  The voice transmission unit 201 transmits an error signal to another site. The image capturing unit 210 captures an image with a camera. The captured image is sent to the image transmission unit 202 and transmitted to another site.

  The image receiving unit 209 receives an image sent from another base. The received image is sent to the image display unit 206. The image display unit 206 displays the sent image on the screen.

  FIG. 10 shows a processing flow of the video conference system. In the acoustic echo canceller adaptation processing S1, a learning signal is emitted from the speaker, and the acoustic echo canceller is adapted. The learning signal is preferably white noise. The learning signal length is desired to be several seconds to several tens of seconds or more. If the learning length is short, the acoustic echo canceller may not be able to learn the room impulse response sufficiently. As described above, the impulse response can be sufficiently learned by setting the learning signal length to several seconds to several tens of seconds or more.

  After learning is completed, a determination S2 is made as to whether or not a connection request has been issued from another site. When a connection request is issued from another site, connection S4 is performed with the other site.

  If no connection request is issued from another site, it is determined in S3 whether a connection request is issued from the local site. The user issues a connection request from his / her base through the GUI.

  If a connection request is issued from the local site, connection S4 is performed with another site. If the connection request is not issued from the own site, the connection to the other site is not made, and the process returns to the determination of S2 whether the connection request is issued from the other site.

  In other words, the video conference system stands by until a connection request is issued from either its own base or another base.

  After connection S4 with another site, reproduction S6, image display S7, audio recording S8, echo cancellation S9, and audio transmission S10 are repeatedly performed from the speaker until the connection is disconnected.

  Reproduce from speaker In S6, the received voice sent from another site is reproduced.

  In the image display S7, an image sent from another base is displayed on the monitor.

  Audio recording S8 records audio from the microphone array at your site.

  In echo cancellation S9, the acoustic echo component is suppressed from the voice of the recorded microphone array.

  In audio transmission S10, an audio signal after acoustic echo component suppression is transmitted. If it is determined in step S11 that the connection has been lost or not, connection S13 is performed from another site, and the video conference system is terminated.

  If it is determined that the connection has not been disconnected, it is determined whether there is a disconnection request from the user at the local site through the GUI, and if there is a disconnection request, disconnection from other sites is performed S13, and the video conference system is terminated.

  FIG. 11 shows sparsity, which is the basic concept of double talk processing, which is the main element of the present invention.

  In the present invention, the voice signal from the microphone array and the reception signal used for pseudo echo generation are all subjected to short-time Fourier transform, wavelet transform, or subband processing to be converted into a frequency domain signal. It is desirable that the frame size at the time of the short-time Fourier transform is the number of points corresponding to about 50 ms.

  For example, when the sampling rate is 32 kHz, the frame size is preferably 2048 points. The voice is steady for about several tens of ms, and by setting such a frame size, it can be assumed that the sparseness is most established in the frequency domain, and the adaptive processing of the acoustic echo canceller operates with high accuracy. It becomes possible to make it.

  The short-time Fourier transform is preferably performed after applying a Hamming window, Hanning window, Blackman window, or the like. The short-time Fourier transform assumes that the signal repeats with an analysis long period. When the window function is not multiplied, a difference occurs between the values at both ends, and a nonexistent frequency is observed after a short-time Fourier transform. By applying the window function in this way, it becomes difficult to observe a non-existent frequency component, and the frequency analysis accuracy can be improved.

  The frame shift is preferably about 1/4 or 1/8 of the frame size, and the finer the frame shift, the better the sound quality of the output sound. However, as the frame shift is made finer, the processing amount becomes larger. Therefore, it is necessary to make the frame shift fine within a range where real-time processing can be performed at the processing speed of the installed computer.

  FIG. 11 shows a lattice in which the horizontal axis is set to time (frame number) and the vertical axis is set to frequency (frequency bin number).

  In the double talk processing of the present invention, for each time-frequency component, it is determined whether the component is an acoustic echo component or a non-acoustic echo component, and the acoustic echo canceller is applied only for the time-frequency determined to be an acoustic echo component. Process.

  It is known that speech is a sparse signal when viewed in the time-frequency domain, and it is rare for multiple speeches to mix on the same time-frequency.

  When both the received signal and the transmitted signal are audio signals, as in a video conference system, only the acoustic echo component is highly accurate by allocating the acoustic echo component or non-acoustic echo component for each time-frequency due to sparsity. Can be extracted.

  FIG. 12 shows a basic configuration of an acoustic echo canceller to which the present invention is applied.

  First, the band dividing unit performs frequency decomposition S101 on the audio signal input to the computer 101 from the microphone array having a plurality of microphone elements, and converts the recorded audio into a frequency domain signal.

  Next, the phase difference calculation unit calculates the inter-element phase difference of the recorded voice.

  Next, the frequency distribution unit determines which voice the band division signal is based on the phase difference of each band output from the phase difference calculation unit. That is, it is determined whether the output signal is a speaker output signal or a speaker signal.

  Then, the acoustic echo canceller removes the voice included in the band division signal S102.

  In S102, the pseudo echo W (z) d (z) is generated by multiplying W (z) by the reference signal d (z). By subtracting W (z) d (z) from the microphone input signal x (x), the acoustic echo can be eliminated from the microphone input signal.

  The acoustic echo canceller adaptation unit determines that it is in the double talk state when there are many components that are speaker output signals in the band division signal, and performs the acoustic echo canceller adaptation processing S104.

  In the adaptive processing of the acoustic echo canceller, the acoustic echo canceller filter W (z) is updated by the NLMS method or the like. In the NLMS method, W (z) = W (z) +2 μX (z) N ′ (z) * / | X (z) | ^ 2 is updated.

  If in the double talk state, the echo canceller adaptation S103 is not performed, and the acoustic echo canceller process is terminated.

  FIG. 13 shows a processing flow of an adaptation process that uses sound source direction information and regards only time-frequency components coming from the speaker direction as acoustic echoes.

  In frequency resolution S201, the recorded voice is converted into a time-frequency domain signal.

  In sound source localization S202, sound source direction estimation is performed based on the modified delay sum array method. The modified delay sum array method used for sound source direction estimation in the present invention determines from which direction the component comes from every time-frequency.

  As shown in FIG. 11, since the sound is a sparse signal when viewed for each time-frequency, it can be assumed that the acoustic echo is divided into main components and the non-acoustic echo is divided into main components for each time-frequency. . Therefore, if a time-frequency component coming from the speaker direction is selected from the sound source localization result estimated for each time-frequency, it can be considered that the component is an acoustic echo as the main component.

  In the modified delay sum array method, sound source localization is performed using the steering vector Aθ (f) in the sound source direction θ. When the number of microphones is M, Aθ (f) is an M-dimensional complex vector.

  Here, f represents a frequency bin number. The time-frequency representation of the input signal is described as X (f, τ). τ is the frame number of the short-time Fourier transform. X (f, τ) is an M-dimensional complex vector, which is a vector having elements of the frame τ and the frequency f of each microphone element.

  In the modified delay sum array method, the virtual sound source direction θ that maximizes | Aθ (f) * X (f, τ) | is estimated as the sound source direction of frame τ and frequency f.

  In the speaker direction specification S203, a histogram is created by accumulating the number of frequencies f estimated that the sound source direction is θ or log | Aθ (f) * X (f, τ) | for each virtual sound source direction θ. Then, the peak of the histogram is calculated within a predetermined speaker direction range (for example, determined from -30 ° to 30 °), and the direction is set as the speaker direction θsp.

  In the determination of S204 whether the sound source direction is the speaker direction, if the estimated sound source direction θ ′ of the frequency f satisfies | θ′−θsp | <β, the frequency f is regarded as a frequency component arriving from the speaker direction. Then, it is regarded as an acoustic echo component, and echo canceller adaptation S205 is performed.

  After performing S204 and S205 at all frequencies, the adaptive processing of the acoustic echo canceller is terminated.

  Next, FIG. 14 shows acoustic echo canceller adaptation processing using information similar to the sound source direction that can be calculated from the pseudo echo.

  Perform frequency decomposition S301 to convert the recorded audio into a frequency domain signal.

  An acoustic echo filter is applied to the converted frequency domain signal to calculate a pseudo echo S302.

  The similarity between the pseudo echo calculated for each frequency f and the input signal is calculated S303. In the similarity calculation process, pseudo echo E (f, τ) is used. E (f, τ) has an M-dimensional complex vector, a frame τ of each microphone element, and a pseudo echo component of frequency f as elements. The 0th element of E (f, τ) is described as E0 (f, τ). It is defined as E ′ (f, τ) = E (f, τ) / E0 (f, τ). further. Let E ″ (f, τ) = E ′ (f, τ) / | E ′ (f, τ) |. The similarity is assumed to be | E ″ (f, τ) * X (f, τ) | / | X (f, τ) |.

  This similarity measures the similarity between the acoustic echo component and the sound source direction of the input signal. When the input signal contains only the acoustic echo component, the similarity is 1. A value obtained by multiplying the similarity by a different threshold value α (f) for each frequency is defined as the final similarity. α (f) = 1 / Σ | E ″ (f, τ) * Aθ (f) |

  When the similarity exceeds a predetermined threshold th (S304), the echo canceller adaptation S305 is performed. When the similarity is lower than the predetermined threshold th, the echo canceller adaptation S305 is not performed.

  After performing S303 to S305 for every frequency, the adaptive processing of the acoustic echo canceller is terminated.

  FIG. 15 shows acoustic echo canceller adaptation processing using information similar to the sound source direction that can be calculated from the pseudo echo. In this processing, whether to apply the acoustic echo canceller is the same for all frequency components in the frame.

  Perform frequency resolution S401 to convert the recorded audio into a frequency domain signal. An acoustic echo filter is applied to the converted frequency domain signal to calculate a pseudo echo S402.

  The similarity between the pseudo echo calculated for each frequency f and the input signal is calculated S403.

  In the similarity calculation process, pseudo echo E (f, τ) is used. E (f, τ) has an M-dimensional complex vector, a frame τ of each microphone element, and a pseudo echo component of frequency f as elements. The 0th element of E (f, τ) is described as E0 (f, τ). It is defined as E ′ (f, τ) = E (f, τ) / E0 (f, τ). further. Let E ″ (f, τ) = E ′ (f, τ) / | E ′ (f, τ) |. The similarity is assumed to be | E ″ (f, τ) * X (f, τ) | / | X (f, τ) |.

  This similarity measures the similarity between the acoustic echo component and the sound source direction of the input signal. When the input signal contains only the acoustic echo component, the similarity is 1. A value obtained by multiplying the similarity by a different threshold value α (f) for each frequency is defined as the final similarity. α (f) = 1 / Σ | E ″ (f, τ) * Aθ (f) | The similarity is calculated for every frequency. Then, the similarity of all frequencies is added S404.

  When the added similarity exceeds a predetermined threshold th (S405), the echo canceller adaptation S406 is performed for all frequency components, and when it falls below, the echo canceller adaptation S406 is not performed and the acoustic echo canceller adaptation is performed. The process ends. When adaptation is performed, the acoustic echo canceller adaptation processing is terminated after the echo canceller adaptation S406.

  FIG. 16 shows a processing flow of processing that uses sound source direction information and regards only time-frequency components coming from the speaker direction as acoustic echoes and adapts them.

  In this process, whether or not the acoustic echo canceller is applied is the same for all frequency components in the frame.

  In frequency resolution S501, the recorded voice is converted into a time-frequency domain signal. In sound source localization S502, sound source direction estimation is performed based on the modified delay sum array method.

  In the modified delay sum array method, sound source localization is performed using the steering vector Aθ (f) in the sound source direction θ. When the number of microphones is M, Aθ (f) is an M-dimensional complex vector.

  Here, f represents a frequency bin number. The time-frequency representation of the input signal is described as X (f, τ). τ is the frame number of the short-time Fourier transform. X (f, τ) is an M-dimensional complex vector, which is a vector having elements of the frame τ and the frequency f of each microphone element.

  In the modified delay sum array method, the virtual sound source direction θ that maximizes | Aθ (f) * X (f, τ) | is estimated as the sound source direction of frame τ and frequency f.

  In speaker direction specification S503, a histogram is created by accumulating the number of frequencies f estimated that the sound source direction is θ or log | Aθ (f) * X (f, τ) | for each virtual sound source direction θ. Then, the peak of the histogram is calculated within a predetermined speaker direction range (for example, determined from -30 ° to 30 °), and the direction is set as the speaker direction θsp.

  In the determination in S504 whether the sound source direction is the speaker direction, if the estimated sound source direction θ ′ of the frequency f satisfies | θ′−θsp | <β, the frequency f is regarded as a frequency component arriving from the speaker direction. Then, the number of frequencies f regarded as the speaker direction or log | Aθ (f) * X (f, τ) | is accumulated to obtain a power spectrum added in the frequency direction. Whether or not the power spectrum added at all frequencies is equal to or greater than a predetermined value is determined in S506. If the power spectrum is equal to or greater than the threshold value, echo canceller adaptation S507 is performed on all frequency components, and the process is terminated.

  FIG. 17 is a diagram showing the effect of the adaptive processing of the present acoustic echo canceller. This is a result of suppressing the acoustic echo in the input signal in which only the acoustic echo exists by using an echo cancellation filter adapted for the voice (double talk voice) being spoken by the speaker at the local site.

  In this case, it is desirable that all input signals be suppressed and silenced. The result of the adaptive control according to the present invention is shown in the upper part.

  The results when adaptive control is not performed are shown in the lower part. The results are shown in such a figure that the darker the light, the smaller the power for each time-frequency. The horizontal axis is time, and the vertical axis is frequency. It can be seen that when adaptive control is performed, the signal power after acoustic echo suppression is particularly small at a high frequency and the acoustic echo suppression performance is high.

  The video conference system using the present invention may be configured such that the acoustic echo canceller is adapted by sounding a white signal or the like from the speaker at the local site before connecting to another site.

  FIG. 18 shows a processing flow when the acoustic echo canceller is adapted in advance.

  The acoustic echo canceller adaptive processing S601 is performed using the data of all frames while the speaker at the local site is on before connecting to another site.

  This corresponds to performing the acoustic echo canceller adaptive processing unconditionally without performing the double talk detection of the present invention.

  Then, another site connection standby S602 is performed, and the system waits until a connection request is received from another site or a user at the local site issues a connection request to another site.

  After the connection with the other site, the acoustic echo canceller adaptive processing with adaptive control S603 by the double talk detection processing of the present invention is repeatedly performed. Then, after the video conference system is disconnected, the process ends.

  FIG. 19 shows a flow of processing for performing nonlinear processing control by VoiceSwitch using the similarity between the input signal and the pseudo echo.

  Perform frequency resolution S701 to convert the recorded audio into a frequency domain signal.

  An acoustic echo filter is applied to the converted frequency domain signal to calculate a pseudo echo S702.

  The degree of similarity between the pseudo echo calculated for each frequency f and the input signal is calculated S703.

  In the similarity calculation process, pseudo echo E (f, τ) is used. E (f, τ) has an M-dimensional complex vector, a frame τ of each microphone element, and a pseudo echo component of frequency f as elements. The 0th element of E (f, τ) is described as E0 (f, τ). It is defined as E ′ (f, τ) = E (f, τ) / E0 (f, τ). further. Let E ″ (f, τ) = E ′ (f, τ) / | E ′ (f, τ) |. The similarity is assumed to be | E ″ (f, τ) * X (f, τ) | / | X (f, τ) |. This similarity measures the similarity between the acoustic echo component and the sound source direction of the input signal. When the input signal contains only the acoustic echo component, the similarity is 1. A value obtained by multiplying the similarity by a different threshold value α (f) for each frequency is defined as the final similarity. α (f) = 1 / Σ | E ″ (f, τ) * Aθ (f) |

  When the similarity exceeds a predetermined threshold th (S704) and the power of the input signal is equal to or higher than the threshold, the corresponding frequency component in the transmitted voice is set to zero. In other cases, the processing is terminated using the signal after echo cancellation as the corresponding frequency component in the transmitted voice.

  FIG. 20 shows a flow of processing for performing nonlinear processing control by VoiceSwitch using the similarity between the input signal and the pseudo echo. Whether to use VoiceSwitch is common to all frequencies.

  Frequency resolution S801 is performed to convert the recorded voice into a frequency domain signal. An acoustic echo filter is applied to the converted frequency domain signal to calculate a pseudo echo S802. The degree of similarity between the pseudo echo calculated for each frequency f and the input signal is calculated S803.

  In the similarity calculation process, pseudo echo E (f, τ) is used. E (f, τ) has an M-dimensional complex vector, a frame τ of each microphone element, and a pseudo echo component of frequency f as elements. The 0th element of E (f, τ) is described as E0 (f, τ). It is defined as E ′ (f, τ) = E (f, τ) / E0 (f, τ). further. Let E ″ (f, τ) = E ′ (f, τ) / | E ′ (f, τ) |. The similarity is assumed to be | E ″ (f, τ) * X (f, τ) | / | X (f, τ) |. This similarity measures the similarity between the acoustic echo component and the sound source direction of the input signal. When the input signal contains only the acoustic echo component, the similarity is 1. A value obtained by multiplying the similarity by a different threshold value α (f) for each frequency is defined as the final similarity. α (f) = 1 / Σ | E ″ (f, τ) * Aθ (f) | Add similarity at all frequencies.

  When the similarity after the addition exceeds a predetermined threshold th (S805) and the power of the input signal is equal to or higher than the threshold, the transmitted voice signal is set to 0. In other cases, the process is terminated using the signal after echo cancellation as the transmitted voice.

  FIG. 21 shows a flow of processing in which the nonlinear suppression coefficient of the residual echo after acoustic echo cancellation is controlled using the similarity between the input signal and the pseudo echo.

  Perform frequency resolution S901 to convert the recorded audio into a frequency domain signal.

  An acoustic echo filter is applied to the converted frequency domain signal to calculate a pseudo echo S902.

  The degree of similarity between the pseudo echo calculated for each frequency f and the input signal is calculated S903.

  In the similarity calculation process, pseudo echo E (f, τ) is used. E (f, τ) has an M-dimensional complex vector, a frame τ of each microphone element, and a pseudo echo component of frequency f as elements. The 0th element of E (f, τ) is described as E0 (f, τ). It is defined as E ′ (f, τ) = E (f, τ) / E0 (f, τ). further. Let E ″ (f, τ) = E ′ (f, τ) / | E ′ (f, τ) |. The similarity is assumed to be | E ″ (f, τ) * X (f, τ) | / | X (f, τ) |. This similarity measures the similarity between the acoustic echo component and the sound source direction of the input signal. When the input signal contains only the acoustic echo component, the similarity is 1. A value obtained by multiplying the similarity by a different threshold value α (f) for each frequency is defined as the final similarity. α (f) = 1 / Σ | E ″ (f, τ) * Aθ (f) |

  When the degree of similarity exceeds a predetermined threshold th (S904) and the power of the input signal is equal to or higher than the threshold, the nonlinear suppression coefficient α is set to a predetermined value α0. In other cases, the nonlinear suppression coefficient α is α1. α0> α1 is set in advance.

The signal after acoustic echo cancellation S907 is n ′ (f, τ), and the pseudo echo component is e (f, τ).
In the nonlinear suppression processing S908, n ′ (f, τ) = Floor (| n ′ (f, τ) |-| e (f, τ) |) arg (n ′ (f, τ)) '(f, τ) is output. Here, Floor (x) is a function that returns x when x is 0 or more, and returns 0 when x is 0 or less. arg (x) is a function that returns the phase component of x. After performing nonlinear suppression processing at all frequencies, the processing ends.

  As described above, according to the present invention, acoustic echo control can be realized by a telephone conference system or a video conference system, and can be applied to an acoustic echo cancellation technique in a double talk state.

The figure which showed the hardware constitutions of this invention. The software block diagram of this invention. The block diagram of speaker number and position specification by GUI of this invention. The block diagram of VoiceSwitch switching processing by sound source localization of the present invention. The whole apparatus which applied this invention to the video conference system. The figure explaining the acoustic echo canceller of a prior art. The figure explaining the acoustic echo canceller using a double talk detection process. The figure which showed the structure of the two-site video conference. 1 is a block diagram of a video conference system to which the present invention is applied. The processing flow figure of the video conference system to which this invention is applied. The figure explaining the acoustic echo component identification method in this invention. The processing flow figure of the echo canceller of this invention, a double talk determination, and an echo canceller adaptation process. The echo canceller adaptive control flowchart for every frequency using a sound source direction. The echo canceller adaptive control flowchart for every frequency using the information similar to a sound source direction. The echo canceller adaptive control flow figure of all the frequency components using the information similar to a sound source direction. The echo canceller adaptive control flowchart for every frequency using a sound source direction. The figure which showed the effect of this invention. The processing flow figure in the case of performing an adaptive process before a video conference start. The VoiceSwitch control flowchart for each frequency using information similar to the sound source direction. The VoiceSwitch control flow diagram of all frequencies using information similar to the sound source direction. The suppression coefficient control flowchart of the nonlinear acoustic echo suppression process for every frequency using the information similar to a sound source direction.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Microphone array, 2 ... A / D converter and D / A converter, 3 ... Central processing unit, 4 ... Memory, 5 ... Hub, 6 ... Speaker, 7 ... External storage medium, 8 ... A / D converter , 9 ... Band division unit, 10 ... Phase difference calculation unit, 11 ... Frequency distribution unit, 12 ... Acoustic echo canceller adaptation unit, 13 ... Pseudo echo generation unit, 14 ... Acoustic echo cancellation unit, 15 ... Speaker number / position setting GUI , 16 ... Echo phase difference calculation processing unit, 17 ... Sound source localization unit, 18 ... VoiceSwitch determination unit, 19 ... Output signal generation unit,
DESCRIPTION OF SYMBOLS 100 ... Video conference system, 101 ... Computer, 102 ... A / DD / A apparatus, 103 ... Hub, 104 ... Image display apparatus, 105 ... Microphone array, 106 ... Speaker, 107 ... Camera, 108 ... Audio cable, 109 ... Audio cable, 110 ... Digital cable, 111 ... Monitor cable, 112 ... LAN cable, 113 ... Digital cable, 201 audio transmission unit 202 202 image transmission unit 203 audio recording unit 204 acoustic echo canceller adaptation unit 205 acoustic echo canceller unit 206 image display unit 207 ..Audio receiving unit, 208... Audio reproducing unit, 209... Image receiving unit, 210. Filter DB, 701 ... Phase difference calculation unit, 702 ... Double talk detection unit, S1 ... Acoustic echo canceller adaptation processing, S2 ... Judgment of connection request from other bases, S3 ... Auto Determining whether a connection request has been issued from the site, S4: Connection to another site, S6: Reproduction processing from speaker, S7: Image display, S8: Audio recording, S9: Echo cancellation, S10 ..Voice transmission, S11... Judgment of disconnection, S12... Judgment of disconnection request from own site, S13... Disconnection from other sites, S101. Echo canceller execution, S103: Judgment whether double talk is in effect, S104: Echo canceller adaptation, S201: Frequency decomposition, S202: Sound source localization, S203: S -Car direction identification, S204 ... Determination of whether the sound source direction is the speaker direction, S205 ... Echo canceller adaptation, S301 ... Frequency resolution, S302 ... Pseudo echo calculation, S303 ... Similarity with input signal Degree calculation, S304... Judgment whether similarity is greater than threshold, S305 ... Echo canceller adaptation, S401 ... Frequency decomposition, S402 ... Pseudo echo calculation, S403 ... Similarity calculation with input signal , S404 ... Similarity addition processing, S405 ... Determination of whether the added similarity is equal to or greater than a threshold, S406 ... All frequency echo canceller adaptation, S501 ... Frequency decomposition, S502 ... Sound source localization, S503: Speaker direction identification, S504: Determination of whether the sound source direction is the speaker direction, S505: Power spectrum addition processing S506: Determination of whether the speaker direction spectrum is equal to or greater than the threshold value, S507: Echo canceller adaptation, S601: All frame adaptation processing, S602: Other site connection standby, S603: Acoustic with adaptive control Echo canceller adaptive processing, S701 ... frequency decomposition, S702 ... pseudo echo calculation, S703 ... similarity calculation with input signal, S704 ... determination of whether similarity is greater than or equal to threshold, S705 ... power S706 ... VoiceSwitch determination, S801 ... frequency decomposition, S802 ... pseudo echo calculation, S803 ... similarity calculation with input signal, S804 ... similarity addition processing, S805: Determination of whether the similarity after addition is equal to or greater than a threshold value, S806: Determination of whether input power is equal to or greater than the threshold value, S807: All bands oiceSwitch determination, S901: frequency decomposition, S902: pseudo echo calculation, S903: similarity calculation with input signal, S904: determination whether similarity is equal to or greater than threshold, S905: power is threshold Determination of whether or not, S906 ... nonlinear suppression coefficient adjustment, S907 ... acoustic echo canceller, S908 ... nonlinear suppression.

Claims (8)

A microphone for voice input,
An AD converter for digitally converting the signal from the microphone;
An information processing apparatus that processes a digital signal from the AD converter and suppresses an acoustic echo component;
An output interface for sending a signal from the information processing apparatus to a network;
An input interface for receiving signals from the network;
A DA converter for analog conversion of a signal from the input interface;
A speaker for outputting a signal from the DA converter as sound;
The microphone is a microphone array having a plurality of microphone elements,
The AD converter is a plurality of AD converters for digitally converting a signal for each microphone element,
The information processing apparatus includes: a band division unit that divides the digital signal into a band; and a phase difference calculation that calculates a phase difference between sounds input to the plurality of microphone elements based on signals from the plurality of AD converters. And a frequency distribution unit that determines whether the sound input to the microphone array from the phase difference output from the phase difference calculation unit is sound from a speaker,
The band dividing unit divides each digital signal digitally converted for each microphone element,
The phase difference calculation unit calculates a phase difference between sounds input to the plurality of microphone elements for each of the divided bands ,
The frequency distribution unit determines whether the band division signal is a speaker output signal or a speaker signal from the phase difference for each band output by the phase difference calculation unit,
It said acoustic echo canceller adaptation unit, for band speaker output signal is determined to be dominant only have an adaptation of the adaptive filter used to suppress contamination content of the voice of the speaker from the signal of the microphone element ,
The acoustic echo canceler unit removes an acoustic echo component from a signal of each microphone element using the adaptive filter.   In order to determine whether or not the output signal is a speaker output signal, the frequency distribution unit measures in advance a transfer function of sound transmitted from the speaker to the microphone array, and the sound is output from the speaker from the measured transfer function. And calculating the phase difference for each band of the microphone array, storing the phase difference for each band in the external storage device, and storing the phase difference between the microphone elements for each band of the band difference signal and the band division signal. The acoustic echo canceller system according to claim 1, further comprising: a frequency distribution unit that determines that the band division signal is a speaker output signal when the phase difference is equal to or less than a predetermined threshold value.   The user interface is characterized in that the user specifies the number of speakers and the physical position relative to the microphone array in advance. Sound is output from the speakers from the number and physical positions of the speakers specified by the user interface. An echo phase difference calculation processing unit that calculates a phase difference for each band of the microphone array, and stores the phase difference for each band in an external storage device, and stores the stored phase difference and the band division signal The acoustic echo canceller system according to claim 1, further comprising: a frequency distribution unit that determines that the band division signal is a speaker output signal when a phase difference between microphone elements for each band is equal to or less than a predetermined threshold value. .   Using the phase difference of each microphone array, calculate a histogram of the sound source direction over the band, and have a sound source localization unit that estimates the sound source direction from the histogram, and estimates that the sound source localization unit has arrived from the sound source direction calculated And the magnitude of the calculated signal is the magnitude of the band division signal determined to be the speaker output signal or the speaker output signal in the band division signal after the acoustic echo canceller. 2. The acoustic echo canceller system according to claim 1, wherein when the magnitude of the band division signal is equal to or smaller than a predetermined magnitude compared to the determined magnitude of the band division signal, all the magnitudes of the band division signal are set to zero. . Furthermore,
Means for estimating the sound source direction;
Means for identifying the speaker direction;
Means for comparing the sound source direction and the speaker direction;
2. The acoustic echo canceller system according to claim 1, wherein when the sound source direction is considered to correspond to the speaker direction, the sound source is regarded as an acoustic echo component and the echo canceller is applied. Furthermore,
Means for applying an acoustic echo filter to the converted frequency domain signal to calculate a pseudo echo;
Means for calculating the similarity between the pseudo echo calculated for each frequency and the input signal,
The acoustic echo canceller system according to claim 1, wherein an echo canceller is applied when the similarity is larger than a predetermined threshold. Furthermore,
Means for applying an acoustic echo filter to the converted frequency domain signal to calculate a pseudo echo;
Means for calculating the similarity between the pseudo echo calculated for each frequency and the input signal;
Means for adding the similarity of all frequencies,
2. The acoustic echo canceller system according to claim 1, wherein the echo canceller is adapted when the added similarity is larger than a predetermined threshold value. Furthermore,
Means for estimating the sound source direction;
A means of identifying the speaker direction;
Means for comparing the sound source direction and the speaker direction;
If the sound source direction is considered to correspond to the speaker direction, means for determining that the sound source is an acoustic echo component;
Means for adding a sound source regarded as a speaker direction in the frequency direction to obtain a power spectrum;
The acoustic echo canceller system according to claim 1, wherein an echo canceller is applied when a power spectrum added at all frequencies is larger than a predetermined threshold.