JP7349410B2 - System and method for reproducing acoustic signals from a predetermined direction of arrival - Google Patents

Description

The present invention relates to technology for detecting and reproducing acoustic signals inherent in environmental sounds. It is particularly suitable for telepresence (video conference) systems via networks.

In many cases, members engage in activities and lives apart from each other in different locations, not just for work or business, but also for parents and children who live far apart. In this case, each party tries to recognize the other party's situation using the telephone and communication functions of smartphones, computers, etc. At this time, it is necessary to communicate spontaneously, and it has been pointed out that members working remotely may feel left out (for example, see Non-Patent Document 1).

In contrast, telepresence systems that constantly stream video and audio between multiple locations have come into use (for example, see Non-Patent Documents 2 and 3). This is the same as a video conferencing system, but instead of being connected only during calls or meetings, it is always connected. According to this system, even if members (parents and children, families, employees) staying in different remote locations can share each other's situation by constantly transmitting environmental sounds and images of the other party's surroundings, they can We can provide an environment where you feel like you are living in the same place. A telepresence system can easily connect not only between domestic and overseas bases, but also between a company and a home or shared office via a network.
For example, according to Skype (registered trademark), a typical videophone service, in order to share the movements of members at remote locations, it is possible to display the status of "attended/away" on the display of the other party's terminal. .

Another conventional technique is to adjust the direction and width of the sound field of acoustic signals picked up by multiple microphones to create a sound field that matches the range of the video specified by the user when displaying an enlarged video. There is also a technique for reproducing this (for example, see Patent Document 1). According to this technology, the direction and width of the sound field of acoustic signals collected by multiple microphones can be adjusted and reproduced.
In addition, there is also a technology in which the situation of a member at a remote site is communicated using a synthesized voice in order to know the situation of a member at a remote site even if the user is not looking at the display while working (for example, see Non-Patent Document 4).
Furthermore, there is a technology for an environmental sound recognition device that mutually recognizes various environmental sounds between remote locations (for example, see Patent Document 2).

JP2019-068210A Patent No. 6085538

Edited by the Ministry of Internal Affairs and Communications, “Research Report on Telework Trends and Productivity, International Strategy Bureau of Information and Communications, Ministry of Internal Affairs and Communications (2010)”, [online], [Retrieved March 10, 2020], Internet <URL: https //www.soumu.go.jp/johotsusintokei/linkdata/h22_06_houkoku.pdf＞ Telepresence: Integrating shared task and person spaces, W Buxton - Proceedings of graphics interface, 1992, [online], [Retrieved March 10, 2020], Internet <URL: https://www.billbuxton.com/TelepShrdSpce .pdf＞ Japan Ergonomics Society Conference Proceedings 406-407, 2009: Always-on audio conferencing system for telework HRI 2018: Fribo: A Social Networking Robot for Increasing Social Connectedness through Sharing Daily Home Activities from Living Noise Data., [online], [Retrieved March 10, 2020], Internet <URL: https://yonsei. pure.elsevier.com/en/publications/fribo-a-social-networking-robot-for-increasing-social-connectedne＞ Institute of Electronics, Information and Communication Engineers "Forest of Knowledge", Group 2 (Image/Sound/Language) - Volume 6 (Acoustic Signal Processing) - Chapter 2 (Sound Source Separation), [online], [Retrieved April 20, 2020] , Internet <URL:http://www.ieice-hbkb.org/files/02/02gun_06hen_02.pdf>

According to the existing telepresence system, a sound signal picked up by a sound pickup device at a first base can be reproduced by a playback device at a second base. At this time, the environmental sounds at the first base are simply recorded, and the second base simply plays back the environmental sounds as they are.

In contrast, the inventors of the present application have found that if the environmental sound of the first base is played back at the second base according to the sound source position of the second base, the user of the second base will We wondered if it would be possible to feel the presence of the users at base 1 in the atmosphere. For example, if the sound of a water faucet is collected at a first base, is it possible to reproduce it so that the sound of the water faucet at the first base arrives from the position of the water supply at the second base? Thought.
Also, considering this, in the case of a playback device equipped with a sound collection function, when a predetermined sound signal is collected, it is possible to play back another sound signal to be played from the direction of arrival of the sound signal. I thought I might be able to do something.

SUMMARY OF THE INVENTION Therefore, an object of the present invention is to provide a system and method for reproducing an acoustic signal to be reproduced or an acoustic signal received from a remote sound collection device from a predetermined arrival direction.

According to the present invention, a sound collection device having a plurality of microphones for picking up environmental sounds, and a playback device having a plurality of microphones and a plurality of speakers for picking up environmental sounds are connected via a network. A system that
The sound collection device is
a first acoustic database that associates and accumulates acoustic signals for each acoustic tag;
a first sound source separation means for detecting one or more acoustic signals inherent in the environmental sound;
a first acoustic tag estimating means for detecting one or more acoustic signals inherent in environmental sounds using a first acoustic database and estimating an acoustic tag of the acoustic signals;
acoustic tag transmitting means for transmitting the acoustic tag to the playback device;
has
The playback device is
a second acoustic database that associates and accumulates acoustic signals for each acoustic tag;
a second sound source separation means that detects one or more acoustic signals inherent in the environmental sound and estimates the direction of arrival of each acoustic signal;
a second acoustic tag estimation means for estimating an acoustic tag of an acoustic signal using a second acoustic database;
and an environmental sound reproduction means for outputting from a plurality of speakers an environmental sound synthesized so that the acoustic signal associated with the acoustic tag received from the sound collection device can be heard from the direction of arrival of the same detected acoustic tag. Featured system.

According to other embodiments of the system of the invention:
It is also preferable that the first sound source separation means of the sound collection device and the second sound source separation means of the reproduction device estimate the arrival direction of the acoustic signal by a blind sound source separation method or a beamforming method using a plurality of microphones.

According to other embodiments of the system of the invention:
It is also preferable that the playback device further includes a second direction of arrival determining means for determining the direction of arrival for each acoustic tag based on the distribution of directions of arrival for a predetermined period in the acoustic signals associated with the acoustic tag.

According to other embodiments of the system of the invention:
Regarding the sound collection device,
The first sound source separation means further estimates the arrival direction of each acoustic signal,
The acoustic tag transmitting means further transmits the direction of arrival to the playback device,
Regarding the playback device,
It is also preferable that the environmental sound reproducing means reproduces an environmental sound that is synthesized from the acoustic signal associated with the acoustic tag to be reproduced so as to be audible from the direction of arrival received from the sound collection device.

According to other embodiments of the system of the invention:
Regarding the sound collection device,
The acoustic tag transmitting means further transmits the acoustic signal output from the first sound source separating means,
Regarding the playback device,
It is also preferable that the environmental sound reproduction means reproduces the acoustic signal received from the sound collection device.

According to other embodiments of the system of the invention:
It is also preferable that the environmental sound reproduction means in the reproduction device selects only the acoustic tags other than the acoustic tag detected by the second acoustic tag estimation means, among the acoustic tags received from the sound collection device, as acoustic tags to be reproduced. .

According to other embodiments of the system of the invention:
Acoustic tags are expressed in such a way that it is possible to calculate the similarity between acoustic signals.
The environmental sound reproduction means in the reproduction device calculates the degree of similarity between the acoustic tag received from the sound collection device and the acoustic tag detected by the second acoustic tag estimation means, and calculates the degree of similarity between the acoustic tag received from the sound collection device and the acoustic tag detected by the second acoustic tag estimating means, and if the degree of similarity is equal to or less than a predetermined threshold ( It is also preferable to select only the acoustic tags that are not similar as the acoustic tags to be reproduced.

According to other embodiments of the system of the invention:
The acoustic tags and acoustic signals accumulated in the second acoustic database of the playback device are part or all of the acoustic tags and acoustic signals accumulated in the first acoustic database of the sound collection device,
Even if the acoustic tag stored in the second acoustic database of the playback device and the acoustic tag accumulated in the first acoustic database of the sound collection device are the same, the acoustic signals may be based on different acoustic signals. preferable.

According to the present invention, a sound collection device having a plurality of microphones for picking up environmental sounds, and a playback device having a plurality of microphones and a plurality of speakers for picking up environmental sounds are connected via a network. A method for reproducing an acoustic signal in a system comprising:
The sound collection device is
Each acoustic tag has a first acoustic database that associates and accumulates acoustic signals.
a first step of detecting one or more acoustic signals inherent in the environmental sound;
a second step of detecting one or more acoustic signals inherent in the environmental sounds using the first acoustic database and estimating an acoustic tag of the acoustic signals;
a third step of transmitting the acoustic tag to the playback device;
Run
The playback device is
Each acoustic tag has a second acoustic database that stores acoustic signals in association with each other;
a fourth step of detecting one or more acoustic signals inherent in the environmental sound and estimating the arrival direction of each acoustic signal;
a fifth step of estimating an acoustic tag of the acoustic signal using the second acoustic database;
and a sixth step of outputting, from a plurality of speakers, an environmental sound synthesized so that the acoustic signal associated with the acoustic tag received from the sound collection device can be heard from the direction of arrival of the same detected acoustic tag. It is characterized by

According to the present invention, there is provided a sound collection method using a sound collection device that collects environmental sounds using a plurality of microphones and is connected to the above-mentioned playback device via a network.
The sound collection device is
a first acoustic database for accumulating acoustic signals for each acoustic tag;
a first step of detecting one or more acoustic signals inherent in the environmental sound;
a second step of detecting one or more acoustic signals inherent in the environmental sounds using the first acoustic database and estimating an acoustic tag of the acoustic signals;
a third step of transmitting the acoustic tag to the playback device;
The playback device is characterized in that it executes the acoustic tag received from the sound collection device so as to set it as a playback target acoustic tag.

According to the system and method of the present invention, an acoustic signal to be reproduced or an acoustic signal received from a remote sound collection device can be reproduced so that it can be heard from a predetermined direction of arrival.
According to the present invention, specifically, the acoustic signal of each sound source on the sound collection side can be reproduced according to the position of each sound source on the reproduction side. Even if the sound collection device and the playback device are located at different bases, members staying remotely can share each other's environmental sounds.

FIG. 2 is a first external view showing the environmental sounds of base B where the playback device is placed. FIG. 2 is a functional configuration diagram of a playback device according to the present invention. FIG. 3 is an explanatory diagram of a second sound source separation unit and a second acoustic tag estimation unit in the playback device. FIG. 2 is an explanatory diagram showing arrival direction detection using a blind sound source separation method. FIG. 3 is an explanatory diagram showing direction of arrival detection using a beamforming method. FIG. 3 is an explanatory diagram of an environmental sound reproduction section in the reproduction device. FIG. 7 is a second external view showing the environmental sounds of base B reproduced by the reproduction device. It is an external view showing the environmental sound of base A where the sound collection device is arranged. 2 is an external view of playback devices arranged in the surrounding environment of base B. FIG. It is a functional block diagram of the sound collection device in this invention. 11 is a functional configuration diagram of a playback device that receives an acoustic tag and a direction of arrival from the sound collection device of FIG. 10. FIG. FIG. 2 is an explanatory diagram showing data transmitted from a sound collection device to a playback device. FIG. 3 is an external view showing an acoustic signal reproduced according to a first pattern. FIG. 7 is an external view showing an acoustic signal reproduced according to a second pattern. FIG. 7 is an external view showing an acoustic signal reproduced according to a third pattern. FIG. 7 is an external view showing an acoustic signal reproduced according to a fourth pattern.

Hereinafter, embodiments of the present invention will be described in detail using the drawings.

According to the present invention, the following two embodiments will be described.
The first embodiment is comprised only of playback devices placed within the base, and will be described later with reference to FIGS. 1 to 7.
The second embodiment includes a sound collection device placed in a first base and a playback device placed in a second base, and will be described later with reference to FIGS. 8 to 13.

<First embodiment>
The playback device detects the direction of arrival of a plurality of acoustic signals from the environmental sounds heard by users within the base, and plays back the sound signal to be played back so that the sound signal can be heard from the direction from which the sound signal arrives.

FIG. 1 is a first external view showing the environmental sounds of base B where the playback device is placed.
According to FIG. 1, the external appearance of user b's surrounding environment is shown, and a water supply, windows, and a washing machine are arranged therein. At this time, user b hears an environmental sound that includes a mixture of the water faucet sound, "squeak", the sound of windows opening and closing, and the "crunching" sound of the washing machine.
Further, according to FIG. 1, a playback device 2 of the present invention is placed within a base B. The playback device 2 is equipped with a plurality of microphones for picking up environmental sounds and a plurality of speakers for playing back an acoustic signal to be played so that it can be heard from a predetermined direction of arrival.

<Playback device 2>
FIG. 2 is a functional configuration diagram of the playback device according to the present invention.

According to FIG. 2, the playback device 2 includes a microphone 201, a speaker 202, a second acoustic database 21, a second sound source separation unit 22, a second acoustic tag estimation unit 23, and a second incoming It has a direction determining section 24 and an environmental sound reproducing section 25. These functional components can be realized by executing a program that causes a computer installed in the device to function. Further, the processing flow of these functional components can also be understood as an environmental sound reproduction method.

[Mike 201]
The microphone 201 includes a plurality of microphones that collect environmental sounds. For example, it may be something like a microphone array. A microphone array can acquire spatial information of sound by signal processing environmental sounds picked up by a plurality of microphones.

[Speaker 202]
The speaker 202 is composed of a plurality of speakers, and reproduces an environmental sound that sounds as if the acoustic signal to be reproduced is coming from a predetermined direction. It is possible to reproduce environmental sounds whose direction of arrival is controlled for each acoustic signal.

[Second acoustic database 21]
The second acoustic database 21 accumulates acoustic signals for each acoustic tag.
Acoustic tag<->acoustic signal An “acoustic tag” is an identifier for specifying an acoustic signal.
The "acoustic signal" is not limited to the acoustic signal itself, but may be a standard pattern of acoustic features such as a time-series frequency spectrum.

[Second sound source separation unit 22]
The second sound source separation unit 22 detects one or more acoustic signals inherent in the environmental sound and estimates the direction of arrival of each acoustic signal.

FIG. 3 is an explanatory diagram of the second sound source separation unit and the second acoustic tag estimation unit in the playback device.
According to FIG. 3, the environmental sound picked up by the microphone 201 is input to the second sound source separation unit 22. This environmental sound includes various acoustic signals such as the following.
"Squeak"
"Krunkrun"
"Shah"
The second sound source separation unit 22 outputs acoustic signals separated and detected for each sound source and their direction of arrival.

The second sound source separation unit 22 may employ a blind sound source separation method or a beamforming method in order to estimate the arrival direction of the acoustic signal. According to these methods, it is possible to detect acoustic signals mixed in environmental sounds and also detect the direction of arrival of each acoustic signal.

(Blind sound source separation method)
FIG. 4 is an explanatory diagram showing arrival direction detection using the blind sound source separation method.
A blind sound source separation method is, for example, when based on independent component analysis (see Non-Patent Document 5), separated signals are made independent of each other under the assumption that multiple sound sources are statistically independent from each other even if they are unknown. Configure a filter like this. Up to (number of microphones - 1) acoustic signals can be detected. Note that the blind sound source separation method does not in principle require information such as knowledge of the type and spatial position of the sound source, extraction of the target sound section, and synthesis conditions, and does not use assumptions about the harmonic structure of the sound source signal.

(Beamforming method)
FIG. 5 is an explanatory diagram showing detection of the direction of arrival using the beamforming method.
The beamforming method is a method in which each microphone detects an acoustic signal of a sound source in a target direction (see, for example, Non-Patent Document 5). Based on the fact that the sound waves propagate differently from the sound source to each microphone, the phase and amplitude are controlled by delays and filters. This reduces the sensitivity of acoustic signals in directions other than the target direction, and ensures the sensitivity (S/N ratio) of the acoustic signals in the target direction.
Specifically, the angle from the microphone array is divided into a plurality of parts (eight divisions according to FIG. 5), and an acoustic signal is collected in each angular range as a target direction.

[Second acoustic tag estimation unit 23]
The second acoustic tag estimation unit 23 uses the second acoustic database 21 to estimate the acoustic tag of the acoustic signal.

The second acoustic tag estimation unit 23 extracts Mel frequency cepstral coefficients (MFCC) as features and identifies acoustic signals using a neural network based on deep learning (for example, see Non-Patent Documents 3 and 4). This is done by stacking hidden layers pre-trained by an autoencoder based on a constrained Boltzmann machine (RBM) to build a multilayer hierarchical network, and adding a discriminative network using the output of the final layer to complete the The acoustic tags are detected using supervised learning.
As a learning stage, the second acoustic tag estimation unit 23 performs learning using teacher data in which acoustic tags and acoustic signals stored in the second acoustic database 21 are associated with each other. In the estimation stage, an acoustic signal from the second sound source separation section 22 is input, and an acoustic tag corresponding to the acoustic signal is output.

According to FIG. 3, the second acoustic tag estimating unit 23 estimates an acoustic tag as follows, for example.
Acoustic tag 101 (water faucet sound)
Acoustic tag 167 (washing machine driving sound)
Acoustic tag 143 (window opening/closing sound)

[Second arrival direction determining unit 24]
The second direction of arrival determining section 24 receives the acoustic tag from the second acoustic tag estimation section 23 and the direction of arrival from the second sound source separation section 22 . Then, the second direction of arrival determining unit 24 determines the direction of arrival for each acoustic signal associated with each acoustic tag from the distribution of directions of arrival over a predetermined period. By analyzing for a predetermined period of time, the direction in which the sound source emitting the acoustic signal associated with each acoustic tag is located is determined.
Then, the second arrival direction determination unit 24 stores the acoustic tag based on the environmental sound around the playback device 2 and the arrival direction in association with each other. For example, let's assume that you store the following links:
Acoustic tag 101 (water faucet sound) <->Direction of arrival 2
Acoustic tag 167 (washing machine driving sound) <-> Direction of arrival 7
Acoustic tag 143 (window opening/closing sound) <->Direction of arrival 1

[Environmental sound reproduction section 25]
The environmental sound reproducing unit 25 refers to the acoustic tag with respect to the second arrival direction determining unit 24 when a trigger for reproducing the acoustic signal to be reproduced occurs. Then, the direction of arrival associated with that acoustic tag is acquired.
Next, the environmental sound reproduction section 25 synthesizes the acoustic signals linked to the acoustic tag to be reproduced so that they can be heard from the direction of arrival of the same acoustic tag determined by the second direction of arrival determination section 24, and Play. The environmental sound is output to multiple speakers 202.
That is, only when the acoustic tag is stored in the second direction of arrival determination section 24, the environmental sound reproduction section 25 refers to the direction of arrival and reproduces the acoustic signal to be reproduced from the direction of arrival.

FIG. 6 is an explanatory diagram of the environmental sound reproduction section in the reproduction device.
According to FIG. 6, the acoustic signal to be reproduced and the acoustic tag 143 are input to the environmental sound reproduction unit 25 at the time of reproduction, which is a trigger.
Acoustic tag 143 (window opening/closing sound) <->Acoustic signal "beep"
At this time, the environmental sound reproducing unit 25 refers to the second direction of arrival determining unit 24 using the acoustic tag 143 and obtains the direction of arrival 1. Then, the environmental sound reproduction unit 25 synthesizes the environmental sounds so as to reproduce the acoustic signal "beep" to be reproduced from the arrival direction 1.

FIG. 7 is a second external view showing the environmental sounds of base B reproduced by the reproduction device.
According to FIG. 7, as the environmental sound of base B, an acoustic signal "beep" is reproduced from the direction of the window. This is so that the reproduction device 2 reproduces the sound signal "beep" to be reproduced so that it arrives from the direction in which the window opening/closing sound based on the collected environmental sound comes.
As described above, according to the first embodiment, it is possible to reproduce the acoustic signal to be reproduced as an environmental sound that can be heard from a sound source (for example, a window) at a position corresponding to the arrival direction of the collected acoustic signal. can.

<Second embodiment>
The sound collection device detects the acoustic tag of each acoustic signal from the environmental sounds heard by users in the first base, and transmits the acoustic tag to the playback device. Further, the sound collection device may further detect the direction of arrival of each acoustic signal and/or the acoustic signal, and may also transmit them to the playback device.
The reproduction device synthesizes and reproduces the acoustic signals associated with one or more acoustic tags received from the sound collection device at the first base as an environmental sound within the second base. At this time, each acoustic signal can be reproduced from a predetermined arrival direction (the arrival direction of the acoustic signal collected by the reproduction device itself, or the arrival direction of the acoustic signal associated with the acoustic tag received from the sound collection device). .

FIG. 8 is an external view showing the environmental sounds of base A where the sound collection device is placed.
According to FIG. 8, the external appearance of the surrounding environment of users a1 and a2 is shown, and a water supply, windows, and a washing machine are arranged. At this time, users a1 and a2 hear environmental sounds that include a mixture of the following acoustic signals.
Water faucet sound "jar"
Window opening/closing sound “bap”
Washing machine driving sound "Grungurun"
Moreover, according to FIG. 8, the sound collection device 1 of the present invention is arranged within the base A. The sound collection device 1 is equipped with a plurality of microphones for collecting environmental sounds, and also communicates with a remote playback device 2 via a network.

FIG. 9 is an external view of playback devices arranged in the surrounding environment of base B.
According to FIG. 9, the external appearance of the surrounding environment of user b is shown, and a water supply, windows, and a washing machine are arranged. Here, the locations of the water supply, windows, and washing machines at base A are naturally different from the locations of the water supply, windows, and washing machines at base B. Therefore, the position of the sound source in the environmental sound of base A is naturally different from the position of the sound source in the environmental sound of base B.

<Sound collection device 1>
FIG. 10 is a functional configuration diagram of the sound collection device according to the present invention.

According to FIG. 10, the sound collection device 1 includes a plurality of microphones 101, a first acoustic database 11, a first sound source separation section 12, a first acoustic tag estimation section 13, and an acoustic tag transmission section 14. and has. These functional components can be realized by executing a program that causes a computer installed in the device to function. Further, the processing flow of these functional components can also be understood as a sound collection and transmission method.

[First acoustic database 11]
The first acoustic database 11 accumulates acoustic signals for each acoustic tag. Basically, the first acoustic database 11 of the sound collection device 1 has the same function as the second acoustic database of the reproduction device 2 described above.

Here, the acoustic tags and acoustic signals accumulated in the second acoustic database 21 of the playback device 2 are part or all of the acoustic tags and acoustic signals accumulated in the first acoustic database 11 of the sound collection device 1. There may be.
For example, if the acoustic tag and acoustic signal accumulated in the first acoustic database 11 of the sound collection device 1 are not accumulated in the second acoustic database 21 of the reproduction device 2, the acoustic signal is simply not reproduced. .
On the other hand, if the acoustic tags and acoustic signals accumulated in the first acoustic database 11 of the sound collection device 1 are also accumulated in the second acoustic database 21 of the reproduction device 2, Environmental sounds synthesized using acoustic signals from the acoustic database 21 are reproduced. That is, if the acoustic tag stored in the second acoustic database 21 of the playback device 2 and the acoustic tag accumulated in the first acoustic database 11 of the sound collection device 1 are different acoustic signals, the second The environmental sound converted by the acoustic signal of the acoustic database 21 will be reproduced.

[First sound source separation unit 12]
The first sound source separation unit 12 detects one or more acoustic signals inherent in the environmental sounds of the base A, and estimates the direction of arrival of each acoustic signal. It has the same function as the second sound source separation section 22 of the reproduction device 2 described above.

[First acoustic tag estimation unit 13]
The first acoustic tag estimation unit 13 uses the first acoustic database 11 to detect one or more acoustic signals inherent in the environmental sound, and estimates the acoustic tag of the acoustic signal. It has the same function as the second acoustic tag estimation section 23 of the reproduction device 2 described above.

[Acoustic tag transmitter 14]
The acoustic tag transmitter 14 transmits the acoustic tag (only the acoustic tag or both the acoustic tag and the direction of arrival) to the playback device 2.
Further, as another embodiment, when the acoustic signal collected by the sound collecting device 1 is played back by the playback device 2 as it is, the acoustic tag transmitting unit 14 also sends the “acoustic signal” itself to the playback device 2.

FIG. 11 is a functional configuration diagram of a playback device that receives an acoustic tag and direction of arrival from the sound collection device of FIG. 10.
According to FIG. 11, the playback device 2 further includes an acoustic tag receiving section 26, and receives the acoustic tag (and direction of arrival) from the sound collection device 1. The received acoustic tag (and direction of arrival) is to be played back by the playback device 2 of FIG. 2 described above. The other functional components are the same as those in the functional configuration diagram of the playback device 2 shown in FIG. 2 described above.

FIG. 12 is an explanatory diagram showing data transmitted from the sound collection device to the playback device.

According to FIG. 12, there are examples of the following four patterns, for example.
(First pattern)
The acoustic tag transmitter 14 of the sound collection device 1 transmits only the “acoustic tag” to the acoustic tag receiver 26 of the playback device 2.
(sound collection device)
Acoustic tag 101 (water faucet sound) ->
Acoustic tag 167 (washing machine driving sound) ->
Acoustic tag 143 (window opening/closing sound) ->
FIG. 13 is an external view showing the acoustic signal reproduced according to the first pattern.
(Playback device)
-> Acoustic tag 101 (water faucet sound): Direction of arrival 3: Acoustic signal "Shar"
-> Acoustic tag 167 (washing machine driving sound): Arrival direction 7: Acoustic signal "Krunkrun"
-> Acoustic tag 143 (window opening/closing sound): Arrival direction 1: Acoustic signal "squeak"
If only acoustic tags are received, the reproduction device 2 reproduces the signal using the direction of arrival detected by itself for each acoustic tag and the acoustic signal registered in the second acoustic database 21 . In the first pattern, another acoustic signal corresponding to each acoustic signal inherent in the environmental sound of base A is reproduced from a direction of arrival corresponding to the surrounding environment of base B.

(Second pattern)
The acoustic tag transmitter 14 of the sound collection device 1 transmits the “acoustic tag” and “direction of arrival” to the acoustic tag receiver 26 of the playback device 2.
(sound collection device)
Acoustic tag 101 (water faucet sound): Direction of arrival 1 ->
Acoustic tag 167 (washing machine driving sound): Direction of arrival 3 ->
Acoustic tag 143 (window opening/closing sound): Direction of arrival 8 ->
FIG. 14 is an external view showing the acoustic signal reproduced by the second pattern.
(Playback device)
-> Acoustic tag 101 (water faucet sound): Direction of arrival 1: Acoustic signal "Shar"
-> Acoustic tag 167 (washing machine driving sound): Arrival direction 3: Acoustic signal "Kurunkurun"
-> Acoustic tag 143 (window opening/closing sound): Arrival direction 8: Acoustic signal "squeak"
Since the reproducing device 2 has received the acoustic tag and the direction of arrival, it reproduces using the acoustic signal registered in the second acoustic database 21. In the second pattern, another acoustic signal corresponding to each acoustic signal inherent in the environmental sound at site A is also reproduced at site B from the same direction of arrival.

(Third pattern)
The acoustic tag transmitter 14 of the sound collection device 1 transmits the “acoustic tag” and “acoustic signal” to the acoustic tag receiver 26 of the playback device 2.
(sound collection device)
Acoustic tag 101 (water faucet sound): Acoustic signal "jar"->
Acoustic tag 167 (washing machine driving sound): Acoustic signal "Grungurun"->
Acoustic tag 143 (window opening/closing sound): Acoustic signal "bap"->
FIG. 15 is an external view showing the acoustic signal reproduced according to the third pattern.
(Playback device)
-> Acoustic tag 101 (water faucet sound): Direction of arrival 3: Acoustic signal "Jah"
-> Acoustic tag 167 (washing machine driving sound): Arrival direction 7: Acoustic signal "Grungurun"
-> Acoustic tag 143 (window opening/closing sound): Arrival direction 1: Acoustic signal "bap"
Since the playback device 2 has received the acoustic tag and the acoustic signal, the playback device 2 plays back each acoustic tag using the direction of arrival detected by itself. In the third pattern, each acoustic signal inherent in the environmental sound of base A is reproduced from the arrival direction according to the surrounding environment of base B.

In particular, according to the third pattern and FIG. 15, user b can hear the "jar" sound of the water tap at base A from the location of the water supply at base B. Additionally, from the position of the window at base B, the sound of the windows at base A opening and closing can be heard. Furthermore, from the location of the washing machine at location B, the driving sound of the washing machine at location A, "Grungurun", can be heard.
As a result, user b can hear environmental sounds as if they were present at the same base as users a1 and a2.

(Fourth pattern)
The acoustic tag transmitting unit 14 of the sound collection device 1 transmits the “acoustic tag”, “direction of arrival”, and “acoustic signal” to the acoustic tag receiving unit 26 of the reproduction device 2.
(sound collection device)
Acoustic tag 101 (water faucet sound): Direction of arrival 1: Acoustic signal "Jar"->
Acoustic tag 167 (washing machine driving sound): Arrival direction 3: Acoustic signal "Grungurun"->
Acoustic tag 143 (window opening/closing sound): Arrival direction 8: Acoustic signal "bap"->
FIG. 16 is an external view showing the acoustic signal reproduced by the fourth pattern.
(Playback device)
-> Acoustic tag 101 (water faucet sound): Direction of arrival 1: Acoustic signal "Jah"
-> Acoustic tag 167 (washing machine driving sound): Arrival direction 3: Acoustic signal "Grungurun"
-> Acoustic tag 143 (window opening/closing sound): Arrival direction 8: Acoustic signal "bap"
Since the reproducing device 2 has received the acoustic tag, direction of arrival, and acoustic signal, it reproduces it as is. In the fourth pattern, each acoustic signal inherent in the environmental sound at base A is reproduced at base B from the same direction of arrival.

As another embodiment, the environmental sound reproduction section 25 in the reproduction device 2 selects only the acoustic tags other than the acoustic tags detected by the second acoustic tag estimation section 23 among the acoustic tags received from the sound collection device 1. It is also preferable to use an acoustic tag to be played back.
(Acoustic tag estimated from the acoustic signal picked up by the microphone of the playback device)
Acoustic tag 143 (window opening/closing sound)
Acoustic tag 167 (washing machine driving sound)
(Acoustic tag received by the playback device from the sound collection device)
-> Acoustic tag 101 (water faucet sound)
-> Acoustic tag 167 (washing machine driving sound)
-> Acoustic tag 143 (window opening/closing sound)
(Sound tag that should be played on the playback device)
Acoustic tag 101 (water faucet sound)
As a result, for example, only the acoustic signals collected at base A but not at base B will be reproduced at base B.

Furthermore, as another embodiment, the acoustic tags may be expressed in such a way that it is possible to calculate the degree of similarity between the acoustic signals.
The environmental sound reproduction section 25 in the reproduction device 2 calculates the degree of similarity between the acoustic tag received from the sound collection device 1 and the acoustic tag detected by the second acoustic tag estimation section 23, and determines whether the degree of similarity is It is also preferable that only acoustic tags that are equal to or less than a predetermined threshold (not similar) are to be reproduced.
(Acoustic tag estimated from the acoustic signal picked up by the microphone of the playback device)
Acoustic tag 101 (water faucet sound)
Acoustic tag 167 (washing machine driving sound)
Acoustic tag 143 (window opening/closing sound)
(Acoustic tag received by the playback device from the sound collection device)
-> Acoustic tag 101 (water faucet sound)
(Sound tag that should be played on the playback device)
Acoustic tag 143 (window opening/closing sound)
Acoustic tag 101 <-(same)-> Acoustic tag 101
Acoustic tag 101 <- (similarity higher than a predetermined threshold) -> Acoustic tag 167
Acoustic tag 101 <- (dissimilar below a predetermined threshold) -> Acoustic tag 143
As a result, for example, only the acoustic signals collected at base A but not at base B will be reproduced at base B.

As described above in detail, according to the system and method of the present invention, an acoustic signal to be reproduced or an acoustic signal received from a remote sound collection device can be reproduced from a predetermined arrival direction.
According to the present invention, specifically, the acoustic signal of each sound source on the sound collection side can be reproduced according to the position of each sound source on the reproduction side. Even if the sound collection device and the playback device are located at different bases, members staying remotely can share each other's environmental sounds.

Regarding the various embodiments of the present invention described above, various changes, modifications, and omissions within the scope of the technical idea and viewpoint of the present invention can be easily made by those skilled in the art. The above description is merely an example and is not intended to be limiting in any way. The invention is limited only by the claims and their equivalents.

1 Sound Collection Device 101 Microphone 11 First Acoustic Database 12 First Sound Source Separator 13 First Acoustic Tag Estimator 14 Acoustic Tag Transmitter 2 Playback Device 201 Microphone 202 Speaker 21 Second Acoustic Database 22 Second Sound Source Separation unit 23 Second acoustic tag estimation unit 24 Second direction of arrival determination unit 25 Environmental sound reproduction unit 26 Acoustic tag reception unit

Claims (9)

A system in which a sound collection device having a plurality of microphones for collecting environmental sounds, and a playback device having a plurality of microphones and a plurality of speakers for collecting environmental sounds are connected via a network,
The sound collection device is
a first acoustic database that associates and accumulates acoustic signals for each acoustic tag;
a first sound source separation means for detecting one or more acoustic signals inherent in the environmental sound;
a first acoustic tag estimating means for detecting one or more acoustic signals inherent in environmental sounds using a first acoustic database and estimating an acoustic tag of the acoustic signals;
acoustic tag transmitting means for transmitting the acoustic tag to the playback device;
has
The playback device is
a second acoustic database that associates and accumulates acoustic signals for each acoustic tag;
a second sound source separation means that detects one or more acoustic signals inherent in the environmental sound and estimates the direction of arrival of each acoustic signal;
a second acoustic tag estimation means for estimating an acoustic tag of an acoustic signal using a second acoustic database;
and an environmental sound reproduction means for outputting from a plurality of speakers an environmental sound synthesized so that the acoustic signal associated with the acoustic tag received from the sound collection device can be heard from the direction of arrival of the same detected acoustic tag. Featured system. The first sound source separation means of the sound collection device and the second sound source separation means of the reproduction device are characterized in that the direction of arrival of the acoustic signal is estimated by a blind sound source separation method or a beamforming method using a plurality of microphones. The system of claim 1. 2. The playback device further comprises a second direction of arrival determining means for determining the direction of arrival for each acoustic tag based on the distribution of directions of arrival for a predetermined period in the acoustic signal associated with the acoustic tag. system described in. Regarding the sound collection device,
The first sound source separation means further estimates the arrival direction of each acoustic signal,
The acoustic tag transmitting means further transmits the direction of arrival to the playback device,
Regarding the playback device,
3. The environmental sound reproduction means reproduces an environmental sound that is synthesized from the acoustic signal associated with the acoustic tag to be reproduced so as to be heard from the direction of arrival received from the sound collection device. system. Regarding the sound collection device,
The acoustic tag transmitting means further transmits the acoustic signal output from the first sound source separating means,
Regarding the playback device,
5. The system according to claim 1 , wherein the environmental sound reproduction means reproduces an acoustic signal received from a sound collection device. The environmental sound reproduction means in the reproduction device is characterized in that, among the acoustic tags received from the sound collection device, only acoustic tags other than the acoustic tag detected by the second acoustic tag estimation means are acoustic tags to be reproduced. A system according to any one of claims 1 to 5 . Acoustic tags are expressed in such a way that it is possible to calculate the similarity between acoustic signals.
The environmental sound reproduction means in the reproduction device calculates the degree of similarity between the acoustic tag received from the sound collection device and the acoustic tag detected by the second acoustic tag estimation means, and calculates the degree of similarity between the acoustic tag received from the sound collection device and the acoustic tag detected by the second acoustic tag estimating means, and if the degree of similarity is equal to or less than a predetermined threshold ( 7. The system according to claim 6 , wherein only acoustic tags that are not similar are selected as acoustic tags to be reproduced. The acoustic tags and acoustic signals accumulated in the second acoustic database of the playback device are part or all of the acoustic tags and acoustic signals accumulated in the first acoustic database of the sound collection device,
Even if the acoustic tag stored in the second acoustic database of the playback device and the acoustic tag accumulated in the first acoustic database of the sound collection device are the same, it can be determined that the acoustic signals are based on different acoustic signals. 8. A system according to any one of claims 1 to 7 . A system in which a sound collection device having a plurality of microphones for collecting environmental sounds and a playback device having a plurality of microphones for collecting environmental sounds and a plurality of speakers are connected via a network. A reproduction method,
The sound collection device is
Each acoustic tag has a first acoustic database that associates and accumulates acoustic signals.
a first step of detecting one or more acoustic signals inherent in the environmental sound;
a second step of detecting one or more acoustic signals inherent in the environmental sounds using the first acoustic database and estimating an acoustic tag of the acoustic signals;
a third step of transmitting the acoustic tag to the playback device;
Run
The playback device is
Each acoustic tag has a second acoustic database that stores acoustic signals in association with each other;
a fourth step of detecting one or more acoustic signals inherent in the environmental sound and estimating the arrival direction of each acoustic signal;
a fifth step of estimating an acoustic tag of the acoustic signal using the second acoustic database;
and a sixth step of outputting, from a plurality of speakers, an environmental sound synthesized so that the acoustic signal associated with the acoustic tag received from the sound collection device can be heard from the direction of arrival of the same detected acoustic tag. A method for reproducing an acoustic signal, characterized by:

Images