JP7571111B2 - COMMUNICATION TERMINAL, INFORMATION PROCESSING APPARATUS, COMMUNICATION METHOD, AND PROGRAM - Google Patents

Description

The present invention relates to a communication terminal, an information processing device, a communication method, and a program.

Conventionally, a technology is known that makes it difficult for a terminal capable of voice communication with a destination terminal to hear the voice of a speaker who is communicating using another nearby terminal (see, for example, Patent Document 1).

Patent No. 7137033

In the technology described in Patent Document 1, the voice of the speaker using the own terminal is separated based on the difference between the voice level of the speaker using the own terminal and the voice level of the speaker not using the own terminal, and the separated voice is transmitted to the destination terminal. When selecting the voice to transmit to the destination terminal in this manner, there is a problem that if the balance of the volumes of multiple speakers contained in the mixed voice of multiple speakers changes due to a change in the relationship between the positions of multiple speakers close to the own terminal and the position of the own terminal, the voice of another speaker may be selected and transmitted to the destination terminal.

The present invention was made in consideration of these points, and aims to improve the probability that the voice of the speaker using the own terminal can be transmitted to the destination terminal.

The communication terminal of the first aspect of the present invention is a communication terminal used by a user, and includes a storage unit that stores feature data including features of the user's voice, a sound collection unit that collects sounds around the communication terminal and converts them into voice data, a voice separation unit that separates multiple voice data by multiple speakers including the user contained in the voice data to generate multiple internally separated voice data, a selection unit that identifies multiple internal similarities that are similarities between each of the multiple internally separated voice data and the feature data, and selects, from the multiple internal similarities, the internally separated voice data that corresponds to the internal similarity that satisfies a predetermined condition as selected separated voice data, and a communication unit that transmits the selected separated voice data to a call destination terminal used by the user's call partner.

The selection unit may select, as the selected separated audio data, the internal separated audio data corresponding to the internal similarity that satisfies the predetermined condition that the internal similarity is the maximum similarity among the multiple internal similarities.

The communication terminal further has an information acquisition unit that acquires, from another communication terminal that collects the same sound as the sound collected by the sound collection unit, a plurality of external separated sound data generated by the other communication terminal in association with a plurality of external similarities that are similarities between each of the plurality of external separated sound data and feature data of another user using the other communication terminal, and the selection unit may, for each of the plurality of internal separated sound data, associate the internal separated sound data with the internal similarity corresponding to the internal separated sound data, and the external separated sound data that is most similar to the internal separated sound data among the plurality of external separated sound data acquired by the information acquisition unit and the external similarity corresponding to the external separated sound data, and determine the selected separated sound data based on a result of comparing the associated internal similarity with the external similarity.

The selection unit may select the internally separated audio data that satisfies the predetermined condition that the internal similarity is greater than the external similarity as the selected separated audio data.

When the internal similarity is equal to or less than the external similarity, the selection unit may select, as the selected separated audio data, the internal separated audio data that is different from the internal separated audio data that corresponds to the internal similarity that does not satisfy the predetermined condition.

The device may further include an image data acquisition unit that acquires facial image data of the user, and the selection unit may select, from among the plurality of internally separated audio data, the internally separated audio data that is synchronized with the timing of changes in the user's face indicated by the facial image data as the selected separated audio data.

The information processing device of the second aspect of the present invention has a storage unit that stores feature data including features of the voice of a user who uses a communication terminal, a voice separation unit that separates multiple voice data by multiple speakers including the user, which are included in voice data based on voices around the communication terminal collected by the communication terminal, to generate multiple separated voice data, a selection unit that identifies a similarity between each of the multiple separated voice data and the feature data, and selects, from the multiple similarities, the separated voice data corresponding to the similarity that satisfies a predetermined condition as selected separated voice data, and a communication unit that transmits the selected separated voice data to a call destination terminal used by the user's call partner.

The storage unit may store a plurality of feature data including voice features of a plurality of users who use a plurality of the communication terminals, the selection unit may determine the selected separated voice data in association with each of the plurality of communication terminals, and the communication unit may transmit the selected separated voice data to a plurality of the call destination terminals corresponding to the plurality of communication terminals.

The communication method of the third aspect of the present invention includes the steps of: separating a plurality of voice data by a plurality of speakers including the user, which are included in voice data based on the voices around the user who is talking to a call partner using a destination terminal, to generate a plurality of internally separated voice data; generating a plurality of internal similarities, which are similarities between each of the plurality of internally separated voice data and feature data including the voice features of the user stored in a storage unit; selecting, from the plurality of internal similarities, the internally separated voice data corresponding to the internal similarity that satisfies a predetermined condition as selected separated voice data; and transmitting the selected separated voice data to the call partner terminal used by the call partner of the user.

The program of the fourth aspect of the present invention is a program for causing a computer to execute the steps of: separating a plurality of voice data by a plurality of speakers including the user, which are included in voice data based on the voices around a user who is talking to a call partner using a destination terminal, to generate a plurality of internal separated voice data; generating a plurality of internal similarities, which are similarities between each of the plurality of internal separated voice data and feature data including the voice features of the user stored in a storage unit; selecting, from the plurality of internal similarities, the internal separated voice data corresponding to the internal similarity that satisfies a predetermined condition as selected separated voice data; and transmitting the selected separated voice data to a destination terminal used by the call partner of the user.

The present invention has the effect of improving the probability that the voice of the speaker using the own terminal can be transmitted to the destination terminal.

1 is a diagram for explaining an overview of a communication system S. FIG. 2 is a diagram showing a configuration of a communication terminal 1. 11 is a diagram for explaining details of an operation performed by the selection unit 163 to determine selected separated audio data using external separated audio data. FIG. 1 is a diagram showing a plurality of internally separated voice data and a plurality of internal similarities transmitted by each communication terminal 1. FIG. 11 is a diagram for explaining a method in which the selection unit 163 of each of the multiple communication terminals 1 determines selected and separated voice data using internal similarity information and external similarity information. FIG. 1 is a sequence diagram showing a process flow in a plurality of communication terminals 1. FIG. 4 is a flowchart showing a process flow in the communication terminal 1. FIG. 11 is a diagram showing a configuration of a communication terminal 1A according to a first modified example. FIG. 11 is a diagram for explaining an overview of a second embodiment. FIG. 2 is a diagram showing a configuration of an information processing device 3.

First Embodiment
[Overview of communication system S]
1 is a diagram for explaining an overview of a communication system S. The communication system S is a system for enabling a user to perform voice communication with another user via a network N using a communication terminal 1 operated by the user. The network N is, for example, a telephone communication network, the Internet, a local area network, or the like.

Figure 1 shows a state in which user U1 is using communication terminal 1 to talk to user U10, who is the call destination using call destination terminal 2. Near user U1, user U2 is talking to a user other than user U10, and the voice of user U2 enters communication terminal 1 along with the voice of user U1. When the voices of user U1 and user U2 enter communication terminal 1 in this way, the voices of user U1 and user U2 are output from call destination terminal 2, causing a problem that user U10 has difficulty hearing the voice of user U1. In the example shown in Figure 1, if communication terminal 1 says "Hello" and user U2 says "Nice to meet you," the voice of "Hello" overlaps with the voice of "Nice to meet you" at call destination terminal 2.

To solve this problem, a technique for separating the voice of user U1 from the voice of user U2 is known. As an example, in the technique described in Japanese Patent No. 7137033, a communication terminal 1 used by user U1 and a communication terminal 1 used by user U2 communicate wirelessly to transmit a signal based on the voice input to each to the other. In the communication terminal 1, based on the difference between the volume ratio of the voice of user U1 and the voice of user U2 contained in the collected mixed voice and the volume ratio of the voice of user U1 and the voice of user U2 contained in the mixed voice received by wireless communication, the collected mixed voice and the mixed voice received by wireless communication are subjected to independent component analysis to separate the voice of user U1 (separated voice Z1 in FIG. 1) from the voice of user U2 (separated voice Z2 in FIG. 1).

The method of separating the voices is arbitrary, and the communication terminal 1 may have multiple microphones and separate the multiple voices based on the voices picked up by the multiple microphones and the distance between the multiple microphones. The communication terminal 1 transmits the voice determined to be the voice of user U1 from the separated voice Z1 and separated voice Z2 to the call destination terminal 2. This results in a state where user U10 using the call destination terminal 2 can hear the voice of user U1 but cannot hear the voice of user U2.

When multiple voices are separated in this manner, there may be cases where the voice of user U1 cannot be transmitted to the destination terminal 2 used by the caller of user U1, due to fluctuations in the volume of the voice uttered by user U2 depending on the state of the surrounding environment, or delays due to reflections. Therefore, the communication terminal 1 according to this embodiment is characterized in that it uses feature data including the features of the voice of user U1 acquired in advance, or feature data including the features of the face of user U1 while he is speaking, to increase the probability of correctly selecting the voice of user U1 from the separated voices Z1 and Z2.

[Configuration of communication terminal 1]
The configuration and operation of the communication terminal 1 will be described in detail below. In this specification, an example will be described in which the communication terminal 1 separates multiple voices using voices received from other communication terminals 1 via wireless communication. In the following description, it is assumed that a user U1 uses the communication terminal 1-1 and a user U2 uses the communication terminal 1-2.

Figure 2 is a diagram showing the configuration of communication terminal 1. Communication terminal 1 has a sound collection unit 11, a first communication unit 12, a second communication unit 13, an audio output unit 14, a storage unit 15, and a control unit 16. The control unit 16 has an audio separation unit 161, an information acquisition unit 162, and a selection unit 163. Here, the configuration of communication terminal 1 will be explained using as an example a case where communication terminal 1 is communication terminal 1-1 used by user U1.

The sound collection unit 11 collects sounds around the communication terminal 1 and converts them into sound data. The sound collection unit 11 has, for example, a microphone. The sound collection unit 11 inputs the sound data to the sound separation unit 161.

The first communication unit 12 wirelessly communicates with the communication terminal 1-2 used by the user U2 using a wireless communication channel (e.g., Bluetooth (registered trademark) or Wi-Fi (registered trademark)). The first communication unit 12 receives mixed voice data collected by the communication terminal 1-2 and a plurality of external separated voice data generated by the communication terminal 1-2 based on the voice collected by the communication terminal 1-2. The first communication unit 12 also receives a plurality of external similarities, which are similarities between the plurality of external separated voice data and feature data of the user U2, who is another user using the communication terminal 1-2, in association with the plurality of external separated voice data. The first communication unit 12 inputs the received mixed voice data to the voice separation unit 161. The first communication unit 12 inputs the received plurality of external separated voice data and the plurality of external similarities to the information acquisition unit 162.

The multiple external separated voice data are voice data generated by the communication terminal 1-2 by separating multiple voices contained in the voice collected by the communication terminal 1-2. The multiple external similarities are represented, for example, by a correlation value or Kullback-Leibler information between information obtained by cepstral analysis of the multiple external separated voice data and information obtained by cepstral analysis of the feature data of the user U2.

The second communication unit 13 has a communication interface for communicating with the call destination terminal 2 used by the call destination user U10 via the network N and transmitting and receiving voice signals. The second communication unit 13 transmits the selected and separated voice data input from the selection unit 163 to the call destination terminal 2, which is the call destination terminal used by user U10, who is the call partner of user U1. In addition, the second communication unit 13 inputs the voice data received from the call destination terminal 2 to the voice output unit 14. Details of the selected and separated voice data will be described later.

The audio output unit 14 converts the audio data received from the call destination terminal 2 via the second communication unit 13 into audio and outputs it. The audio output unit 14 is, for example, a speaker, earphones, headphones, etc.

The storage unit 15 has storage media such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The storage unit 15 stores programs executed by the control unit 16. The storage unit 15 also stores various data used to separate multiple voices. The storage unit 15 stores, for example, feature data including voice features of a pre-registered user U1.

The control unit 16 has, for example, a CPU (Central Processing Unit). The control unit 16 executes the programs stored in the storage unit 15 to function as an audio separation unit 161, an information acquisition unit 162, and a selection unit 163.

The audio separation unit 161 separates multiple pieces of audio data by multiple speakers including user U1, which are included in the audio data input from the sound collection unit 11, to generate multiple pieces of internally separated audio data. As described above, the audio separation unit 161 may separate the multiple pieces of audio data included in the audio data input from the sound collection unit 11 using any method. The audio separation unit 161 inputs the multiple pieces of internally separated audio data that it has generated to the selection unit 163. The audio separation unit 161 inputs the multiple pieces of internally separated audio data to the selection unit 163, for example, in association with information for identifying each of the multiple pieces of internally separated audio data. The audio separation unit 161 may input the multiple pieces of internally separated audio data to the selection unit 163 at different times.

The information acquisition unit 162 acquires, via the first communication unit 12, from another communication terminal 1 (e.g., communication terminal 1-2) that collects the same sound as the sound collected by the sound collection unit 11, a plurality of external separated sound data generated by the other communication terminal 1, and a plurality of external similarities, which are the similarities between each of the plurality of external separated sound data and the feature data of another user U2 who uses the other communication terminal 1, in association with each other. The information acquisition unit 162 associates the acquired plurality of external separated sound data with the plurality of external similarities and inputs them to the selection unit 163. The information acquisition unit 162 may associate the acquired plurality of external separated sound data with the plurality of external similarities and store them in the storage unit 15.

The selection unit 163 identifies multiple internal similarities, which are the similarities between each of the multiple internally separated audio data input from the audio separation unit 161 and the feature data of the user U1 stored in the storage unit 15. Similar to the multiple external similarities described above, the multiple internal similarities are represented by, for example, a correlation value or Kullback-Leibler information between information obtained by cepstral analysis of the multiple internally separated audio data and information obtained by cepstral analysis of the feature data of the user U1.

The selection unit 163 selects, as the selected separated audio data, the internal separated audio data corresponding to an internal similarity that satisfies a predetermined condition from among the multiple identified internal similarities. The predetermined condition is, for example, the largest value from among the multiple internal similarities. In this case, the selection unit 163 selects, as the selected separated audio data, the internal separated audio data corresponding to an internal similarity that satisfies the predetermined condition of being the largest similarity from among the multiple internal similarities. The predetermined condition may further include a condition that the similarity is equal to or greater than a threshold value.

The selection unit 163 inputs the selected separated voice data to the second communication unit 13, and transmits the selected separated voice data to the call destination terminal 2 via the network N. By transmitting the internal separated voice data that is most similar to the characteristic data of user U1 to the call destination terminal 2 in this manner, the user U10 using the call destination terminal 2 cannot hear any voice other than the voice of user U1, making it easier for user U10 to talk to user U1.

However, if the voice characteristics of user U1 and user U2 are similar, the selection unit 163 of the communication terminal 1-1 used by user U1 may erroneously select separated voice data based on the voice of user U2. Therefore, the selection unit 163 may determine the selected separated voice data by further using multiple external separated voice data acquired via the information acquisition unit 162.

Specifically, for each of the multiple internal separated audio data, the selection unit 163 associates the internal separated audio data with the internal similarity corresponding to the internal separated audio data, and the external separated audio data that is most similar to the internal separated audio data among the multiple external separated audio data acquired by the information acquisition unit 162 with the external similarity corresponding to the external separated audio data. Then, the selection unit 163 selects the selected separated audio data to be transmitted to the call destination terminal 2 based on the result of comparing the associated internal similarity with the external similarity. Below, the operation of the selection unit 163 to determine the selected separated audio data using the external separated audio data will be described in detail.

Figures 3 to 5 are diagrams for explaining the details of the operation of the selection unit 163 when determining selected separated voice data using external separated voice data. Here, the operation of the selection unit 163 is explained when users U2 and U3 are present in the vicinity of user U1. User U1 using communication terminal 1-1 utters the voice "Hello," user U2 using communication terminal 1-2 utters the voice "Nice to meet you" at the same time that user U1 utters the voice, and user U3 using communication terminal 1-3 also utters the voice "How are you?" at the same time that user U1 utters the voice.

The selection units 163 of each of communication terminals 1-1, 1-2, and 1-3 share, via wireless communication, information indicating multiple internal similarities corresponding to the multiple internally separated audio data input from the audio separation unit 161 (hereinafter referred to as "internal similarity information"). Specifically, for example, communication terminal 1-1 transmits internal similarity information to communication terminals 1-2 and 1-3 with which wireless communication is possible. Communication terminals 1-2 and 1-3 similarly transmit internal similarity information to the other communication terminals 1. Each communication terminal 1 manages the internal similarity information received from the other communication terminals 1 as external similarity information.

Figure 4 is a diagram showing multiple internal separated audio data and multiple internal similarities transmitted by each communication terminal 1. Figure 4(a) shows internal similarity information transmitted by communication terminal 1-1, Figure 4(b) shows internal similarity information transmitted by communication terminal 1-2, and Figure 4(c) shows internal similarity information transmitted by communication terminal 1-3. The internal similarity information includes information (circles in Figure 4) indicating internal separated audio data selected from the multiple internal separated audio data.

The selection unit 163 determines the selected separated audio data using the external similarity information, which is the internal similarity information received from the other communication terminal 1 in this manner. Specifically, the selection unit 163 selects, as the selected separated audio data, internal separated audio data that satisfies a predetermined condition that the internal similarity is greater than the external similarity. When the internal similarity of the own terminal is equal to or less than the external similarity, the selection unit 163 selects, as the selected separated audio data, internal separated audio data that is different from the internal separated audio data that corresponds to the internal similarity that does not satisfy the above-mentioned predetermined condition.

Figure 5 is a diagram for explaining a method in which the selection unit 163 of each of multiple communication terminals 1 determines selected separated audio data using internal similarity information and external similarity information. Figure 5(a) shows information (provisional selection result) indicating the internal similarity calculated in each communication terminal 1 and the internal separated audio data provisionally selected in each communication terminal 1. Table A corresponds to communication terminal 1-1, table B corresponds to communication terminal 1-2, and table C corresponds to communication terminal 1-3.

In table A, the similarity between the feature data of user U1 and "Hello" is the highest at 0.8, and it can be seen that communication terminal 1-1 has provisionally selected "Hello." In table B, the similarity between the feature data of user U2 and "Nice to meet you" is the highest at 0.6, and it can be seen that communication terminal 1-2 has provisionally selected "Nice to meet you." In table C, the similarity between the feature data of user U3 and "Nice to meet you" is the highest at 0.5, and it can be seen that communication terminal 1-3 has also provisionally selected "Nice to meet you."

In such a case, the selection unit 163 of communication terminal 1-1 determines the provisionally selected internal separated audio data as the selected separated audio data, since the other communication terminal 1 has not selected the internal separated audio data that it has provisionally selected. On the other hand, the selection units 163 of communication terminals 1-2 and 1-3 compare the internal similarity and external similarity corresponding to the selected internal separated audio data, since the internal separated audio data that they have provisionally selected has been provisionally selected by the other communication terminal 1. In the example shown in FIG. 5(a), the similarity of 0.6 in table B is compared with the similarity of 0.5 in table C. As a result, the internal similarity of table B is greater than the external similarity of table C, so the selection unit 163 of communication terminal 1-2 determines the provisionally selected internal separated audio data as the selected separated audio data.

The selection unit 163 of the communication terminal 1-3 does not determine the provisionally selected internal separated voice data as the selected separated voice data because the internal similarity is smaller than the external similarity of the other communication terminal 1-2, but determines the internal separated voice data with the next highest internal similarity as the selected separated voice data. In the example shown in FIG. 5(a), the selection unit 163 of the communication terminal 1-3 determines "How are you?", which has the second highest internal similarity, as the selected separated voice data.

Figure 5(b) shows the result of the selection unit 163 of each communication terminal 1 determining the selected and separated voice data in this manner. By the selection unit 163 operating in this manner, the probability that the correct voice of user U will be selected is increased even if the voice characteristics of user U using communication terminal 1 are similar to the voice characteristics of another user U.

[Processing flow in multiple communication terminals 1]
Fig. 6 is a sequence diagram showing the flow of processing in a plurality of communication terminals 1. The sequence diagram shown in Fig. 6 starts from the point when communication terminals 1-1, 1-2, and 1-3 start collecting sound. In Fig. 6, it is assumed that communication terminal 1-1 communicates with call destination terminal 2-1 via network N, and communication terminal 1-2 communicates with call destination terminal 2-2 via network N.

First, communication terminal 1-1, communication terminal 1-2, and communication terminal 1-3 acquire mixed audio including the voices of user U1, user U2, and user U3 from other communication terminals 1 (S1). Next, the audio separation unit 161 of each communication terminal 1 executes audio separation processing based on the acquired mixed audio (S2). Next, the selection unit 163 of each communication terminal 1 calculates the similarity between the feature data stored in the storage unit 15 and multiple internally separated audio data (S3), and provisionally selects one internally separated audio (S4).

Then, communication terminal 1-1, communication terminal 1-2, and communication terminal 1-3 share similarity information indicating the multiple separated audio data and the similarity (S5). That is, the information acquisition unit 162 of each communication terminal 1 acquires the internal similarity information transmitted by the other communication terminal 1 and indicating the multiple internal separated audio data and the internal similarity as external similarity information indicating the multiple external separated audio data and the external similarity.

The selection unit 163 of each communication terminal 1 calculates a correlation value between the multiple externally separated audio data acquired by the information acquisition unit 162 from the other communication terminals 1 and the multiple internally separated audio data input from the audio separation unit 161 of the terminal itself, and links each internally separated audio data to the externally separated audio data with the highest correlation value (S6). The selection unit 163 then identifies the external similarity corresponding to each internally separated audio data, and creates a table as shown in FIG. 5.

As explained with reference to FIG. 5, the selection unit 163 of each communication terminal 1 determines whether the provisionally selected internal separated audio data is correct by determining whether the internal similarity corresponding to the internal separated audio data that it has provisionally selected is greater than the external similarity corresponding to that internal separated audio data (S7). If the selection unit 163 determines that the internal similarity is greater than the external similarity and that the provisionally selected internal separated audio data is correct, it selects the provisionally selected internal separated audio data as the selected separated audio data. If the selection unit 163 determines that the provisionally selected internal separated audio data is incorrect, it selects other internal separated audio data as the selected separated audio data (i.e., audio data to be transmitted) (S8).

The selection unit 163 of communication terminal 1-1 transmits the selected separated voice data to the call destination terminal 2-1 (S9), and the selection unit 163 of communication terminal 1-2 transmits the selected separated voice data to the call destination terminal 2-2 (S10). Through this procedure, each of the multiple communication terminals 1 can transmit the voice of a user U using the multiple communication terminals 1 to the call destination terminal 2 used by the other party in the call.

[Processing flow in communication terminal 1]
Fig. 7 is a flowchart showing the flow of processing in the communication terminal 1. The flowchart shown in Fig. 7 starts at the point when the user U1 who uses the communication terminal 1 performs an operation to start a call with the user U10.

The audio separation unit 161 acquires audio data via the sound collection unit 11, and also acquires mixed audio data generated by the other communication terminal 1 collecting audio via the first communication unit 12 (S11). The audio separation unit 161 generates multiple internally separated audio data based on the acquired audio data (S12).

Then, the selection unit 163 calculates the similarity between each of the internal separated audio data and the feature data by comparing the multiple internal separated audio data with the feature data stored in the storage unit 15 (S13). The selection unit 163 provisionally selects the internal separated audio data with the highest similarity (S14). The selection unit 163 transmits internal similarity information indicating the internal separated audio data and the internal similarity to the other communication terminal 1 via the first communication unit 12 (S15).

The selection unit 163 also acquires external similarity information indicating the external separated audio data and the external similarity from the other communication terminal 1 via the first communication unit 12 and the information acquisition unit 162 (S16). The selection unit 163 links the internal separated audio data with the external separated audio data by calculating a correlation value for each combination of multiple internal separated audio data and multiple external separated audio data, thereby creating a table such as that shown in FIG. 5 (S17).

The selection unit 163 determines whether the maximum similarity among the multiple similarities calculated in S13 (i.e., the similarity corresponding to the provisionally selected internal separated audio data) is greater than the external similarity associated with the external separated audio data corresponding to the internal separated audio data obtained from another communication terminal 1 (S18). If the selection unit 163 determines in S18 that the maximum internal similarity is greater than the external similarity (YES in S18), it selects the internal separated audio data corresponding to the maximum internal similarity as the selected separated audio data (S19).

When the selection unit 163 determines in S18 that the maximum internal similarity is smaller than the external similarity (NO in S18), it selects the internal separated audio data corresponding to the next largest internal similarity as selected separated audio data (S20). The selection unit 163 transmits the selected separated audio data to the call destination terminal 2 via the second communication unit 13 (S21). The communication terminal 1 repeats the processes from S11 to S21 at a predetermined time interval. The predetermined time interval is, for example, one second.

If the selection unit 163 determines in S18 that the maximum internal similarity and the external similarity are equal, the selection unit 163 executes the process of S19 under the condition that, for example, the difference between the maximum internal similarity and the second internal similarity is greater than the difference between the maximum external similarity and the second external similarity of the other communication terminal 1.

If the selection unit 163 determines in S18 that the maximum internal similarity and the external similarity are equal, the selection unit 163 may transmit the provisionally selected internal voice data to the call destination terminal 2. If the selection unit 163 determines in S18 that the maximum internal similarity and the external similarity are equal, the selection unit 163 may not determine the selected separated voice data, and may not transmit the voice data to the call destination terminal 2 for, for example, one second, which is the processing time interval.

[Modification]
Fig. 8 is a diagram showing the configuration of a communication terminal 1A according to a first modified example. The communication terminal 1A shown in Fig. 8 differs from the communication terminal 1 shown in Fig. 2 in that it further includes an image data acquisition unit 17, but is otherwise the same. The image data acquisition unit 17 acquires face image data of a user U who uses the communication terminal 1. The image data acquisition unit 17 includes, for example, an image sensor.

The selection unit 163 selects, from among the multiple internal separated audio data, internal separated audio data that is synchronized with the timing of changes in the face of user U indicated by the facial image data as the selected separated audio data. For example, the selection unit 163 converts the multiple internal separated audio data into frequency domain data and selects internal separated audio data in which the facial image data changes in frequency in synchronization with the opening and closing of user U's mouth or changes in the shape of the mouth. The selection unit 163 may use feature data including features of user U's mouth movement (or lip movement) based on the inter-frame difference of the facial image data. By the selection unit 163 operating in this manner, the probability of selecting appropriate internal separated audio data is further increased.

[Effects of communication terminal 1]
As described above, the communication terminal 1 according to the present embodiment identifies a plurality of internal similarities, which are the similarities between each of a plurality of internally separated voice data generated by separating collected voice for each speaker and feature data including voice features of the user U of the communication terminal 1, and transmits the internally separated voice data corresponding to the internal similarity that satisfies a predetermined condition among the plurality of internal similarities to the call destination terminal 2. By configuring the communication terminal 1 in this way, the probability that the internally separated voice data corresponding to the voice of the user U of the communication terminal 1 will be selected increases, and the probability that the voice of the speaker using the communication terminal 1 can be transmitted to the call destination terminal increases.

Second Embodiment
In the first embodiment, the communication terminal 1 separated multiple voices included in the input mixed voice and selected the voice of the user U who uses the communication terminal 1. In contrast, the second embodiment differs from the first embodiment in that the information processing device 3 acquires mixed voice data from the communication terminal 1, separates multiple voices included in the mixed voice to generate selected separated voice data, and transmits the selected separated voice data to the call destination terminal 2. The information processing device 3 is a computer capable of communicating with the communication terminal 1 and the call destination terminal 2 via, for example, a network N.

Figure 9 is a diagram for explaining an overview of the second embodiment. Communication terminal 1-1, communication terminal 1-2, and communication terminal 1-3 each transmit voice data generated by collecting sound to information processing device 3. Information processing device 3 stores feature data of user U of each of the multiple communication terminals 1, and selects the internal separated voice data that is most similar to the feature data from multiple internal separated voice data based on voice data received from each of the multiple communication terminals 1. Information processing device 3 transmits the selected internal separated voice data to call destination terminal 2 communicating with each of the multiple communication terminals 1. Information processing device 3 transmits internal separated voice data corresponding to "hello" uttered by user U1 to call destination terminal 2-1 communicating with communication terminal 1-1, for example.

Figure 10 is a diagram showing the configuration of the information processing device 3. The information processing device 3 has a communication unit 31, a storage unit 32, and a control unit 33. The control unit 33 has an audio separation unit 331 and a selection unit 332.

The communication unit 31 has a communication interface for communicating with the communication terminal 1 and the call destination terminal 2 via the network N. The communication unit 31 receives voice data transmitted from the communication terminal 1 and inputs the received voice data to the voice separation unit 331. The communication unit 31 transmits the selected separated voice data generated by the selection unit 332 to the call destination terminal 2. In other words, the communication unit 31 transmits the selected separated voice data to the call destination terminal 2, which is the call destination terminal used by the call partner of the user U. The communication unit 31 may transmit the selected separated voice data to multiple call destination terminals 2 corresponding to multiple communication terminals 1.

The storage unit 32 has storage media such as ROM, RAM, and SSD (Solid State Drive). The storage unit 32 stores programs executed by the control unit 33. The storage unit 32 also stores multiple feature data including voice features of multiple users who use multiple communication terminals 1. The storage unit 32 stores the feature data of user U, for example, in association with identification information for identifying the communication terminal 1 or the user U who uses the communication terminal 1. The storage unit 32 may also store identification information of the call destination terminal 2, in association with each of the multiple communication terminals 1.

The control unit 33 executes a program stored in the storage unit 32 to function as a voice separation unit 331 and a selection unit 332. The voice separation unit 331 has a function equivalent to that of the voice separation unit 161 in the communication terminal 1 described in the first embodiment. That is, the voice separation unit 331 separates multiple voice data by multiple speakers, including the user U, contained in voice data based on voices around the communication terminal 1 collected by the communication terminal 1, to generate multiple separated voice data. However, the voice separation unit 331 generates multiple separated voice data for each of the multiple voice data received from the multiple communication terminals 1. The voice separation unit 331 notifies the selection unit 332 of the multiple separated voice data in association with the identification information of the communication terminal 1 or the user U.

The selection unit 332 has a function equivalent to that of the selection unit 163 in the communication terminal 1 described in the first embodiment. That is, the selection unit 332 specifies the similarity between each of the multiple separated voice data and the feature data, and selects, from the multiple similarities, the separated voice data corresponding to the similarity that satisfies a predetermined condition as the selected separated voice data. For example, based on the multiple separated voice data corresponding to the multiple communication terminals 1 notified by the voice separation unit 331, the selection unit 332 selects, from the multiple separated voice data corresponding to each communication terminal 1, the separated voice data that has the greatest similarity to the feature data of the user U of the communication terminal 1 as the selected separated voice data. The selection unit 332 determines the selected separated voice data in association with each of the multiple communication terminals 1.

When the selected separated audio data corresponding to the multiple communication terminals 1 becomes the same separated audio data, the selection unit 332 determines the selected separated audio data so that the separated audio data corresponding to each of the multiple communication terminals 1 is different, according to the procedure described with reference to FIG. 5. The selection unit 332 transmits the selected separated audio data corresponding to each of the multiple communication terminals 1 to the call destination terminal 2 used by the other user U with whom the user U of each of the multiple communication terminals 1 calls.

In this way, by generating separated voice data similar to the characteristic data of the user U using the communication terminal 1 from the voice data received by the information processing device 3 from multiple communication terminals 1, it is possible to reduce the processing load on the communication terminal 1 while increasing the probability that the voice of the speaker using the communication terminal 1 can be transmitted to the destination terminal.

Furthermore, this invention will make it possible to contribute to Goal 9 of the United Nations' Sustainable Development Goals (SDGs), which is "Build resilient infrastructure, promote inclusive and sustainable industrialization, and promote innovation and infrastructure."

Although the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist of the invention. For example, all or part of the device can be configured by distributing or integrating functionally or physically in any unit. In addition, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effect of the new embodiment resulting from the combination also has the effect of the original embodiment.

REFERENCE SIGNS LIST 1 Communication terminal 2 Call destination terminal 3 Information processing device 11 Sound collection unit 12 First communication unit 13 Second communication unit 14 Audio output unit 15 Storage unit 16 Control unit 17 Image data acquisition unit 31 Communication unit 32 Storage unit 33 Control unit 151 Storage unit 161 Audio separation unit 162 Information acquisition unit 163 Selection unit 331 Audio separation unit 332 Selection unit

Claims (10)

A communication terminal used by a user,
A storage unit that stores feature data including features of the user's voice;
a sound collection unit that collects sounds around the communication terminal and converts the sounds into sound data;
a voice separation unit that separates a plurality of voice data by a plurality of speakers including the user, which are included in the voice data, to generate a plurality of internally separated voice data;
a selection unit that identifies a plurality of internal similarities that are similarities between each of the plurality of internal separated voice data and the feature data, and selects, from the plurality of internal similarities, the internal separated voice data corresponding to the internal similarity that satisfies a predetermined condition as selected separated voice data;
a communication unit for transmitting the selected and separated voice data to a destination terminal used by a call partner of the user;
A communication terminal having the above configuration. the selection unit selects, as selected separated audio data, the internal separated audio data corresponding to the internal similarity that satisfies the predetermined condition that the internal similarity is the maximum similarity among the plurality of internal similarities.
The communication terminal according to claim 1. The present invention further includes an information acquisition unit that acquires, from another communication terminal that collects the same sound as the sound collected by the sound collection unit, a plurality of external separated sound data generated by the other communication terminal, and a plurality of external similarities, which are similarities between each of the plurality of external separated sound data and feature data of another user who uses the other communication terminal, in association with each other,
the selection unit associates, for each of the plurality of internal separated audio data, the internal similarity corresponding to the internal separated audio data, and the external separated audio data that is most similar to the internal separated audio data among the plurality of external separated audio data acquired by the information acquisition unit, and the external similarity corresponding to the external separated audio data, and determines the selected separated audio data based on a result of comparing the associated internal similarity with the external similarity;
The communication terminal according to claim 1. The selection unit selects the internal separated speech data that satisfies the predetermined condition that the internal similarity is greater than the external similarity as the selected separated speech data.
The communication terminal according to claim 3. When the internal similarity is equal to or less than the external similarity, the selection unit selects, as the selected separated audio data, the internal separated audio data that is different from the internal separated audio data that corresponds to the internal similarity that does not satisfy the predetermined condition.
The communication terminal according to claim 4. An image data acquisition unit that acquires face image data of the user,
the selection unit selects, from the plurality of internal separated audio data, the internal separated audio data synchronized with a timing of a change in the face of the user indicated by the face image data as selected separated audio data;
The communication terminal according to claim 1. A storage unit that stores feature data including features of a voice of a user who uses a communication terminal;
a voice separation unit that separates a plurality of voice data by a plurality of speakers including the user, which are included in voice data based on voices around the communication terminal collected by the communication terminal, to generate a plurality of separated voice data;
a selection unit that specifies a similarity between each of the plurality of separated speech data and the feature data, and selects, from the plurality of similarities, the separated speech data corresponding to the similarity that satisfies a predetermined condition as selected separated speech data;
a communication unit for transmitting the selected and separated voice data to a destination terminal used by a call partner of the user;
An information processing device having the above configuration. the storage unit stores a plurality of feature data including voice features of a plurality of users who use a plurality of the communication terminals;
The selection unit determines the selected and separated audio data in association with each of the plurality of communication terminals;
the communication unit transmits the selected and separated voice data to a plurality of the call destination terminals corresponding to the plurality of communication terminals.
The information processing device according to claim 7. The computer executes
A step of generating a plurality of internally separated voice data by separating a plurality of voice data by a plurality of speakers including the user, the plurality of voice data being included in voice data based on voices around the user who is communicating with the communication partner using the communication destination terminal;
generating a plurality of internal similarities which are similarities between each of the plurality of internal separated voice data and feature data including features of the user's voice stored in a storage unit;
selecting, as selected separated speech data, the internal separated speech data corresponding to the internal similarity that satisfies a predetermined condition from among the plurality of internal similarities;
transmitting the selected and separated voice data to a destination terminal used by a call partner of the user;
A communication method comprising: On the computer,
A step of generating a plurality of internally separated voice data by separating a plurality of voice data by a plurality of speakers including the user, the plurality of voice data being included in voice data based on voices around the user who is communicating with the communication partner using the communication destination terminal;
generating a plurality of internal similarities which are similarities between each of the plurality of internal separated voice data and feature data including features of the user's voice stored in a storage unit;
selecting, as selected separated speech data, the internal separated speech data corresponding to the internal similarity that satisfies a predetermined condition from among the plurality of internal similarities;
transmitting the selected and separated voice data to a destination terminal used by a call partner of the user;
A program for executing the above.

Images