JP7713415B2 - Conversation Analysis Device - Google Patents

Description

The present invention relates to a conversation analysis device.

Patent Document 1 discloses a technique for determining whether or not a conversation has been established based on the meaning of speech. Patent Document 2 discloses a technique for calculating the degree of conversation establishment, which indicates the degree to which a conversation has been established, based on the proportion of intervals in which one speaker's conversational voice is active and the other speaker's conversational voice is silent, and the proportion of intervals in which both are active or silent.

JP 2014-178621 A International Publication No. 2012/042768

In recent years, various robots that converse with users through voice have been proposed. If the user does not respond sincerely to questions from the robot, for example, when the user replies in a joking manner or gives a casual response, it cannot be said that a conversation between the robot and the user has been established. However, in a technology that determines whether a conversation has been established based on the meaning of the user's speech or the proportion of sound or silence in each of the robot's conversational voice and the user's conversational voice, it may be erroneously determined that a conversation has been established if the user is not responding sincerely.

In order to solve the above problems, a conversation analysis device according to a preferred aspect of the present disclosure includes a voice control unit, a first estimation unit, a second estimation unit, and a first determination unit. The voice control unit acquires voice data representing the voice of a reply from a user in a conversation between a robot and a user. The first estimation unit estimates the emotion implied in the reply based on the tone of voice represented by the voice data. The second estimation unit estimates the emotion implied in the reply based on the wording represented by the voice data. The first determination unit determines whether or not the conversation is established based on a degree of agreement indicating the degree of agreement between the emotion estimated by the first estimation unit and the emotion estimated by the second estimation unit.

The conversation analysis device disclosed herein reduces the chance of erroneously determining that a conversation is taking place even when the user is not responding sincerely to a robot.

1 is a block diagram showing an example of the configuration of a conversation system 1 according to an embodiment. FIG. 2 is a front view showing an example of the appearance of the robot 30. 1 is a block diagram showing an example of the configuration of a user device 10. FIG. FIG. 2 is a block diagram showing an example of the configuration of a robot 30. FIG. 2 is a block diagram showing an example of the configuration of a conversation analysis device 40. FIG. 2 is a diagram showing an example of a topic table TBL1. FIG. 2 is a diagram showing an example of a word table TBL2. FIG. 4 is a functional block diagram for explaining functions realized by a processing device 410 of a conversation analysis device 40 according to a program PR4. FIG. 2 is a diagram showing an example of a conversation between a robot 30 and a conversation target U1. 13 is a flowchart showing the flow of a conversation analysis method executed by the processing device 410 according to the program PR4.

A. EMBODIMENTS
(A-1: Overall configuration)
FIG. 1 is a block diagram showing a configuration example of a conversation system 1 according to an embodiment of the present disclosure. As shown in FIG. 1, the conversation system 1 includes a conversation analysis device 40, a user device 10, and a robot 30. A conversation target U1 is a user who uses the robot 30. The robot 30 has a conversation with the conversation target U1 under the control of the conversation analysis device 40. A typical example of the conversation target U1 is a child such as an elementary school student. The robot 30 is purchased for the conversation target U1 by a guardian U2 of the conversation target U1. The robot 30 is placed, for example, in a study room of the conversation target U1. The user device 10 communicates with the conversation analysis device 40 via a communication network NW. The user device 10 is used by the guardian U2.

2 is a front view showing an example of the appearance of the robot 30. The robot 30 has a head 31 and a torso 32. The robot 30 operates by battery power. A speaker 370 and a human sensor 350 are arranged in the torso 32. A microphone 360 (not shown) is arranged inside the head 31. The robot 30 detects the approach of the conversation target U1 using the human sensor 350. The robot 30 switches the operation mode based on the detection result by the human sensor 350. The operation modes of the robot 30 include a conversation mode in which the robot 30 converses with the conversation target U1, and a sleep mode in which the robot 30 does not converse with the conversation target U1. The power consumption of the robot 30 in the sleep mode is smaller than that of the robot 30 in the conversation mode. In the sleep mode, power is supplied to the human sensor 350, so that the approach of the conversation target U1 to the robot 30 is detected, while the supply of power to other components is restricted. By having a conversation mode and a sleep mode, the robot 30 can converse when the conversation target U1 is nearby, but can save power consumption when the conversation target U1 is not nearby.

In this embodiment, the robot 30 is used for the emotional education of the conversation target U1 through conversation. In this embodiment, the conversation analysis device 40 analyzes the voice of the conversation target U1's response in conversation with the robot 30, thereby grasping the growth factors of the conversation target U1. The growth factors of the conversation target U1 refer to things with which the conversation target U1 has low empathy and where future growth is expected.

(A-2: User device)
FIG. 3 is a block diagram showing an example of the configuration of the user device 10. Specific examples of the user device 10 include a smartphone or a tablet terminal. The user device 10 includes a processing device 110, a storage device 120, a display panel 130, a communication device 140, and an input device 150. The elements of the user device 10 are connected to each other by a single or multiple buses for communicating information. Note that the term "apparatus" in this specification may be replaced with other terms such as a circuit, a device, or a unit. The elements of the user device 10 may be composed of a single or multiple devices. Some elements of the user device 10 may be omitted.

The processing device 110 is a processor that controls the entire user device 10. The processing device 110 is composed of, for example, one or more chips. The processing device 110 is composed of, for example, a central processing unit (CPU) including an interface with peripheral devices, an arithmetic unit, and a register. Note that some or all of the functions of the processing device 110 may be realized by hardware such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The processing device 110 executes various processes in parallel or sequentially. The processing device 110 functions as the control center of the user device 10 by operating according to the program PR1 stored in the storage device 120.

The storage device 120 is a recording medium that can be read by the processing device 110. The storage device 120 stores a plurality of programs including the program PR1, and various information used by the processing device 110. The storage device 120 may be configured with at least one of, for example, a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), an EEPROM (Electrically Erasable Programmable ROM), a RAM (Random Access Memory), and the like. The program PR1 may be transmitted from another device such as the conversation analysis device 40 via the communication network NW. Specific examples of information stored in the storage device 120 and used by the processing device 110 include first identification information that uniquely identifies the robot 30 in the communication network NW, and second identification information that uniquely identifies the conversation analysis device 40 in the communication network NW. A specific example of the first identification information is a communication address assigned to the robot 30. Similarly, a specific example of the second identification information is a communication address assigned to the conversation analysis device 40.

The display panel 130 is a device that displays images. Specific examples of the display panel 130 include a liquid crystal display panel and an organic EL (Electro Luminescence) display panel.

The communication device 140 is hardware (transmitting/receiving device) for communicating with other devices. The communication device 140 is also called, for example, a network device, a network controller, a network card, a communication module, etc.

The input device 150 is a device that accepts input from the outside. A specific example of the input device 150 is a touch panel that is integrated with the display panel 130. The input device 150 may include a plurality of operators that can be operated by the user. The input device 150 outputs input information according to the operation of the user, i.e., the guardian U2, to the processing device 110. In this embodiment, the guardian U2 can specify an object to be determined as to whether or not it is a growth element for the conversation target U1 by operating the input device 150. The processing device 110 transmits information indicating the object specified by the guardian U2 and the first identification information to the device indicated by the second identification information, i.e., the conversation analysis device 40, using the communication device 140. Hereinafter, the information indicating the object specified by the guardian U2 is referred to as determination target information.

(A-3: Robot)
4 is a block diagram showing an example of the configuration of the robot 30. The robot 30 includes a processing device 310, a storage device 320, a motor 330, a communication device 340, a human sensor 350, a microphone 360, and a speaker 370. The elements of the robot 30 are connected to each other by one or more buses for communicating information.

The processing device 310 is a processor that controls the entire robot 30. The processing device 310 is composed of one or more chips. The processing device 310 functions as the control center of the robot 30 by operating according to the program PR3 stored in the memory device 320.

The storage device 320 is a recording medium that can be read by the processing device 310. The storage device 320 stores a number of programs including the program PR3, and various information used by the processing device 310. The storage device 320 may be configured, for example, with at least one of a ROM, an EPROM, an EEPROM, and a RAM.

The motor 330 operates under the control of the processing device 310. By driving the motor 330, the head 31 or the arm of the robot 30 shown in FIG. 2 moves. The communication device 340 is hardware (transmitting/receiving device) for communicating with other devices. The communication device 340 communicates with the conversation analysis device 40. The human presence sensor 350 detects the approach of a person and outputs a detection signal indicating the detection result to the processing device 310. When the detection signal is input from the human presence sensor 350, the processing device 310 switches the operation mode to the conversation mode and transmits a start signal indicating the start of operation in the conversation mode to the conversation analysis device 40. After switching the operation mode to the conversation mode, if no detection signal is input from the human presence sensor 350 for a predetermined time, the processing device 310 switches the operation mode from the conversation mode to the sleep mode. In addition, when the processing device 310 switches the operation mode from the conversation mode to the sleep mode, it transmits an end signal indicating the end of operation in the conversation mode to the conversation analysis device 40.

The microphone 360 converts sound into an electrical signal to generate an audio signal representing the voice of the conversation target U1. The audio signal generated by the microphone 360 is converted into audio data by an AD converter (not shown). This audio data is output to the processing device 310. The processing device 310 transmits the audio data to the conversation analysis device 40 via the communication device 340.

Under the control of the processing device 310, the speaker 370 outputs the voice of the robot 30 to the conversation target U1. The speaker 370 is equipped with a DA converter. Although details will be described later, in this embodiment, voice data representing the voice of the robot in a conversation with the conversation target U1 is generated by the conversation analysis device 40. The voice data generated by the conversation analysis device 40 is transmitted to the robot 30 via the communication network NW. The processing device 310 outputs the voice data received via the communication network NW to the speaker 370. The speaker 370 converts the voice data output from the processing device 310 into a voice signal using a DA converter. The speaker 370 is driven by this voice signal.

(A-4: Conversation analysis device)
5 is a block diagram showing an example of the configuration of the conversation analysis device 40. The conversation analysis device 40 is, for example, a server. The conversation analysis device 40 includes a processing device 410, a storage device 420, a display panel 430, a communication device 440, and an input device 450. The elements of the conversation analysis device 40 are connected to each other by a single or multiple buses for communicating information. The hardware configuration of the conversation analysis device 40 is similar to the hardware configuration of the user device 10. However, it is preferable that the processing speed of the processing device 410 is faster than the processing speed of the processing device 110. In addition, it is preferable that the storage capacity of the storage device 420 is larger than the storage capacity of the storage device 120.

The processing device 410 functions as the control center of the conversation analysis device 40 by operating according to the program PR4. The storage device 420 stores a plurality of programs including the program PR4, a topic table TBL1, a word table TBL2, and various information used by the processing device 410. A specific example of the various information used by the processing device 410 is a pair of the first identification information and the judgment target information received from the user device 10.

FIG. 6 is a diagram showing an example of the topic table TBL1. As shown in FIG. 6, the topic table stores a plurality of pairs of theme data and story data. The theme data is text data representing an object to be determined as a growth element. As shown in FIG. 6, a specific example of theme data is text data representing character strings such as "kindness" and "politeness". Story data is text data representing a story related to an object represented by the theme data associated with the story data. As shown in FIG. 6, in the topic table TBL1 of this embodiment, the theme data representing "kindness" is associated with story data A. Also, in this embodiment, the theme data representing "politeness" is associated with story data B. The topic table TBL1 shown in FIG. 6 stores two types of theme data, "kindness" and "politeness". However, instead of or in addition to "kindness" and "politeness", theme data representing "friendship" and "courage", etc. and story data corresponding to the theme data may be stored in the topic table TBL1.

In this embodiment, story data A represents a story about kindness, for example, "Someone walking in front of me dropped it. When I picked it up..."

In addition, story data B in this embodiment represents a story about courtesy, for example, "He greeted me loudly saying 'Hello,' so I replied 'Hello,' and..."

Figure 7 is a diagram showing an example of the word table TBL2. As shown in Figure 7, the word table TBL2 stores a plurality of pairs of word data and emotion data. The word data is text data representing words assumed to be used in the response of the conversation target U1 in a conversation with the robot 30. For example, the word table TBL2 shown in Figure 7 stores word data representing each of the words "fun", "kind", "boring", and "sad". The emotion data is text data representing emotions assumed to be included in a response including a word represented by the word data associated with the emotion data. In this embodiment, the word data "fun" is associated with emotion data indicating "joy". In addition, in this embodiment, the word data "kind" is associated with emotion data indicating "slight joy". In this embodiment, the word data "boring" is associated with emotion data indicating "anger". In this embodiment, the word data "sad" is associated with emotion data indicating "sadness". The details will be described later, but the word table TBL2 is used to estimate the emotion implied in the reply of the conversation target U1 in a conversation with the robot 30.

Figure 8 is a block diagram showing the functions realized by the processing device 410 in accordance with program PR4. The processing device 410 reads out and executes program PR4 from the storage device 420, thereby functioning as a presentation unit 411, a voice control unit 412, a second determination unit 413, a first estimation unit 414, a second estimation unit 415, a first determination unit 416, and a third estimation unit 417.

Prior to the start of a conversation between the robot 30 and the conversation target U1, the presentation unit 411 presents to the conversation target U1 a story related to a subject to be determined as to whether it is a growth element. When the presentation unit 411 receives a start signal SS from the robot 30 via the communication network NW, it reads out the subject information stored in the storage device 420 in association with the first identification information corresponding to the sender of the start signal SS. Next, the presentation unit 411 reads out story data stored in the topic table TBL1 in association with theme data indicating the same subject as the read out subject information. For example, if the subject to be determined is "kindness", the presentation unit 411 reads out story data A from the topic table TBL1.

Next, the presentation unit 411 synthesizes a voice saying "Will you listen to my story today?" and voice data DS representing the voice reading out the story represented by the read story data. The presentation unit 411 transmits the synthesized voice data DS to the robot 30. FIG. 9 is a diagram showing an example of a conversation between the robot 30 and the conversation target U1. The robot 30 receives the voice data DS and outputs a voice Q11 saying "Will you listen to my story today?" as shown in FIG. 9, and then outputs a voice Q12 saying "The person walking in front of me...". The story represented by the story data A is presented to the conversation target U1 by outputting the voice Q12 reading out the story represented by the story data A from the robot 30.

After the presentation of the story by the presentation unit 411, the voice control unit 412 causes the robot 30 to carry out a conversation with the conversation target U1 until an end signal SE is received from the robot 30. The voice control unit 412 generates voice data Dv2 representing the voice of a question from the robot 30 in the conversation with the conversation target U1. The voice control unit 412 transmits the generated voice data Dv2 to the robot 30 via the communication network NW. The robot 30 that receives the voice data Dv2 outputs the voice represented by the voice data Dv2. For example, when the voice data Dv2 representing the voice Q13 of "What did you think when you heard this story?" is transmitted from the conversation analysis device 40 to the robot 30, the robot 30 outputs the voice Q13 as shown in FIG. 9. The voice Q13 may be a voice of "Did you feel warm?" The voice control unit 412 also acquires the voice data Dv1 representing the voice of the conversation target U1's response to the question from the robot 30 from the robot 30. For example, in response to a question by voice Q13, as shown in FIG. 9, if a response A11 of "I thought it was kind" is given, the voice control unit 412 acquires voice data Dv1 representing the response A11 from the robot 30.

The second determination unit 413 determines that a conversation has not been established if the voice control unit 412 does not acquire voice data Dv1 representing a response within a predetermined time, such as 30 seconds, from the time when the voice data Dv2 is transmitted to the robot 30. This is because not responding to a question is an indication that the conversation target U1 has no intention of conversing with the robot 30.

Even if the voice control unit 412 acquires voice data Dv1 representing a response within a predetermined time from the time when the voice data Dv2 is transmitted to the robot 30, the second determination unit 413 may determine that a conversation has not been established if the response represented by the voice data Dv1 is a topic-changing response. Specific examples of topic-changing responses include a response that explicitly encourages a change of topic, such as "Let's talk about something else instead," or a response regarding a thing unrelated to the presented story, such as "XX is interesting, isn't it?" (XX is a thing unrelated to the story presented by the presentation unit 411). This is because changing the topic is an indication that the conversation target U1 does not have the intention to converse with the robot 30 regarding a thing that is to be determined as being a growth element or not.

The first estimation unit 414 estimates the emotion contained in the response represented by the voice data Dv1 acquired by the voice control unit 412 based on the tone of voice represented by the voice data Dv1. The tone of voice refers to the change in pitch over time in the voice, the change in sound intensity over time in the voice, and the change in sound length over time in the voice. The first estimation unit 414 generates tone data representing the tone of the voice represented by the voice data Dv1 by analyzing the voice data Dv1 acquired by the voice control unit 412. Specific examples of tone data include data representing 12 dimensions of MFCC (Mel-Frequency Cepstrum Coefficients), loudness, fundamental frequency (F0), voice probability, zero-crossing rate, HNR (Harmonics-to-Noise-Ratio), their first derivatives, and part or all of the second derivatives of MFCC and loudness.

Next, the first estimation unit 414 generates a first emotion vector V1 based on the tone of voice data generated from the voice data Dv1, the first emotion vector V1 having as its components the evaluation values of the emotions "happiness," "anger," "sorrow," and "happiness" contained in the reply voice represented by the voice data Dv1. Voice generally contains the emotions of the person who uttered the voice. The first emotion vector V1 is a four-dimensional vector that represents the strength of each emotion of "happiness," "anger," "sorrow," and "happiness" contained in the reply voice represented by the voice data Dv1. As an algorithm for estimating the emotions contained in the voice based on the tone of voice, an existing tone of voice analysis algorithm may be used as appropriate.

The second estimation unit 415 estimates the emotion contained in the response based on the wording of the response represented by the voice data Dv1. The second estimation unit 415 performs voice recognition on the voice data Dv1 to generate text data representing the response sentence represented by the voice data Dv1. Next, the second estimation unit 415 performs morphological analysis or the like on the sentence represented by the text data generated from the voice data Dv1 to extract words that constitute the document. Then, the second estimation unit 415 identifies the corresponding emotion for each word extracted from the sentence represented by the text data by referring to the word table TBL2. Then, the second estimation unit 415 generates a second emotion vector V2 representing the emotion contained in the response represented by the voice data Dv1 based on the emotion identified for each word extracted from the sentence represented by the text data.

The person who produces the voice generally selects the words that compose the voice in accordance with the emotions they are feeling at the time. The second emotion vector V2 is a four-dimensional vector that represents the intensity of each emotion of "joy," "anger," "sorrow," and "happiness" contained in the reply words represented by the voice data Dv1. As an algorithm for inferring emotions from the words in the voice, an existing emotion analysis algorithm may be used as appropriate.

The first determination unit 416 determines whether or not a conversation between the robot 30 and the conversation target U1 has been established based on the degree of agreement between the emotion estimated from the tone of voice of the reply indicated by the voice data Dv1 and the emotion estimated from the wording of the reply indicated by the voice data Dv1. The reason why it is possible to determine whether or not a conversation between the robot 30 and the conversation target U1 has been established based on the emotion estimated from the tone of voice of the reply of the conversation target U1 and the emotion estimated from the wording of the reply of the conversation target U1 is as follows.

If the conversation target U1 is responding sincerely to the question from the robot 30, i.e., if a conversation between the robot 30 and the conversation target U1 has been established, there should be no discrepancy between the emotion inferred from the tone of voice of the response and the emotion inferred from the wording of the response, i.e., the degree of agreement between the two should be high. For example, suppose that in response to a question from voice Q13, as shown in FIG. 9, a response A11 of "I thought it was kind," is returned, and the emotion inferred from the tone of voice of response A11 is "joy." In this case, the conversation target U1 is responding to the question from voice Q13 with a genuine emotion of "joy," and it is considered that a conversation between the robot 30 and the conversation target U1 has been established.

On the other hand, if the emotion estimated from the tone of the response does not match highly with the emotion estimated from the wording of the response, it is considered that the conversation target U1 is responding playfully or giving a casual response to the question from the robot 30. For example, in response to a question by voice Q13, as shown in FIG. 9, the response A11 of "I thought it was kind" is returned, and the emotion estimated from the tone of the response A11 is "cold" or "emotionless". In this case, it is considered that the conversation target U1 is responding playfully to the question from the robot 30. Also, if a sad story is presented by the presentation unit 411 and the response is "sad", but the tone of the response contains the emotion of "fun", it is considered that the conversation target U1 is responding playfully. If the conversation target U1 is responding playfully or giving a casual response to the question from the robot 30, it cannot be said that a conversation between the robot 30 and the conversation target U1 is established. This is why it is possible to determine whether or not a conversation between the robot 30 and the conversation target U1 has been established based on the degree of agreement between the emotion inferred from the tone of voice of the conversation target U1's reply and the emotion inferred from the wording of the conversation target U1's reply.

In this embodiment, the first determination unit 416 calculates the angle θ (0≦θ≦180°) between the first emotion vector V1 and the second emotion vector V2 as a value indicating the degree of agreement between the emotion estimated from the tone of voice of the response and the emotion estimated from the wording of the response. The angle θ is calculated as the arccosine of the inner product of the first emotion vector V1 and the second emotion vector V2 divided by the product of the norm of the first emotion vector V1 and the norm of the second emotion vector V2. If the angle θ is less than a predetermined threshold (e.g., 30°), the first determination unit 416 determines that the conversation between the robot 30 and the conversation target U1 is established. Conversely, if the angle θ is equal to or greater than the threshold, the first determination unit 416 determines that the conversation between the robot 30 and the conversation target U1 is not established. In this embodiment, the angle between the first emotion vector V1 and the second emotion vector V2 is used as a value indicating the degree of agreement between the emotion estimated from the tone of voice of the reply and the emotion estimated from the wording of the reply, but the cosine value of the angle may be used. The cosine value of the angle between the first emotion vector V1 and the second emotion vector V2 is a value obtained by dividing the inner product of the first emotion vector V1 and the second emotion vector V2 by the product of the norm of the first emotion vector V1 and the norm of the second emotion vector V2. In an aspect in which the cosine value of the angle between the first emotion vector V1 and the second emotion vector V2 is used as a value indicating the degree of agreement, the first determination unit 416 may determine that a conversation between the robot 30 and the conversation target U1 has been established when the value is equal to or greater than a predetermined threshold value (e.g., 0.6). In addition, the distance between the end point of the first emotion vector V1 and the end point of the second emotion vector V2 may be used as a value indicating the degree of agreement between the emotion estimated from the tone of voice of the reply and the emotion estimated from the wording of the reply.

When the first determination unit 416 determines that a conversation has been established, the third estimation unit 417 estimates a degree of empathy indicating the strength of empathy of the conversation target U1 toward the thing indicated by the determination target information based on the first emotion vector V1 and the second emotion vector V2. In this embodiment, the third estimation unit 417 calculates the degree of empathy by weighting and adding the norm of the first emotion vector V1 and the norm of the second emotion vector V2, but the degree of empathy may also be the inner product of the first emotion vector V1 and the second emotion vector V2. The third estimation unit 417 may determine whether the emotion of the conversation target U1 toward the thing indicated by the determination target thing is a positive emotion or a negative emotion based on the first emotion vector V1 and the second emotion vector V2, and calculate the degree of empathy according to the degree of positivity.

If the calculated degree of empathy is less than a predetermined threshold, the third estimation unit 417 stores the judgment target information in the storage device 420 as growth element information indicating growth elements that are expected to further grow the conversation target U1. Note that, if the calculated degree of empathy is equal to or greater than a predetermined threshold, the third estimation unit 417 may store the judgment target information in the storage device 420 as strength information indicating the strengths of the conversation target U1. Furthermore, the third estimation unit 417 may store the degree of empathy in the storage device 420 in association with the growth element information or strength information.

The processing device 410 also executes the conversation analysis method of the present disclosure by executing program PR4. FIG. 10 is a flowchart showing the flow of the conversation analysis method executed by the processing device 410 in accordance with program PR4. As shown in FIG. 10, this conversation analysis method includes the processes of steps SA110 to SA150.

In step SA110, the processing device 410 functions as a presentation unit 411. In step SA110, the processing device 410, upon receiving a start signal SS, causes the robot 30 to output audio of a story about an object to be determined as to whether it is a growth element or not.

In step SA120, which follows step SA110, the processing device 410 functions as a voice control unit 412. In step SA120, the processing device 410 determines whether or not an end signal SE has been received from the robot 30. If an end signal SE has been received from the robot 30, the determination result of step SA120 is "Yes". If the determination result of step SA120 is "Yes", the processing device 410 ends the execution of this conversation analysis method. Conversely, if the determination result of step SA120 is "No", the processing device 410 executes the processes from step SA130 onwards.

In step SA130, the processing device 410 functions as the voice control unit 412. In step SA130, the processing device 410 causes the robot 30 to carry out a conversation with the conversation target U1. That is, in step SA130, the processing device 410 causes the robot 30 to output a voice question to the conversation target U1, while acquiring from the robot 30 voice data representing the voice of the response from the conversation target U1.

In step SA140 following step SA130, the processing device 410 functions as the second determination unit 413, the first estimation unit 414, the second estimation unit 415, and the first determination unit 416. In step SA140, the processing device 410 first functions as the second determination unit 413. If the voice data Dv1 representing a reply is not acquired within a predetermined time from the time when the voice data Dv2 representing the voice of the question is transmitted to the robot 30, the processing device 410 determines that a conversation between the robot 30 and the conversation target U1 has not been established. If the voice data Dv1 representing a reply is not acquired within a predetermined time from the time when the voice data Dv2 representing the voice of the question is transmitted to the robot 30, the determination result of step SA130 becomes "No".

When the processing device 410 acquires the voice data Dv1 representing the reply within a predetermined time from the time when the voice data Dv2 representing the voice of the question is transmitted to the robot 30, the processing device 410 functions as the first estimation unit 414, the second estimation unit 415, and the first determination unit 416. The processing device 410 calculates the above-mentioned first emotion vector V1 and the second emotion vector V2 based on the acquired voice data Dv1. Then, the processing device 410 determines whether or not a conversation between the robot 30 and the conversation target U1 has been established based on the degree of agreement between the emotion represented by the first emotion vector V1 and the emotion represented by the second emotion vector V2. If it is determined that a conversation between the robot 30 and the conversation target U1 has been established, the determination result of step SA140 is "Yes". Conversely, if it is determined that a conversation between the robot 30 and the conversation target U1 has not been established, the determination result of step SA140 is "No".

If the determination result of step SA140 is "Yes", the processing device 410 executes the process of step SA150 and then executes the process of step SA110 again. On the other hand, if the determination result of step SA140 is "No", the processing device 410 executes the process of step SA110 again without executing the process of step SA150.

In step SA150, the processing device 410 functions as a third estimation unit 417. In step SA150, the processing device 410 calculates a degree of empathy indicating the strength of empathy of the conversation target U1 toward the thing indicated by the judgment target information, based on the first emotion vector V1 and the second emotion vector V2. If the calculated degree of empathy is less than a predetermined threshold, the processing device 410 stores the judgment target information in the storage device 420 as growth element information.

As described above, the conversation analysis device 40 determines whether or not a conversation with the robot 30 has been established based on the degree of agreement between the emotion inferred from the tone of voice of the conversation target U1's reply in the conversation with the robot 30 and the emotion inferred from the wording of the reply, thereby reducing the misjudgment that a conversation has been established when the conversation target U1 is not responding sincerely. In addition, the conversation analysis device 40 can estimate the strength of empathy of the conversation target U1 toward the thing indicated by the judgment target information based on the reply when it is determined that a conversation with the robot 30 has been established, thereby reducing the misjudgment of growth factors.

(B: Modified Example)
The present disclosure is not limited to the above-described exemplary embodiments. Specific modified aspects are as follows. Two or more aspects selected from the following examples may be combined.
(B-1: Modification 1)
When the degree of empathy of the conversation target U1 for any of the plurality of things is less than a threshold, the processing device 410 may determine that the conversation target U1 is not good at expressing emotions. The processing device 410 may generate a vocabulary database by writing the vocabulary used by the conversation target U1 in the reply and the emotion estimated from the tone of the reply in the storage device 420 in association with each other. For example, it is assumed that the vocabulary "dangerous" is commonly associated with both "fun" and "fear" in the vocabulary database. In this way, when the same vocabulary is used for a plurality of emotions, a vocabulary shortage (or a lack of expressiveness) is estimated for the conversation target U1. In the aspect of generating a vocabulary database, the processing device 410 may estimate the presence or absence of a vocabulary shortage (or a lack of expressiveness) for the conversation target U1 from the contents stored in the vocabulary database. Then, when it is estimated that the conversation target U1 has a lack of vocabulary (or a lack of expressiveness), the processing device 410 may refer to the stored contents of the word table TBL2 and suggest to the conversation target U1 rephrasing the word with other words. For example, when expressing "fun," it may be suggested to use "tanoshii" (fun) or "exciting" (exciting), etc., instead of "yabai" (awesome).

In addition, in the aspect of generating a vocabulary database, the processing device 410 may store at least one of information indicating the age of the conversation target U1, information indicating the area in which the conversation target U1 lives, and information indicating the developmental status of the conversation target U1 in the vocabulary database in association with the vocabulary used by the conversation target U1 in the reply and the emotion estimated from the reply. In the vocabulary database generated by this aspect, the usage of vocabulary specific to the age, residential area, or developmental status of the conversation target U1 (or the distribution of vocabulary) is databased. By referring to this vocabulary database, when creating a commercial or the like targeting a conversation target U1 of a specific age or a conversation target U1 living in a specific area, it is possible to select vocabulary that is likely to gain sympathy and create the commercial or the like. In addition, by using this vocabulary database, it is possible to estimate the age, residential area, or developmental status of the conversation target U1 based on the vocabulary used by the conversation target U1 in a conversation about a subject to be determined as being a growth element or not, that is, to profile the conversation target U1.

(B-2: Modification 2)
The presentation unit 411, the second determination unit 413, and the third estimation unit 417 are not essential and may be omitted. In short, the conversation analysis device of the present disclosure may include the voice control unit 412, the first estimation unit 414, the second estimation unit 415, and the first determination unit 416. The voice control unit 412 acquires voice data Dv1 representing the voice of the response of the conversation target U1 in the conversation between the robot 30 and the conversation target U1 user. The first estimation unit 414 estimates the emotion contained in the response indicated by the voice data Dv1 based on the tone of the voice represented by the voice data Dv1. The second estimation unit 415 estimates the emotion contained in the response indicated by the voice data Dv1 based on the wording of the response represented by the voice data Dv1. The first determination unit 416 determines whether or not a conversation between the robot 30 and the conversation target U1 is established based on the degree of agreement indicating the degree of agreement between the emotion estimated by the first estimation unit 414 and the emotion estimated by the second estimation unit 415. The conversation analysis device of this aspect can reduce the erroneous determination that a conversation has been established when the conversation target U1 is not responding sincerely.

(B-3: Modification 3)
In the above embodiment, the conversation analysis device 40 and the robot 30 are separate devices, but the conversation analysis device 40 may be included in the robot 30. Furthermore, the program PR4 in the above embodiment may be manufactured or sold separately.

(C: Other)
(1) In the above-described embodiment, the storage devices 120, 320, and 420 are exemplified by ROM and RAM, but the storage devices may be flexible disks, magneto-optical disks (e.g., compact disks, digital versatile disks, Blu-ray (registered trademark) disks), smart cards, flash memory devices (e.g., cards, sticks, key drives), CD-ROMs (Compact Disc-ROMs), registers, removable disks, hard disks, floppy (registered trademark) disks, magnetic strips, databases, servers, or other suitable storage media. The program may also be transmitted from a network via a telecommunications line. The program may also be transmitted from a communications network via a telecommunications line.

(2) In the above-described embodiments, the information, signals, etc. described may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

(3) In the above-described embodiment, the input and output information, etc. may be stored in a specific location (e.g., memory) or may be managed using a management table. The input and output information, etc. may be overwritten, updated, or added to. The output information, etc. may be deleted. The input information, etc. may be transmitted to another device.

(4) In the above-described embodiment, the determination may be made based on a value represented by one bit (0 or 1), a Boolean value (true or false), or a comparison of numerical values (e.g., a comparison with a predetermined value).

(5) The order of the process procedures, sequences, flow charts, etc. illustrated in the above-described embodiments may be changed as long as it is not inconsistent. For example, the methods described in this disclosure present elements of various steps using an exemplary order and are not limited to the particular order presented.

(6) Each function illustrated in FIG. 8 is realized by any combination of at least one of hardware and software. Furthermore, there are no particular limitations on the method of realizing each functional block. That is, each functional block may be realized using one device that is physically or logically coupled, or may be realized using two or more devices that are physically or logically separated and directly or indirectly connected (e.g., using wires, wirelessly, etc.). A functional block may be realized by combining software with the one device or the multiple devices.

(7) The programs exemplified in the above-described embodiments should be broadly construed to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., regardless of whether software is called software, firmware, middleware, microcode, hardware description language, or by other names.

Software, instructions, information, etc. may also be transmitted and received via a transmission medium. For example, if the software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and/or wireless technologies (such as infrared, microwave), then these wired and/or wireless technologies are included within the definition of a transmission medium.

(8) In each of the above embodiments, the terms "system" and "network" are used interchangeably.

(9) The information, parameters, etc. described in this disclosure may be expressed using absolute values, may be expressed using relative values from a predetermined value, or may be expressed using other corresponding information.

(10) In the above-described embodiments, the user device 10 may be a mobile station (MS). A mobile station may also be referred to by those skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable term. In this disclosure, the terms "mobile station", "user terminal", "user equipment (UE)", "terminal", etc. may be used interchangeably.

(11) In the above-mentioned embodiments, the terms "connected" and "coupled" or any variation thereof refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "access". As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and light (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

(12) In the above embodiments, the term "based on" does not mean "based only on," unless otherwise specified. In other words, the term "based on" means both "based only on" and "based at least on."

(13) The terms "determining" and "determining" as used in this disclosure may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, search, inquiry (e.g., searching in a table, database, or other data structure), ascertaining something as "judging" or "determining", etc. Also, "determining" and "determining" may include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), etc. Additionally, "judgment" and "decision" can include considering resolving, selecting, choosing, establishing, comparing, etc., to have been "judged" or "decided." In other words, "judgment" and "decision" can include considering some action to have been "judged" or "decided." Additionally, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

(14) In the above embodiments, when the terms "include," "including," and variations thereof are used, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or" as used in this disclosure is not intended to be an exclusive or.

(15) In this disclosure, where articles have been added by translation, such as a, an, and the in English, this disclosure may include that the nouns following these articles are plural.

(16) In this disclosure, the term "A and B are different" may mean "A and B are different from each other." In addition, the term may mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."

(17) Each aspect/embodiment described in this disclosure may be used alone, in combination, or switched depending on the execution. In addition, notification of specific information (e.g., notification that "X is the case") is not limited to being done explicitly, but may be done implicitly (e.g., not notifying the specific information).

(D: Aspects understood from the above-mentioned embodiments or modifications)
Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described in the present disclosure. The present disclosure can be implemented as modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure. The following aspects can be understood from at least one of the above-mentioned embodiments or modifications.

The conversation analysis device according to the first aspect includes a voice control unit, a first estimation unit, a second estimation unit, and a first determination unit. The voice control unit acquires voice data representing the voice of the user's reply in a conversation between the robot and the user. The first estimation unit estimates the emotion contained in the user's reply based on the tone of voice of the reply, i.e., the tone of voice represented by the voice data acquired by the voice control unit. The second estimation unit estimates the emotion contained in the user's reply based on the wording of the reply, i.e., the wording represented by the voice data acquired by the voice control unit. The first determination unit determines whether or not a conversation between the user and the robot has been established based on a degree of agreement indicating the degree of agreement between the emotion estimated by the first estimation unit and the emotion estimated by the second estimation unit. The conversation analysis device according to the first aspect determines whether or not a conversation between the user and the robot has been established based on the degree of agreement between the emotion estimated from the tone of voice of the user's reply to the robot and the emotion estimated from the wording of the reply, thereby reducing erroneous determinations when the user is not responding sincerely.

The conversation analysis device in the example of the first aspect (second aspect) may further include a second determination unit that determines that a conversation has not been established when there is no response to a question posed to the user, or when the response changes the topic. The conversation analysis device according to the second aspect can determine that a conversation between the user and the robot has not been established when there is no response to a question posed to the user, or when the response changes the topic.

In an example of the first or second aspect (third aspect), the topic in the conversation between the robot and the user may be a matter for which the strength of the user's empathy is to be determined. The conversation analysis device in the third aspect may further include a third estimation unit as described below. When the first determination unit determines that a conversation is established, the third estimation unit estimates a degree of empathy indicating the strength of the user's empathy for the matter for which the strength of the user's empathy is to be determined, based on the emotion estimated by the first estimation unit and the emotion estimated by the second estimation unit. The conversation analysis device of the third aspect can estimate the strength of the user's empathy for the matter that was the topic of the conversation between the robot and the user.

The conversation analysis device in the example of the third aspect (fourth aspect) may further include a presentation unit that presents to the user, prior to the start of a conversation between the robot and the user, a story related to the subject that is the topic of the conversation between the robot and the user, i.e., the subject that is the subject of the determination of the strength of the user's empathy. The conversation analysis device of the fourth aspect can present to the user, prior to the conversation, a story related to the subject that is the topic of the conversation between the robot and the user, i.e., the subject that is the subject of the determination of the strength of the user's empathy.

1...conversation system, 10...user device, 30...robot, 40...conversation analysis device, 411...presentation unit, 412...voice control unit, 413...second determination unit, 414...first estimation unit, 415...second estimation unit, 416...first determination unit, 417...third estimation unit.

Claims (4)

a voice control unit that acquires voice data representing a response voice of the user in a conversation between the robot and the user;
a first estimation unit that estimates an emotion contained in the reply based on a tone of voice represented by the voice data;
a second estimation unit that estimates an emotion contained in the reply based on a phrase expressed by the voice data;
a first determination unit that determines whether the conversation has been established based on a degree of agreement between the emotion estimated by the first estimation unit and the emotion estimated by the second estimation unit;
A conversation analysis device comprising: The conversation analysis device according to claim 1, further comprising a second determination unit that determines that the conversation has not been established if there is no response to a question posed to the user, or if the response changes the topic. The topic in the conversation is a subject for determining the strength of empathy of the user,
3. The conversation analysis device according to claim 1, further comprising: a third estimation unit that, when the first determination unit determines that the conversation is established, estimates a degree of empathy indicating a strength of the user's empathy for the thing based on the emotion estimated by the first estimation unit and the emotion estimated by the second estimation unit. The conversation analysis device according to claim 3, further comprising a presentation unit that presents a story related to the subject to the user prior to the start of the conversation.