JP7616290B2 - Robot, response method and program - Google Patents

Description

The present invention relates to a robot , a response method , and a program.

A robot that operates autonomously and converses with humans has been proposed. For example, Patent Document 1 describes a natural language processing device that enables appropriate dialogue with humans. The natural language processing device of Patent Document 1 includes a sequential analysis processing unit that executes analysis processing sequentially and in parallel in each analysis processing unit each time a portion of a natural language sentence in an analyzable unit is input, and an output unit that obtains output such as a dialogue response sentence based on the analysis results in each analysis processing unit of the sequential analysis processing unit. Each processing unit prepared in the sequential analysis processing unit obtains the previous analysis result immediately before or before that in its own processing unit and the previous analysis result immediately before or before that in other processing units, and obtains the analysis result while looking ahead and referring to the obtained analysis result.

JP 2017-102771 A

Generally, robots that converse with humans generate and speak a response sentence only after they have finished listening to what the person is saying, so if the robot is in the middle of an utterance it does not respond, and the speaker feels as if the robot is not listening at all. In the natural language processing device of Patent Document 1, each time a portion of a sentence is input, each analysis processing unit performs analysis processing sequentially and in parallel in an attempt to return a response as quickly as possible.

However, when this natural language processing device is applied to a robot configured to be able to interact with a user, the analysis process is performed using the results predicted by a look-ahead process that reads ahead the next input data, and there is a possibility that the results of this analysis process will be incorrect. If the analysis result is incorrect, an inappropriate response sentence will be generated in response to the sentence input by the user, and ultimately, it will be impossible to respond appropriately to the user's utterance.

The present invention has been made in consideration of the above-mentioned circumstances, and has an object to make it possible to make a response that can prevent the speaker from feeling that the robot is not listening at all, even when a response in a response sentence is suppressed .

In order to achieve the above object, the robot of the present invention is a robot capable of responding by voice and by gesture, and includes a first identification means for starting a predetermined identification process in response to a start of an utterance by a speaker to identify a category of a content that the speaker is trying to convey through the utterance, and a control means for controlling the robot to execute a gesture that is preset in association with the identified category as soon as the category is identified by the identification process regardless of whether the speaker has finished the utterance or not, and the control means is configured to control the robot to be silent until the speaker finishes the utterance by controlling the robot to wait for the speaker to finish the utterance before executing the voice response, even in a case where the control means controls the robot to execute the gesture response before the speaker finishes the utterance , and to control the robot to execute a gesture that differs for each category.
Further, a response method according to the present invention is a response method executed by a robot capable of executing voice responses and gesture responses, comprising: a specifying step of starting a predetermined specification process in response to a start of an utterance by a speaker to specify a category of a content that the speaker is trying to convey through the utterance; and a control step of controlling the robot to execute a gesture that is preset in association with the specified category as soon as the category is specified by the specification process regardless of whether the utterance by the speaker has ended or not, wherein the control step is configured to control the robot to execute a gesture that is preset in association with the specified category after waiting for the utterance by the speaker to end, even in a case where the robot is controlled to execute the gesture response before the utterance by the speaker ends, so that the robot is controlled to be silent until the utterance by the speaker ends, and the robot is controlled to execute a gesture that differs for each category.
The program according to the present invention causes a computer of a robot capable of responding by voice and by gesture to function as: an identification means which starts a predetermined identification process in response to the start of an utterance by a speaker to identify a category of the content that the speaker is trying to convey through the utterance; and a control means which controls the robot to execute a gesture that is preset in association with the identified category as soon as the category is identified by the identification process regardless of whether the speaker has finished the utterance or not. The control means is configured to control the robot to be silent until the speaker finishes the utterance by controlling the robot to wait for the speaker to finish the utterance before executing the voice response, even in a case where the control means controls the robot to execute the gesture response before the speaker finishes the utterance, and to control the robot to execute a gesture that differs for each category.

According to the present invention, even when responses in response sentences are suppressed, a response can be made that can prevent the speaker from feeling that the robot is not listening at all.

1 is a diagram showing a schematic configuration of a robot to which a control device according to an embodiment of the present invention is applied; 2 is a block diagram showing a functional configuration of a control device of a robot according to an embodiment. FIG. FIG. 13 is a front view showing the motion of gesture number 2 that the control device in the embodiment causes the robot to perform. FIG. 13 is a front view showing the motion of gesture number 3 that the control device in the embodiment causes the robot to perform. FIG. 13 is a front view showing the motion of gesture number 4 that the control device in the embodiment causes the robot to perform. FIG. 13 is a front view showing the motion of gesture number 5 that the control device in the embodiment causes the robot to perform. FIG. 13 is a front view showing the gesture of gesture number 6 that the control device in the embodiment causes the robot to perform. 5 is a flowchart showing a conversation recording process of the control device according to the embodiment. 5 is a diagram showing an example of a conversation record recorded in the conversation recording process shown in FIG. 4. 4 is a flowchart showing an analytical learning process of the control device according to the embodiment. FIG. 13 is a diagram showing an example in which all spoken sentences corresponding to a predetermined target are read out from a conversation record. FIG. 13 is a diagram illustrating an example of a unique phoneme sequence table. FIG. 13 is a diagram showing an example of correspondence between sentences and gestures. 10 is a flowchart showing a response gesture database registration process of the control device according to the embodiment. 11 is a timing chart showing an example of the time when a speech sentence is uttered. 11 is a diagram showing an example of a response gesture database used in the response gesture database registration process shown in FIG. 10 . 4 is a flowchart showing a predicted response control process of the control device according to the embodiment. FIG. 13 is a diagram illustrating an example of a response time list. 4 is a flowchart showing a linguistic response control process of the control device according to the embodiment. 10 is a timing chart showing an example of a response of a robot according to an embodiment.

The following describes an embodiment of the present invention with reference to the drawings. Note that the same or corresponding parts in the drawings are given the same reference numerals.

Embodiment
1 is a diagram showing a schematic configuration of a robot 1 to which a control device 2 according to an embodiment of the present invention is applied. The robot 1 has a three-dimensional shape that resembles a human (child) in appearance. The robot 1 includes a head 101, a body 102, and arms 103. The head 101 and the arms 103 are parts that can be moved by a gesture actuator 7, which is a drive device built into the robot 1. The head 101 is attached to the body 102 by a neck joint 5 so as to be able to bend, extend, rotate, and flex laterally. The arms 103 are attached to the body 102 by a shoulder joint 6 so as to be able to bend, extend, and adduce and abduct.

The robot 1 is equipped with a microphone 3 for picking up sound, a speaker 4 for outputting sound, a gesture actuator 7 for moving the head 101 and arms 103, and a control device 2. This robot 1 can converse with a person by performing voice recognition of an utterance from a predetermined target, generating a response sentence to the utterance, and speaking the response sentence through voice synthesis. When conversing with a predetermined target, the robot 1 can also respond with non-verbal behavior, that is, by moving the head 101 and arms 103. In the robot 1, multiple different gesture actions are set as such non-verbal behavior, and these multiple gesture actions include actions such as nodding and raising and lowering the arms.

Robot 1 performs various actions in response to external stimuli, such as a call or contact from a specific object that exists outside the device itself. This allows robot 1 to communicate with and interact with the specific object. A specific object is an object that exists outside robot 1 and that communicates and interacts with robot 1. Examples of specific objects include the user who owns robot 1, people around the user (such as the user's close relatives or friends), and other robots that can speak. A specific object can also be called a communication object, communication partner, exchange object, exchange partner, etc.

Figure 2 is a block diagram showing the functional configuration of the control device 2. The control device 2 is electrically connected to the microphone 3 and speaker 4, and receives a voice signal from the microphone 3 and speaks a response sentence from the speaker 4. The control device 2 also controls the gesture operation unit 7 to cause the robot 1 to execute the above gesture actions. The gesture operation unit 7 includes an actuator, and drives, for example, the head 101 and the arm 103 of the robot 1 shown in Figure 1.

Robot 1 is equipped with sensors (potentiometers) at joints 5 that detect the rotation angles of flexion/extension, rotation, and lateral bending of head 101, and gesture actuator 7 causes head 101 to perform a predetermined movement by feedback control using the detection values of the sensors at joints 5. Similarly, robot 1 is equipped with sensors (potentiometers) at joints 6 that detect the rotation angles of flexion/extension, adduction/abduction of arm 103, and gesture actuator 7 causes arm 103 to perform a predetermined movement by feedback control using the detection values of the sensors at joints 6.

Figures 3A to 3E are front views showing examples of gesture actions to be executed by the robot 1. Figure 3A shows the gesture action of gesture number 2 for responding to the speech of a predetermined target, "Good morning." The gesture action of gesture number 2 is an action in which the head 101 is turned forward (or slightly above the front) by the joints 5 and the gesture actuator 7, and the left and right arms 103 are raised above the shoulders by the joints 6 and the gesture actuator 7.

Figure 3B shows the gesture action of gesture number 3 for responding to the utterance "hello" from a predetermined target. The gesture action of gesture number 3 is an action in which the head 101 is turned forward (or slightly above the front) by the joint 5 and the gesture actuator 7, and the left and right arms 103 are raised in front of the head 101 by the joint 6 and the gesture actuator 7.

Figure 3C shows the gesture action of gesture number 4 for responding to a predetermined target's speech of "Bye-bye," "Goodbye," or "Goodbye." The gesture action of gesture number 4 is an action in which the head 101 is tilted slightly to the left by the joint 5 and the gesture actuator 7, and the right arm 103 is raised close to the head 101 while the left arm 103 is kept down by the joint 6 and the gesture actuator 7. In the gesture action of gesture number 4, the right arm 103 may be raised and swung from side to side.

Figure 3D shows the gesture action of gesture number 5 for responding to the utterance "I'm home" by a predetermined target. The gesture action of gesture number 5 is an action in which the head 101 is turned forward (or slightly above the front) by the joint 5 and the gesture actuator 7, and the left and right arms 103 are raised to shoulder height by the joint 6 and the gesture actuator 7.

Figure 3E shows the gesture action of gesture number 6 for responding to the speech "good night" from a specific target. The gesture action of gesture number 6 is an action of pointing the head 101 downward by the joint 5 and the gesture actuator 7 while keeping the left and right arms 103 downward by the joint 6 and the gesture actuator 7.

As shown in FIG. 2, the control device 2 includes a control unit 20, a storage unit 30, and an I/O interface for inputting and outputting signals to and from the microphone 3, the speaker 4, and the gesture operation unit 7. The control unit 20 is composed of a CPU (Central Processing Unit) and the like, and by executing a program stored in the storage unit 30, it realizes the functions of each unit (voice acquisition unit 21, identification unit 22, partial analysis unit 23, gesture response control unit 24, speech analysis unit 25, response sentence generation unit 26, language response control unit 27, learning unit 28, and identification unit 29) described later, and controls the operation of the robot 1. The storage unit 30 is composed of a ROM (Read Only Memory) and a RAM (Random Access Memory), and the ROM is composed of a part or all of an electrically rewritable memory (such as a flash memory). The robot 1 may include, for example, an imaging device for recognizing the face of a predetermined target, and the control unit 20 may communicate with the imaging device via the I/O interface to obtain image data and the like.

The voice acquisition unit 21, which functions as an acquisition means, performs A/D conversion on the voice signal input from the microphone 3 by a specific subject by sampling it at a specific frequency, and generates digital data using, for example, linear PCM (Pulse Code Modulation). The voice acquisition unit 21 further converts the digital data using a short-time Fourier transform (STFT) to acquire a spectrogram. The voice acquisition unit 21 sends the acquired spectrogram to the identification unit 22.

The speech acquisition unit 21 also sequentially analyzes the spectrogram to acquire a string of phonemes that indicate the speech content. The phonemes are acquired by segmental phone labeling. Segmental phone labeling is a method of decomposing a speech signal into components such as consonants and vowels that are thought to constitute the speech signal, and assigning labels that represent each component. The speech acquisition unit 21 decomposes the speech signal into components based on the formant transitions obtained from the spectrogram, and the spectrogram pattern and its changes, and selects and assigns labels that match the patterns of the components. The process of acquiring a string of phonemes (labeling speech data in phoneme units) can be performed, for example, using the open source Julius phoneme segmentation kit. The speech acquisition unit 21 converts the segmental phonetic labels into phonemes, and sequentially sends the acquired phonemes and the time between phonemes or the time when the phonemes were spoken to the learning unit 28 and the speech analysis unit 25.

The identification unit 22 extracts speech feature data, such as an i-vector, from the spectrogram sent from the speech acquisition unit 21, and identifies the specific target by determining which of the speech feature data of multiple specific targets stored in the storage unit 30 the extracted speech feature data matches, and acquires the target ID of the identified specific target. The identification unit 22 sends the acquired target ID to the learning unit 28 and the speech analysis unit 25.

The learning unit 28, which functions as a learning means, learns, for each predetermined target, the shortest phoneme sequence that can identify the content of the utterance input from the microphone 3, from the phoneme sequence sent from the voice acquisition unit 21 and the sentence of the voice recognition result by the utterance analysis unit 25. The partial analysis unit 23, which functions as a first analysis means, partially analyzes the utterance input from the predetermined target, and if a part of the phoneme sequence in the utterance matches the phoneme sequence learned by the learning unit 28 (the shortest phoneme sequence described later), sends that information to the gesture response control unit 24. In addition, the identification unit 29, which functions as an identification means, identifies a gesture action corresponding to the input utterance according to the voice recognition result by the utterance analysis unit 25. If a part of the phoneme sequence in the input utterance matches the shortest phoneme sequence, the gesture response control unit 24 selects a gesture action corresponding to the matching shortest phoneme sequence, and causes the gesture operation unit 7 to perform the response.

The gesture response control unit 24 sends to the gesture actuation unit 7 an action sequence that describes the action start angle, action angular velocity, action angular acceleration, stop angle, and stop time of each of the flexion/extension, rotation, and lateral bending of the head 101, and the flexion/extension and adduction/abduction of the arms 103, which correspond to the gesture actions of, for example, Figures 3A to 3E. The gesture actuation unit 7 generates control signals for driving the head 101 and the arms 103 in accordance with the detection values of the sensors of the joints 5 and 6 in accordance with the action sequence sent from the gesture response control unit 24, and drives the head 101 and the arms 103 by inputting the generated signals to the actuators.

The speech analysis unit 25, functioning as the second analysis means, uses the sequence of phonemes sent from the speech acquisition unit 21 to analyze the input utterance for a section longer than the section of the utterance analyzed by the partial analysis unit 23, functioning as the first analysis means, and performs speech recognition. At that time, the speech analysis unit 25 refers to a dictionary database stored in the storage unit 30 and performs morphological analysis, etc., to perform speech recognition on the speech data sent from the speech acquisition unit 21. The speech analysis unit 25 sends the analysis result to the response sentence generation unit 26. The response sentence generation unit 26 generates a response sentence appropriate to the content of the utterance for each speaker who has spoken. The response sentence generation unit 26 sends the generated response sentence to the language response control unit 27, and the language response control unit 27 causes the response sentence to be spoken from the speaker 4 by voice synthesis.

The control unit 20 executes a conversation recording process, an analysis learning process, and a predictive response control process in order to respond to the user's speech as quickly as possible with a non-verbal response using a gesture action. Below, the conversation recording process, the analysis learning process, and the predictive response control process will be described in order.

Figure 4 is a flowchart showing the conversation recording process according to the embodiment. In the conversation recording process, the control device 2 records the contents of the conversation between the robot 1 and each specific target registered in the robot 1, as well as the recognized short sentences. The conversation recording process is repeated each time a series of conversations takes place, and the conversation record data is accumulated.

The control unit 20 starts the conversation recording process at the same time as the start of a conversation between the robot 1 and a predetermined target. For example, the control unit 20 starts the voice recognition process when the voice acquisition unit 21 detects a voice with a voice level exceeding a predetermined threshold, and determines that a conversation has started if the voice recognition is successful. In addition, if there is a lot of noise in the voice, the voice recognition result may be separated into words, and if one word is recognized, it may be determined that a conversation has started.

First, the identification unit 22 identifies a specific object as described above (step S401).

Next, the speech analysis unit 25 determines whether the conversation has ended (step S402). The end of the conversation can be determined, for example, by determining that the length of the silent period exceeds a predetermined length, by determining that the context has ended, or, in the case of facial recognition using a camera, by determining that the speaker's face cannot be recognized, or by a combination of these. The end of the context can be determined, for example, by the speech analysis unit 25 when there is no next utterance from the predetermined target for a predetermined time after an answer to a question is spoken, or when it detects an utterance from the predetermined target declaring the end of the conversation, such as "bye-bye" or "see you later."

When the conversation has not ended (step S402; N), the speech analysis unit 25 performs speech recognition on the speech data acquired by the speech acquisition unit 21 from the speech of the predetermined target by performing morphological analysis, etc., with reference to the dictionary database stored in the storage unit 30 (step S403). The speech analysis unit 25 then records a conversation record in which the recognition result and the speech data are associated in the storage unit 30 as a conversation record database (step S404), and returns to step S402. As shown in FIG. 5, the conversation record includes the time, the conversation partner (target ID), the utterance of the predetermined target during the conversation, the phoneme string included in the utterance of the predetermined target during the conversation, and the speech data. Here, the time is the time when the conversation started or ended. The utterance sentence is a character string that represents the content of the uttered sentence. The phoneme string is a string of phonemes of the utterance sentence. The voice data column is information that specifies the voice data itself or a file in which the voice data is recorded. The voice data is used later to analyze the length of the utterance of the phoneme string.

On the other hand, when the conversation has ended (step S402; Y), the utterance analysis unit 25 ends the conversation recording process. In this manner, the utterance analysis unit 25 stores the conversation record in which the recognition result of the utterance of the specified target is associated with the voice data as a conversation record database in the storage unit 30.

Next, the analysis learning process will be described with reference to FIG. 6. This analysis learning process is a process for generating a response gesture database used when controlling gesture actions and recording it in the storage unit 30. The analysis learning process is executed, for example, when a predetermined number of conversations are newly stored in the conversation record database, or each time a predetermined period of time has passed.

The learning unit 28 first clears the response gesture database generated in the previous analysis and learning process (S600). The analysis and learning is performed for each of the predetermined targets registered in the robot 1. Next, from the conversation record database shown in FIG. 5, as shown in FIG. 7, all of the multiple spoken sentences corresponding to the target ID (ID=1) of the first registered predetermined target are read out, and the multiple spoken sentences that have been read out are stored in a predetermined storage area of the RAM (step S601).

Next, the learning unit 28 determines whether or not analysis learning for controlling gesture operations, which will be described later, has been completed for all the predetermined targets (step S602). When analysis learning has not been completed for all the predetermined targets (step S602; N), a table (unique phoneme sequence table) is created by deleting the overlapping utterance sentences (e.g., "Good morning") from the multiple utterance sentences (as shown in FIG. 7) stored in step S601, leaving only the first one (e.g., "Good morning" on 9/9/2018 9:01) and the other utterance sentences (e.g., "Good morning" on 9/12/2018 8:00, 9/14/2018 8:00, and 9/14/2018 9:00), and storing the table in a predetermined storage area of the RAM (step S603). This unique phoneme sequence table is temporary for analytical learning of one target ID, and does not require time and voice data. Since the target ID is numbered for each predetermined target, it is sufficient to have a correspondence between the spoken sentence and the phoneme sequence, as shown in Figure 8. The learning unit 28 then reads out the first spoken sentence from the unique phoneme sequence table (step S604).

Next, it is determined whether or not all the spoken sentences have been read from the unique phoneme sequence table (step S605). When all the spoken sentences have not been read from the unique phoneme sequence table (step S605; N), the identification unit 29 identifies the gesture action corresponding to the spoken sentence read in step S604 using the gesture action database shown in FIG. 9 stored in the storage unit 30 (step S606). As shown in FIG. 9, this gesture action database stores gesture corresponding sentences and gesture action numbers in association with each other, and in step S606, the number of the gesture action corresponding to the gesture corresponding sentence that matches the spoken sentence read in step S604 is identified as the number representing the corresponding gesture action. For example, for the gesture corresponding sentence "Good morning", the gesture number "2" representing the gesture action shown in FIG. 3A is identified.

Then, the learning unit 28 determines whether or not a corresponding gesture action has been identified in step S606 (step S607). If a gesture action has been identified in step S606 (step S607; Y), a response gesture database registration process (described later, shown in FIG. 10) is executed (step S608). Then, the speech sentence following the speech sentence read in step S604 is read from the unique phoneme sequence table (step S609), and steps S605 to S608 are executed again. On the other hand, if a corresponding gesture action cannot be identified in step S606 (step S607; N), step S608 is skipped, and step S609 and subsequent steps are executed without executing the response gesture database registration process.

When all the spoken sentences have been read from the unique phoneme sequence table as a result of repeatedly executing steps S605 to S609 (step S605; Y), all the spoken sentences corresponding to the target ID next to the first target ID that was the target to be read in step S601 are read from the spoken sentences stored in the conversation record database shown in FIG. 5, and the read spoken sentences are stored in a specified memory area of the RAM (step S610). Next, steps S602 and onwards are executed again. When the analytical learning by the above-mentioned steps S603 to S609 is completed for all the specified targets (step S602; Y), the analytical learning process is terminated.

Next, the response gesture database registration process in step S608 in FIG. 6 will be described with reference to FIG. 10. Through this response gesture database registration process, the learning unit 28 identifies the shortest phoneme string as the minimum (shortest) phoneme string that can identify the spoken sentence read out in step S604 or S609 in FIG. 6. For example, for the sentence "good morning," the phoneme string "oha" is identified.

First, the learning unit 28 sets a counter N as a local variable to 1 (step S1000), and reads out the first N phoneme strings of the spoken sentence read out in step S604 or S609 of FIG. 6 (step S1001). Then, it is determined whether the length of the phoneme string of the spoken sentence under consideration is equal to the length N of the read phoneme string (step S1002). If the length of the phoneme string of the spoken sentence is equal to the length N of the read phoneme string (step S1002; Y), it is determined that there is no shortest phoneme string capable of identifying the spoken sentence, and the process returns to the flowchart of FIG. 6 (step S609) without storing anything in the response gesture database.

When the length of the phoneme sequence of the spoken sentence is not equal to the length N of the read phoneme sequence (step S1002; N), the learning unit 28 searches the unique phoneme sequence table for a spoken sentence other than the focused sentence whose phoneme sequence from the beginning matches the read phoneme sequence (step S1003).

Then, the learning unit 28 determines whether or not a matching phoneme string was found in step S1003 (step S1004). If a matching phoneme string was found (step S1004; Y), the counter N is incremented by 1 (step S1005), and the phoneme string from the beginning to the Nth phoneme string of the utterance sentence of interest is read (step S1006). Then, the process returns to step S1002, and the process from step S1002 is executed again.

If, as a result of repeatedly executing steps S1002 to S1006, there is no utterance other than the focused sentence in which the phoneme sequence from the beginning matches the read phoneme sequence in the unique phoneme sequence table (step S1004; N), the read phoneme sequence up to the Nth phoneme sequence is recorded in the RAM of the memory unit 30 as the shortest phoneme sequence capable of identifying the content of the focused sentence (hereinafter referred to as the "shortest phoneme sequence") (step S1007).

Next, the learning unit 28 refers to a predetermined area in which multiple spoken sentences (as shown in FIG. 7) corresponding to the target ID of a predetermined target read out in step S601 or step S610 in FIG. 6 are stored, and measures the average spoken length of the corresponding phoneme strings (shortest phoneme strings) up to the Nth phoneme string for all spoken sentences that are the same as the spoken sentence of interest (step S1008). At this time, the number of times the same spoken sentence occurs is counted.

In step S1008, all the voice data of the same utterance sentence as the utterance sentence of interest is extracted, and the lengths of the phoneme sections of the shortest phoneme sequence identified in step S1007 (the time from the beginning to the end of the shortest phoneme sequence) are extracted, and the average time is calculated. For example, assume that the shortest phoneme sequence of the utterance sentence "Good morning" is "oha". FIG. 11 shows voice data of the same utterance sentence "Good morning" of the same specified target uttered at different times, arranged vertically with the same start timing. The learning unit 28 extracts the voice data of the utterance sentence "Good morning" from the voice data of multiple utterance sentences corresponding to the target ID of the specified target shown in FIG. 7, as shown in FIG. 11, and measures the average length of the utterance of the shortest phoneme sequence "oha" by averaging the time from the start of the voice data to the end of the phoneme "a", for example, t1 and t2 in FIG. 11.

Then, the learning unit 28 stores the spoken sentence, the shortest phoneme sequence identified in step S1007, the time when the shortest phoneme sequence was spoken measured in step S1008 (which may be the time when the shortest phoneme sequence was spoken plus the detection time (e.g., 20 ms)), the gesture action number identified in step S606 of FIG. 6, and the number of occurrences of the spoken sentence in the response gesture database shown in FIG. 12 (step S1009), and returns to the flowchart of FIG. 6 (step S609).

In Figures 5, 9, and 12, greetings such as "Good morning" are shown as examples, but the spoken sentences and gesture-enabled sentences may each include interjections, interjections, or exclamations such as "Um," "Erm," "Oh," and "Oh," respectively.

Next, the predictive response control process will be described with reference to FIG. 13. The predictive response control process is started, for example, when the voice acquisition unit 21 detects a voice with a voice level exceeding a predetermined threshold. When the control unit 20 starts predictive response control, it first identifies a predetermined target with the identification unit 22 (step S1300). Next, the partial analysis unit 23 reads out the shortest phoneme sequence length for the identified predetermined target from the response gesture database, and uses the read out shortest phoneme sequence length to generate a response time list as shown in FIG. 14 (step S1301). In the response time list shown in FIG. 14, the response times are listed in order from shortest to oldest.

Then, the control unit 20 judges whether the conversation between the predetermined target and the robot 1 has ended (step S1302). The end of the conversation can be judged in the same manner as step S402 of the conversation recording process (FIG. 4). If the conversation has not ended (step S1302; N), it judges whether the predetermined target has started speaking (step S1303), and waits until the target starts speaking (step S1303; N). The start of speaking is detected, for example, when the sound level reaches or exceeds a threshold value, or by facial recognition of the predetermined target in the case of facial recognition using a camera. If the predetermined target has started speaking (step S1303; Y), the process from step S1304 onward is executed.
On the other hand, when the conversation ends (step S1302; Y), the prediction response control process ends. As described above, the process from step S1304 onward is executed every time a speech by a predetermined target is started.
After step S1304, the control unit 20 performs the following process the same number of times as the number of predicted response times recorded in the predicted response time list.

The partial analysis unit 23 reads the first response time from the response time list generated in step S1301 (step S1304), determines whether the response time has elapsed since it was determined in step S1303 that the target speech started (step S1305), and waits until the response time has elapsed (step S1305; N). When the response time has elapsed (step S1305; Y), the partial analysis unit 23 extracts the target speech input from the microphone 3 from the target speech started until the response time has elapsed (step S1306). Then, the partial analysis unit 23 determines whether silence is detected in the extracted speech (step S1307). If the extracted target speech includes silence, for example, with a level below a threshold, for a certain period of time (for example, 100 ms) or more (step S1307; Y), the process returns to step S1302.

On the other hand, if the extracted voice of the specified target does not include silent voice (step S1307; N), the partial analysis unit 23 converts the voice extracted in step S1306 into a phoneme string (step S1308). Then, the partial analysis unit 23 determines whether or not a phoneme string matching the phoneme string converted in step S1308 is present among the multiple shortest phoneme strings stored in the response gesture database corresponding to the specified target identified in step S1300 (step S1309). If a shortest phoneme string matching the converted phoneme string is present (step S1309; Y), the robot 1 is caused to execute a gesture movement corresponding to this matching shortest phoneme string (step S1312), and the process returns to step S1302.

For example, suppose the phoneme string converted in step S1308 is "oha." The partial analysis unit 23 searches for the phoneme string "oha" among the shortest phoneme strings in the response gesture database shown in FIG. 12, and since the shortest phoneme string "oha" matches, it obtains the corresponding gesture number "2" and sends it to the gesture response control unit 24. The gesture response control unit 24 then causes the robot 1 to perform the gesture action corresponding to gesture number 2 shown in FIG. 3A.

On the other hand, if there is no shortest phoneme sequence that matches the converted phoneme sequence (step S1309; N), it is determined whether or not all response times have been read from the response time list (step S1310). If not all response times have been read from the response time list (step S1310; N), the partial analysis unit 23 reads the next response time from the response time list (step S1311) and executes steps S1305 and after again. Then, if all response times have been read from the response time list (step S1310; Y), the process returns to step S1302.

The above describes the predictive response control process. In parallel with performing the gesture action in this predictive response control process, the control unit 20 performs the language response control process, which will be described next. This language response control process will be described with reference to FIG. 15. As with the predictive response control process, the language response control process is started, for example, when the voice acquisition unit 21 of the control unit 20 detects a voice with a voice level exceeding a predetermined threshold. When the control unit 20 starts the language response control process, first, the control unit 20 identifies a predetermined target with the identification unit 22 (step S1501). Next, the control unit 20 determines whether the conversation between the predetermined target and the robot 1 has ended, in the same manner as step S402 of the conversation recording process (FIG. 4) (step S1502).

If the conversation has not ended (step S1502; N), similar to step S1303 of the prediction response control process (FIG. 13), it is determined whether or not the predetermined target has started speaking (step S1503), and the process waits until the target starts speaking (step S1503; N). On the other hand, if the conversation has ended (step S1502; Y), the language response control process is terminated.

When the target starts speaking (step S1503; Y), the speech analysis unit 25 acquires the target's speech phonemes from the speech acquisition unit 21 (step S1504) and determines whether the target's speech has ended (step S1505). Whether the speech has ended can be determined, for example, by whether the voice level of the voice data acquired by the speech acquisition unit 21 remains below a predetermined threshold for a predetermined period of time (e.g., 600 ms). As long as the speech has not ended (step S1505; N), acquisition of the speech phonemes (step S1504) is repeated.

When the target has finished speaking (step S1505; Y), the speech analysis unit 25 converts the acquired sequence of phonemes into a spoken sentence (step S1506), and from the converted spoken sentence, references the dictionary database stored in the storage unit 30, performs syntactic analysis, and obtains the words and syntax contained in the spoken sentence (step S1507).

Next, the response sentence generation unit 26 generates a response sentence to the utterance of the predetermined target based on the words and syntax of the utterance sentence by referring to the response sentence database stored in the storage unit 30 (step S1508). Then, the language response control unit 27 vocalizes the response sentence from the speaker 4 by voice synthesis (step S1509), and the process returns to step S1502.

The above describes the linguistic response control process. Next, an example of the operation of the predictive response control process will be described with reference to FIG. 16. FIG. 16 shows an example of the operation when a phoneme string of a voice extracted according to a response time read from the response time list in FIG. 14 matches the shortest phoneme string in the response gesture database in FIG. 12, and a gesture action corresponding to the matching shortest phoneme string is executed. In this example, the utterance "Good morning" is input by a specified target.

Then, it is assumed that the speech of the specified target input from when the specified target starts speaking (time: T0~) until the first response time (100 ms) in the response time list has elapsed (time: T1) is extracted (step S1306 in FIG. 13) and the extracted speech is converted into a phoneme string (step S1308 in FIG. 13), resulting in "oha." This converted phoneme string of "oha" matches the shortest phoneme string "oha" of the spoken sentence "Good morning" in the response gesture database in FIG. 12 (step S1309: Y). Therefore, the robot 1 is caused to execute a gesture action with gesture number "2" corresponding to this shortest phoneme string "oha" (step S1312).

After that, when a certain time (e.g., 600 ms) has elapsed without any voice from the predetermined target (a state in which the voice level of the voice data acquired by the voice acquisition unit 21 is below a predetermined threshold) (step S1505 in FIG. 15; Y), the language response control unit 27 controls the robot 1 to make the robot 1 execute a speech response using language (steps S1506 to S1509). The certain time (e.g., 600 ms) is a time for confirming that the predetermined target has finished speaking and generating a response sentence. In this way, before making the robot 1 execute a speech response, the speech sentence of the predetermined target, i.e., the content of the speech, is predicted using the shortest phoneme sequence, and the robot 1 is made to execute a gesture movement accordingly, so that the predetermined target can have the feeling that the robot 1 is listening to his/her speech before the speech response of the robot 1 is executed.

The non-verbal responses that the control device 2 causes the robot 1 to perform are not limited to the movements of the head 101 and arms 103. Non-verbal responses, such as gesture actions, include not only the movements of the head 101 and arms 103, but also facial expressions, such as opening and closing eyelids, raising and lowering eyebrows, and movements of the eyes, nose, or mouth, or waving hands or changing the shape of the hand, such as fisting hands or opening and raising hands. Other non-verbal responses may include changing the display mode of the eyes of a display provided on the robot 1.

The conversation records that are the subject of analytical learning include at least the most recently recorded speech sentences, but the period for which these conversation records were recorded does not need to be a fixed period. For example, the most recent specified period when analytical learning is performed may be any period of conversation records, such as the most recent day, the most recent week, the most recent month, etc. When the period of the conversation records that are the subject of analytical learning is changed for each analytical learning, the conversation records that were the subject of the previous analytical learning and the conversation records that are the subject of the new analytical learning may overlap in part in their target periods, or they may not overlap at all.

As described above, according to this embodiment, a partial match of an utterance by a specific target can cause the robot to perform non-verbal behavior corresponding to that utterance, so that a response can be made without the time required for at least detecting the end of the utterance, recognizing the voice, and generating a response sentence, and a response to the utterance of the specific target can be made quickly and appropriately.

In addition, since the robot 1 returns a predetermined response to a predetermined target using non-verbal behavior, it does not interfere with the conversation of the target (the conversation itself proceeds normally). Therefore, there is less chance of an incorrect response compared to a case where the robot 1 quickly responds to an utterance with a response sentence. In addition, even if the non-verbal behavior (gestures, etc.) of the robot 1 is not an appropriate response to a spoken sentence by a predetermined target, it does not have a significant impact on the conversation.

In the analysis and learning process according to the embodiment, for a spoken sentence associated with a gesture movement, if the length of the shortest phoneme sequence that does not match the phoneme sequence from the beginning in a unique phoneme sequence table from which duplicates in the conversation record have been removed is shorter than the length of the spoken sentence, it is recorded as the shortest phoneme sequence that identifies the spoken sentence, allowing a response to be made in a shorter time than if the entire spoken sentence were analyzed before responding.

The control device 2 includes an identification unit 22 that identifies a specific target, associates the target ID of the identified specific target with a spoken sentence in the speech record, identifies and records the shortest phoneme sequence that identifies the spoken sentence associated with a gesture action for each specific target in an analysis and learning process, and creates a response gesture database for each specific target. Then, in a conversation with a specific target, the specific target is identified and the response gesture database for the specific target is used to identify the spoken sentence with the shortest phoneme sequence, making it possible to make a gesture response that matches the speech of the specific target, and to quickly return a gesture response.

For example, in the embodiment, as shown in FIG. 8, if the spoken sentence contains "good morning" and "good night," the shortest phoneme sequence that identifies "good morning" is "oha," and therefore "oha" has been mainly used as an example of the shortest phoneme sequence. However, if the identified target is a person who speaks "hayaenassu" in a dialect instead of the standard "good morning," the shortest phoneme sequence that identifies "hayaenassu" could be "ha," "haya," "hayae," etc. If the shortest phoneme sequence is "ha," a faster response is possible compared to a person whose shortest phoneme sequence is "oha." In this way, the control device 2 performs a conversation recording process and an analysis and learning process for each identification target, thereby enabling it to perform a predictive response control process that is optimal for the identification target.

In the embodiment, the sound is cut out over a time period equal to the length of the shortest phoneme string and the phoneme strings are compared, but this is not limiting and various modifications are possible. For example, the sound may be cut out sequentially, and a determination may be made as to whether or not there is a matching phoneme string in the predictive response control process based on whether or not the phoneme string from the beginning of speech matches the shortest phoneme string. Alternatively, instead of converting to a phoneme string, the sound from the beginning of speech may be directly compared with a reference sound stored in association with a gesture movement, and if the sounds are similar, a gesture corresponding to the reference sound may be performed.

In the embodiment, the response gesture database is registered by calculating the average time of the length of the phoneme string, but if the aggregated time varies greatly, it is possible not to register it. In the embodiment, registration is performed without considering the number of occurrences or the variation in the length of the phoneme string, but registration may be performed when the number of occurrences and the variation in the length of the phoneme string reach a statistically significant frequency.

In the embodiment, the learning unit 28 learns the correspondence between the shortest phoneme sequence that identifies a speech sentence for each predetermined target and a gesture action. However, instead of learning a response gesture database for each predetermined target, for example, a response gesture database created before shipping from the factory may be stored in the ROM (or non-volatile RAM) of the robot 1. To create a response gesture database in advance, a voice conversation database that collects various speech sentences or a database that collects voice conversations with the robot (a database that collects a lot of voice conversations) is prepared, and the analysis learning of FIG. 6 is performed using this prepared database. In that case, the length of the shortest phoneme sequence registered in the response gesture database can be a time that includes the average or standard deviation of the expected general target. In addition, the response gesture database may include the general occurrence probability of the speech sentence instead of the number of occurrences. Alternatively, the response gesture database may be stored in a rewritable ROM or non-volatile RAM, and the number of occurrences may be counted from the utterance of the predetermined target while the robot 1 is operating, and the number of occurrences item in the response gesture database may be updated.

In the embodiment, the content of the utterance is defined as an "utterance sentence", but the content of the utterance is not limited to a sentence. For example, "utterance categories" (here, "purposes of utterance") such as "greetings" ("good morning", "hello", etc.), "thanks" ("thank you", "I'm grateful", etc.), "questions" ("tell me a bit", "can I ask you something", etc.), and "evaluations" ("good", "you understand well", etc.) may be defined as the content of the utterance. In this case, the control device 2 can determine the non-verbal behavioral responses of the robot 1 for each of these "utterance categories" and learn the correspondence between the shortest phoneme sequence that identifies the "utterance category" and the non-verbal behavioral responses. Non-verbal behaviors can be, for example, a bowing gesture for a "greeting", a shaking hand gesture for a "thank you" or "evaluation", and a tilting of the head 101 for a "question".

The spoken sentences associated with the gestures (gesture-corresponding sentences) are not limited to Japanese, and may be in a foreign language. The control device 2 can perform analysis learning processing and predictive response processing using a phoneme set for each language. For example, English spoken sentences can be associated with gesture movements, and the above-mentioned analysis learning processing and predictive response control processing can be performed using an English phoneme set.

In addition, the number of non-verbal behaviors corresponding to one spoken sentence is not limited to one. For example, a gesture action group including multiple gesture actions may be associated with one spoken sentence (gesture-corresponding sentence), and when the shortest phoneme string of the spoken sentence is detected, one gesture action may be selected from the corresponding gesture action group and executed by the robot 1. In this case, the selection of a gesture action from the gesture action group may be made with a fixed probability or randomly, or a gesture action may be selected from the gesture action group according to changes in speech, such as the pitch of the sound when the shortest phoneme string is spoken, the volume of the speech, the position of the accent in the phoneme string, and differences in intonation of the phoneme string. Furthermore, the probability of selecting a gesture action from the gesture action group may be changed according to changes in speech, and a gesture action may be selected with the changed probability.

In addition to the above-mentioned changes and modifications, various other modifications and variations are possible. For example, the shape of the robot 1 is not limited to the shape shown in FIG. 1. For example, it may be shaped to resemble a pet, such as a dog or a cat. The robot 1 may also be shaped like a stuffed toy or an anime character.

Alternatively, the robot 1 may be an avatar displayed on the screen of a smartphone or tablet. When the robot 1 is an avatar, the control device 2 can be realized by an application program installed on the smartphone or tablet. The control device 2 acquires an audio signal from a microphone 3 provided on the smartphone or tablet on whose screen the avatar is displayed, causes the avatar displayed on the screen to respond non-verbally, and causes the response sentence to be spoken from a speaker 4 provided on the smartphone or tablet.

Instead of a CPU, the control device 2 may have dedicated hardware such as an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or various control circuits as the control unit 20, and the dedicated hardware may function as each unit shown in FIG. 2. In this case, the functions of each unit may be realized by individual hardware, or the functions of each unit may be realized together by a single piece of hardware. Also, some of the functions of each unit may be realized by dedicated hardware, and the other parts may be realized by software or firmware.

The programs that realize the functions of the control device 2 can be stored and applied on a computer-readable storage medium, such as a flexible disk, a CD (Compact Disc)-ROM, a DVD (Digital Versatile Disc)-ROM, or a memory card. Furthermore, the programs can be superimposed on carrier waves and applied via a communication medium such as the Internet. For example, the programs can be posted and distributed on a bulletin board system (BBS) on a communication network. The above-mentioned processes can then be performed by starting up the programs and executing them under the control of an OS (Operating System) in the same way as other application programs.

Although the preferred embodiment of the present invention has been described above, the present invention is not limited to such a specific embodiment, and the present invention includes the inventions described in the claims and their equivalents. The inventions described in the original claims of this application are described below.

(Appendix 1)
A control device for a robot capable of responding to an object, comprising:
An acquisition means for acquiring a speech of the target;
a first analysis means for partially analyzing the speech acquired by the acquisition means while the subject is speaking;
a first control means for controlling a first response by the robot using a non-verbal behavior toward the target in accordance with an analysis result by the first analysis means;
a second analysis means for analyzing the target speech acquired by the acquisition means in a section longer than the section of the speech analyzed by the first analysis means;
a generation means for generating a response sentence to the utterance of the target in response to an analysis result by the second analysis means;
a second control means for controlling a second response using language by the robot based on the response sentence generated by the generation means;
A robot control device comprising:

(Appendix 2)
A robot control device as described in Appendix 1, characterized in that the length of the phoneme sequence of the utterance analyzed by the first analysis means is set shorter than the length of the phoneme sequence of the utterance analyzed by the second analysis means.

(Appendix 3)
the first response comprises a plurality of first responses;
a storage means for storing a plurality of reference phoneme sequences corresponding to the plurality of first responses;
the first analysis means determines whether or not a phoneme sequence of a part of the utterance acquired by the acquisition means while the target is speaking matches any one of the plurality of reference phoneme sequences;
The control device for a robot described in Appendix 1 or 2, characterized in that, when the first analysis means determines that the part of the phoneme sequence matches any one of the plurality of reference phoneme sequences, the first control means causes the robot to execute a first response among the plurality of first responses that corresponds to the any one of the plurality of reference phoneme sequences that has been determined to match.

(Appendix 4)
The robot control device described in Appendix 3, characterized in that each of the multiple reference phoneme sequences is set to the shortest phoneme sequence capable of identifying the content of the target's speech.

(Appendix 5)
the robot is capable of executing the first response and the second response to a plurality of targets different from each other as the target;
further comprising an identification means for identifying each of the plurality of objects;
The acquisition means acquires, for each of the identified targets, an utterance of the target,
an analysis means for analyzing the content of the acquired target utterance;
A determination means for determining a first response corresponding to an analysis result by the analysis means from the plurality of first responses;
and a learning means for learning, for each identified target, the shortest phoneme sequence capable of identifying the content of the target's utterance corresponding to the analysis result based on the analysis result by the analysis means, by associating the shortest phoneme sequence with the identified first response as the reference phoneme sequence based on the analysis result by the analysis means.

(Appendix 6)
The robot has a drivable movable part,
A robot control device as described in any one of appendices 1 to 5, characterized in that the first response is a response by a gesture movement realized by driving the movable part of the robot, and the second response is a response by speaking the response sentence to the target.

(Appendix 7)
A robot configured to be able to execute the first response and the second response to a target, comprising the robot control device described in any one of appendices 1 to 6.

(Appendix 8)
A method for controlling a robot that is responsive to an object, the method comprising:
An acquisition step of acquiring a target utterance;
a first analysis step of partially analyzing the speech acquired by the acquisition step while the subject is speaking;
a first control step of controlling a first response by the robot using a non-verbal behavior toward the target in accordance with an analysis result by the first analysis step;
a second analysis step of analyzing the target utterance acquired by the acquisition step, for a section longer than the section of the utterance analyzed by the first analysis step;
a generation step of generating a response sentence to the utterance of the target according to an analysis result in the second analysis step;
a second control step of controlling a second response using language by the robot based on the response sentence generated in the generation step;
A robot control method comprising:

(Appendix 9)
A computer that controls a robot that can respond to an object,
An acquisition means for acquiring a speech of the target;
a first analysis means for partially analyzing the speech acquired by the acquisition means while the subject is speaking;
a first control means for controlling a first response by the robot using a non-verbal behavior toward the target in accordance with a result of the analysis by the first analysis means;
a second analysis means for analyzing the target speech acquired by the acquisition means over a section longer than the section of the speech analyzed by the first analysis means;
a generation means for generating a response sentence to the utterance of the target in accordance with an analysis result by the second analysis means; and a second control means for controlling a second response using language by the robot based on the response sentence generated by the generation means.
A program to function as a

1...Robot, 2...Control device, 3...Microphone, 4...Speaker, 5, 6...Joints, 7...Gesture actuation unit, 20...Control unit, 21...Speech acquisition unit, 22...Identification unit, 23...Partial analysis unit, 24...Gesture response control unit, 25...Speech analysis unit, 26...Response sentence generation unit, 27...Language response control unit, 28...Learning unit, 29...Identification unit, 30...Memory unit, 101...Head, 102...Torso, 103...Arms

Claims (5)

A robot capable of responding by voice and by gesture,
a first identification means for starting a predetermined identification process in response to a start of an utterance by a speaker in order to identify a category of a content that the speaker is trying to convey through the utterance;
a control means for controlling the robot to execute a gesture that is preset in association with the identified category as soon as the category is identified by the identification process, regardless of whether the speaker has finished speaking or not;
Equipped with
The control means
Even when controlling the robot to execute the gesture response before the speaker finishes speaking, the robot is controlled to execute the voice response after waiting for the speaker to finish speaking, thereby controlling the robot to be silent until the speaker finishes speaking,
The robot is configured to be controllable to perform different gestures for each category.
A robot characterized by: the identification process is a process of identifying the category by performing a forward match search of a predetermined database with a phoneme string, which is a time series of phonemes that are sequentially pronounced as the utterance, each time the phoneme string extends by a predetermined elapsed time due to the pronunciation;
2. The robot according to claim 1 . A second identification means for identifying the speaker is provided,
the identification process performs a forward match search for the phoneme string in the database associated with the speaker identified by the second identification means;
3. The robot according to claim 2 . A response method executed by a robot capable of responding by voice and by gesture, comprising:
a step of starting a predetermined identification process in response to a start of speech by a speaker in order to identify a category of a content that the speaker intends to convey by the speech;
a control step of controlling the robot to execute a gesture that is preset in association with the identified category as soon as the category is identified by the identification process, regardless of whether the speaker has finished speaking or not;
having
The control step includes:
Even when controlling the robot to execute the gesture response before the speaker finishes speaking, the robot is controlled to execute the voice response after waiting for the speaker to finish speaking, thereby controlling the robot to be silent until the speaker finishes speaking,
The robot is configured to be controllable to perform different gestures for each category.
A response method comprising: A robot computer capable of responding by voice and by gestures,
a determination means for starting a predetermined determination process in response to a start of speech by a speaker in order to determine a category of the content that the speaker is trying to convey through the speech;
a control means for controlling the robot to execute a gesture that is preset in association with the identified category as soon as the category is identified by the identification process, regardless of whether the speaker has finished speaking or not;
Functioning as a
The control means
Even when controlling the robot to execute the gesture response before the speaker finishes speaking, the robot is controlled to execute the voice response after waiting for the speaker to finish speaking, thereby controlling the robot to be silent until the speaker finishes speaking,
The robot is configured to be controllable to perform different gestures for each category.
A program characterized by:

Images