JP7653828B2 - Robots and robot systems - Google Patents

Description

The present invention relates to a robot and a robot system.

Patent Document 1 describes a robot that aims to promote or suppress interaction between a device and a subject. When the robot leaves the child, it is made to utter a voice that notifies the child that the robot is leaving, such as, for example, "I'm a bit tired, so I'm going to have a rest" or "I've played a bit too much, so I'm going to have a rest." Also, when the robot stops interacting with the child, it is made to utter a voice that notifies the child that the robot is going to sleep, such as, for example, "I'm getting sleepy" or "Grrr (snoring sound)," or that expresses the state of being asleep.
Patent Document 2 describes a robot that aims to look after and take care of children. The robot acquires information indicating the child's situation, and based on the acquired situation, refers to a behavior memory unit that associates and stores the situation with the behavior that the robot should take when the situation occurs, and determines and controls the robot's behavior.

JP 2018-176383 A JP 2005-305631 A

The raison d'être of robots, especially communication robots, is that they are close to parents and family members, something that cannot be replaced by ordinary electrical appliances. For this reason, with the trend towards nuclear families, the declining birthrate, and the increasing use of robots in ordinary homes, it is expected that young children, who are highly dependent on their parents and family, will become more dependent on robots. As a result, it is expected that robots will become indispensable for childcare and the growth of children, even from the perspective of parents. Patent Documents 1 and 2 both propose robots that interact with children by speaking. However, the speech of robots such as those in Patent Documents 1 and 2 is merely one-sided or simply tailored to the content of the child's speech. If the speech of a robot to a child is merely one-sided or simply tailored to the content of the child's speech, it is no different from an ordinary electrical appliance.

The objective of the present invention is to provide a robot whose speech content and characteristics directed at children can contribute to childcare and growth, something that could not be achieved with conventional robots that only speak one-sidedly or that simply match the speech of the user.

The invention described in claim 1 is a robot comprising an extraction means for detecting that a voice spoken by a first user contains a voice portion with a predetermined characteristic and extracting the voice portion, an estimation means for estimating an emotional pattern of the first user based on the characteristic of the voice portion extracted by the extraction means, and a determination means for determining a form of voice to be output to a second user in accordance with the emotional pattern estimated by the estimation means.
The invention described in claim 2 is the robot described in claim 1, wherein the determination means determines, as the form of the voice to be output to the second user, a voice that has the same content as the speech of the first user but has a change in the emotion pattern.
The invention described in claim 3 is the robot described in claim 1, further comprising a storage means for storing in advance a plurality of voice manner patterns corresponding to positive emotions and negative emotions of the first user, wherein the estimation means estimates, as an estimation of the emotion pattern of the first user, whether the emotion pattern of the first user falls into either the positive emotion pattern or the negative emotion pattern, and the determination means selects and determines a voice manner pattern to be output to the second user from the plurality of voice manner patterns stored in the storage means in accordance with the emotion pattern of the first user estimated by the estimation means.
The invention described in claim 4 is the robot described in claim 3, wherein the determination means selects and determines a positive emotion pattern as the voice style pattern to be output to the second user regardless of a result of the estimation means' estimation of the emotion pattern of the first user.
The invention described in claim 5 is the robot described in claim 1, wherein the extraction means further extracts parts of contextual voice before and after parts of the voice of the first user's utterance that have a predetermined feature, and the estimation means estimates the emotion pattern of the first user by further taking into consideration the features of the parts of contextual voice before and after the parts of the voice with the predetermined feature extracted by the extraction means.
The invention described in claim 6 is a robot system comprising: an extraction means for extracting a part of a voice having a predetermined characteristic when a robot detects that a voice spoken by a first user includes the part of the voice; an estimation means for estimating an emotional pattern of the first user based on the characteristic of the part of the voice extracted by the extraction means; a determination means for determining a type of voice to be output by the robot to a second user in accordance with the emotional pattern estimated by the estimation means; and an output control means for controlling the robot to output a voice having the type of voice determined by the determination means.
The invention described in claim 7 is the robot system described in claim 6, wherein the determination means determines, as the form of the voice to be output to the second user, a voice that is identical in content to the speech of the first user but has a change in the emotion pattern.
The invention described in claim 8 is the robot system described in claim 6, further comprising a storage means for pre-storing a plurality of voice behavior patterns corresponding to each of positive emotions and negative emotions of the first user, wherein the estimation means estimates, as an estimation of the emotion pattern of the first user, whether the emotion pattern of the first user falls into either the positive emotion pattern or the negative emotion pattern, and the determination means selects and determines a voice behavior pattern to be output to the second user from the plurality of voice behavior patterns stored in the storage means in accordance with the emotion pattern of the first user estimated by the estimation means.
The invention described in claim 9 is the robot system described in claim 8, wherein the determination means selects and determines a positive emotion pattern as the voice style pattern to be output to the second user regardless of the result of the estimation means' estimation of the emotion pattern of the first user.
The invention described in claim 10 is the robot system described in claim 6, wherein the extraction means further extracts contextual voice parts before and after the voice parts having predetermined features from the voice of the first user's speech, and the estimation means estimates the emotion pattern of the first user by further taking into consideration the features of the contextual voice parts before and after the voice parts having the predetermined features extracted by the extraction means.

The present invention provides a robot that can contribute to childcare and the development of children by providing speech content and features directed at children that cannot be achieved with conventional robots that only speak one-sidedly or that simply match the speech of the user.

1 is a diagram showing a hardware configuration of a child-rearing support robot system to which the present embodiment is applied. FIG. 2 is a diagram illustrating a hardware configuration of the child care support robot. FIG. 2 is a diagram illustrating a functional configuration of a child care support robot. FIG. 2 is a diagram illustrating a functional configuration of a server. 13 is a flowchart showing a process flow of the childcare support robot. 13 is a flowchart showing a flow of processing by the server. FIG. 11 is a diagram showing a specific example of the correspondence between emotion patterns, specific voice contents, and output modes stored in a database. Fig. 2A is a diagram showing a specific example of timing to which the childcare support robot system in Fig. 1 is applied, and Fig. 2B is a diagram showing a specific example of variations in the form of sounds output from the childcare support robot.

[Hardware configuration of the childcare support robot system]
Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.
FIG. 1 is a diagram showing a hardware configuration of a child-rearing support robot system 1 to which the present embodiment is applied.
The childcare support robot system 1 is a system that realizes a "childcare support robot service" (hereinafter referred to as "this service") provided by a service provider to a user Ua who is the parent of a user Ub. The childcare support robot system 1 that realizes this service has a childcare support robot 10, a user terminal 50, and a server 70, all connected to a network 90 such as the Internet. This service is a service that supports childcare using the childcare support robot 10. The childcare support robot 10 detects that the speech of the user Ua and the speech of the user Ub who is the child of the user Ua each contain a specific voice that is a part of the voice having a predetermined characteristic amount. Then, the childcare support robot 10 uses this as a trigger to output a voice Vr corresponding to the emotion of the speaker (user Ua or user Ub) contained in the specific voice to the user Ub.

The user Ua who uses this service may not only be the parent of user Ub, but may also be a person in a position to manage or educate user Ub, such as a grandparent or caregiver. In this case, user Ub may be a person who is managed or educated by user Ua, such as user Ua's grandchild or care recipient. In either case, user Ub is assumed to be a preschool child, elementary school student, junior high school student, or high school student. In this specification, when there is no need to distinguish between user Ua and user Ub, both will be collectively referred to as "user U."

When the childcare support robot 10 detects that a specific voice is included in either the voice Va spoken by the user Ua to the user Ub or the voice Vb spoken by the user Ub, it extracts voice data of the specific voice. The childcare support robot 10 estimates the emotional pattern contained in the specific voice by analyzing the features of the extracted voice data. Specifically, when the childcare support robot 10 detects that a specific voice is included in part or all of the voice Va, it analyzes the features of the specific voice to estimate the emotional pattern of the user Ua when he spoke the voice Va. Furthermore, when the childcare support robot 10 detects that a specific voice is included in part or all of the voice Vb, it analyzes the features of the specific voice to estimate the emotional pattern of the user Ub when he spoke the voice Vb.

The speaker's emotional patterns can be classified into positive emotional patterns and negative emotional patterns. Positive emotional patterns include, for example, "joy," "calmness," "happiness," and "interest." Negative emotional patterns include, for example, "sorrow," "anger," "fear," and "disgust." Another example is "surprise," which can be both a positive and negative emotion. Note that the emotional patterns are not limited to these.

Here, the childcare support robot 10 may extract a specific voice and estimate the emotional pattern of the speaker of the specific voice without any particular limitation, and any method may be adopted. For example, a commonly used method of analyzing voice data may be adopted. Specifically, a method may be adopted in which AI (artificial intelligence) performs machine learning or deep learning on the pitch, fundamental frequency, voice waveform periodicity, vocal tract resonance frequency, formant frequency band, voice volume, voice quality and tone, speed, frequency of waveform peaks, etc. of the spoken voice. Here, the analysis of the voice data does not only target the specific voice, but also targets the voice data of the context before and after the specific voice. This is because even if the same word is spoken in the same way, the meaning may be completely different when the context before and after the word is taken into account.

When the childcare support robot 10 estimates the emotional pattern of the person who uttered the specific voice, it determines the mode (hereinafter referred to as the "output mode") when outputting the voice Vr to the user Ub according to the result of the estimation. The output mode is configured to include the content of the voice Vr to be output and the pattern of the emotion contained in the voice Vr. The output mode is managed in a database in association with each of a plurality of predetermined specific voices. Specifically, the output mode is stored in a specific voice DB 41 provided in an area of the memory unit 13 of the childcare support robot 10 in FIG. 2 described later, and a specific voice DB 81 provided in an area of the memory unit (not shown) of the server 70 having the same function as the memory unit 13. Specific examples of patterns of output modes stored as a database will be described later with reference to FIG. 7.

As a specific example of this service, assume that user Ua utters a voice Va saying "Hurry up!" in a manner that strongly scolds user Ub. In this case, the childcare support robot 10 converts the voice Va saying "Hurry up!" into a gentler voice Vr saying "Let's hurry up," and outputs it to user Ub. In other words, if the emotional pattern conveyed in the speech from user Ua to user Ub is negative, the childcare support robot 10 outputs a voice Vr that eases the negative emotional impact that being scolded has on user Ub and stabilizes his/her emotions.

For example, assume that user Ua utters a voice Va praising user Ub, such as "Amazing!" In this case, the childcare support robot 10 outputs a voice Vr that echoes the user's words, such as "Amazing!" In other words, if the emotional pattern conveyed in the speech from user Ua to user Ub is positive, the childcare support robot 10 outputs a voice Vr that amplifies the positive impact that being praised has on user Ub's mind.

When the voice Vr is output from the childcare support robot 10, the user Ub may react. When the user Ub utters the voice Vb that is the reaction, the presence or absence of a reaction and the data of the voice Vb that is the reaction are acquired as feedback information. This feedback information is information for retroactively verifying the validity of the output manner of the voice Vr output from the childcare support robot 10. In other words, it is information for verifying whether the output manner of the voice Vr output from the childcare support robot 10 was appropriate or completely off the mark. For this reason, the feedback information can be used as information that is the subject of machine learning by AI (artificial intelligence), for example.

Specifically, for example, if the user Ub utters a skeptical voice Vb such as "What are you saying?" or "What's that?" immediately after the voice Vr is output from the child care support robot 10 to the user Ub, the content of that voice is acquired as feedback information. This feedback information is information indicating that the validity of the voice Vr output from the child care support robot 10 is low. In contrast, for example, if the user Ub utters a positive voice Vb such as "I got it" or "Thank you" immediately after the voice Vr is output from the child care support robot 10 to the user Ub, the content of that voice is acquired as feedback information. This feedback information is information indicating that the validity of the voice Vr output from the child care support robot 10 is high.

The feedback information is managed in a database in association with each of a number of predetermined specific voices. Specifically, the feedback information is stored in a specific voice DB 41 provided in an area of the memory unit 13 of the child care support robot 10 (see FIG. 2, which will be described later), and in a specific voice DB 81 provided in an area of the memory unit (not shown) of a server 70 that has the same function as the memory unit 13.

As described above, the childcare support robot 10 can perform machine learning using AI (artificial intelligence) using the feedback information stored in the specific voice DB 41. The results of the machine learning are used as basis information for determining the output mode. In this way, by repeating the routine of obtaining feedback information, machine learning, determining the output mode, outputting the voice Vr, and obtaining feedback information again, the accuracy of the voice Vr output from the childcare support robot 10 can be improved. Note that machine learning targeting the feedback information can also be configured to be performed on the server 70 side, which will be described later. However, when the childcare support robot 10 is made to function as a standalone robot, the childcare support robot 10 performs machine learning targeting the feedback information.

Regardless of whether the emotional pattern of the user Ua is positive or negative, the childcare support robot 10 can output a voice Vr in a positive manner to the user Ub. The childcare support robot 10 can output not only a positive voice Vr, but also a negative voice Vr depending on the situation at the time. Specifically, for example, if the user Ua scolds the user Ub while suppressing his/her emotions, the childcare support robot 10 can output a voice Vr with a slightly stronger admonishment to the user Ub. This allows the user Ub to understand the meaning of the user Ua's attitude. For example, if the voice Vb uttered by the user Ub contains content that is disrespectful to an elder person, the childcare support robot 10 can output a voice Vr with a content that reprimands the user Ub on behalf of the user Ua.

As described above, the childcare support robot 10 basically determines the output mode based on the results of analyzing data on a specific voice and the voice in the surrounding context. At that time, the childcare support robot 10 takes into consideration the user information of the user Ua and the user Ub. "User information" refers to information on the user Ua and the user Ub. The user information includes personal information such as the nicknames, addresses (prefectures), ages, and genders of the users Ua and Ub.

The user information is acquired by being input into the user terminal 50. There are also cases where users Ua and Ub have registered user information separately in services other than this service, and there are provisions between this service and the other service regarding the flexibility of user information for user Ub. In such cases, with user U's consent, information that has already been registered as user information in the other service is also acquired as user information in this service. Specifically, the user information is acquired from an external server (not shown), such as a server that operates the other service. The child care support robot 10 stores and manages the acquired user information in a user information DB 42 provided in one area of the memory unit 13 in FIG. 2, which will be described later.

Here, an example of a case where user information is taken into consideration when determining the output mode will be described. For example, assume that the user information includes information such as the nickname of user Ub "XX-chan", address (prefecture) "Osaka Prefecture", age "5 years old", and gender "female". In this case, when outputting voice Vr to user Ub in determining the output mode, the child care support robot 10 takes into consideration the nickname "XX-chan", the phrase specific to the Kansai region, and the character for a preschool girl. Since user information is taken into consideration in determining the output mode in this way, the child care support robot 10 is able to output a more familiar voice Vr that includes personal information of user Ub.

In addition, when a part of the output mode is specified as a result of the input operation by the user Ua, the childcare support robot 10 accepts this. Then, the childcare support robot 10 outputs the voice Vr in an output mode that includes the content of the accepted specification. The reason why the user Ua can specify "part" of the output mode is as follows. That is, if the user Ua can specify "all" of the modes of the voice Vr output from the childcare support robot 10 in advance, the relationship between the voice Va and the voice Vb uttered by the user Ua and the user Ub, respectively, and the voice Vr output from the childcare support robot 10 will be weak. As a result, this is because there are cases where the original purpose of "having the content and characteristics uttered by the robot contribute to childcare and the growth of children" cannot be achieved. As an example of specifying a part of the output mode, there is a specification of an "output mode" that allows the specification of a variation of the output mode. The details of the output mode will be described later with reference to FIG. 8 (B).

As shown in FIG. 1, the childcare support robot 10 can function as a networked robot that constitutes the childcare support robot system 1, but in places with poor communication environments, it can also function temporarily as a standalone robot. It can also function from the beginning as a standalone robot that does not have communication capabilities.

The user terminal 50 constituting the childcare support robot system 1 is composed of a smartphone, tablet, personal computer, etc. The user terminal 50 is held by the user Ua who is a parent, or placed in the space in which the user Ua lives. The user terminal 50 accepts input operations from the user Ua. For example, the user terminal 50 accepts input operations for making various settings for the childcare support robot 10, and input operations for registering user information in this service. Note that the "input operation" referred to here includes manual input operations into a user interface displayed on the screen of the user terminal 50, voice input operations into a microphone, etc., and input operations by gestures into a camera, etc.

When the user terminal 50 is configured as a smartphone, tablet, or personal computer, dedicated application software (hereinafter referred to as a "dedicated app") that enables the use of this service can be installed on the user terminal 50. Even if a dedicated app is not installed, the service can be used by accessing a dedicated website that enables the use of this service using the browser function of the user terminal 50.

The server 70 constituting the childcare support robot system 1 is an information processing device that controls the entire system, and provides, for example, a dedicated application to the user terminal 50. The server 70 also accepts the designation of a part of the output mode by an input operation to the user terminal 50. The server 70 also stores and manages various information in a database. Specifically, the server 70 stores and manages feedback information acquired from the childcare support robot 10 in a specific voice DB 81 provided in one area of a storage unit (not shown) having a function similar to that of the storage unit 13 in FIG. 2 described later. The server 70 also stores and manages user information acquired from the user terminal 50, etc. in a user information DB 82 provided in one area of a storage unit (not shown) having a function similar to that of the storage unit 13 in FIG. 2 described later. The server 70 also transmits part or all of the user information stored and managed in the user information DB 82 to the childcare support robot 10 and the user terminal 50 as necessary.

In addition, when the childcare support robot 10 functions as a networked robot as shown in FIG. 1, the server 70 can perform machine learning using AI (artificial intelligence) using feedback information stored in the specific voice DB 81. The results of the machine learning are used as basis information for the childcare support robot 10 to determine the output mode. In this way, machine learning using feedback information can be configured to be performed on the server 70 side, or on the childcare support robot 10 side.

[Configuration of childcare support robot]
FIG. 2 is a diagram showing the hardware configuration of the child care support robot 10. As shown in FIG.
The childcare support robot 10 is a so-called communication robot with a built-in computer device. The childcare support robot 10 has a control unit 11, which is a central processing unit (CPU) that controls the entire robot itself, a memory 12, such as a random access memory (RAM) that is used as a working area during calculations, and a storage unit 13, which is a storage device, such as a hard disk drive (HDD) or a semiconductor memory, that is used to store programs and various setting data. The childcare support robot 10 also has a communication unit 14 that transmits and receives data via a network 90. The childcare support robot 10 also has an operation unit 15, such as a touch panel, that accepts input operations from the user U, a display unit 16, such as a liquid crystal display, that displays images and text information to the user U, and a display control unit 17 that controls the display unit 16. The childcare support robot 10 also has a recording unit 18, such as a microphone for recording sounds made by the user, and a sound output unit 19, such as a speaker for outputting sound to the user U. The childcare support robot 10 also has an imaging unit 20, such as a camera, that captures the state of the user U's gestures and obtains them as still or moving image information.

The hardware configuration of the user terminal 50 constituting the childcare support robot system 1 has a similar configuration to the hardware configuration of the childcare support robot 10 shown in FIG. 2. The hardware configuration of the server 70 constituting the childcare support robot system 1 has a similar configuration to the hardware configuration of the childcare support robot 10 shown in FIG. 2, except for the recording unit 18, the audio output unit 19, and the imaging unit 20. For this reason, a description of the hardware configuration of the user terminal 50 and the server 70 will be omitted.

FIG. 3 is a diagram showing the functional configuration of the child-rearing support robot 10.
The childcare support robot 10 has a voice acquisition unit 31 that acquires voice data uttered by the user U, and a specific voice extraction unit 32 that extracts specific voice data from the acquired voice data. The childcare support robot 10 also has an emotion pattern estimation unit 33 that estimates the emotion patterns of the users Ua and Ub who have uttered the voices, and an output mode storage unit 34 that stores the output mode patterns. The childcare support robot 10 also has a user information acquisition unit 35 that acquires user information, a user information storage unit 36 that stores the user information, and an output mode reception unit 37 that receives a designation of a part of the output mode. The childcare support robot 10 also has an output mode determination unit 38 that determines the output mode, a conversion unit 39 that converts the specific voice data, and an output control unit 40 that controls the output of the voices.

When at least one of the voice Va spoken by the user Ua and the voice Vb spoken by the user Ub is input, the voice is recorded by the recording unit 18 in FIG. 2 described above. The voice acquisition unit 31 acquires information on each of the voice Va and voice Vb recorded by the recording unit 18.

The specific voice extraction unit 32 detects that the voices Va and Vb of the respective utterances of the users Ua and Ub contain a voice portion with a predetermined characteristic amount, and extracts this as a specific voice. The specific voice extraction unit 32 also extracts the voice portions of the context before and after the specific voice portion.

The emotion pattern estimation unit 33 estimates the emotion pattern of the user U who uttered the specific voice based on the features of the specific voice extracted by the specific voice extraction unit 32. Specifically, for example, the emotion pattern estimation unit 33 estimates whether the emotion pattern of the user U who uttered the specific voice corresponds to a positive emotion pattern or a negative emotion pattern. The emotion pattern estimation unit 33 can also estimate the emotion patterns of each of the users Ua and Ub by taking into account the features of the parts of the voice in the context before and after the specific voice extracted by the specific voice extraction unit 32.

The output mode storage unit 34 stores, as output mode patterns, voice mode patterns corresponding to each of the positive emotions and negative emotions of the user U who has uttered the voice Va or the voice Vb, which includes a specific voice in part or in whole. Specifically, the output mode storage unit 34 stores a plurality of output mode patterns in the specific voice DB 41 in association with the specific voice.

The user information acquisition unit 35 acquires user information for each of the users Ua and Ub. Specifically, the user information acquisition unit 35 acquires user information for each of the users Ua and Ub that is input by an input operation. The user information acquisition unit 35 also acquires the usage history of the service for each of the users Ua and Ub as user information. The user information acquisition unit 35 also acquires, from information that has already been registered as user information for each of the users Ua and Ub in other services, information for which consent has been obtained from the user U, as user information for each of the users Ua and Ub in the service.

The user information storage unit 36 stores and manages the user information of each of the users Ua and Ub acquired by the user information acquisition unit 35 in the user information DB 42. Specifically, the user information storage unit 36 associates the user information of each of the users Ua and Ub with identification information that allows each of the users Ua and Ub to be uniquely identified, and manages the user information in a manner that allows the identification information to be extracted at any time as a key.

The output mode receiving unit 37 receives a designation for a part of the output mode. Specifically, when an input operation for designating a part of the output mode is performed, the output mode receiving unit 37 receives the designation. Examples of this input operation include manual input operation into a user interface displayed on the screen of the user terminal 50, voice input operation into a microphone or the like, input operation by gesture into a camera or the like, and the like.

The output mode determination unit 38 determines the output mode of the voice Vr to be output to the user Ub according to the emotion pattern of the user U who uttered the specific voice, which is estimated by the emotion pattern estimation unit 33. Specifically, for example, the output mode determination unit 38 determines a voice Vr whose content is the same as the speech of the user Ua and whose emotion pattern has been converted, as the output mode determination. Also, for example, the output mode determination unit 38 selects and determines the output mode pattern of the voice Vr to be output to the user Ub from among the multiple output mode patterns of the voice Vr stored in the output mode storage unit 34 according to the emotion pattern estimated by the emotion pattern estimation unit 33.

The output mode determination unit 38 may select and determine a positive emotion pattern as the output mode pattern of the voice Vr to be output to the user Ub, regardless of the result of the emotion pattern estimation by the emotion pattern estimation unit 33. The output mode determination unit 38 may determine the output mode, for example, based on predetermined selection criteria or the result of AI (artificial intelligence) learning about the usage history of the service by the user U. In addition, if part of the output mode has already been determined by the input operation of the user Ua, the output mode determination unit 38 reflects the determined content when determining the output mode.

The conversion unit 39 appropriately converts the output mode of the specific voice extracted by the specific voice extraction unit 32. Specifically, the conversion unit 39 converts the output mode of the extracted specific voice into the output mode determined by the output mode determination unit 38. Specifically, for example, when the result of the determination by the output mode determination unit 38 is in accordance with some of the specified contents received by the output mode reception unit 37, the conversion unit 39 converts the output mode when the voice Vr is output from the child care support robot 10 into the output mode specified by the output mode reception unit 37.

The specific method by which the conversion unit 39 converts the specific voice is not particularly limited, and may be, for example, a commonly used voice editing method. In this case, the conversion unit 39 converts the specific voice by, for example, waveform editing, noise removal, volume adjustment, frequency adjustment, sound pressure adjustment, breath removal, pitch adjustment, intonation adjustment, tone adjustment, etc.

The output control unit 40 controls to output the audio Vr in the output mode determined by the output mode determination unit 38. Specifically, the output control unit 40 controls to output the audio Vr, which has been appropriately converted so as to have the output mode determined by the output mode determination unit 38, to the audio output unit 19 in FIG. 2. The timing at which the output control unit 40 outputs the audio Vr is not particularly limited. For example, when specific audio information is extracted, the output mode may be determined and converted immediately, and the audio may be output from the audio output unit 19. Also, for example, after specific audio information is extracted, the output mode may be determined and converted within a preset time, and the audio may be output from the audio output unit 19. In this case, the "preset time" may be set arbitrarily by the user U, and may be set, for example, to "after 1 second" or "after 3 seconds".

[Server configuration]
FIG. 4 is a diagram showing the functional configuration of the server 70.
The server 70 has a user information acquisition unit 71 that acquires user information, a user information storage unit 72 that stores the user information, and a user information transmission control unit 73 that controls the transmission of a part or all of the user information to the child care support robot 10. The server 70 also has an output mode reception unit 74 that receives an input operation that specifies a part of the output mode, and an output mode determination unit 75 that determines a part of the output mode. The server 70 also has an output mode transmission control unit 76 that controls the transmission of the determined content to the child care support robot 10.

The user information acquisition unit 71 acquires the user information of each of the users Ua and Ub input by the user Ua's input operation. The user information acquisition unit 71 also acquires the usage history of the service by each of the users Ua and Ub as the user information of the users Ua and Ub. The user information acquisition unit 71 also acquires, from the information that has already been registered as the user information of each of the users Ua and Ub in another service, information for which the user U has given consent, as the user information of each of the users Ua and Ub in the service.

The user information storage unit 72 stores and manages the user information of each of the users Ua and Ub acquired by the user information acquisition unit 71 in the user information DB 82. Specifically, the user information storage unit 72 associates the user information with identification information that allows each of the users Ua and Ub to be uniquely identified, and manages the user information in a manner that allows the identification information to be used as a key to be extracted at any time.

The user information transmission control unit 73 controls the transmission of some or all of the user information stored in the user information DB 82 to the child care support robot 10. The user Ua can arbitrarily set whether or not to transmit user information from the server 70 to the child care support robot 10 and the items of user information to be transmitted.

The output mode receiving unit 74 receives the designation of a part of the output mode. Specifically, when an input operation for designating a part of the output mode is performed, the output mode receiving unit 74 receives the designation. This input operation can be a manual input operation to the user interface of the user terminal 50, an input operation by voice to a microphone or the like, an input operation by gesture to a camera or the like, and the like. An example of a "designation of a part of the output mode" is the designation of an "output mode" that allows the designation of variations in the output mode. Details of the output mode will be described later with reference to FIG. 8 (B).

When the output mode determination unit 75 receives a designation for a part of the output mode from the output mode reception unit 74, the output mode determination unit 75 makes a decision according to the designation. The output mode determination unit 75 may also determine a part of the output mode based on, for example, predetermined criteria or the results of machine learning using AI (artificial intelligence). When a part of the output mode is determined by the output mode determination unit 75, the output mode transmission control unit 76 controls the transmission of the determined content to the child care support robot 10.

[Childcare support robot processing]
Next, the processing of the child care support robot 10 will be described with reference to FIG.
FIG. 5 is a flowchart showing the flow of processing by the child-rearing support robot 10.
When the user information of each of the users Ua and Ub is input by the input operation of the user Ua (YES in step 101), the user information acquisition unit 35 acquires the user information input by the input operation of the user Ua (step 102). The user information acquired by the user information acquisition unit 35 is stored and managed in the user information DB 42 by the user information storage unit 36. On the other hand, if no user information has been input (NO in step 101), the process proceeds to the judgment in step 103.

When part of the output mode is designated by the input operation of the user Ua (YES in step 103), the output mode acceptance unit 37 accepts the designation, and the output mode determination unit 38 determines part of the output mode in accordance with the designation (step 104). On the other hand, when part of the output mode is not designated (NO in step 103), the process proceeds to the judgment in step 105.

When at least one of the voice Va spoken by user Ua and the voice Vb spoken by user Ub is input (YES in step 105), the recording unit 18 records the voice, and the voice acquisition unit 31 acquires information on each of the voices Va and Vb recorded by the recording unit 18 (step 106). On the other hand, when at least one of the voice Va spoken by user Ua and the voice Vb spoken by user Ub is not input (NO in step 105), the judgment in step 105 is repeated until at least one of the voice Va spoken by user Ua and the voice Vb spoken by user Ub is input.

The specific voice extraction unit 32 detects that the voices Va and Vb of the users Ua and Ub each contain a voice portion with a predetermined characteristic amount, and extracts this as a specific voice (step 107). The emotion pattern estimation unit 33 estimates the emotion pattern of the user U who spoke the specific voice, based on the characteristic amount of the specific voice extracted by the specific voice extraction unit 32 in step 107 (step 108).

The output mode determination unit 38 determines the output mode of the voice Vr to be output to the user Ub according to the emotional pattern of the user U who uttered the specific voice, estimated by the emotional pattern estimation unit 33 in step 108 (step 109). Here, if the output mode reception unit 37 receives a designation of a part of the output mode, a determination is made according to the designation.

The conversion unit 39 converts the output mode of the extracted specific voice into the output mode determined by the output mode determination unit 38 (step 110). Here, if the result of the determination by the output mode determination unit 38 in step 109 is in accordance with the part of the specification content accepted by the output mode acceptance unit 37, the conversion unit 39 converts into the output mode in which the part of the specification is accepted by the output mode acceptance unit 37. Also, for example, if the result of the determination by the output mode determination unit 38 is in accordance with a setting to take user information into consideration when deciding the output mode, the conversion unit 39 converts the output mode into an output mode in which the user information is taken into consideration. The output control unit 40 controls to output the voice Vr in the output mode determined by the output mode determination unit 38 (step 111). This ends the process.

[Server processing]
Next, the process of the server 70 will be described with reference to FIG.
FIG. 6 is a flowchart showing the flow of processing by the server 70.
When the user information of the user Ua and the user Ub is input by the input operation of the user Ua (YES in step 701), the user information acquisition unit 71 acquires the user information of the user Ua and the user Ub input by the input operation of the user Ua (step 702). The user information of the user Ua and the user Ub acquired by the user information acquisition unit 71 is stored and managed in the user information DB 82 by the user information storage unit 72. The user information transmission control unit 73 controls to transmit a part or all of the user information of the user Ua and the user Ub stored in the user information DB 82 to the child care support robot 10 (step 703). On the other hand, when the user information of the user Ua and the user Ub has not been input (NO in step 701), the process proceeds to the judgment in step 704.

When part of the output mode is specified by the input operation of the user Ua (YES in step 704), the output mode receiving unit 74 receives the specification, and the output mode determining unit 75 determines part of the output mode in accordance with the specification (step 705). When part of the output mode is determined, the output mode transmission control unit 76 controls the transmission of the determined content to the child care support robot 10 (step 706). On the other hand, when part of the output mode is not specified (NO in step 704), the process ends without determining part of the output mode.

According to the childcare support robot system 1 having the above configuration, the childcare support robot 10 sometimes becomes a presence that cares for the mental health of the user Ub, who is the child, and sometimes becomes a presence that supports the user Ua, who is the parent. In other words, the childcare support robot 10 becomes a member of the family of the user Ua and the user Ub, and is able to support childcare together with the user Ua.

[Specific examples]
Next, a specific example of this service will be described with reference to FIGS.
FIG. 7 is a diagram showing a specific example of the correspondence between emotion patterns, specific voice contents, and output modes stored in the database.

The information shown in FIG. 7 is a portion of the information stored in specific voice DB41 and specific voice DB81. As described above, feedback information is stored in specific voice DB41 and specific voice DB81, but FIG. 7 only shows the correspondence between emotion patterns, specific voice content, and output modes as information for explaining specific examples.

As shown in FIG. 7, the emotional patterns of the speakers (users Ua and Ub) include those that indicate "positive" emotions and those that indicate "negative" emotions. Among these, the "positive" emotions include those that indicate "joy", those that indicate "calm", and others (not shown) that indicate emotions such as "happiness" and "interest". Specific voice contents that correspond to those that indicate "joy" include, for example, "Amazing!" and "Great!", and the output modes of the childcare support robot 10 that correspond to each of these include, for example, "Amazing!" and "Great!". Specific voice contents that correspond to those that indicate "calm" include, for example, "Okay?" and "Got it?", and the output modes of the childcare support robot 10 that correspond to each of these include, for example, "Listen carefully", and "You understand well, don't you?".

Furthermore, the "negative" emotions include emotions such as "sorrow", "anger", and others (not shown), such as "fear" and "disgust". Specific voice contents corresponding to emotions such as "sorrow" include, for example, "That's terrible!" and "(crying)", and the output forms of the childcare support robot 10 corresponding to each of these include, for example, "What's wrong?" and "Don't cry". Specific voice contents corresponding to emotions such as "anger" include, for example, "Hey!" and "No!", and the output forms of the childcare support robot 10 corresponding to each of these include, for example, "That's not good" and "No."

Now, for example, suppose that user Ua utters the voice Va "Amazing!" with the emotion of "joy" to praise user Ub. Then, the child care support robot 10 outputs the voice Vr "Amazing!" to user Ub in an output mode that expresses the emotion of "joy". Also, for example, suppose that user Ua utters the voice Va "Great!" with the emotion of "joy" to praise user Ub. Then, the child care support robot 10 outputs the voice Vr "Great!" to user Ub in an output mode that expresses the emotion of "joy".

For example, suppose that user Ua utters the voice Va "Okay?" with a feeling of "calmness" to encourage user Ub to listen attentively. In response, the child care support robot 10 outputs the voice Vr "Listen carefully" to user Ub in an output mode that expresses the feeling of "calmness". For example, suppose that user Ua utters the voice Va "Do you understand?" with a feeling of "calmness" to admonish user Ub. In response, the child care support robot 10 outputs the voice Vr "You understand well, don't you?" to user Ub in an output mode that expresses the feeling of "calmness".

For example, suppose that user Ub utters a voice Vb saying "That's terrible!" to user Ua with the emotion of "sorrow". In response, the child care support robot 10 outputs a voice Vr saying "What's wrong?" to user Ub in an output mode expressing the emotion of "calm". For example, suppose that user Ub utters a voice Vb indicating crying (sobbing) with the emotion of "sorrow" rather than using specific words. In response, the child care support robot 10 outputs a voice Vr saying "Don't cry" to user Ub in an output mode expressing the emotion of "calm".

For example, suppose that user Ua utters a voice Va saying "Hey!" with the emotion of "anger" as if strongly scolding user Ub. The childcare support robot 10 then outputs a voice Vr saying "That's not good" to user Ub in an output mode expressing the emotion of "calm". For example, suppose that user Ua utters a voice Va saying "No!" with the emotion of "anger" as if strongly scolding user Ub. The childcare support robot 10 then outputs a voice Vr saying "No" to user Ub in an output mode expressing the emotion of "calm".

FIG. 8A is a diagram showing a specific example of the timing at which the child-rearing support robot system of FIG. 1 is applied.
8A, examples of the timings when the childcare support robot system 1 is applied include the timing when the user Ub, who is a child, succeeds in something, the timing when the user Ub fails in something, the timing when the user Ub is crying, etc. When the user Ub succeeds in something and the user Ua, who is a mother, praises the user Ub, the childcare support robot 10 outputs a voice Vr praising the user Ub, as if to join in. This can amplify the positive impact that the praise has on the user Ub's mind.

Furthermore, when user Ub makes a mistake and user Ua, the mother, scolds user Ub harshly, the child-rearing support robot 10 does not jump on the bandwagon, but outputs a voice Vr that comforts user Ub. This can ease the negative emotional impact that being scolded has on user Ub, and stabilize his/her emotions. When user Ub makes a mistake and user Ua, the mother, scolds user Ub while suppressing her emotions, the child-rearing support robot 10 can output a voice Vr that admonishes user Ub a little more forcefully. This can make user Ub understand the meaning of user Ua's attitude. In this way, by having user Ua and child-rearing support robot 10 express opposing emotions, effective child-rearing can be achieved through a combination of tension and relaxation.

In addition, if the user Ub cries for some reason, the childcare support robot 10 outputs a voice Vr that unconditionally comforts the user Ub. This allows the childcare support robot 10 to be an emotional support for the user Ua and the user Ub as a member of their family. Note that the timings shown in FIG. 8(A) are merely examples. The childcare support robot system 1 is applied at any timing when someone around the user Ub, who is a child, should lend a helping hand as the user Ub goes about his or her daily life.

FIG. 8(B) is a diagram showing specific examples of variations in the form of sounds output from the child care support robot.
As described above, the childcare support robot 10 outputs a voice Vr to the user Ub to support the user Ua in raising the user Ub, according to the emotions contained in the voices Va and Vb uttered by the user Ua and the user Ub, respectively. At this time, various variations are assumed as the output form of the voice Vr output from the childcare support robot 10. The variations of the output form may be preselected by the input operation of the user Ua as "output modes" with names given to each output form, as shown in FIG. 8B, for example. The output modes include "soft", "hard", "grandfather", "grandmother", "teacher", "neighborhood aunt", and "favorite character". The output modes are displayed on the user terminal 50 in a form selectable by the user Ua.

When "soft" is selected from the output modes shown in FIG. 8(B), the voice Vr is output in a softer manner than usual. For example, in the example of FIG. 7 described above, when the user Ua says "Amazing!", the childcare support robot 10 outputs the voice Vr "Amazing!" to piggyback on this, but if "soft" has been selected in advance, the voice Vr is output as "Amazing!". Also, when "hard" is selected from the output modes shown in FIG. 8(B), the voice Vr is output in a harder manner than usual. For example, in the example of FIG. 7 described above, when the user Ua says "Okay?", the childcare support robot 10 outputs the voice Vr "Listen carefully," but if "hard" has been selected in advance, the voice Vr is output as "Please listen carefully."

When "grandfather" is selected from the output modes shown in FIG. 8(B), the voice Vr is output in a manner that a typical grandfather would likely speak. For example, in the example of FIG. 7 described above, when the user Ua utters "Great!", the child care support robot 10 outputs the voice Vr "Great!" to piggyback on this, but if "grandfather" has been selected in advance, the voice Vr is output as "That's impressive!". Note that, when "grandmother," "teacher," or "neighborhood aunt" is selected from the output modes shown in FIG. 8(B), the voice Vr is similarly output in a manner that a typical grandmother, teacher, or neighborhood aunt would likely speak.

When "Favorite Character" is selected from the output modes shown in FIG. 8(B), the voice Vr is output in a manner that would be spoken by a favorite anime or manga character of the user Ub. In this case, a separately recorded voice of the target voice actor is used. For example, in the example of FIG. 7 described above, when the user Ub is crying, the child care support robot 10 outputs a voice Vr saying "Don't cry" in the voice of the favorite character of the user Ub. This is expected to improve the effect of caring for the user Ub's mind. When selecting "Favorite Character", the user Ua specifies in advance one or more favorite characters of the user Ub when inputting user information into the user terminal 50.

Although the present embodiment has been described above, the present invention is not limited to the above-described present embodiment. Furthermore, the effects of the present invention are not limited to those described in the above-described present embodiment. For example, users Ua and Ub include those who have not yet started using the service but wish to use the service.

For example, in the above-described embodiment, the input operation modes by the user Ua include manual input operation, voice input operation, and gesture input operation, but the input operation is not limited to these, and any input operation that can input the intention of the user Ua can be adopted. Also, in the above-described embodiment, the user U is not able to specify all of the output modes in advance, but the user U can also specify all of the output modes in advance.

In addition, in the above-described embodiment, the elements that constitute the output manner when the voice Vr is output from the childcare support robot 10 are "content" and "emotion pattern," but in addition to these, other elements may also be included, such as the timing (pause) at which the voice Vr is output, and the volume and speed at which it is output.

In addition, in the above embodiment, the user Ua performs input operations to configure various settings for the childcare support robot 10 using a dedicated app installed on the user terminal 50, but this is not limited to the present invention. For example, the input may be made directly to the childcare support robot 10.

In the above embodiment, the conversion is performed while maintaining the features of the voice Va uttered by the user Ua as much as possible, but by further utilizing AI (artificial intelligence), it is also possible to perform conversion that significantly changes the features of the voice Va uttered by the user Ua. Specifically, the schedule of the user Ub may be managed as the user information of the user Ub, and the voice Vr including more detailed information may be output from the childcare support robot 10. In this case, for example, when the user Ua utters a voice Va saying "Hurry up!" to the user Ub, the childcare support robot 10 may convert the voice Va into a voice Vr including detailed information such as "I'll leave the house by 8 o'clock, so let's finish changing clothes by 7:50" and output it.

1...Childcare support robot system, 10...Childcare support robot, 31...Speech acquisition unit, 32...Specific voice extraction unit, 33...Emotion pattern estimation unit, 34...Output mode storage unit, 35...User information acquisition unit, 36...User information storage unit, 37...Output mode reception unit, 38...Output mode determination unit, 39...Conversion unit, 40...Output control unit, 50...User terminal, 70...Server, 71...User information acquisition unit, 72...User information storage unit, 73...User information transmission control unit, 74...Output mode reception unit, 75...Output mode determination unit, 76...Output mode transmission control unit, 90...Network

Claims (10)

an extraction means for detecting that a voice of a direct speech from a first user to a second user includes a portion of the voice having at least one of predetermined features of pitch, fundamental frequency, voice waveform periodicity, vocal tract resonance frequency, formant frequency band, voice volume, voice quality, voice tone, speed, and frequency of occurrence of waveform peaks, and extracting the portion of the voice as a specific voice;
an estimation means for estimating an emotion pattern of the first user based on the feature amount of the specific voice extracted by the extraction means;
a determination means for determining a manner of a voice to be output to a second user in accordance with the emotion pattern estimated by the estimation means;
A robot comprising: the determining means determines, as the mode of the voice to be output to the second user, a voice having the same content as the speech of the first user and having a changed emotion pattern;
The robot according to claim 1. a storage unit configured to store in advance a plurality of patterns of the voice manner corresponding to each of the positive emotions and the negative emotions of the first user;
the estimation means estimates, as an estimation of an emotion pattern of the first user, whether the emotion pattern of the first user corresponds to the positive emotion pattern or the negative emotion pattern;
the determining means selects and determines a pattern of the voice manner to be output to the second user from among the plurality of voice manner patterns stored in the storage means in accordance with a pattern of the emotion of the first user estimated by the estimating means.
The robot according to claim 1. the determining means selects and determines a positive emotion pattern as a speech aspect pattern to be output to the second user regardless of a result of the estimation by the estimating means of a pattern of the emotion of the first user.
The robot according to claim 3 . The extraction means further extracts a part of a context of a part of a voice having a predetermined feature from the voice of the first user,
The estimation means estimates an emotion pattern of the first user by further considering features of parts of a voice in a context before and after the part of the voice having the predetermined feature extracted by the extraction means.
The robot according to claim 1. an extraction means for extracting, when the robot detects that a voice portion of a direct speech from a first user to a second user includes at least one of predetermined features of pitch, fundamental frequency, voice waveform periodicity, vocal tract resonance frequency, formant frequency band, voice volume, voice quality, voice tone, speed, and frequency of appearance of waveform peaks, as a specific voice;
an estimation means for estimating an emotion pattern of the first user based on the feature amount of the specific voice extracted by the extraction means;
a determination means for determining a form of a voice to be output by the robot to a second user in accordance with the emotion pattern estimated by the estimation means;
an output control means for controlling the robot to output a voice having the voice style determined by the determination means;
A robot system comprising: The determining means determines, as the mode of the voice to be output to the second user, a voice having the same content as the speech of the first user and having a changed emotion pattern.
The robot system according to claim 6 . a storage unit configured to store in advance a plurality of patterns of the voice manner corresponding to each of the positive emotions and the negative emotions of the first user;
the estimation means estimates, as an estimation of an emotion pattern of the first user, whether the emotion pattern of the first user corresponds to the positive emotion pattern or the negative emotion pattern;
The determination means selects and determines a pattern of the voice manner to be output to the second user from among the plurality of voice manner patterns stored in the storage means in accordance with a pattern of the emotion of the first user estimated by the estimation means.
The robot system according to claim 6 . the determining means selects and determines a positive emotion pattern as a speech aspect pattern to be output to the second user regardless of a result of the estimating means of the emotion pattern of the first user.
The robot system according to claim 8 . The extraction means further extracts a part of a context of a part of a voice having a predetermined feature from the voice of the first user,
The estimation means estimates an emotion pattern of the first user by further considering features of parts of a voice in a context before and after the part of the voice having the predetermined feature extracted by the extraction means.
The robot system according to claim 6 .