JP7735466B2 - Behavior Control System - Google Patents

Description

The present invention relates to a behavior control system.

Patent Document 1 discloses technology for determining appropriate robot behavior in response to a user's state. The conventional technology in Patent Document 1 recognizes the user's reaction when the robot performs a specific action, and if the robot is unable to determine its behavior in response to the recognized user reaction, it updates the robot's behavior by receiving information from a server about behavior appropriate to the recognized user's state.

Patent No. 6053847

However, conventional technology leaves room for improvement in enabling robots to perform appropriate actions in response to user actions.

The behavior control system of the present invention is a robot behavior control system applied to a vending machine, and includes a user emotion determination unit that determines an emotion value indicating the user's emotion, a robot emotion determination unit that determines an emotion value indicating the robot's emotion, and a behavior determination unit that determines the robot's behavior based on at least one of the user's emotion value and the robot's emotion value and information obtained by a sentence generation model with a dialogue function that allows the user and the robot to converse, and the behavior determination unit determines the product that the robot will recommend to the user.

This invention makes it possible to have a robot perform appropriate actions based on the user's situation.

FIG. 1 is a diagram illustrating an example of a system 5 according to an embodiment of the present invention. FIG. 2 is a diagram schematically illustrating the functional configuration of the robot 100. FIG. 2 is a diagram schematically illustrating an example of an operation flow by the robot 100. FIG. 1 is a diagram schematically illustrating an example of the hardware configuration of a computer 1200. FIG. 3 shows an emotion map 300 onto which multiple emotions are mapped. FIG. 9 shows an emotion map 900 onto which multiple emotions are mapped. FIG. 2 is a diagram illustrating an emotion table of the robot 100. FIG. 2 is a diagram illustrating an emotion table of a user 10. FIG. 10 is a flowchart illustrating an example of the operation of the robot 100 when applied to a vending machine.

The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention as claimed. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

A. Present Embodiment FIG. 1 schematically illustrates an example of a system 5 according to this embodiment. The system 5 includes a robot 100, a robot 101, a robot 102, and a server 300. A user 10, a user 11, and a user 12 are users of the robots 100, 101, and 102, respectively. The users 10, 11, and 12 are, for example, family members of the house in which the robot 100 is located, or strangers visiting the house. The robots 100, 101, and 102 can also be placed at the reception desk of a store or office, and used to greet visiting customers. In the description of this embodiment, the robots 101 and 102 have substantially the same functions as the robot 100. Therefore, the system 5 will be described mainly focusing on the functions of the robot 100.

The robot 100 converses with the user 10 and provides images to the user 10. At this time, the robot 100 cooperates with a server 300 or the like, with which it can communicate via a communication network 20, to converse with the user 10 and provide images, etc. to the user 10. For example, the robot 100 not only learns appropriate conversations by itself, but also cooperates with the server 300 to learn how to have more appropriate conversations with the user 10. The robot 100 also records captured video data of the user 10 on the server 300, and requests video data, etc. from the server 300 as needed, providing it to the user 10. While Figure 1 illustrates a mobile robot capable of autonomous movement as the robot 100, the robot 100 can be applied to any device (e.g., an in-vehicle device or a mobile device) equipped with an AI emotion engine, sentence generation model, etc., as described below.

The robot 100 is equipped with an AI emotion engine (described below) that has the function of generating simulated human emotions, and determines and stores emotion values that represent the type of emotion it feels. For example, the robot 100 has emotion values that represent the strength of each of the following emotions: "joy," "anger," "sadness," "pleasure," "discomfort," "relief," "anxiety," "sadness," "excitement," "worry," "relief," "fulfillment," "emptiness," and "neutral." For example, when the robot 100 is conversing with the user 10 when its excitement emotion value is high, it will speak at a fast speed. In this way, the robot 100 can express its emotions through its actions.

The robot 100 may also be configured to determine the behavior of the robot 100 that corresponds to the emotions of the user 10 by matching a sentence generation model (chat engine) with an AI emotion engine (emotion engine). Specifically, the robot 100 may be configured to recognize the behavior of the user 10, determine the emotions of the user 10 regarding the user's behavior, and determine the behavior of the robot 100 that corresponds to the determined emotions.

More specifically, when the robot 100 recognizes the user's 10 behavior, it uses a pre-configured chat engine to automatically generate the behavior that the robot 100 should take in response to the user's 10 behavior. A chat engine can be interpreted as algorithms and calculations for automatic text-based dialogue processing. Chat engines are publicly known, as disclosed in, for example, JP 2018-081444 A and chatGPT (Internet search <URL: https://openai.com/blog/chatgpt>), and therefore a detailed description thereof will be omitted. Such chat engines are configured using a large-scale language model (LLM). As described above, this embodiment combines a large-scale language model and an emotion engine to reflect the emotions of the user 10 and the robot 100, as well as various linguistic information, in the robot's 100 behavior. In other words, according to this embodiment, a synergistic effect can be achieved by combining a chat engine and an emotion engine.

The robot 100 also has the function of recognizing the actions of the user 10. The robot 100 recognizes the actions of the user 10 by analyzing facial images of the user 10 acquired using the camera function and the voice of the user 10 acquired using the microphone function. The robot 100 determines the actions to be performed by the robot 100 based on the recognized actions of the user 10, etc.

The robot 100 stores rules that define the actions that the robot 100 will take based on the emotions of the user 10, the emotions of the robot 100, and the actions of the user 10, and performs various actions in accordance with the rules.

Specifically, the robot 100 has reaction rules for determining the behavior of the robot 100 based on the emotions of the user 10, the emotions of the robot 100, and the behavior of the user 10. For example, the reaction rules define the behavior of the robot 100 as "laughing" when the user 10's behavior is "laughing." Furthermore, the reaction rules define the behavior of the robot 100 as "apologizing" when the user 10's behavior is "angry." Furthermore, the reaction rules define the behavior of the robot 100 as "answering" when the user 10's behavior is "asking a question." Furthermore, the reaction rules define the behavior of the robot 100 as "calling out" when the user 10's behavior is "sad."

When the robot 100 recognizes that the user's 10 behavior is "angry" based on the reaction rules, it selects the behavior of "apologizing" defined in the reaction rules as the behavior to be performed by the robot 100. For example, when the robot 100 selects the behavior of "apologizing," it performs the motion of "apologizing" and outputs a sound representing the word "apologize."

It is also defined that when the robot's 100 emotion is "normal" (i.e., "happy" = 0, "anger" = 0, "sad" = 0, "happy" = 0) and the condition that the user's 10 state is "alone and lonely" is met, the robot's 100 emotion will change to "worried" and it will be able to perform the action of "calling out."

When the robot 100 recognizes, based on the reaction rules, that its current emotion is "normal" and that the user 10 appears lonely, it increases the emotion value of the robot 100, "sad." Furthermore, the robot 100 selects the behavior of "calling out" defined in the reaction rules as the behavior to be performed toward the user 10. For example, when the robot 100 selects the behavior of "calling out," it converts the words "What's wrong?", which express concern, into a worried voice and outputs it.

The robot 100 also transmits user reaction information to the server 300 indicating that this action has elicited a positive reaction from the user 10. The user reaction information includes, for example, the user action of "getting angry," the robot 100 action of "apologizing," the fact that the user 10's reaction was positive, and the attributes of the user 10.

The server 300 stores the user reaction information received from the robot 100. The server 300 receives and stores user reaction information not only from the robot 100, but also from the robots 101 and 102. The server 300 then analyzes the user reaction information from the robots 100, 101, and 102 and updates the reaction rules.

The robot 100 receives the updated reaction rules from the server 300 by querying the server 300 for the updated reaction rules. The robot 100 incorporates the updated reaction rules into the reaction rules stored in the robot 100. This allows the robot 100 to incorporate the reaction rules acquired by the robots 101, 102, etc. into its own reaction rules.

Figure 2 shows an outline of the functional configuration of the robot 100. The robot 100 has a sensor unit 200, a sensor module unit 210, a storage unit 220, a user state recognition unit 230, a user emotion determination unit 231, a robot emotion determination unit 232, a behavior recognition unit 234, a behavior determination unit 236, a memory control unit 238, a behavior control unit 250, a control target 252, and a communication processing unit 280.

The control object 252 includes the display device, speaker, and eye LEDs mounted on the robot 100, as well as motors that drive the arms, hands, and feet. The posture and gestures of the robot 100 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 100 can be expressed by controlling these motors. In addition, the facial expressions of the robot 100 can also be expressed by controlling the illumination state of the LEDs in the eyes of the robot 100. The posture, gestures, and facial expressions of the robot 100 are examples of the attitude of the robot 100.

The sensor unit 200 includes a microphone 201, a 3D depth sensor 202, a 2D camera 203, and a distance sensor 204. The microphone 201 continuously detects sound and outputs audio data. The microphone 201 may be provided on the head of the robot 100 and may have a binaural recording function. The 3D depth sensor 202 detects the contours of an object by continuously emitting an infrared pattern and analyzing the infrared pattern from infrared images continuously captured by the infrared camera. The 2D camera 203 is an example of an image sensor. The 2D camera 203 captures images using visible light and generates visible light video information. The distance sensor 204 detects the distance to an object by emitting, for example, a laser or ultrasonic waves. The sensor unit 200 may also include a clock, a gyro sensor, a touch sensor, a sensor for motor feedback, etc.

Note that, of the components of the robot 100 shown in FIG. 2, the components excluding the control target 252 and the sensor unit 200 are examples of components that the behavior control system of the robot 100 has. The behavior control system of the robot 100 controls the control target 252.

The storage unit 220 includes reaction rules 221 and history data 222. The history data 222 includes the user 10's past emotional values and behavioral history. This emotional value and behavioral history is recorded for each user 10, for example, by being associated with the user 10's identification information. At least a portion of the storage unit 220 is implemented as a storage medium such as a memory. It may also include a person DB that stores the user 10's facial image, attribute information of the user 10, etc. Note that the functions of the components of the robot 100 shown in FIG. 2, excluding the control target 252, sensor unit 200, and storage unit 220, can be realized by the CPU operating based on a program. For example, the functions of these components can be implemented as CPU operations using operating system (OS) and programs running on the OS.

The sensor module unit 210 includes a voice emotion recognition unit 211, a speech understanding unit 212, a facial expression recognition unit 213, and a face recognition unit 214. Information detected by the sensor unit 200 is input to the sensor module unit 210. The sensor module unit 210 analyzes the information detected by the sensor unit 200 and outputs the analysis results to the user state recognition unit 230.

The voice emotion recognition unit 211 of the sensor module unit 210 analyzes the voice of the user 10 detected by the microphone 201 and recognizes the emotions of the user 10. For example, the voice emotion recognition unit 211 extracts features such as frequency components of the voice and recognizes the voice and emotions of the user 10 based on the extracted features. The speech understanding unit 212 analyzes the voice of the user 10 detected by the microphone 201 and outputs text information representing the content of the user 10's utterance.

The facial expression recognition unit 213 recognizes the facial expression and emotions of the user 10 from the image of the user 10 captured by the 2D camera 203. For example, the facial expression recognition unit 213 recognizes the facial expression and emotions of the user 10 based on the shape, positional relationship, etc. of the eyes and mouth.

The face recognition unit 214 recognizes the face of the user 10. The face recognition unit 214 recognizes the user 10 by matching a face image stored in a person DB (not shown) with the face image of the user 10 captured by the 2D camera 203.

The user state recognition unit 230 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 210. For example, it uses the analysis results of the sensor module unit 210 to perform processing mainly related to perception. For example, it generates perceptual information such as "Dad is alone" or "There is a 90% chance that Dad is not smiling." It then performs processing to understand the meaning of the generated perceptual information. For example, it generates semantic information such as "Dad is alone and looks lonely."

The user emotion determination unit 231 includes an emotion recognition engine 231a, and determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized state of the user 10 are input into a pre-trained neural network to obtain an emotion value indicating the emotion of the user 10.

Here, the emotion value indicating the emotion of user 10 is a value that indicates the positive or negative state of the user's emotion. For example, if the user's emotion is a cheerful emotion accompanied by a sense of pleasure or comfort, such as "joy," "pleasure," "comfort," "relief," "excitement," "relief," and "fulfillment," the value will be positive, and the more cheerful the emotion, the larger the value. If the user's emotion is a negative emotion, such as "anger," "sorrow," "discomfort," "anxiety," "sorrow," "worry," and "emptiness," the value will be negative, and the more unpleasant the emotion, the larger the absolute value of the negative value. If the user's emotion is none of the above ("neutral"), the value will be 0.

The robot emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230. The robot emotion determination unit 232 includes an endocrine control unit 232a and an emotion generation engine 232b. The endocrine control unit 232a adjusts the parameters of the neural network used in the emotion generation engine 232b using the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230. For example, the endocrine control unit 232a adjusts parameters corresponding to the amount of dopamine released. Dopamine is an example of an endocrine substance. Endocrine substances refer to substances secreted in the body that transmit signals, such as neurotransmitters and hormones. However, the endocrine substances of the robot 100 itself are a type of information that affects the behavior of the robot 100, and do not mean that the robot 100 actually produces endocrine substances. Note that the use of endocrine secretions to determine a robot's own emotions is publicly known, as disclosed in, for example, Japanese Patent Application Laid-Open No. 2018-81583, and therefore a detailed explanation will be omitted.

The emotion generation engine 232b uses a neural network to determine an emotion value that indicates the emotion of the robot 100 based on the information analyzed by the sensor module unit 210, the state of the user 10 recognized by the user state recognition unit 230, and parameters adjusted by the endocrine control unit 232a. The emotion value of the robot 100 includes emotion values for each of multiple emotion categories, and in this embodiment, values (0 to 5) indicating the strength of each of "joy," "anger," "sadness," and "happiness" are assumed.

To give a specific example, if the user state recognition unit 230 recognizes that the user 10 looks lonely, the endocrine control unit 232a controls to increase the parameter corresponding to the emotion value of "sadness" of the robot 100, causing the robot emotion determination unit 232 to increase the emotion value of "sadness" of the robot 100. Also, if the user state recognition unit 230 recognizes that the user 10 is smiling, the endocrine control unit 232a controls to increase the parameter corresponding to the emotion value of "joy" of the robot 100, causing the robot emotion determination unit 232 to increase the emotion value of "joy" of the robot 100.

The robot emotion determination unit 232 may determine the emotion value indicating the emotion of the robot 100 by further considering the state of the robot 100. For example, when the robot 100 has a low battery or when the environment surrounding the robot 100 is completely dark, the endocrine control unit 232a may perform control to increase a parameter corresponding to the emotion value of "sadness" of the robot 100, thereby causing the robot emotion determination unit 232 to increase the emotion value of "sadness" of the robot 100. Furthermore, when the user 10 continues to talk to the robot 100 despite a low battery, the endocrine control unit 232a may perform control to increase a parameter corresponding to the emotion value of "anger" of the robot 100, thereby causing the robot emotion determination unit 232 to increase the emotion value of "anger" of the robot 100.

The behavior recognition unit 234 recognizes the behavior of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized state of the user 10 are input into a pre-trained neural network, the probability of each of multiple predetermined behavior categories (e.g., "laughing," "angry," "asking a question," "sad") is obtained, and the behavior category with the highest probability is recognized as the behavior of the user 10.

As described above, in this embodiment, the robot 100 identifies the user 10 and then acquires the content of the user's utterance. However, when acquiring and using the content of the utterance, the robot 100 obtains the necessary consent from the user 10 in accordance with laws and regulations. Furthermore, the behavior control system of the robot 100 according to this embodiment takes into consideration the protection of the user's 10 personal information and privacy.

The behavior determination unit 236 determines a behavior corresponding to the behavior of the user 10 recognized by the behavior recognition unit 234 based on the current emotion value of the user 10 determined by the user emotion determination unit 231, history data 222 of past emotion values determined by the user emotion determination unit 231 before the current emotion value of the user 10 was determined, and the emotion value of the robot 100 determined by the robot emotion determination unit 232. In this embodiment, the behavior determination unit 236 describes a case in which the behavior determination unit 236 uses one most recent emotion value included in the history data 222 as the user 10's past emotion value, but the disclosed technology is not limited to this aspect. For example, the behavior determination unit 236 may use multiple most recent emotion values as the user 10's past emotion value, or may use an emotion value from a unit period ago, such as one day ago. Furthermore, the behavior determination unit 236 may determine a behavior corresponding to the user 10's behavior by further considering not only the robot 100's current emotion value but also the history of the robot 100's past emotion values. The behavior determined by the behavior determination unit 236 includes gestures performed by the robot 100 or speech content of the robot 100.

The behavior determination unit 236 according to this embodiment determines the behavior of the robot 100 as a behavior corresponding to the behavior of the user 10, based on a combination of the user 10's past and current emotional values, the emotional value of the robot 100, the user 10's behavior, and the reaction rules 221. For example, if the user 10's past emotional value is a positive value and the current emotional value is a negative value, the behavior determination unit 236 determines the behavior corresponding to the user 10's behavior as a behavior that will change the user 10's emotional value to a positive value.

The reaction rules 221 define the behavior of the robot 100 according to a combination of the user 10's past and current emotional values, the robot 100's emotional value, and the user 10's behavior. For example, if the user 10's past emotional value is a positive value and the current emotional value is a negative value, and the user 10's behavior is sad, the robot 100's behavior is defined as a combination of gestures and speech content when asking a question to encourage the user 10 with gestures.

For example, the reaction rule 221 defines behaviors of the robot 100 for all combinations of patterns of the robot 100's emotional values (1296 patterns, which are the fourth power of six values from "0" to "5" for "joy,""anger,""sadness," and "happiness"), patterns of combinations of the user 10's past emotional values and current emotional values, and behavioral patterns of the user 10. That is, for each pattern of the robot 100's emotional values, behaviors of the robot 100 are defined in accordance with the behavioral patterns of the user 10 for each of a plurality of combinations of the user 10's past emotional values and current emotional values, such as negative and negative values, negative and positive values, positive and negative values, positive and positive values, negative and normal values, and normal and normal values. Note that, for example, when the user 10 makes an utterance intending to continue a conversation from a past topic, such as "I want to talk about that topic we talked about last time," the behavior determination unit 236 may transition to an operation mode that determines the behavior of the robot 100 using the history data 222.
The reaction rule 221 may define at least one of a gesture and a statement as the behavior of the robot 100 for each of the patterns (1296 patterns) of the emotional value of the robot 100. Alternatively, the reaction rule 221 may define at least one of a gesture and a statement as the behavior of the robot 100 for each group of patterns of the emotional value of the robot 100.

The strength of each gesture included in the behavior of the robot 100 defined in the reaction rules 221 is predetermined. The strength of each utterance included in the behavior of the robot 100 defined in the reaction rules 221 is predetermined.

The memory control unit 238 determines whether or not to store data including the behavior of the user 10 in the history data 222 based on the predetermined behavior intensity for the behavior determined by the behavior determination unit 236 and the emotion value of the robot 100 determined by the robot emotion determination unit 232.
Specifically, if the total intensity value, which is the sum of the sum of the emotion values for each of the multiple emotion classifications of the robot 100, the predetermined intensity for the gestures included in the behavior determined by the behavior determination unit 236, and the predetermined intensity for the speech content included in the behavior determined by the behavior determination unit 236, is equal to or greater than a threshold value, it is determined that data including the behavior of the user 10 is to be stored in the history data 222.

When the storage control unit 238 decides to store data including the behavior of the user 10 in the history data 222, it stores in the history data 222 the behavior determined by the behavior determination unit 236, information analyzed by the sensor module unit 210 from the present time up to a certain period of time ago (for example, all surrounding information such as data on the sound, images, smells, etc. of the scene), and the state of the user 10 recognized by the user state recognition unit 230 (for example, the user 10's facial expression, emotions, etc.).

The behavior control unit 250 controls the control target 252 based on the behavior determined by the behavior determination unit 236. For example, when the behavior determination unit 236 determines an behavior that includes speaking, the behavior control unit 250 outputs sound from a speaker included in the control target 252. At this time, the behavior control unit 250 may determine the speaking speed of the sound based on the emotional value of the robot 100. For example, the behavior control unit 250 determines a faster speaking speed the higher the emotional value of the robot 100. In this way, the behavior control unit 250 determines the execution form of the behavior determined by the behavior determination unit 236 based on the emotional value determined by the robot emotion determination unit 232.

The behavior control unit 250 may recognize a change in the user 10's emotions in response to the execution of the behavior determined by the behavior determination unit 236. For example, the change in emotions may be recognized based on the user 10's voice or facial expression. Alternatively, the change in emotions may be recognized based on the detection of an impact by a touch sensor included in the sensor unit 200. If an impact is detected by a touch sensor included in the sensor unit 200, the behavior control unit 250 may recognize that the user 10's emotions have worsened, or if the detection result of the touch sensor included in the sensor unit 200 indicates that the user 10 is smiling or happy, the behavior control unit 250 may recognize that the user 10's emotions have improved. Information indicating the user 10's reaction is output to the communication processing unit 280.

Furthermore, after the behavior control unit 250 executes the behavior determined by the behavior determination unit 236 in an execution mode determined according to the emotion of the robot 100, it controls the robot emotion determination unit 232 to further change the emotion value of the robot 100 based on the user's reaction to the execution of the behavior. Specifically, the robot emotion determination unit 232 increases the emotion value of "joy" of the robot 100 when the user's reaction to the behavior determined by the behavior determination unit 236 being performed on the user in the execution mode determined by the behavior control unit 250 is not negative. Furthermore, the robot emotion determination unit 232 increases the emotion value of "sad" of the robot 100 when the user's reaction to the behavior determined by the behavior determination unit 236 being performed on the user in the execution mode determined by the behavior control unit 250 is negative.

Furthermore, the behavior control unit 250 expresses the emotion of the robot 100 based on the determined emotion value of the robot 100. For example, if the emotion value of "happiness" of the robot 100 is increased, the behavior control unit 250 controls the control object 252 to make the robot 100 perform a happy gesture. Furthermore, if the emotion value of "sadness" of the robot 100 is increased, the behavior control unit 250 controls the control object 252 to make the robot 100 assume a droopy posture.

The communication processing unit 280 is responsible for communication with the server 300. As described above, the communication processing unit 280 transmits user reaction information to the server 300. The communication processing unit 280 also receives updated reaction rules from the server 300. When the communication processing unit 280 receives updated reaction rules from the server 300, it updates the reaction rules 221.

The server 300 communicates between the robots 100, 101, and 102 and the server 300, receives user reaction information sent from the robot 100, and updates the reaction rules based on reaction rules that include actions that have received positive reactions. The functions of the server 300 may be implemented by one or more computers. At least some of the functions of the server 300 may be implemented by a virtual machine. At least some of the functions of the server 300 may be implemented in the cloud.

Figure 3 shows an example of an operational flow for determining the behavior of the robot 100. The operational flow shown in Figure 3 is executed repeatedly. At this time, it is assumed that information analyzed by the sensor module unit 210 has been input. Note that "S" in the operational flow indicates the step being executed.

First, in step S100, the user state recognition unit 230 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 210.

In step S102, the user emotion determination unit 231 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230.

In step S103, the robot emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230. The robot emotion determination unit 232 adds the determined emotion value of the user 10 to the history data 222.

In step S104, the behavior recognition unit 234 recognizes the behavior classification of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230.

In step S106, the behavior determination unit 236 determines the behavior of the robot 100 based on a combination of the current emotion value of the user 10 determined in step S102 and the past emotion values included in the history data 222, the emotion value of the robot 100, the behavior of the user 10 recognized by the behavior recognition unit 234, and the reaction rules 221.

In step S108, the behavior control unit 250 controls the control target 252 based on the behavior determined by the behavior determination unit 236.

In step S110, the memory control unit 238 calculates a total intensity value based on the predetermined action intensity for the action determined by the action determination unit 236 and the emotion value of the robot 100 determined by the robot emotion determination unit 232.

In step S112, the storage control unit 238 determines whether the total intensity value is equal to or greater than the threshold value. If the total intensity value is less than the threshold value, the processing ends without storing data including the user's 10's behavior in the history data 222. On the other hand, if the total intensity value is equal to or greater than the threshold value, the processing proceeds to step S114.

In step S114, the behavior determined by the behavior determination unit 236, the information analyzed by the sensor module unit 210 from the present time up to a certain period of time ago, and the state of the user 10 recognized by the user state recognition unit 230 are stored in the history data 222.

As described above, the robot 100 determines an emotion value indicating the emotion of the robot 100 based on the user state, and determines whether or not to store data including the behavior of the user 10 in the history data 222 based on the emotion value of the robot 100. This reduces the capacity of the history data 222 that stores data including the behavior of the user 10. For example, if the robot 100 determines that the user's state 10 years from now will be the same as it was 10 years ago, the robot 100 can read the history data 222 from 10 years ago and present the user 10 with all peripheral information, such as the state of the user 10 from 10 years ago (e.g., the user's facial expressions, emotions, etc.), as well as data on the sound, images, smells, etc. of the situation.

Furthermore, the robot 100 can be made to perform an appropriate action in response to the user's 10 actions. Conventionally, the user's actions were classified and actions, including the robot's facial expressions and posture, were determined. In contrast, the robot 100 determines the user's 10 current emotional value and performs an action on the user 10 based on the past and current emotional values. Therefore, for example, if the user 10 was cheerful yesterday but is depressed today, the robot 100 can utter an utterance such as, "You were cheerful yesterday, but what's up with you today?" The robot 100 can also utter an utterance using gestures. For example, if the user 10 was depressed yesterday but is cheerful today, the robot 100 can utter an utterance such as, "You were depressed yesterday, but you seem cheerful today, don't you?" For example, if the user 10 was cheerful yesterday but is more cheerful today than yesterday, the robot 100 can utter an utterance such as, "You're more cheerful today than yesterday. Has anything better happened than yesterday?" Furthermore, for example, when the emotion value is equal to or greater than 0 and the range of fluctuation in the emotion value continues to be within a certain range, the robot 100 can say something like, "You've been feeling stable lately, which is good."

For example, the robot 100 can ask the user 10, "Did you finish the homework you told me about yesterday?" and, if the user 10 replies, "Yes, I did," make a positive utterance such as "Great!" and perform a positive gesture such as clapping or a thumbs-up. For example, if the user 10 utters, "The presentation you gave the day before yesterday went well," the robot 100 can make a positive utterance such as "Good job!" and perform the above-mentioned positive gesture. In this way, the robot 100 can be expected to make the user 10 feel a sense of affinity with the robot 100 by performing actions based on the state history of the user 10.

In the above embodiment, the robot 100 recognizes the user 10 using a facial image of the user 10, but the disclosed technology is not limited to this. For example, the robot 100 may recognize the user 10 using a voice uttered by the user 10, the user 10's email address, the user 10's SNS ID, or an ID card with a built-in wireless IC tag that the user 10 possesses.

The robot 100 is an example of an electronic device equipped with a behavior control system. The application of the behavior control system is not limited to the robot 100, and the behavior control system can be applied to a variety of electronic devices. The functions of the server 300 may be implemented by one or more computers. At least some of the functions of the server 300 may be implemented by a virtual machine. At least some of the functions of the server 300 may be implemented in the cloud.

Figure 4 shows a schematic diagram of an example of the hardware configuration of a computer 1200 functioning as the robot 100 and the server 300. A program installed on the computer 1200 can cause the computer 1200 to function as one or more "parts" of an apparatus according to the present embodiment, or to perform operations or one or more "parts" associated with an apparatus according to the present embodiment, and/or to perform a process or steps of the process according to the present embodiment. Such a program may be executed by the CPU 1212 to cause the computer 1200 to perform specific operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

The computer 1200 according to this embodiment includes a CPU 1212, RAM 1214, and graphics controller 1216, which are interconnected by a host controller 1210. The computer 1200 also includes input/output units such as a communications interface 1222, a storage device 1224, a DVD drive 1226, and an IC card drive, which are connected to the host controller 1210 via an input/output controller 1220. The DVD drive 1226 may be a DVD-ROM drive, a DVD-RAM drive, or the like. The storage device 1224 may be a hard disk drive, a solid-state drive, or the like. The computer 1200 also includes a ROM 1230 and legacy input/output units such as a keyboard, which are connected to the input/output controller 1220 via an input/output chip 1240.

The CPU 1212 operates according to programs stored in the ROM 1230 and RAM 1214, thereby controlling each unit. The graphics controller 1216 acquires image data generated by the CPU 1212 into a frame buffer provided in the RAM 1214 or into the graphics controller itself, and causes the image data to be displayed on the display device 1218.

The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive 1226 reads programs or data from a DVD-ROM 1227 or the like and provides them to the storage device 1224. The IC card drive reads programs and data from an IC card and/or writes programs and data to an IC card.

ROM 1230 stores therein a boot program and the like that is executed by computer 1200 upon activation, and/or programs that depend on the hardware of computer 1200. I/O chip 1240 may also connect various I/O units to I/O controller 1220 via USB ports, parallel ports, serial ports, keyboard ports, mouse ports, etc.

The programs are provided by a computer-readable storage medium such as a DVD-ROM 1227 or an IC card. The programs are read from the computer-readable storage medium, installed in storage device 1224, RAM 1214, or ROM 1230, which are also examples of computer-readable storage media, and executed by CPU 1212. The information processing described in these programs is read by computer 1200, resulting in cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be configured by implementing the operation or processing of information in accordance with the use of computer 1200.

For example, when communication is performed between computer 1200 and an external device, CPU 1212 may execute a communication program loaded into RAM 1214 and instruct communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of CPU 1212, communication interface 1222 reads transmission data stored in a transmission buffer area provided in RAM 1214, storage device 1224, DVD-ROM 1227, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes received data received from the network to a reception buffer area or the like provided on the recording medium.

The CPU 1212 may also cause all or a necessary portion of a file or database stored on an external recording medium such as the storage device 1224, DVD drive 1226 (DVD-ROM 1227), IC card, etc. to be read into the RAM 1214, and perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write the processed data back to the external recording medium.

Various types of information, such as various types of programs, data, tables, and databases, may be stored on the recording medium and may undergo information processing. CPU 1212 may perform various types of processing on data read from RAM 1214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search/replacement, etc., as described throughout this disclosure and specified by the program's instruction sequence, and write the results back to RAM 1214. CPU 1212 may also search for information in files, databases, etc. on the recording medium. For example, if multiple entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored on the recording medium, CPU 1212 may search for an entry whose attribute value of the first attribute matches a specified condition from among the multiple entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

The programs or software modules described above may be stored on computer-readable storage media on or near computer 1200. Recording media such as a hard disk or RAM provided within a server system connected to a dedicated communications network or the Internet can also be used as computer-readable storage media, thereby providing the programs to computer 1200 via the network.

The blocks in the flowcharts and block diagrams in this embodiment may represent stages of a process in which an operation is performed or "parts" of a device responsible for performing the operation. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and/or a processor provided with computer-readable instructions stored on a computer-readable storage medium. Dedicated circuitry may include digital and/or analog hardware circuitry, and may include integrated circuits (ICs) and/or discrete circuits. Programmable circuitry may include reconfigurable hardware circuitry including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements, such as field programmable gate arrays (FPGAs) and programmable logic arrays (PLAs).

A computer-readable storage medium may include any tangible device capable of storing instructions executed by a suitable device, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture, including instructions that can be executed to create means for performing the operations specified in the flowchart or block diagram. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, etc. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disc (DVD), Blu-ray disc, memory stick, integrated circuit card, etc.

The computer-readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, JAVA, C++, etc., and conventional procedural programming languages such as the "C" programming language or similar programming languages.

The computer-readable instructions may be provided locally or over a wide area network (WAN) such as a local area network (LAN), the Internet, etc. to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, or to a programmable circuit, such that the processor or programmable circuit executes the computer-readable instructions to generate means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

(Determining emotions using emotion maps)
The user emotion determination unit 231 may determine the user's emotion in accordance with a specific mapping. Specifically, the user emotion determination unit 231 may use the emotion recognition engine 231a to determine the user's emotion in accordance with an emotion map (see FIG. 5), which is a specific mapping. Similarly, the robot emotion determination unit 232 may use the emotion generation engine 232b to determine the emotion of the robot 100 in accordance with the emotion map (see FIG. 5), which is a specific mapping. Note that when there is no need to particularly distinguish between the emotion recognition engine 231a and the emotion generation engine 232b, they will be collectively referred to as emotion engines.

Figure 5 shows an emotion map 300 on which multiple emotions are mapped. In emotion map 300, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both emotions and mental states. Emotions that are generally generated from reactions that occur in the brain are arranged on the left side of the concentric circles. Emotions that are generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions that are generally generated from reactions that occur in the brain and are induced by situational judgment are arranged above and below the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the top side of the concentric circles, and the emotion of "discomfort" is arranged on the bottom side. In this way, in emotion map 300, multiple emotions are mapped based on the structure by which emotions are generated, with emotions that tend to occur simultaneously being mapped close together.

(1) For example, if the emotion engine detects emotions at approximately 100 msec, the frequency with which the robot 100 determines its reaction behavior (e.g., a backchannel response) may be set at a timing at least as frequent as the emotion engine's detection frequency (100 msec), or may be set at a timing earlier than this. The emotion engine's detection frequency may be interpreted as the sampling rate.

By detecting emotions within about 100 ms and immediately performing a corresponding reaction (e.g., a nod), unnatural reactions are eliminated, enabling a natural, well-read dialogue. The reaction (e.g., a nod) is performed according to the directionality and degree (strength) of the mandala in the robot's 100 emotion map 300. The detection frequency (sampling rate) of the emotion recognition engine 231a is not limited to 100 ms, and may be changed depending on the situation (e.g., when playing sports), the user's age, etc.

(2) The directionality and intensity of emotions may be set in advance in reference to the emotion map 300, and the movement of the interjections and the strength of the interjections may be set. For example, if the robot 100 feels a sense of stability or security, the robot 100 may nod and continue listening. If the robot 100 feels anxious, confused, or suspicious, the robot 100 may tilt its head or stop shaking its head.

These emotions are distributed in the 3 o'clock direction on emotion map 300, and usually fluctuate between relief and anxiety. In the right half of emotion map 300, situational awareness takes precedence over internal sensations, resulting in a sense of calm.

(3) If the robot 100 feels good after being praised, the filler "ah" may be inserted before the line, and if the robot 100 feels pain after receiving harsh words, the filler "ugh!" may be inserted before the line. A physical reaction, such as the robot 100 crouching down while saying "ugh!", may also be included. These emotions are distributed around 9 o'clock on the emotion map 300.

(4) In the left half of the emotion map 300, internal sensations (reactions) take precedence over situational awareness. This can give the impression of an unconscious reaction.

When the robot 100 feels an internal sense (reaction) of satisfaction and also feels a favorable impression in its situational awareness, the robot 100 may nod deeply while looking at the other person, or may say "uh-huh." In this way, the robot 100 may generate behavior that shows a balanced favorable impression toward the other person, that is, acceptance and tolerance toward the other person. Such emotions are distributed around 12 o'clock on the emotion map 300.

Conversely, even when robot 100 is aware of an internal sensation (reaction) of discomfort, it may shake its head when it feels disgust, or turn the LEDs in its eyes red and glare at the other person when it feels hatred. These emotions are distributed around 6 o'clock on emotion map 300.

(5) The inside of emotion map 300 represents the mind, and the outside of emotion map 300 represents behavior, so the further out you go on emotion map 300, the more visible (expressed in behavior) the emotion becomes.

(6) When listening to someone while feeling a sense of security, which is distributed around 3 o'clock on the emotion map 300, the robot 100 may nod its head lightly and say "hmm," but when it comes to feelings of love, which are distributed around 12 o'clock, it may nod its head strongly.

The user emotion determination unit 231 inputs the information analyzed by the sensor module unit 210 and the recognized state of the user 10 into a pre-trained neural network, and uses the emotion recognition engine 231a to obtain emotion values indicating each emotion shown in the emotion map 300, thereby determining the emotion of the user 10. This neural network is pre-trained based on multiple pieces of training data that are combinations of the information analyzed by the sensor module unit 210, the recognized state of the user 10, and emotion values indicating each emotion shown in the emotion map 300. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in Figure 6. Figure 6 shows an example in which multiple emotions, such as "relieved," "calm," and "reassuring," have similar emotion values.

Meanwhile, the robot emotion determination unit 232 inputs the information analyzed by the sensor module unit 210, the state of the user 10 recognized by the user state recognition unit 230, and the state of the robot 100 into a pre-trained neural network, obtains emotion values indicating each emotion shown in the emotion map 300 using the emotion generation engine 232b, and determines the emotion of the robot 100. This neural network is pre-trained based on multiple learning data that are combinations of the information analyzed by the sensor module unit 210, the recognized state of the user 10, the state of the robot 100, and emotion values indicating each emotion shown in the emotion map 300. For example, the neural network is trained based on learning data indicating that when the robot 100 is recognized as being stroked by the user 10 based on the output of a touch sensor (not shown), the emotion value of "happy" is "3," and based on learning data indicating that when the robot 100 is recognized as being hit by the user 10 based on the output of an acceleration sensor (not shown), the emotion value of "anger" is "3." Furthermore, this neural network is trained so that emotions that are placed close to each other have similar values, as shown in the emotion map 900 in Figure 6.

The behavior determination unit 236 generates the robot's behavior by adding fixed sentences to ask about the robot's behavior corresponding to the user's behavior to text representing the user's behavior, the user's emotions, and the robot's emotions, and inputting this into the dialogue function.

For example, the behavior determination unit 236 uses an emotion table such as that shown in FIG. 7 to obtain text representing the state of the robot 100 from the emotion of the robot 100 determined by the robot emotion determination unit 232. Here, in the emotion table, an index number is assigned to each emotion value for each type of emotion, and text representing the state of the robot 100 is stored for each index number.

If the emotion of the robot 100 determined by the robot emotion determination unit 232 corresponds to index number "2", the text "very happy" is obtained. Note that if the emotion of the robot 100 corresponds to multiple index numbers, multiple pieces of text representing the state of the robot 100 are obtained.

An emotion table such as that shown in Figure 8 is also prepared for the emotions of the user 10. Here, if the user's behavior is "speaking good mood," the robot 100's emotion is index number "2," and the user 10's emotion is index number "3," the following is input to the chat engine: "The robot is in a very happy state. The user is in a normal happy state. The user spoke to the user saying 'good mood.' How would you respond as the robot?" and the robot's behavior content is obtained. The behavior decision unit 236 decides the robot's behavior from this behavior content.

In this way, robot 100 can change its behavior according to the index number that corresponds to the robot's emotions, which gives the user the impression that the robot has a heart and encourages them to take actions such as talking to the robot.

The behavior determination unit 236 may also generate the robot's behavior by adding text representing the contents of the history data 222, as well as text representing the user's behavior, the user's emotions, and the robot's emotions, and then adding a fixed sentence for asking about the robot's behavior corresponding to the user's behavior, and inputting this into a sentence generation model with a dialogue function. This allows the robot 100 to change its behavior in accordance with the history data representing the user's emotions and behavior, giving the user the impression that the robot has individuality and encouraging them to take actions such as talking to the robot. The history data may also include the robot's emotions and behavior.

(Example)
Next, an example in which the robot 100 of this embodiment is applied to a vending machine will be described. The sensor unit 200 of the robot 100 as a vending machine includes, for example, a camera and a microphone for interacting with a user and monitoring the user's condition and the surrounding environment. It also includes a thermometer, a hygrometer, and an anemometer for acquiring information about the surrounding temperature and weather. The control target 252 of the robot 100 also includes a speaker and a display for interacting with the user.

Next, the operation of the robot 100 will be explained using the flowchart in Figure 9. Through the following steps S1 to S5-1 and S5-2, the robot 100 suggests recommended products while conversing with the user 10, based on the user's preferences, situation, and reactions. Here, the explanation will be based on the assumption that the robot 100 is a vending machine for beverages (products).

(Step S1)
The robot 100 acquires the state of the user 10, the emotional value of the user 10, the emotional value of the robot 100, and the history data 222. Specifically, the same processes as in steps S100 to S103 above are performed to acquire the state of the user 10, the emotional value of the user 10, the emotional value of the robot 100, and the history data 222.

(Step S2)
The robot 100 acquires the user's beverage preferences. Specifically, the behavior determination unit 236 determines, as the behavior of the robot 100, utterances that ask the user 10 questions about their beverage preferences (e.g., "What would you like to drink today?", "Would you like it hot or iced?", etc.), and the behavior control unit 250 controls the control target 252 (speaker) to make utterances that ask these questions to the user 10. The robot 100 may also analyze the user's preferences not only from the user's answers to the questions but also from the user's facial expressions acquired by a camera and the user's tone of voice during the conversation. The user state recognition unit 230 recognizes the user's preferences based on information analyzed by the sensor module unit 210 (e.g., the user's answers).

(Step S3)
The robot 100 determines a product to recommend to the user. Specifically, the behavior determination unit 236 adds a fixed sentence, "What behavior would you recommend the user to do at this time?" to text representing the user's 10 beverage preferences, the user's emotions, the robot's 100's emotions, and the content stored in the history data 222, and inputs the resulting sentence into a sentence generation model to obtain recommended content related to behavior on the auction site. At this time, by taking into consideration not only the user's 10 beverage preferences but also the user's emotions and the history data 222, it is possible to suggest a product suitable for the user 10. Furthermore, by taking into consideration the robot's 100's emotions, it is possible to make the user 10 realize that the robot 100 has emotions.

The emotion value of the robot 100 can be determined based on an emotion map such as the example shown in Figure 5. The robot's emotion may fluctuate depending on the surrounding circumstances, the time of day, and the user's state and emotion. Specifically, if the user 10 is a regular customer, the robot's emotion may become positive and it may behave in a way that proactively suggests new products. Whether the user 10 is a regular customer may be determined based on the history data 222. Specifically, the history data 222 may store facial information and voice characteristics of customers who have made purchases, and the determination may be made by comparing this information. The more frequently the user 10 uses the vending machine, the more positive the robot 100 becomes, and the robot 100 may speak to the user 10 in a friendly tone, or may make an active utterance such as "Thank you for your continued patronage!" when the user 10 stops in front of the vending machine.

The robot 100 may also be configured to prefer offering hot drinks on cold days (e.g., when the ambient temperature is below 10 degrees) and cold drinks on hot days (e.g., when the ambient temperature is above 25 degrees).

Furthermore, if it is determined from the facial expression or tone of voice of the user 10 that the user's 10 emotions are positive, the robot 100 may also be made to be positive, and if it is determined that the user's 10 emotions are negative, the robot 100 may also be made to be negative, so as to synchronize with the user's emotions.

Furthermore, if multiple items in the vending machine are sold out or if many items are being heated or cooled, the robot 100's emotions may become negative.

The robot 100 may also acquire history data 222 and, if the user 10 is a regular customer, suggest products that the user has previously purchased. The history data may include information such as the date and time of purchase, the product purchased (hot or iced), and the temperature on that day. Note that, if the product previously purchased by the user 10 is currently being heated or cooled, other similar products may be suggested.

The robot 100 determines the product to suggest to the user 10 based on information obtained through dialogue with the user 10, the emotions of the user 10, the emotions of the robot 100, past history, etc. Specifically, the behavior decision unit 236 adds a fixed sentence, "What behavior would you recommend the user to do at this time?" to text representing the user 10's beverage preferences, the emotions of the user 10, the emotions of the robot 100, the surrounding situation, and the contents of the history data 222, and inputs this into a sentence generation model to obtain the recommended behavior. As a result, recommended behavior such as "I recommend the new product XX" or "I recommend the usual △△" can be obtained. In this way, by considering not only the information obtained through dialogue with the user, but also the user's emotions and the history data 222, it is possible to suggest behavior that is appropriate for the user.

Furthermore, by taking the robot's 100's emotions into consideration when making suggestions, the user 10 can sense that the robot 100 has emotions. For example, by suggesting a cold carbonated drink to a user 10 who always buys hot coffee, the user 10 can sense that the robot 100 has the emotion, "It's hot today, so I want to recommend a cold drink." Furthermore, by recommending a new product to a regular customer who comes every day, the user 100 can sense that the robot 100 has the emotion, "Since you always come, I want to recommend a new product."

(Step S4)
The robot 100 proposes the selected product to the user and acquires the user's reaction. Specifically, the behavior control unit 250 controls the control object 252 (speaker) to make a suggestion to the user, such as "How about our new product, XX Coffee?" The robot may also make a suggestion to the user, such as "XX Coffee is a charcoal-roasted coffee overseen by a coffee master at a famous coffee shop." Furthermore, the robot may recommend products based on the user's past purchase history as a regular customer. For example, if the user frequently purchases coffee drinks, the robot may make a suggestion such as "I think you like coffee, so I recommend our new product, XX Coffee, today." This allows the user 10 to feel that the suggestion was made taking into consideration the user's preferences and past purchases.

The user state recognition unit 230 recognizes the user's state based on the information analyzed by the sensor module unit 210, and the user emotion determination unit 231 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the user's state recognized by the user state recognition unit 230. The behavior determination unit 236 determines whether the user's reaction is positive or not based on the state of the user 10 recognized by the user state recognition unit 230 and the emotion value indicating the user's emotion, and determines whether to suggest a different behavior to the user 10 as the behavior of the robot 100.

(Step S5-1)
If the user 10 responds positively to the suggestion, the robot 100 confirms the action content. Whether the user 10 responded positively may be determined based on the state of the user 10 recognized by the user state recognition unit 230 (the content of the user's reply, facial expression, tone of voice, etc.) or the emotion of the user 10. If it is determined that the action content of the robot 100 is to be confirmed, the action control unit 250 may control the purchase button of the target product to blink, thereby supporting the user 10 in making a purchase.

(Step S5-2)
If the user 10's reaction is negative, the robot 100 determines another product to suggest to the user 10. Whether the user 10's reaction is negative may be determined based on the state of the user 10 recognized by the user state recognition unit 230 (such as the content of the user's 10 response, facial expression, and tone of voice) or the emotion of the user 10. When it is determined that another action content should be suggested to the user 10 as the action of the robot 100, the action determination unit 236 adds a fixed sentence, "Are there any other actions you would recommend to the user?" to the text representing the user's 10 beverage preferences, the user's 10 emotion, the robot's 100 emotion, and the content stored in the history data 222, and inputs the resulting sentence into the sentence generation model to obtain the recommended product. Then, the process returns to step S4, and the processes of steps S4 to S5-2 are repeated until a positive reaction is obtained from the user 10 and the suggested action content is confirmed.

As described above, the robot 100 can suggest appropriate products based on the preferences, situation, and reactions of the user 10. Furthermore, the content of the suggestions is also influenced by the emotions of the robot 100 itself, so the user 10 can feel as if the recommended products are being suggested by someone with emotions.

The above example assumes a beverage vending machine, but the present invention can be applied to vending machines for any product. It is also possible to record each user's purchased products in association with the user's attribute information (gender, age, etc.), and when deciding which products to suggest, suggest products that are frequently purchased by users with the same attributes.

In the above example, the robot 100 is implemented in a vending machine, but it may also be implemented in a humanoid robot or a stuffed toy robot. Alternatively, it may be applied to a control device connected wirelessly or by wire to a controlled device (speaker or camera) mounted on a humanoid robot or a stuffed toy robot.

In the case of a humanoid robot, the humanoid robot stands next to the vending machine and converses with the user 10 to suggest products. If the user 10 is a child who cannot reach the purchase button, the humanoid robot may operate the purchase button on their behalf. The humanoid robot may also take products dispensed from the vending machine and hand them to the user 10. A stuffed animal robot may be placed around the vending machine, such as on top of or beside it. In the case of a stuffed animal robot, the humanoid robot may also converse with the user 10 to suggest products.

The present invention has been described above using embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be clear to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the claims that such modifications and improvements can also be included within the technical scope of the present invention.

The order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before" or "prior to," and it should be noted that processes can be performed in any order unless the output of a previous process is used in a subsequent process. Even if the operational flow in the claims, specifications, and drawings is described using terms such as "first," "next," etc. for convenience, this does not mean that the processes must be performed in that order.

5 System, 10, 11, 12 User, 20 Communication network, 100, 101, 102 Robot, 200 Sensor unit, 201 Microphone, 202 Depth sensor, 203 Camera, 204 Distance sensor, 210 Sensor module unit, 211 Voice emotion recognition unit, 212 Speech understanding unit, 213 Facial expression recognition unit, 214 Face recognition unit, 220 Storage unit, 221 Response rules, 222 History data, 230 User state recognition unit, 232 Emotion determination unit, 234 Behavior recognition unit, 236 Behavior determination unit, 238 Memory control unit, 250 Behavior control unit, 252 Control target, 280 Communication processing unit, 300 Server, 1200 Computer, 1210 Host controller, 1212 CPU, 1214 RAM, 1216 Graphics controller, 1218 Display device, 1220 I/O controller, 1222 communication interface, 1224 storage device, 1226 DVD drive, 1227 DVD-ROM, 1230 ROM, 1240 I/O chip

Claims (3)

A robot behavior control system applied to a vending machine,
a user emotion determination unit that determines an emotion value indicating the emotion of the user;
a robot emotion determination unit that determines an emotion value indicating an emotion of the robot;
a behavior determination unit that determines a behavior of the robot based on at least one of the emotion value of the user and the emotion value of the robot, and information acquired by a sentence generation model having a dialogue function that allows the user and the robot to have a dialogue,
The behavior determination unit
The robot determines a product to be recommended to the user ;
The robot emotion determination unit
If the user is a regular customer,
Set positive emotions that suggest new products or products that match the weather conditions of the day.
The behavior determination unit
A behavior control system that suggests new products or products with temperatures that suit the weather conditions of the day, based on products previously purchased by the user . a user state recognition unit that recognizes the state of the user to whom the product is proposed,
The behavior determination unit
2. The behavior control system according to claim 1, wherein, when it is determined that the user's reaction is negative based on the recognized state of the user and the emotional value of the user determined by the user emotion determination unit, a different product is determined based on at least one of the emotional value of the user and the emotional value of the robot, and information acquired by the sentence generation model. 3. The behavior control system according to claim 1, wherein the robot is mounted on a stuffed toy or is connected wirelessly or by wire to a control target device mounted on the stuffed toy.