JP7446178B2 - Behavior control device, behavior control method, and program - Google Patents

Description

The present invention relates to a behavior control device, a behavior control method, and a program.

Today, smart speakers and communication robots are being developed. Such systems focus on functions such as turning lights on or off, accessing calendars, reading email, and setting appointments based on commands. In such a system, the input of instructions is limited to, for example, selection using a touch panel or predetermined voice commands, making it difficult to build relationships with people.

For this reason, a system that allows for relationships with people is desired. For example, Patent Document 1 proposes a system in which a companion device interacts with a person and a person is involved in interaction and operation with the device. In the technology described in Patent Document 1, a companion device detects the user's utterances and actions, expresses them through movement, graphics, sound, light, and fragrance, and provides a friendly presence.

And since robots operate in environments where humans live, in order for robots to become more useful, they need to be able to quickly learn from ordinary people through natural interactions between robots and humans. . Reinforcement learning from human evaluation feedback can make it easier for non-technical people to teach robots to do tasks. Learning from demonstration often leads to faster learning than evaluative feedback.

Special Publication No. 2019-521449

However, in conventional learning based on human evaluation feedback, the robot learns through trial and error, which can make the robot's learning dangerous and expensive. Furthermore, while learning from conventional robot demonstrations limits the trainee's performance, learning from human rewards can generally exceed the trainee's performance. For this reason, with conventional technology, it has been difficult to learn through interaction between robots and humans.

The present invention has been made in view of the above problems, and provides a behavior control device, a behavior control method, and a program that can autonomously learn through interaction between the device and a human. With the goal.

(1) In order to achieve the above object, a behavior control device according to one aspect of the present invention includes a learning unit that generates a reward function by inverse reinforcement learning based on a demonstrated result, a learning unit that generates a reward function, and a human An agent that selects an action based on information fed back from the environment.

(2) Furthermore, in the behavior control device according to one aspect of the present invention, the agent corrects the behavior using a reward function learned by the learning unit, and based on information fed back from the person and the environment, A predictive reward model may be learned.

(3) Furthermore, in the behavior control device according to one aspect of the present invention, the agent includes a reward learning management unit, an allocation evaluation unit, and a behavior selection unit, and the allocation evaluation unit is configured to receive information from the person. Based on the feedback and feedback from the environment, the probability of the previously selected action is calculated, and the state, action, probability of the previously selected action, and supervised learning sample are calculated, and the reward learning management unit calculates the probability of the previously selected action, and the reward learning management unit calculates the probability of the previously selected action obtaining the supervised learning sample output by the allocation evaluation unit, learning the predictive reward model, and updating the reward function using the learned predictive reward model; The behavior selection unit may select the behavior based on information fed back from the person and the environment and the reward learning management unit.

(4) In the behavior control device according to one aspect of the present invention, the agent may detect the direction of the person's voice, the direction of the person's face, the direction of the person's body, and the direction of the person's body in the current orientation of the device. By estimating the state of the environment represented by the orientation and selecting the action with the reward function with the largest predicted reward value, the action in which the device turns its face toward the person of interest may be selected.

ｈは任意の時間ステップｔでエージェントが受け取った人間の報酬ラベルであり、αは学習率であり、ｓは状態表現であり、ａは選択された行動であり、ω^→＝（ω_０，…，ω_ｍ－１）^Ｔは列パラメータベクトルであり、｛（ｓ_０，ａ_０），…，（ｓ_ｎ，ａ_ｎ）｝は状態・動作ペアであり、φ_ｉ（ｘ^→）を基底関数とするφ（ｘ^→）＝（φ_０（ｘ^→），…，φ_ｍ－１（ｘ^→））^Ｔであり、ｍはｉ＝０，…，ｍ－１でありパラメータの総数であり、δ_ｔは時間差誤差であり次式である、ようにしてもよい。

(5) Furthermore, in the behavior control device according to one aspect of the present invention, the reward learning management unit uses the calculated probability h^ and the state-behavior pair as a supervised learning sample, and calculates the least squares gradient. Based on this, we learn a function R^H (s, a) that approximates the expected value of the reward of the person receiving the interaction experience by updating the parameters using the following formula,

h is the human reward label received by the agent at any time step t, α is the learning rate, s is the state representation, a is the selected action, and ω ^→ = (ω ₀ ,... , ω _m−1 ) ^T is a column parameter vector, {(s ₀ , a ₀ ), ..., (s _n , a _n )} is a state-action pair, and φ _i (x ^→ ) is a basis function Let φ(x ^→ ) = (φ ₀ (x ^→ ), ..., φ _m-1 (x ^→ )) ^T , and m is the total number of parameters with i=0, ..., m-1, δ _t may be a time difference error expressed by the following equation.

(6) In order to achieve the above object, in the behavior control method according to one aspect of the present invention, the learning unit generates a reward function by inverse reinforcement learning based on the demonstrated result, and the agent generates a reward function using the reward function. and choose actions based on information fed back from people and the environment.

(7) In order to achieve the above object, a program according to one aspect of the present invention causes a computer to generate a reward function by inverse reinforcement learning based on the demonstrated result, and combines the generated reward function with a human and the behavior is selected based on information fed back from the environment.

According to (1) to (7), autonomous learning is possible through interaction between the device and a human. According to (1) to (7), the robot can learn from the demonstration and evaluation feedback provided by humans, and the number of human evaluations needed to obtain optimal behavior, especially the number of errors (expected). actions that are not performed) can be reduced.

FIG. 1 is a diagram schematically showing an autonomous learning method by a robot according to an embodiment. FIG. 1 is a block diagram showing a configuration example of a robot according to an embodiment. 1 is a diagram showing an example of the external shape of a robot according to an embodiment. FIG. 3 is a diagram for explaining the definition of face angles between a robot and a user. FIG. 3 is a diagram for explaining an action set of a robot according to an embodiment. FIG. 2 is a diagram for explaining an interactive reinforcement learning framework. FIG. 1 is a diagram schematically showing an IRL-TAMER framework according to an embodiment. FIG. 3 is a diagram illustrating an example of a processing algorithm performed by an agent according to an embodiment. FIG. 2 illustrates a framework for interactive RL from human social feedback according to an embodiment. FIG. 3 is a diagram illustrating a CNN model for real-time emotion classification. FIG. 3 is a diagram showing an example of visual display of real-time gesture recognition. FIG. 2 is a schematic diagram of feedback signals received by the TAMER agent from the real-time emotion recognition module and real-time gesture recognition. FIG. 3 is a diagram showing a screenshot of a LoopMaze task. FIG. 2 is a diagram showing a screenshot of a Tetris task. FIG. 7 is an experimental result for the LoopMaze task, and is a diagram showing the total number of time steps and the number of times feedback was received in each episode by the keyboard agent and the emotion agent. FIG. 7 is an experimental result for a Tetris task, showing the total number of time steps, the number of feedback received, and the number of cleared lines by the keyboard agent and emotion agent. FIG. 7 is an experimental result for the LoopMaze task, and is a diagram showing the number of time steps and the number of times feedback was received in each episode by the keyboard agent and gesture agent. FIG. 7 is an experimental result for a Tetris task, showing the number of time steps, the number of feedback received, and the number of cleared lines by the keyboard agent and gesture agent.

Embodiments of the present invention will be described below with reference to the drawings. Note that in the drawings used in the following explanation, the scale of each member is changed as appropriate in order to make each member a recognizable size.

<Summary>
FIG. 1 is a diagram schematically showing an autonomous learning method by a robot 1 according to the present embodiment. In the following embodiments, an example will be described in which the robot 1 autonomously learns the orientation of a human's face and acts based on the learned result. As will be described later, the robot 1 includes, for example, a photographing section and a sound collecting section, which is a microphone array, on the front surface. The robot 1 detects the human face, voice, and body orientation based on the information acquired by the imaging unit and the sound collection unit. The robot 1 acts using a learning model based on its state. The user observes the direction of the robot 1's face and provides rewards (evaluation feedback) and demonstrations (face direction, speech, facial expressions, etc.) to teach desired behavior.

In the example of FIG. 1, the robot 1 says to the user, "Where should I look?" The human Hu performs a demonstration such as turning his face in the viewing direction in response to the robot 1's utterances. Robot 1 observes the behavior of human Hu. The learning model detects the human Hu's face, voice, and body orientation, and learns the detected results as actions in response to the robot 1's utterances. Furthermore, the learning model receives the detected behavior as input and outputs the behavior. The robot 1 changes the direction of its face, for example, in accordance with the motion instructions output by the learning model. The learning model further learns this behavior. In addition, in the following example, an example in which the behavior control device is applied to the robot 1 will be described, but the application target is not limited to the robot 1.

<Configuration example of robot 1>
Next, a configuration example of the robot 1 will be explained.
FIG. 2 is a block diagram showing a configuration example of the robot 1 according to the present embodiment. As shown in FIG. 2, the robot 1 includes an operation section 101, a photographing section 102, a sensor 103, a sound collection section 104, a behavior control device 100, a storage section 106, a database 107, a display section 111, a speaker 112, an actuator 113, and a robot. A sensor 115 is provided.

The behavior control device 100 includes a recognition unit 105 (recognition means) and an agent 300.
The agent 300 includes a learning section 301 (inverse reinforcement learning section), a reward learning management section 302, an allocation evaluation section 303, and a behavior selection section 304.
The action selection section 304 includes an image generation section 3041, a sound generation section 3042, a drive section 3043, and an output section 3044.

<Function and operation of robot 1>
Next, the functions and operations of each functional section of the robot 1 will be explained with reference to FIG. 2.

The operation unit 101 is, for example, a keyboard. The operation unit 101 detects the result of an operation performed by a user, and outputs the detected operation result to the recognition unit 105.

The imaging unit 102 is, for example, a CMOS (Complementary Metal Oxide Semiconductor) imaging device, a CCD (Charge Coupled Device) imaging device, or the like. The photographing unit 102 outputs photographed images (still images, continuous still images, and moving images) to the recognition unit 105. Note that the robot 1 may include a plurality of imaging units 102. In this case, the imaging unit 102 may be attached to the front and rear of the housing of the robot 1, for example.

The sensor 103 is, for example, a motion sensor that detects movements such as gestures of the user, as described later. The sensor 103 outputs the detected value to the recognition unit 105.

The sound collection unit 104 is, for example, a microphone array composed of a plurality of microphones. The sound collection unit 104 outputs the acoustic signals collected by the plurality of microphones to the recognition unit 105. Note that the sound collection unit 104 may output each of the acoustic signals picked up by the microphone to the recognition unit 105 after being sampled with the same sampling signal and converting the analog signals into digital signals.

The robot sensor 115 is a gyro sensor that detects the inclination of the head or housing of the robot 1, an acceleration sensor that detects the movement of the head or housing of the robot 1, or the like. The robot sensor 115 outputs the detected value to the recognition unit 105.

The storage unit 106 stores, for example, items to be recognized by the recognition unit 105, various values (threshold values, constants) used during recognition, algorithms for recognition, and the like.

The database 107 stores, for example, a language model database, an acoustic model database, a dialogue corpus database, and acoustic features used in speech recognition, a comparison image database and image features used in image recognition, and the like. Note that each data and feature amount will be described later. Note that the database 107 may be placed on a cloud or may be connected via a network.

The recognition unit 105 extracts an image of the user's face from the photographed image and recognizes the user's facial expression using a well-known method, thereby recognizing the user's emotion. The recognition unit 105 recognizes the user's gestures by tracking the user's movements using a well-known method based on the detection value acquired by the sensor 103. The recognition unit 105 recognizes the content of the operation based on the operation result. The recognition unit 105 recognizes the direction of the user's voice by performing voice recognition processing on the user's voice signal from the collected acoustic signal. The recognition unit 105 detects the orientation, state, etc. of each part of the robot 1 based on the detection values detected by the robot sensor 115. The recognition unit 105 determines the direction of the user's voice based on the acquired information, as will be described later. The direction of the user's face, the direction of the user's body, and the direction of the robot 1's face are detected. The recognition unit 105 detects, recognizes, and outputs information on the results of the recognition to the agent 300.

The agent 300 detects, recognizes, and acquires information on the results of the recognition from the recognition unit 105. The agent 300 generates an agent using the generated reward function and the acquired information (information fed back from people and the environment), and uses the generated agent to perform actions (speech, gestures, image output, output). generate. The agent 300 also modifies its behavior using the reward function learned by the learning unit 301, and learns a predictive reward model based on information fed back from the person and the environment. The agent 300 estimates the state of the environment represented by the direction of the person's voice, the direction of the person's face, the direction of the person's body, and the orientation of the device in the current orientation of the device, and selects the one with the highest predicted reward value. By selecting an action that has a large reward function, the action that causes the device to turn its face (head) toward the person of interest is selected. Note that in the following description, the action selected by the agent is the action of turning the robot's face in the direction expected by the user, but the action selected is not limited to this.

The learning unit 301 performs learning using information on detection, recognition, and recognition results output by the recognition unit 105 to generate an agent. Furthermore, the learning unit 301 generates a reward function by inverse reinforcement learning based on the demonstrated results. Note that the learning method will be described later.

The reward learning management unit 302 acquires the reward function generated by the learning unit 301, acquires the supervised learning sample output by the allocation evaluation unit 303, learns a predictive reward model, and uses the learned predictive reward model. and update the reward function.

The assignment evaluation unit 303 calculates the probability of the previously selected action based on the feedback from the person and the environment, and uses the probability of the previously selected action as the state, the action, the probability of the previously selected action, and a supervised learning sample.

The behavior selection unit 304 selects a behavior based on information fed back from the person and the environment and the reward learning management unit 302. The selected action is at least one action such as image output, sound output, and driving of the head or the housing. In the embodiment, an action of changing the direction of the face (head) of the robot 1 will be explained as an example.

The image generation unit 3041 generates an output image (still image, continuous still image, or video) to be displayed on the display unit 111 based on the learned result and the acquired information, and displays the generated output image. It is displayed on the display unit 111.

The audio generation unit 3042 generates an output audio signal to be output to the speaker 112 based on the learned result and the acquired information, and causes the speaker 112 to output the generated output audio signal.

The drive unit 3043 generates a drive signal for driving the actuator 113 based on the learned result and the acquired information, and drives the actuator 113 with the generated drive signal.

The output unit 3044 generates, for example, an instruction based on the learned result and the acquired information, and outputs the generated instruction to the external device 2. The external device 2 is, for example, a personal computer, a game device, a tablet terminal, or the like.

The display unit 111 is a liquid crystal image display device, an organic EL (Electro Luminescence) image display device, or the like. The display unit 111 displays the output image output by the image generation unit 3041.

The speaker 112 outputs the output audio signal output by the audio generation section 3042.

The actuator 113 drives the operating section according to a drive signal output by the drive section 3043.

<Example of external shape of robot 1>
Next, an example of the external shape of the robot 1 will be explained.
FIG. 3 is a diagram showing an example of the external shape of the robot 1 according to the present embodiment. In the example of the front view g101 and side view g102 of FIG. 3, the robot 1 includes three display units 111 (111a, 111b, 111c). Further, in the example of FIG. 3, the photographing section 102a is attached to the top of the display section 111a, and the photographing section 102b is attached to the top of the display section 111b. The display units 111a and 111b correspond to human eyes and present image information. The speaker 112 is attached to the housing 120 near a display section 111c that displays an image corresponding to a human mouth. The sound collection section 104 is attached to the housing 120.

The robot 1 also includes a boom 121. The boom 121 is movably attached to the housing 120 via a movable part 131. A horizontal bar 122 is rotatably attached to the boom 121 via a movable part 132.
Furthermore, the display section 111a is rotatably attached to the horizontal bar 122 via a movable section 133, and the display section 111b is rotatably attached via a movable section 134.
Note that the outer shape of the robot 1 shown in FIG. 3 is an example, and is not limited to this. For example, the robot 1 may be a bipedal robot.

<Face angle between robot and human>
Here, the face angle between the robot and the user will be explained.
FIG. 4 is a diagram for explaining the definition of the face angle between the robot 1 and the user.
The state expression used in the embodiment is composed of four feature quantities. The four feature quantities are user Hu's voice direction, user Hu's face direction, user Hu's body direction, and robot 1's face direction.

The diagram of the area indicated by the symbol g151 is a diagram for explaining the voice direction of the user Hu. The first feature amount is the user Hu's voice direction α _a (angular range is [-π, π]). The user Hu's voice direction α _a is the angle of the user Hu's voice direction a with respect to the front direction α of the robot 1 . Note that the voice direction is detected by, for example, voice recognition processing.

The diagram of the area indicated by the symbol g161 is a diagram for explaining the direction of the user Hu's face. The second feature amount is the orientation α _f of the user Hu's face (the angular range is [-π, π]). The user Hu's face direction α _f is the angle of the user Hu's face direction f with respect to the front direction α of the robot 1 . Note that the direction of the user Hu's face is detected, for example, by image processing. This feature amount is used to reinforce the feature of the user Hu's voice direction.

The diagram of the area indicated by the symbol g171 is a diagram for explaining the orientation of the user Hu's body. The third feature amount is the orientation α _b of the user Hu's face (the angular range is [-π, π]). The user Hu's body orientation α _b is the angle of the user Hu's body orientation b with respect to the front direction α of the robot 1 . Note that the direction of the user Hu's face is detected, for example, by image processing. This feature amount is used to complement the feature amount of the user Hu's voice direction and face direction.

The diagram of the area indicated by the symbol g181 is a diagram for explaining the direction of the face of the robot 1. The fourth feature amount is the orientation θ _c of the face of the robot 1 (the angular range is [-π, π]). The direction of the face of the robot 1 θ _c is the angle from the direction of the face of the robot 1 to the direction of the position of the person. The facial orientation θ _c of the robot 1 is acquired by the behavior control device 100 based on the motion instruction.

The actions of the robot 1 can be a list of angular commands that the robot 1 can perform. In the embodiment, as shown in FIG. 5, an action set [-φ _a ,0,φ _a ] for tilting the behavior of the robot 1's face direction is used. Note that φ _a is a positive angle with a small degree. FIG. 5 is a diagram for explaining the action set of the robot 1 according to this embodiment.

When the robot 1 selects the φ _a command, it means that the robot 1 moves the current face direction by an angle φ _a to the left. When the robot 1 selects the -φ _a command, it means that the robot 1 moves the current face direction by an angle ψ _a to the right. If robot 1 selects the 0 command, it means that robot 1 remains in the current face orientation.

The user's evaluation feedback regarding the facial movements of the robot 1 will be explained. In this embodiment, while the robot 1 is learning from user Hu's evaluation feedback, the user Hu transmits the evaluation of the robot 1's behavior as a voice reply and maps it to a numerical reward value.

Here, we define a set of feedback used in the embodiment. In the embodiment, feedbacks such as "fairly good", "good", "bad", and "fairly bad" are defined and mapped to +2, +1, -1, and -2, respectively. For example, if the user Hu thinks that the action selected by the robot 1 is correct, the user Hu answers "good", which is mapped to +1. Furthermore, if the user Hu thinks that the motion selected by the robot 1 is of higher quality, the user Hu answers "quite good", which is mapped to +2.

<Interactive reinforcement learning>
First, an overview of interactive reinforcement learning will be explained.
In standard reinforcement learning, an agent interacts with its environment to learn how to perform sequential decision-making tasks. This sequential decision task is modeled as a Markov decision process and is called {S, A, T, R, γ}. S and A are sets of possible states and actions, respectively. T is a transition function T: S×A×S→R (set of all real numbers), which gives the probability of transitioning to state s _t+1 under state s _t and action. γ determines the present value of future rewards and is called the discount rate. R is a reward function and is T:S×A×S→R (set of all real numbers). The reward is a function of s _t , _at and s _{t +1} or only s _t , at. For agent learning, there are typically two relevant value functions.

The first value function is the state value function V ^π (s), which is related only to the initial state s of the agent under the policy π as shown in equation (1) below.

The second value function is an action/value function called the state/action pair value Q ^π (s, a), and is the expected return after taking action a in state s in the following equation (2).

Interactive reinforcement learning is a variation of standard reinforcement learning. In interactive reinforcement learning, the reward signal is not only based on the state of the world and the agent's actions, but also on real-time interaction with a human trainer. In this case, the trainer can change the value of the reward signal by providing a clear goal state, or interact continuously in a continuous process. In interactive reinforcement learning, as shown in Figure 6, each time an agent performs an action in a state, the trainer provides evaluation feedback that tells the agent the quality of the selected action based on the trainer's experience. FIG. 6 is a diagram for explaining the framework of interactive reinforcement learning.

<Learning framework>
In contrast, this embodiment uses an agent based on the TAMER (Training an Agent Manually via Evaluative Reinforcement) framework. The TAMER framework is an approach that myopically learns by directly modeling human rewards. Here, "myopic" means that the agent only takes immediate rewards into account, ie, sets the discount factor γ to zero.

As shown in Figure 7, the learning mechanism is the interaction of humans, the environment, and the TAMER agent. In the TAMER framework, a human teacher observes the agent's behavior and rewards it based on its quality assessment.
First, an outline of the learning framework will be explained using FIG. 7. FIG. 7 is a diagram schematically showing the IRL-TAMER framework according to this embodiment. IRL-TAMER used in this embodiment learns from human performance using inverse reinforcement learning (IRL), and learns from human rewards using TAMER. IRL-TAMER consists of two algorithms as shown in FIG. 7, and is executed in the following order.

- Step 1: The IRL learns the reward function from the demonstration provided by the trainer (block 201 on the left). In step 1, reverse reinforcement learning is performed from the demon station. Note that the processing of this block is performed by the learning unit 301.

- Step 2: TAMER learns a predictive reward model from human rating feedback (block 311 on the right). In step 2, the TAMER agent learns from the evaluation feedback. Note that the process of this block 311 is performed by the agent 300.

<Step 1; IRL algorithm>
Next, the algorithm of step 1 will be explained.
A human teacher demonstrates a sequence of state-action pairs. The state and action pairs include {(s ₀ , a ₀ ), ..., (s _n , a _n )}. Here, (s ₀ , a ₀ ) indicates the state and behavior at the beginning. (s _n , a _n ) indicates the state and action at the end. Note that the person who performs the motion on the robot 1 is a person other than the trainer. Then, the robot 1 records the demonstration. Here, the state s represents the above-mentioned four feature quantities (person's voice direction α _a , face direction α _f , body direction α _b , and robot's face direction θ _c ) as feature variables. Further, the action is one of the action set [-φ _a , 0, φ _a ].

The IRL algorithm 211 provides the recorded demonstration to the inverse reinforcement learning module as a reward function R=ω·φ(s) (212). Here, ω is a parameter weight vector, and φ(s) is a vector of base features on the state of the IRL.

The reward function R learned from the demonstration via the IRL algorithm 211 is used to initialize the weight vector w of the reward function R _H of TAMER. This allows the trainee to fine-tune the motion of the robot 1 using the human evaluation feedback h.

<Step 2; TAMER agent learning algorithm>
Next, the learning algorithm of the TAMER agent in step 2 will be explained. As shown in FIG. 7, the agent 300 includes a reward learning management section 302, an allocation evaluation section 303, and a behavior selection section 304.

First, an outline of the processing of the agent 300 (hereinafter also referred to as TAMER agent 300) using TAMER will be explained.
The TAMER agent 300 determines, in the current face orientation θ _c of the robot 1, the voice direction α _a of the person (human Hu), the human face orientation α _f , the human body orientation α _b , and the robot face orientation θ Estimate the state s of the environment represented by _c . Then, the TAMER agent 300 uses the action selection unit 304 to select action a (angle command) having the reward function R _H (s, a).
Note that the action selection unit 304 maximizes the reward for the human Hu due to the immediate action of the robot 1 by selecting the action for which the human's predicted reward value is the largest.

The trainer (human Hu) observes the state s of the robot 1 and the selected motion φ _a (environment 312), evaluates its quality, and provides feedback. The allocation evaluation unit 303 acquires the evaluation h fed back in this way.

The assignment evaluation unit 303 receives the evaluation feedback h given by the trainer and calculates the probability (credit) h of the previously selected action. The assignment evaluation unit 303 is used to deal with a time delay in human reward due to evaluating the behavior of the robot 1 and giving the reward. The allocation evaluation unit 303 is trained to predict a human reward model R^ _H , and provides a reward to the agent within an MDP (Markov decision process) specified as {S, A, T, R^ _H , γ}. .

Specifically, the allocation evaluation unit 303 uses the following equation (3) to learn a function R^ _H (s, a) that approximates the expected value of the reward of the person receiving the interaction experience. Note that S is a set of states in the environment, and A is a set of actions that the agent 300 can execute.

In ^equation (3), here, ω ^→ = (ω ₀ , ..., ω _m-1 ) ^T is a column parameter vector, and _φ (x ^→ ) = (φ ₀ (x ^→ ), ..., φ _m-1 (x ^→ )) ^T , where m is i=0, ..., m-1 and is the total number of parameters.

The TAMER agent 300 learns the human user's reward function and attempts to maximize the human's reward by argmaxaR^ _H (s,a). In the TAMER agent, the optimal policy is defined by the human user.
Compared to other methods of learning from human rewards, TAMER has the following three features.

I. In TAMER, units are assigned to accommodate delays in human evaluation.
II. The TAMER agent learns a human reward model (R^H).
III. At each time step, the TAMER agent chooses the action (argmaxaR^ _H (s,a)) that is predicted to directly derive the maximum reward, without considering the effects on future states.

Specifically, when a human observes an agent's behavior, the human brain requires a certain reaction time to issue a corresponding feedback signal. However, during this time, the agent may have already started a new exploration, causing some delay in human feedback. To solve this problem, the agent needs to distribute each human reward signal to multiple recent state-action pairs that contribute to the label of the state-action pair for each sample. TAMER uses the established regression algorithm R _H :S×A→R to simulate a hypothetical human reward function. Note that the TAMER framework does not include any specific models or supervised learning algorithms for approximating the human reward function; all decisions are made, for example, by the designer. Also, in the AMER framework, the label for a state-behavior sample is . Everything is made up of human rewards. Furthermore, when an action selection is made under state s, the TAMER agent directly adopts the strategy a=argmaxaR^ _H (s, a). Such myopic learning corresponds to performing reinforcement learning with a discount coefficient γ=0.

In the TAMER framework, the role of agents is somewhat unspecified in nature. That's because it must learn from human rewards to maximize performance on tasks that cannot be measured.
To this end, embodiments hypothesize that the agent will best fulfill its role by learning a model of the trainer's rewards and selecting the behavior that the model predicts will lead to the most rewards. erected.

The reward learning management unit 302 acquires the calculated probability h^, the state representation s, and the selected action a as one supervised learning sample (s, a, h^), and calculates the reward function R using a supervised learning algorithm. Learn _H (s, a). The reward learning management unit 302 updates R _H (s, a) with (s, a, h^).
The reward learning management unit 302 is a credit granting device for dealing with a time delay in receiving a person's reward due to evaluating the behavior of the agent 300 and providing it. TAMER defines a probability density function f(t) to estimate the probability of a teacher's feedback delay. The probability density function f(t) provides the probability that feedback occurs within any particular time interval, and to calculate the probability (credit) that a single reward signal is targeted for a single time step used. At the current time step t, the probability of each previous time step tk is calculated as follows (4).

Note that when humans give multiple rewards, the label h^ of each previous time step (state-action pair) is the sum of all probabilities calculated for each person's reward using equation (4). .
The reward learning management unit 302 uses h^ and the state-behavior pair as a supervised learning sample, and updates the parameters as shown in equations (5) and (6) based on the least squares gradient. ^ Learn _H (s, a). Note that in equations (5) and (6), α is the learning rate and δ _t is the time difference error.

Note that in equations (5) and (6), h is the human reward label received by the agent at any time step t.

The behavior selection unit 304 selects another behavior (proposed angle command) φ _a using the updated reward function R _H (s, a). The trajectory generated through human demonstration and planning consists of a sequence of state-action pairs {(s ₀ , a ₀ ), ..., (s _n , a _n )}, which are processed using the inverse RL algorithm. supplied to The agent 300 learns the human reward function and determines the behavior of the robot 1 that maximizes the human reward by argmax _a R^ _H (s, a) using, for example, the following equation (7) or the following equation (8). and select.

The agent 300 takes actions, detects rewards, updates the predictive reward function model, and starts a new cycle until the robot 1 learns the optimal action. Note that the selected action is, for example, the expected direction of the robot 1's face. This allows the robot 1 to face the person Hu who is paying attention.

<Processing algorithm>
Next, an example of a processing algorithm performed by the agent will be explained. FIG. 8 is a diagram illustrating an example of a processing algorithm performed by an agent according to this embodiment.

- Procedure 1: The agent 300 initializes the reward function R, the human reward function R^ _H , the action value function Q(s, a), or the state value function V(s).

・Step 2: A human performs a demonstration. The agent 300 obtains and records the demonstration performed.

- Step 4, Step 5: The agent 300 records the demo, generates a trajectory through planning, and optimizes the reward function R through inverse RL. The trajectory generated through human demonstration and planning consists of a sequence of state-action pairs {(s ₀ , a ₀ ), ..., (s _n , a _n )}, which are processed using the inverse RL algorithm. supplied to

- Step 6: The agent 300 uses the reward function R learned from the demonstration through inverse RL (reinforcement learning) to seed the human reward function R^ _H in TAMER.

- Step 8: The trainer can modify the agent's 300 policy by providing human evaluation feedback. The agent 300 determines whether or not the person's reward h has been received.

- Steps 9 and 10: For example, when the agent 300 receives a human reward, the agent 300 uses the received reward to update the human reward function R^ _H .

- Step 11, Step 12, Step 13: The agent 300 selects and executes one action with the reward function R.

- Step 14: The agent 300 repeats until the trainer is satisfied with the behavior of the agent 300.

<Framework for interactive RL from human social feedback>
In the example below, rather than having the user manipulate controls to train the agent using explicit feedback signals, human social signals, such as facial expressions and gestures, can be used to communicate between the agent and the human. Provide to the process of interaction between users.
As a result, according to the present embodiment, a user who has no experience in agent training can train an agent based on the agent's behavioral preferences without learning complicated training rules. Furthermore, according to the present embodiment, it is possible to provide an approach that allows feedback to be provided in accordance with the user's expectations.

FIG. 9 is a diagram illustrating a framework for interactive RL from human social feedback according to this embodiment. As shown in Figure 9, human social feedback is trained in two ways.
The first method is to train the agent using facial feedback. The second method is to directly introduce gesture feedback instead of keyboard feedback.

The agent 300 learns human Hu's rewards from facial and gesture signals and obtains an optimal policy. In this embodiment, a module 401 that realizes real-time face recognition with the imaging unit 102, a module 403 that realizes online gesture recognition with a motion sensor, and a module that maps human feedback to reward signals through socket communication between different processes ( 401 to 403, agent 300), and includes a plurality of modules with different functions. In this embodiment, real-time face recognition is introduced into the TAMER agent that constitutes the emotional agent. Furthermore, in this embodiment, a gesture agent is realized by combining an online gesture recognition module with a TAMER Agent. Note that the motion sensor is a sensor that tracks hands and fingers, for example.

First, real-time emotion recognition will be explained.
Facial expressions are the result of the movement or state of one or more facial muscles. These movements express the observer's personal emotions. And facial expressions are a form of nonverbal communication. Facial expressions are the main means of expressing social information between humans and are usually used to convey emotions. In this embodiment, a convolutional neutral network (CNN) framework as shown in FIG. 10, for example, is used to design real-time emotion recognition. FIG. 10 is a diagram showing a CNN model for real-time emotion classification. The framework of FIG. 10 is a fully convolutional neural network with four residual depth-separable convolutions for the acquired image 501. A batch normalization operation (502, 503, 504, 505, 506, 509) and a ReLU activation function are connected to each convolution. The last layer (507, 510, 511) applies global average pooling and softmax activation functions to generate predicted values.

The face recognition module of this embodiment recognizes, for example, seven emotions: "happy," "sad," "angry," "scared," "surprised," "neutral," and "disgusted." can. In addition, in the experiment, emotions were classified into "happy (positive)" emotions and "unhappy (negative)" emotions. In the embodiment, "anger," "sadness," and "fear" are labeled as "unhappy" emotions. In the experiment, users directly observed the behavior of agent 300 and communicated their preferences using expressions such as "happy" and "unhappy." Note that the "happy (positive)" facial expression is, for example, a smiling facial expression, and the "unhappy (negative)" facial expression is, for example, an angry facial expression.

In this embodiment, instead of continuously giving reward signals for each agent's state or action, feedback can be given at the trainer's discretion when the trainer feels it is necessary.
In the experiment, since facial expression recognition is continuous, the interval between capturing facial expressions was set to 2 seconds to make it easier for the module to operate at the same speed as the agent, and facial feedback was extracted. In addition, in the experiment, if the user did not want to provide feedback, the user was "neutral" or in a place where the imaging unit 102 could not take pictures, so that the agent did not receive feedback.

Next, online gesture recognition will be explained.
Gestures are a natural way of communicating in everyday life, and are often used, for example, by people who are deaf or have speech disabilities. From the perspective of human-computer interaction, gestures as a communicative language are useful for users with speech impairments using service robots, underwater work where it is inconvenient to use dedicated input devices, and noise that greatly interferes with verbal communication. It has a very wide range of applications, such as environments with many environments.
In the embodiment, three types of gestures are recognized using a motion sensor.

The first model is "Easy Gesture Play Input," which detects one neutral state and five basic gestures and labels them using simple signal processing. The neutral state is "Yes" and is represented by, for example, a stationary thumb raised. In the embodiment, basic gestures include "No" (e.g. thumbs down), "Great" (e.g. thumbs flipped up), "Stop" (waving hand), and "Left swipe" (e.g. "Right swipe" (for example, swing sharply to the right).

The second model uses machine learning algorithms to recognize and classify gestural activity to trigger a response on the robot 1 (eg, telepresence or social robot).

The third model is a Kinematic Feature Mapping Control Input that maps gesture movements to the robot based on ultrasound principles.

In the experiment, based on the gesture recognition experiment conducted in advance using the first model, "Great" was selected to provide positive feedback, and "Stop" was selected to express negative feedback to the agent. I made it possible to select. FIG. 11 is a diagram illustrating a visual display example of real-time gesture recognition. Reference numeral g201 in FIG. 11 indicates an example of a "Great" gesture, and reference numeral g202 in FIG. 11 indicates an example of a "Stop" gesture.

Next, socket communication used in the embodiment will be explained.
In the experiment, the TAMER agent was operated using Java (registered trademark) script, and the real-time facial expression recognition module and online gesture recognition module were implemented using Python (registered trademark). Note that the implementation may use other programming languages or scripts.

In the experiment, the socket method for the TCP communication mechanism shown in FIG. 12 was adopted in order to achieve safe and reliable data transfer between two processes. FIG. 12 is a schematic diagram of feedback signals received by the TAMER agent from the real-time emotion recognition module and real-time gesture recognition. An outline of the socket communication used in the experiment will be explained with reference to FIG.
In the process of data transmission, the agent is the server side, and the real-time recognition module is the communicating client. The client passes all recognized results to the server without delay, but the server selectively receives data. In the experiment, the server was configured to receive and read data from the client's output stream every two seconds.

<Experiment>
Next, an example of the results of an experiment using the agent of this embodiment will be described.
In the experiment, a human (for example, a user) observes the actions performed by the robot 1 in the current environment. Then, the user gives positive feedback if the user agrees with the action selected by the agent of the robot 1, and gives negative feedback if the user thinks that the selected action does not meet expectations. The user trained the agent by providing different facial expressions or gestures as reward signals compared to predefined keypad feedback.

In the experiment, the TAMER agent was trained with three types of feedback.
The first, a comparative example, is a keyboard agent that learns using explicit feedback from the keyboard. The second is an emotional agent that learns using facial expressions. The third is a gesture agent that provides feedback using gestures.
In the experiment, three agents were tested using two reinforcement learning benchmark tasks (LoopMaze and Tetris (registered trademark)). In addition, in the experiment, each agent was trained 10 times for each task. Note that the experimental results are the average value of data collected in 10 trials.

For the keyboard agent, humans observed the agent's actions and provided feedback by pressing designated keyboard keys. In the experiment, pressing the v key increased the reward by +1, pressing the n key decreased the reward by -1, and the reward values could be stacked by clicking the key many times.

For the emotional agent, humans observed the agent's performance during the first two episodes and trained it to learn the first policy using keyboard feedback. In the experiment, facial feedback was introduced into the learning model starting from the third episode. If the action selected by the agent is as expected, the human will show a smiling face and output a positive feedback signal to the agent. If the agent's chosen action is unsatisfactory, humans output negative feedback signals to the agent by expressing emotions such as anger, fear, and sadness. Note that the emotional agent learns from the expressions to further adjust the initial policy.

In the gesture agent, a human observes the agent's actions and provides feedback using two types of gestures (“Great” and “Stop”). If the action selected by the agent was as expected, the human would give a thumbs up and bounce to express "Great". When humans thought that the agent's chosen action was inappropriate, they would wave their hands and say "Stop" as a negative feedback signal to the agent.

[LoopMaze]
In an experiment using the maze game LoopMaze, the training session was stopped when the agent reached the goal 25 times (i.e., 25 episodes), and the maximum training time step for each episode was set to 2000. FIG. 13 is a diagram showing a screenshot of the LoopMaze task. In FIG. 13, symbols g311 and g312 represent walls, g321 represents an agent, and g322 represents the moving direction of the agent.

The LoopMaze task includes 30 states. In a task, an agent in each state can move up, down, left, or right, and if an action selected in a certain state causes the agent to hit a wall, no movement will occur. The agent's goal is to learn as quickly as possible the optimal policy that leads from the starting state g301 to the goal state g302. The shortest path from the starting state to the goal state requires 19 actions. The actions available to an agent in LoopMaze depend on the target state and its position relative to the last selected action.

[Tetris]
In Tetris, a falling object puzzle, there are ^2,200 states, and four adjacent blocks are selected at once to form a Tetris piece. The action set has four options: down, left, right, and rotation. When Tetris pieces line up in a row, the blocks filling that row are automatically eliminated. For Tetris tasks, the best policy is to line up the falling blocks in a line, constantly clearing out rows, and eventually letting the game run indefinitely. Therefore, within the limits, the more rows an agent can clear in one episode, the better the performance of this policy.

In the experiment, 20 episodes of training were performed using Tetris. In one episode, the upper limit number of time steps was set to 10,000, and a new episode was started until one training session was completed. In the experiment, in order to compare the performance of the keyboard agent and the emotional agent, we analyzed the mean and variance of the number of execution time steps and the total amount of feedback in each episode. FIG. 14 is a diagram showing a screenshot of a Tetris task.

In the Tetris task, the human provided feedback (positive or negative) about the placement of Tetris pieces in the previous step while any Tetris pieces were falling. Also, when a falling block completely falls, the square part turns black, but we have made sure that the feedback given at this time works most effectively.

[Experimental result]
First, we will explain the experimental results of learning from facial feedback and gesture feedback in both the LoopMaze task and the Tetris task.
In an agent without emotional feedback, a user's preferences can only be realized through the reward process of pressing a keyboard key. In contrast, according to the present embodiment, when the correct action is selected, the user simply appears happy, and when the unexpected action is selected, the user appears sad or angry. people can express their feelings. A satisfied agent can be obtained simply by expressing preferences through facial expressions, and the cognitive burden on the user (for example, speaking) can be significantly reduced. In the experiment, the agent was trained to obtain an initial policy using keyboard feedback, then improve the formed policy by introducing facial feedback, and finally obtain the optimal policy. The reason for conducting the experiment in this way is that the facial expression recognition rate of the agent used in the experiment was 66%, and a stable policy could not be learned.
Note that in LoopMaze, the learning performance of two agents (keyboard agent and emotion agent) is evaluated using two indicators: the number of time steps performed by the agent and the total number of feedback provided by the human trainer.

FIG. 15 is an experimental result for the LoopMaze task, and is a diagram showing the total number of time steps and the number of times feedback was received in each episode by the keyboard agent and emotional agent.
Note that each plot was created by averaging data obtained from 10 independent runs, with each run consisting of 25 episodes. In this experiment, we evaluated the learning performance of the two agents using two metrics: the number of time steps performed by the agents and the total number of feedback provided by the human trainer. As for the results obtained, as learning progresses, it is desirable to gradually reduce the number of time steps from the initial state to the target state and to reduce the number of times feedback is received. Note that if the agent is perfect, it will reach the goal state in 20 time steps per episode without giving any human-generated rewards.

Graph g401 is a diagram showing the total number of time steps in each episode by the keyboard agent and emotional agent to complete one episode of the LoopMaze task. Graph g411 is a diagram showing the number of times feedback is received in each episode from the keyboard agent and emotional agent when the user successfully navigates from the start state to the goal state. In graph g401 and graph g411, the horizontal axis is the number of episodes, in the graph g401, the vertical axis is the total number of time steps (times), and in graph g411, the vertical axis is the number of times (times) feedback was received. Furthermore, symbol g402 indicates the experimental results for the keyboard agent, symbol g403 indicates the experimental results for the emotion agent, and symbol g404 indicates the standard error of the mean value.

As shown in FIG. 15, at the initial stage when facial expression feedback was introduced, the number of time steps to reach the target state and the amount of feedback slightly increased. However, after three or four training sessions, the number of time steps and amount of feedback for the emotion agent basically match those for the keyboard agent, and both agents have learned almost optimal policies. As shown in Figure 15, after the learning period, the emotional agent was able to learn similar or better policies than the keyboard agent.

FIG. 16 shows the experimental results for the Tetris task, showing the total number of time steps, the number of feedback received, and the number of cleared lines by the keyboard agent and emotional agent.
In the Tetris task, agents are generated using random combinations of four small squares, and the time and probability of each combination are uncertain. The better the agent is at this task, the longer it will run and the more states and actions it can experience. Under human training, the agent can eventually learn to eliminate entire rows of Tetris pieces and continue playing the game. Therefore, the expectation for this task is that the number of time steps and the number of Tetris lines eliminated gradually increases while requiring a reduction in human feedback. In addition, in the experiment, the agent was trained using keyboard feedback in the first two episodes, and facial feedback was introduced in the third episode, for a total of 20 episodes of training. In the experiment, we evaluate the learning performance of two agents (keyboard agent and emotion agent) using three indicators. The three measures are the number of steps in the agent's execution time, the number of feedback received, and the number of Tetris lines eliminated within the episode.

Graph g451 is a diagram showing the number of time steps in each episode by the keyboard agent and the emotional agent in each episode of the experiment. Graph g461 is a diagram showing the number of times feedback was received in each episode from the keyboard agent and the emotional agent in each episode of the experiment. Graph g471 is a diagram showing the number of lines read by the keyboard agent and emotion agent in each episode of the experiment.
In the graphs with symbols g451, g461, and g471, the horizontal axis is the number of episodes, in the graph with symbol g451, the vertical axis is the number of time steps (times), and in the graph with symbol g461, the vertical axis is the number of times feedback was received (times). In the graph with symbol g471, the vertical axis is the number of cleared lines (rows).

As shown in Figure 16, in the first two episodes, the number of time steps, the number of feedback received, and the number of line clears are almost the same between the emotion agent and the keyboard agent, which is a reasonable number that can clear dozens of lines in the game. acquired the policy. After 7 episodes of training after 4 episodes of training, the number of time steps and the number of cleared lines of the emotional agent were similar to those of the keyboard agent, although the number of feedbacks increased slightly.

Next, we compared the learning performance of the keyboard agent and gesture agent in the LoopMaze task and Tetris task.
FIG. 17 is an experimental result for the LoopMaze task, and is a diagram showing the number of time steps and the number of times feedback was received in each episode by the keyboard agent and gesture agent.
Graph g501 is a diagram showing the number of time steps in each episode by the keyboard agent and gesture agent to finish one episode of the LoopMaze task. A graph g511 is a diagram showing the number of times feedback is received in each episode by the keyboard agent and the gesture agent. In the graph g501 and the graph g511, the horizontal axis is the number of episodes, the vertical axis in the graph g501 is the number of time steps (times), and the vertical axis in the graph g511 is the number of times (times) feedback is received. Further, symbol g502 indicates the experimental result of the keyboard agent, symbol g503 indicates the experimental result of the gesture agent, and symbol g504 indicates the standard error of the mean value.

In this experiment, 10 independent rounds of experiments were conducted, with 20 episodes running in each round. Similar to learning from facial feedback, in the LoopMaze task, the number of time steps and the total number of feedback provided by the human user are indicators of the agent's performance. Gesture feedback is expected to reduce the amount of feedback an agent receives during training.

As shown in Figure 17, in the first three episodes, the gesture agent needs to experience more time steps than the keyboard agent. However, gesture agents receive significantly less feedback than keyboard agents. After three episodes of training, both agents obtained the best policy.

Next, we compared the learning performance of gesture agents and keyboard agents in the Tetris task. Here, we evaluate the learning performance of the two agents using the same three metrics as for learning from facial feedback: the number of steps in the agent's execution time, the number of feedback received, and the number of Tetris rows eliminated within an episode. was evaluated.

FIG. 18 is an experimental result for the Tetris task, and is a diagram showing the number of time steps, the number of feedback received, and the number of cleared lines by the keyboard agent and gesture agent. Note that in the experiment, we repeated 10 independent rounds, running 20 episodes in each round, and each episode measured by the number of time steps, the number of total feedback received, and the number of Tetris lines eliminated. performance was averaged.

Graph g551 is a diagram showing the number of time steps in each episode by the keyboard agent and gesture agent in each episode of the Tetris task. Graph g561 is a diagram showing the number of times feedback is received in each episode by the keyboard agent and the gesture agent. Graph g571 is a diagram showing the number of lines read in each episode by the keyboard agent and gesture agent.
In the graphs with symbols g551, g561, and g571, the horizontal axis is the number of episodes, in the graph with symbol g551, the vertical axis is the number of time steps (times), and in the graph with symbol g561, the vertical axis is the number of times feedback was received (times). In the graph with symbol g571, the vertical axis is the number of cleared lines (rows).

As shown in Figure 18, in the first four episodes, the gesture agent received much less gesture feedback than the keyboard agent, took slightly fewer time steps, and received slightly fewer lines in each episode. It has been cleared. Even after that, the gesture agent received less feedback than the keyboard agent. After nine episodes, the learning performance of both agents was quite similar, and the number of feedbacks required to adjust the learned policy was also comparable.

This means that agents can better understand human intent from gestures, even without predefined keystrokes, and can learn from limited or fewer feedback to provide information. can be captured accurately. In other words, according to experiments using this embodiment, it was confirmed that by using human social signals, it is possible to effectively improve the learning efficiency of an agent that requires little human feedback (sufficient recognition accuracy). did it. And, as shown in our experiments, agents that learn from gesture feedback can achieve similar performance to agents that learn from keyboard feedback, but receive much less feedback. Furthermore, the experimental results show that when learning from facial feedback, it is possible to obtain performance similar to learning from keyboard feedback.

Additionally, the above experimental results show that using human social signals (e.g., facial expressions, gestures) effectively improves the learning efficiency of agents with little human feedback (with sufficient recognition accuracy). This eliminates the need for the user to learn the learning method in advance, and can reduce the user's cognitive burden and workload. As a result, according to the present embodiment, there is no need for humans to learn training methods in advance, thereby providing a natural method for ordinary people to train agents on how to perform tasks according to their preferences. Can be done.

Note that in the above-mentioned experimental results, an example in which an emotional agent and a gesture agent are used as an example of the agent is explained, but the present invention is not limited to this. Agents may also use other humans' social signals. The agent may be, for example, a voice emotion agent that uses voice emotions.

As described above, in this embodiment, the robot learns from human demonstrations via IRL and then from human rewards via TAMER. This IRL-TAMER consists of two algorithms that are executed in the following order: I and II.
I. IRL learns the reward function from demonstrations provided by human trainers,
II. TAMER learns predictive reward models from human rating feedback.
Note that the feedback provided by human demonstrations includes recognition of human facial expressions through images, recognition of human gestures, and the like.

As a result, according to the present embodiment, it is possible to enable the robot to autonomously learn behavior from both demonstrations and evaluation feedback provided by humans (for example, users, trainers).
As a result, the present embodiment allows the robot to learn from the demonstration and evaluation feedback provided by the human, and the number of human evaluations required to obtain optimal behavior, in particular the number of errors (expected behavior) can be reduced.

Note that although the embodiment has been described using the robot 1 as an example, the agent and the like can also be applied to other devices, such as an in-vehicle navigation device, a smartphone, a tablet terminal, and the like. For example, when applied to a smartphone, a still image of the robot 1 as shown in FIG. 3 may be displayed on the display section of the smartphone. Alternatively, the gestures of the robot 1 may be displayed as an animation on the display section of the smartphone.

Note that a program for realizing all or part of the functions of the behavior control device 100 according to the present invention is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed. By doing so, all or part of the processing performed by the behavior control device 100 may be performed. Note that the "computer system" herein includes hardware such as an OS and peripheral devices. Furthermore, the term "computer system" includes a WWW system equipped with a home page providing environment (or display environment). Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording medium" refers to volatile memory (RAM) inside a computer system that serves as a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. This also includes programs that are retained for a certain period of time.

Further, the program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in a transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. Moreover, the above-mentioned program may be for realizing a part of the above-mentioned functions. Furthermore, it may be a so-called difference file (difference program) that can realize the above-described functions in combination with a program already recorded in the computer system.

Although the mode for implementing the present invention has been described above using embodiments, the present invention is not limited to these embodiments in any way, and various modifications and substitutions can be made without departing from the gist of the present invention. can be added.

DESCRIPTION OF SYMBOLS 1...Robot, 101...Operation unit, 102...Photography unit, 103...Sensor, 104...Sound collection unit, 100...Behavior control device, 106...Storage unit, 107...Database, 111...Display unit, 112...Speaker, 113... Actuator, 115... Robot sensor, 105... Recognition unit, 300... Agent, 301... Learning unit, 302... Reward learning management unit, 303... Allocation evaluation unit, 304... Behavior selection unit, 3041... Image generation unit, 3042... Sound generation section, 3043...drive section, 3044...output section, 211...IRL algorithm

Claims (7)

The demonstrated results of the sequence of pairs of states and actions of the trainer are given as a reward function to an inverse reinforcement learning module by an inverse reinforcement learning algorithm; a learning section that learns a reward function;
The agent selects an action that has a reward function using the weight vector as an initial value, and the trainer evaluates the state of the self-apparatus and the selected action. Calculate the probability, learn the approximate expected value of the reward value of the person receiving the interaction experience, learn to maximize the reward function of the person, and based on the learned result, calculate the predicted reward value of the person An agent that selects the action with the largest
A behavior control device comprising: The agent is
modifying the behavior using a reward function learned by the learning unit;
learning a predictive reward model based on feedback information from the trainer ;
The behavior control device according to claim 1. The agent includes a reward learning management section, an allocation evaluation section, and an action selection section,
The assignment evaluation unit calculates the probability of the previously selected action based on the feedback from the trainer , and sets the state, the action, the probability of the previously selected action, and a supervised learning sample;
The reward learning management unit acquires the reward function generated by the learning unit, acquires the supervised learning sample output by the allocation evaluation unit, learns the predictive reward model, and calculates the learned prediction. updating the reward function using a reward model;
The behavior selection unit selects the behavior based on information fed back from the trainer and the reward learning management unit.
The behavior control device according to claim 2. The agent is
Estimates the state of the environment expressed by the direction of the person's voice, the direction of the person's face, the orientation of the person's body, and the orientation of the device in the current orientation of the device, and selects a reward function with the largest predicted reward value. By selecting an action to hold, the device selects an action to turn its face toward the person of interest.
The behavior control device according to any one of claims 1 to 3. The reward learning management unit uses the calculated probability h^ and the state-behavior pair as a supervised learning sample, and updates the parameters using the following equation based on the least squares gradient to create an interaction experience. Learn a function R^H (s, a) that approximates the expected value of the reward of the recipient,

h is the human reward label received by the agent at any time step t, α is the learning rate, s is the state representation, a is the selected action, and ω ^→ = (ω ₀ ,... , ω _m−1 ) ^T is a column parameter vector, {(s ₀ , a ₀ ), ..., (s _n , a _n )} is a state-action pair, and φ _i (x ^→ ) is a basis function Let φ(x ^→ ) = (φ ₀ (x ^→ ), ..., φ _m-1 (x ^→ )) ^T , and m is the total number of parameters with i=0, ..., m-1,
δ _t is the time difference error and is the following formula,

The behavior control device according to claim 3. a learning unit, wherein the demonstrated result of the sequence of pairs of states and actions of the trainer is given as a reward function to the inverse reinforcement learning module by an inverse reinforcement learning algorithm; learning the reward function including a weight vector ;
The agent selects an action that has a reward function using the weight vector in the agent as an initial value, and uses the previous evaluation to which the trainer evaluates the state of the self-apparatus and the selected action as feedback. calculates the probability of the selected action, learns the approximate expected value of the reward value of the recipient in the interaction experience, learns to maximize the reward function of said person, and based on the learned result, select the action with the highest predicted reward value ,
Behavioral control methods. to the computer,
The demonstrated result of the sequence of trainer state and action pairs is given as a reward function to the inverse reinforcement learning module by an inverse reinforcement learning algorithm, and the demonstrated result is passed through the inverse reinforcement learning algorithm from the demonstrated result to the inverse reinforcement learning module. Learn the reward function ,
The agent selects an action that has a reward function using the weight vector as an initial value, and the trainer evaluates the state of the self-apparatus and the selected action. Calculate the probability, learn the approximate expected value of the reward value of the person receiving the interaction experience, learn to maximize the reward function of the person, and based on the learned result, calculate the predicted reward value of the person Let them choose the action with the largest
program.

Images