JP3465236B2 - Robust reinforcement learning method - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for learning a behavior rule that is robust against environmental uncertainty in reinforcement learning. More specifically, robots,
Control of physical systems such as automobiles and aircraft, information retrieval, user response, resource allocation, on behalf of human beings
It is a reinforcement learning method and a reinforcement-learned controller capable of coping with various environmental disturbances and deviations between the assumed model and the actual environment in a computer program that conducts market transactions and the like. In this specification, in order to deal with characters that cannot be used in the patent specification, the following measures are taken.

[Outer 1]

[0002]

2. Description of the Related Art A conventional reinforcement learning method will be described.
FIG. 2 is a block diagram of a circuit used in the conventional learning method. FIG. 3 is a block diagram of feedback between the controlled object and the controller, and is a diagram for explaining the H∞ control theory. The environment 1 represents a general system that is an object of action including a controlled object, and outputs a state signal x (t) to the action generator 2 and the state evaluator 3. The action generator 2 outputs the action signal u (t) to the environment 1 when the state signal x (t) is input. Further, the reward signal r (t) is input from the environment 1 to the state evaluator 3. The state evaluator 3 is an expected value of the weighted sum of the rewards r (t), which is obtained from the current state x (t) for the future when the reward r (t) corresponding to the achievement level of the goal is input. That is, the evaluation function V (x (t)) is predicted, the predicted value is used to generate a prediction error signal δ (t), and the prediction error signal δ (t) is output to the behavior generator 2. When the prediction error signal δ (t) is input from the state evaluator 3, the action generator 2 expects the weighted sum of the rewards r (t) obtained from the current state x (t) for the future. That is,
Learning is performed so that the evaluation function V (x (t)) is maximized, and the input / output relationship is changed. However, the evaluation function V (x
(t)) is continuous,

[Equation 1] In this learning method, the optimal behavior is learned in one environment, but the behavior is not guaranteed in another environment. Further, in order to adapt to a different environment, it is necessary to re-learn, and a new re-learning time is required.

Next, conventional H∞ control will be described.
In FIG. 3, the observed signal y (t) from the controlled object G is the controller
Input to K. The observation signal y (t) is input to the controller K.
Then, the action signal u (t) which is a control signal is output to the controlled object G.
Force In addition, the disturbance signal w (t) and the line
Motion signal u (t) is input and these signals are input
And output the evaluation signal z (t) and the observation signal y (t).
It The evaluation signal z for evaluating the influence of disturbance
(t) and the input to the controller K by observing the controlled object G
It is the same as the observed signal y (t) which is the feedback signal
It is possible, but it is also possible to make them different.
And H∞ control, which is a typical formulation of robust control
The problem request is unknown in the feedback system shown in Fig. 3.
Evaluation by disturbance signal w (t) caused by disturbance or model error
Stabilizes the output while suppressing the influence on the signal z (t) for use
To do. That is, approach the evaluation signal z (t) = 0.
Is. Specifically, the H∞ norm causes system disturbance.
The sensitivity is measured to be less than the robustness standard value γ
Such a controller K is designed. The norm is a kind of large
It is an index of the strength, and the disturbance signal w (t) is used to evaluate the evaluation signal z
Transfer function matrix to (t) is T _ZWAnd the H ∞ nor
Mu, ‖T_ZW‖∞ is defined as the following (Equation 2).

[Equation 2] However, sup _w represents the upper limit for the disturbance signal w (t),
When the disturbance signal w (t) is changed, ‖z‖ ₂ / ‖w‖
₂ indicates that it does not become larger than _{sup w (‖z‖ 2 / ‖w‖ 2} ). Further, ‖z‖ ₂ and ‖w‖ ₂ are L ₂ norms of the evaluation signal z (t) and the disturbance signal w (t), respectively, and are defined by the following (Equation 3) and (Equation 4).

[Equation 3]

[Equation 4]

Here, the evaluation function V is defined as in the following (formula 5).

[Equation 5] This is maximized for the action signal u (t), and the disturbance signal w
Consider the minimization problem for (t). As a result, V
If a solution satisfying ≧ 0 is obtained, the evaluation signal z (t) can be stabilized under the condition of (Equation 2).

Among the tasks of reinforcement learning, the task of which the reward signal is the proximity to a predetermined target point or target trajectory can be considered as a task of learning control. In the H∞ control, which is a typical method of the robust control, the susceptibility to the influence of the system disturbance is evaluated by the H∞ norm, and a controller that suppresses the H∞ norm of the feedback system below a certain level is designed. , Guarantees robustness against disturbances and model errors. However, the analytical construction method of the controller is shown for the linear system, and for the nonlinear system, there is generally no method for constructing the controller analytically except for a limited system. In the nonlinear system, the method of approximating the state-value function considering the unknown disturbance by using the multilayer neural network and realizing the robust control has been proposed, but these are the regions where the adaptive range of the controller can be linearly approximated. Limited to nearby areas and learning limited to offline batch learning. Moreover, the model of the environment is required for these learnings. Furthermore, a method of discretizing states and using dynamic programming has also been proposed, but the process of discretizing states and the process of off-line calculation are required to construct a controller, and an environmental model is also required. And Further, the robust controller described so far functions only as a regulator (control in which the target point is the origin and the state of the controlled object is brought to the origin).

By the way, in the conventional reinforcement learning method using the minimax strategy, learning of Othello and backgammon is performed. There is also an example applied to the flight follow-up problem of fighters and missiles. On the other hand, the present invention is different in that the controller is configured excluding the disturbance generator when the controller is actually applied after learning is performed assuming a virtual enemy. This relies on the new idea of constructing robust controllers by reinforcement learning by treating disturbances as virtual enemies. In addition, (Expression 8-2) described later
Unlike the method in which a robust behavioral rule is obtained by reinforcement learning by weighting the prediction error of the evaluation function defined by, and learning, the present invention has a clear relation with H∞ control and an objective function. . Therefore, in terms of seeking the worst disturbance, it is possible to more reliably obtain a robust controller.

As will be described later, according to the embodiment of the present invention, the reinforcement learning method, which is a nonlinear, online, model-independent dynamic optimization method, is used to minimize the sensitivity to the worst disturbance in robust control. A robust reinforcement learning method is realized by introducing the principle of, and by applying it to the control task, a nonlinear, online, model-independent robust learning control is realized. Even if the approximate model of the environment is known or unknown, the model-dependent reinforcement learning method is expanded and applied by learning the model of the environment to efficiently construct a nonlinear robust controller online. can do.

The advantage of being able to do this is that if the environment is non-stationary, if a nonlinear robust controller or environment model constructed by learning is used, it can be dealt with without re-learning if the environment changes to some extent. That is what you can do. Further, the conventional robust control has been formulated only for the regulator problem that causes the output to converge to a certain target value, but the robust reinforcement learning method of the present invention can be applied to any evaluation criterion.

[0009]

As described above, in the conventional reinforcement learning, it is necessary to re-learn when the environment changes, and the conventional robust control method is used when the environment model is unknown. Couldn't handle. further,
No specific method of control law applicable to nonlinear systems is given.

Therefore, an object of the present invention is to provide a robust reinforcement learning method and a robust controller which can cope with a case where an environment model is unknown and which can be applied to a non-linear system and which is resistant to a change in environment. .

[0011]

Therefore, the means for solving the problems adopted by the present invention is a behavior generator that outputs a behavior signal to a controlled object or environment and has a learning function, and a disturbance signal to the controlled object or environment. Is provided with a disturbance generator having a learning function and outputs an evaluation signal that is a reward signal in which the reward according to the degree of achievement of the target is added to the reward according to the ability to withstand the disturbance from the disturbance generator. The behavior generator learns to maximize (or minimize) the expected sum of the weighted sums of the evaluation signals generated from the current state to the future, while the disturbance generator generates the sum of weights of the evaluation signals. Is a reinforcement learning method characterized by learning to minimize (or maximize) the expected value of, and in the learning method, the expected value of the sum of evaluation signals obtained from the current state to the future is calculated. A state evaluator for measurement, and a prediction error signal thereof is used for learning of at least one of a state evaluator, a behavior generator, and a disturbance generator. At least one of the behavior generator and the disturbance generator is, as a function approximation means,
A reinforcement learning method characterized by comprising a reference table showing an input / output relationship, wherein at least one of the state evaluator, the action generator and the disturbance generator is a linear model or a polynomial model as a function approximation means. It is a reinforcement learning method characterized by comprising, at least one of the state evaluator, the action generator and the disturbance generator,
A reinforcement learning method characterized by comprising a multi-layer neural network as a function approximating means.
It is a control method using only the state evaluator and the action generator that have been learned in advance or the action generator, the method is applied to an environment model realized by computer simulation, and the state evaluator learned by it A behavioral generator or a control method characterized in that only the behavioral generator is applied to a real environment, at least one of the state evaluator or the behavioral generator, as a function approximating means, a reference table showing an input / output relationship. A robust controller characterized in that at least one of the state evaluator or the behavior generator comprises a linear model, a polynomial model or a multilayer neural network as a function approximating means. It features a robust controller.

[0012]

BEST MODE FOR CARRYING OUT THE INVENTION Next, an embodiment of a reinforcement learning method and a robust controller according to the present invention will be described. FIG. 1 is a block diagram of a circuit used in the robust reinforcement learning method according to the embodiment of the present invention. FIG. 4 is an explanatory diagram of the first specific example.
(A) is a schematic diagram and (b) is a graph of the angle change of a pendulum having a length of 1 m. FIG. 5 is a graph of an angle change of the first specific example, (a) is a graph of a pendulum having a length of 0.5 m,
(B) is a graph of a pendulum having a length of 2 m. FIG. 6 is a schematic view of the second specific example. 7A and 7B are graphs of the position change of the second specific example, where FIG. 7A is a graph when a conveyed product having a mass of 1 kg is used, and FIG. 7B is a graph when a conveyed product having a mass of 3 kg is used.

According to the present invention, in order to realize robust reinforcement learning against disturbances and changes in the environment, the reward s corresponding to the ability to withstand the disturbances is given to the reward r (t) according to the achievement level of the target. Evaluation signal q (t) which is a new reward with (t) added
When is defined as

[0014]

[Equation 6] The maximum / minimum problem with the evaluation signal q (t) as a reward is solved in the framework of the reinforcement learning method. Therefore, it is understood that the above-mentioned problem setting of the H∞ control is one of the implementation examples of the present invention. In consideration of the above, a circuit used in the learning method of robust reinforcement learning is constructed as shown in FIG. 1 as follows.

In the description of FIG. 1, the same components as those of the conventional circuit of FIG. 2 are designated by the same reference numerals, and the description thereof will be omitted. In FIG. 1, a disturbance generator 4 is provided. Then, the observation signal y (t) from the environment 1 becomes a state signal x (t) via the state estimator, and is input to the behavior generator 2, the state evaluator 3, and the disturbance generator 4. When the state signal x (t) is input, the disturbance generator 4 outputs the disturbance signal w (t) to the environment 1 and the state evaluator 3. The state evaluator 3 is based on the target reward signal r (t) from the environment 1 and the disturbance reward signal s (t) generated based on the disturbance signal w (t) from the disturbance generator 4 The expected value of the sum of weights of the evaluation signal q (t) obtained for the future from the state x (t) of is predicted, the prediction error signal δ (t) is generated based on the predicted value, and the behavior generator 2 And the disturbance generator 4
Output to. In this way, the state evaluator 3 adds the target reward signal r (t) to the disturbance reward signal s (t) to obtain the evaluation signal q.
After obtaining (t), the prediction error signal δ (t) is generated and output. Then, when the prediction error signal δ (t) is input, the behavior generator 2 maximizes the expected value of the weighted sum of the evaluation signal q (t) obtained from the current state x (t) toward the future. Learn to do so and change its input / output relationship. On the other hand, when the prediction error signal δ (t) is input to the disturbance generator 4, the current state x
Learning is performed to minimize the expected value of the weighted sum of the evaluation signal q (t) obtained from (t) toward the future, and the input / output relationship is changed.

The state evaluator 3 calculates an evaluation signal q obtained in the future from the present state x (t) defined by (Equation 7).
Predict the expected value V (x (t)) of (t). However, τ is a time constant for evaluation.

[Equation 7] The behavior generator 2 and the disturbance generator 4 have the expected value V (x
(t)) so that the action signal u is maximized and minimized, respectively.
(t), the disturbance signal w (t) is learned. A reference table, a linear model, a polynomial model, a multilayer neural network, etc. can be used as the behavior generator 2, the state evaluator 3, and the disturbance generator 4.

At the time of performing robust reinforcement learning,
The state evaluator 3, the action generator 2, and the disturbance generator 4 are operated at the same time, but at the stage where the actually learned action rule is used as a control target or environment, the state evaluator 3 and the action generator 2 or the action Only the generator 2 is used for operation. In this action generator 2, when the state signal x (t) is directly obtained as the observation signal y (t), it is used, but generally, the state signal x (t) is estimated by an observer, Kalman filter, etc. When using and learning
Environment 1 can be a model or an actual environment. In the case of an actual environment, the action signal u (t) and the disturbance signal w (t) are transferred to the environment 1 via a driving source such as an actuator or an operating means such as a command to a low-level control program. Is entered. On the other hand, the reward signal r (t) and the state signal x
(t) is output from the environment 1 via a detection means such as a sensor.

The state evaluator 3 then evaluates the evaluation function V (x
(t)) is realized as an approximator V (x (t); v) having parameters v = {v ₁ , v ₂ , ..., V _i , ...} Models, polynomial models and multilayer neural networks can be used.

In this way, the state evaluator 3 obtains the target reward signal r (t) from the environment 1, the means to obtain the disturbance signal w (t) from the disturbance generator 4 and the target reward signal. Means for obtaining the evaluation signal q (t) with the addition of the above, and means for predicting the expected value of the sum of the evaluation signal q (t) obtained from the current state toward the future, and generating the prediction error signal δ (t). And have. In addition, the behavior generator 2 receives the state signal x (t) from the environment 1.
, A means for obtaining the prediction error signal δ (t) from the state evaluator 3, a means for outputting the action signal u (t) to the environment 1, and an evaluation signal q obtained from the present state toward the future.
It has means for learning so that the expected value of the sum of (t) is maximized. Then, the disturbance generator 4 outputs a disturbance signal w (t) to the environment 1, a means for obtaining the state signal x (t) from the environment 1, a means for obtaining the prediction error signal δ (t) from the state evaluator 3. And means for learning so that the expected value of the sum of the evaluation signals q (t) obtained from the current state toward the future is minimized.

Hereinafter, the learning of the evaluation function in the discrete system, the learning of the evaluation function in the continuous system, the action determining method in the discrete system, and the action determining method in the continuous system will be described in this order. Learning Evaluation Function in Discrete System: Consider a dynamic control target according to the following probability distribution P. P (x _{T + 1} | x _T , u _T , w _T ) where x _T is a state variable, u _T is a control input, and w _T is a disturbance input. At this time, the evaluation function V _{T to be obtained} is expressed by the following equation.

[Equation 8] However, q _T is an evaluation signal obtained immediately, α (0 ≦ α
≦ 1) represents the evaluation attenuation rate. Therefore, the prediction error [delta] _T of the state evaluation value is expressed by the following equation. δ _T = q _T + αV _{T + 1} −V _T (Equation 8-2)

Therefore, using this prediction error δ _T and the contribution history e _iT of each parameter represented by the following (Equation 9),

[Equation 9] The parameter update amount Δv _i is expressed by the following equation. Δv _i = ηδ _T e _iT where λ is the attenuation rate of the history of the contribution of the parameter, η
Represents the learning rate. Also, the history e of contribution of each parameter
_iT is updated using the following equation from the definition of (Equation 9).

[Equation 10]

Learning of evaluation function in continuous system: Consider the following dynamic control target (time change dx / d of state variable x (t))
t is state variable x (t), control input u (t), disturbance input w
(t), considered as a function of the noise input n (t)). dx / dt = f (x (t), u (t), w (t)) + n (t) However, at this time, the evaluation function V (t) to be obtained is represented by the following equation.

[Equation 11] However, q (t) is the reward immediately obtained, and τ is the time constant of the evaluation value. Therefore, the prediction error δ of the state evaluation value
(t) is expressed by the following equation. δ (t) = q (t)-(1 / τ) x V (t) + dV (t) / dt

Prediction error δ of the state evaluation value obtained here
Using (t) and the contribution history of each parameter expressed by the following equation,

[Equation 12] However, k is a time constant of the history of the contribution degree of the parameter. The update amount dv _i / dt of the parameter of the state evaluator (represented by the time derivative of the parameter v _{i in} the continuous system) is expressed by the following equation. dv _i / dt = ηδ (t) e _i (t) where η represents the learning rate.

Also, the history e of the contribution of each parameter
update amount de _i (t) / dt of _i (t) is updated using the following equation by the definition of (Equation 12).

[Equation 13]

Behavior determination method in discrete system: (in the case of model independence) Behavior a in state s is determined according to the probability distribution Pr shown by the following equation.

[Equation 14] However, A (s, a) is a parameter for action determination, and represents the ease of action a in state s.
Further, β is a parameter indicating the randomness of the action.
At this time, the parameter update amount ΔA _u (s _T , a of the action generator)
_uT ) and the disturbance generator parameter update amount ΔA _w (s _T , a
_wT) is respectively represented by the following equation using the prediction error [delta] _T (Equation 8-2). _{ΔA u (s T, a uT} ) = η A U δ T ΔA w (s T, a wT) However ^{_{= -η A W η A δ T}} , η A U, η A W represents a learning rate. Also, a _uT ,
_awT represents the actions of the action generator and the disturbance generator at time T, respectively.

As a model-independent learning method, robust reinforcement learning can be realized by learning the action value function. That is, using the prediction error δ _T of the action value function expressed by the following equation,

[Equation 15] The update amount ΔQ (s _T , a _uT, a _wT ) of the action value function is given by the following equation. ΔQ (s _T , a _uT, a _wT ) = η _Q δ _T where η _Q is a learning rate and α is an evaluation decay rate.

In the state s, the action a _{wT j} of the disturbance generator is determined with respect to the actions a _uj of all the action generators according to the probability distribution Pr _w shown by the following equation.

[Equation 16] In this case, by following the probability distribution Pr _w , the action a _w of the disturbance generator having a small action value is selected with high probability. This causes the disturbance generator to generate the worst disturbance for achieving the intended task. However, β is a parameter indicating the randomness of the action.

Next, according to a probability distribution Pr _u represented by the following formula, in the state s, determined by using an action a _WTj for a _uj disturbance generator which already determines an activity a _uT action generator.

[Equation 17] However, when the action generator selects the action a _uT = a _uj , the disturbance generator selects the action a _wT = a _wTj . In this case, by following the probability distribution Pr _u , the action a _uj of the action generator having a large action value is selected with high probability. This causes the behavior generator to generate the highest behavior output for achieving the desired task. However, β is a parameter indicating the randomness of the action.

Behavior determination method in discrete system: (in case of model dependence) In state X _T , the behavior generator behavior is u and the disturbance generator behavior is w, and as a result, the state X _{T + 1} is reached. Suppose If the evaluation signal q _{T + 1} obtained at that time is used, the action u _T of the action generator is calculated using the model of the environment.

[Equation 18] Where α is the attenuation factor of the evaluation, P (x _{T + 1} | x _T , u,
w) when the output to the disturbance generator action generator action u in state x _T has output the disturbance w is the probability of transition to a state x _{T + 1.}

Behavior determination method in continuous system: (in the case of model independence) u (t),
If the behavior of the disturbance generator is w (t), then u (t) = A (x (t); v ^Au ) + n _u (t) w (t) = A (x (t); v ^Aw ) It is expressed as + n _w (t). However, n _u (t) and n _w (t) represent noise inputs for searching. Each action is realized as an approximator A (x (t); v ^A ) having parameters v ^A = {v ₁ ^A , v ₂ ^A , ... _, V _i ^A _, ...} Models, polynomial models, multilayer neural networks, etc. can be used. Also, each parameter is updated as follows using the above-mentioned prediction error signal δ (t).

[Formula 19] (In the case of model dependence) When a model of the environment can be used, the gradient of the state evaluator can be used to perform learning more efficiently than in the case of model independence. Here, the environment model and the reward model are expressed by the following equations.

[Equation 20] Then, a gradient of the evaluation function, the input gain g ₁ obtained from the environment model (x), using g ₂ a (x), and an optimal output u _op action generator, optimal output w _op of the disturbance generator, respectively It is expressed by the following equation.

[Equation 21] The input gains g ₁ (x) and g ₂ (x) are not necessarily known, but can be obtained by learning the state evaluation and the environment model at the same time.

Concrete Example 1: Swinging up of a Simple Pendulum An example is shown in which robust reinforcement learning is applied to control of a simple pendulum as shown in FIG. 4, and a simple pendulum is swung up using a learned controller. Single pendulum has mass m = 1 [kg] and length L
In [m], the state variable is x = (θ, dθ / dt),
It is represented by the pendulum angle and angular velocity. The control command u = T is a drive torque at the center of the pendulum rotation axis. Therefore, considering the correspondence with (Expression 18-1, Expression 18-2, Expression 18-3), the respective functions forming the equation of motion of the pendulum are given as follows.

[Equation 22] Therefore, the evaluation signal is represented by the following function.

Q (t) = cos (θ) -1-0.08u ² + γ ² w ^{2 The} state variable is x = (θ, dθ / dt), and the mass of the pendulum during learning is m = 1 [kg]. , And the length L = 1 [m], the behavior generator and the disturbance generator are as follows from (Equation 19-1, Equation 19-2, Equation 19-3).

[Equation 23] Here, since the state variables are continuous, a multilayer neural network was used as the function approximating means of the state evaluator.

In this way, the behavior generator learned is
Adopted as a non-linear robust controller, length L = 0.5,1.0,
It was applied to a system having three different lengths of 2.0 [m]. Moreover, the same experiment was conducted for the controller learned by the conventional reinforcement learning. The results are shown below.

As shown by the solid lines in FIGS. 4 (b) and 5, the nonlinear robust controller succeeds in swinging up the single pendulum in all environments. However, the vertical axis of the graph represents the rotation angle of the pendulum, and the horizontal axis represents time. The alternate long and short dash line shows the state of being swung up. And since the solid line converges to the straight line indicated by the one-dot chain line,
It can be seen that the swing was successful.

On the other hand, as shown by the broken line, the conventional controller learned by the normal reinforcement learning cannot raise the pendulum except in the same environment as the environment used at the time of learning. As shown in FIG. 4 (b), when swinging up in the learning environment (pendulum length L = 1.0 [m]), it can be seen that the swinging up trajectory converges on the alternate long and short dash line. As shown, except for the learning environment (pendulum length L = 0.5 [m], 2.
When swinging up under the environment of 0 [m]), the swinging trajectory does not converge to the alternate long and short dash line, and it can be seen that swinging up is not possible.

Concrete Example 2: Conveyance of luggage in a non-linear force field Here, it is considered that the conveyed article 12 is carried on the linear motion actuator 11 as shown in FIG. However, it is assumed that the controlled object has non-linearity due to the gradient and that the actuator 11 is downsized so that a large output is not output. The equation of motion of the controlled object is represented by the following equation, where the gradient is θ, the friction coefficient μ = 0.01, and the gravitational acceleration g = 9.8 [m / s ² ].

[Equation 24] However, the state variable is x = (a, da / dt), a represents the horizontal position of the actuator 11, and da / dt represents the speed. Further, F is a force given by the actuator 11,
M is the mass of the conveyed product 12, and m is the mass of the actuator 11. When the horizontal position is a, the gradient θ is
θ = arctan (cos (πa)).

Therefore, (Equation 18-1, Equation 18-2,
Considering the correspondence with Expression 18-3), the respective functions forming the equation of motion of the transport actuator 11 are given as follows.

[Equation 25] Here, the following function was used as the reward corresponding to (Equation 6). ^{q (t) = 1.0 -0.02u 2} + γ 2 w 2 (if 0.4 ≦ a ≦ 0.6) = - 0.5-0.02u 2 + γ 2 w 2 (otherwise)

Since the state variable is x = (a, da / dt), the mass of the actuator 11 at the time of learning is m = 1 [kg], and the mass of the transported object 12 is M = 1 [kg], Formula 19
-2) From (Equation 19-3), the behavior generator and the disturbance generator are as follows.

[Equation 26] Here, since the state variables are continuous, a multilayer neural network was used as the function approximating means of the state evaluator.

Then, it was learned to move to a certain target point (illustrated by a chain line in FIG. 7). In addition,
In FIG. 7, the vertical axis represents the position of the actuator 11, and the horizontal axis represents time.

In this way, the learned behavior generator is
Adopted as a non-linear robust controller, the case where the load 12 with the same mass (M = 1 [kg]) as during learning was placed and the load 12 (M = 3 [kg]) that was heavier than during learning was placed. In each case, the results of a simulation experiment of transporting from a point 15 [m] apart to a target point are shown by solid lines in FIG. 7 (a) and FIG. 7 (b). 7 (a) and 7 (b), since the trajectory of the actuator 11 converges on the alternate long and short dash line indicating the target point, the mass of the conveyed product 12 is M = 1 [kg], M = It can be seen that the conveyance can be performed in both cases of 3 [kg].

On the other hand, the same experiment was conducted for the conventional controller learned by the normal reinforcement learning, and the result is shown by the broken line in FIG. As shown by the broken line in FIG.
When the mass of the transported object 12 is M = 1 [kg], the trajectory of the actuator 11 converges on the alternate long and short dash line indicating the target point, which indicates that the transport to the target point is successful. . However, as shown by the broken line in FIG. 7B, when the mass of the conveyed product 12 is M = 3 [kg], the trajectory of the actuator 11 does not converge to the alternate long and short dash line indicating the target point. From this, it can be seen that the transportation to the target point has not been completed. As described above, the controller obtained by using the robust reinforcement learning method of the present invention can cope with the mass of the transported object 12 within a certain range.

Although the embodiments of the present invention have been described above, various embodiments can be implemented within the scope of the spirit of the present invention.

[0043]

As described above, according to the present invention, an evaluation signal which is a reward signal in which a reward according to the degree of achievement of a goal and a reward according to the ability to withstand disturbance are added is generated, and the present evaluation signal is generated. The behavior generator learns to maximize the expected value of the sum of the evaluation signals obtained from the state toward the future, while the disturbance generator learns to minimize the expected value of the sum of the evaluation signals. It can cope with the case where the model is unknown, and can be resistant to environmental changes. Further, it can be applied to a non-linear control target or environment.

[Brief description of drawings]

FIG. 1 is a block diagram of a circuit used in a robust reinforcement learning method according to an embodiment of the present invention.

FIG. 2 is a block diagram of a circuit used in a conventional reinforcement learning method.

FIG. 3 is a block diagram of a control system of a control target and a controller for explaining the H∞ control theory.

FIG. 4 is an explanatory diagram of the first specific example, (a) is a schematic diagram,
(B) is a graph of an angle change of a pendulum having a length of 1 m.

5A and 5B are graphs of changes in angle of the first specific example, where FIG. 5A is a graph in which a pendulum having a length of 0.5 m is used as a control target, and FIG. 5B is a control target of a pendulum having a length of 2 m. It is a graph when it is used as.

FIG. 6 is a schematic view of a second specific example.

7A and 7B are graphs of changes in the position of the linear actuator of the second specific example, where FIG. 7A is a graph when a conveyed product having a mass of 1 kg is used, and FIG. 7B is a case where a conveyed product having a mass of 3 kg is used. Is a graph of.

[Explanation of symbols]

q (t) evaluation signal r (t) target reward signal s (t) disturbance reward signal u (t) action signal w (t) disturbance signal 1 environment 2 Behavior generator 3 State evaluator 4 Disturbance generator

Continuation of front page (56) Reference JP-A-3-208102 (JP, A) Kazuyuki Samejima, Kenji Dotani, Mitsuo Kawahito, Evaluation assignment among modules in multiple model-based reinforcement learning, IEICE technology Research Report, Japan, The Institute of Electronics, Information and Communication Engineers, August 14, 2000, Vol. 100 No. 191, P.I. 51-58 Douji Kenji, Metaparameter control and neuromodulators in behavioral learning, Mathematical science, Japan, Science Co., Ltd., June 1, 2000, Volume 38, No. 6, p. 19-p. 24 Kazuyuki Samejima, Kenji Dotani, Mitsuo Kawato, Hierarchical reinforcement learning using state partitioning based on local linear model, IEICE technical report, Japan, The Institute of Electronics, Information and Communication Engineers, March 2000. 13th, Vol. 99, No. 684, p. 173-p. 180 Kazumitsu Miyazaki, Reinforcement Learning in Discrete Markov Decision Process, Journal of Japan Fuzzy Society, Japan, Japan Fuzzy Society, August 15, 1997, Vol. 9, No. 4, p17-p. 20 (58) Fields surveyed (Int.Cl. ⁷ , DB name) G05B 13/02 G05B 13/04 G06N 3/00

Claims (9)

(57) [Claims] 1. A behavior generator that outputs a behavior signal to a controlled object or environment and has a learning function, and a disturbance generator that outputs a disturbance signal to a controlled object or environment and that has a learning function. Generates an evaluation signal that is a reward signal in which the reward according to the ability to withstand the disturbance from the disturbance generator is added to the reward according to the degree of achievement, and the weight of the evaluation signal obtained from the current state to the future The behavior generator learns to maximize (or minimize) the expected value of the sum, while the disturbance generator learns to minimize (or maximize) the expected value of the sum of the evaluation signals. Robust reinforcement learning method. 2. In the learning method, a state evaluator for predicting an expected value of a sum of evaluation signals obtained from the present state toward the future is provided, and the prediction error signal is used as a state evaluator, an action generator, And a robust reinforcement learning method for use in learning at least one of a disturbance generator. 3. At least one of the state evaluator, the behavior generator, and the disturbance generator is provided with a reference table indicating an input / output relationship as a function approximating means. Robust reinforcement learning method described. 4. At least one of the state evaluator, the behavior generator, and the disturbance generator is equipped with a linear model or a polynomial model as a function approximating means. Robust reinforcement learning method. 5. The robust system according to claim 1, wherein at least one of the state evaluator, the action generator and the disturbance generator comprises a multilayer neural network as a function approximating means. Reinforcement learning method. 6. A control method using only the state evaluator and the behavior generator that have been preliminarily learned according to the method of claim 1 or claim 2. 7. The method according to claim 1 or 2 is applied to an environment model realized by computer simulation, and only the state evaluator and the action generator or the action generator learned by the method are applied to an actual environment. The control method according to claim 6, wherein: 8. The robust control according to claim 6, wherein at least one of the state evaluator and the action generator comprises a reference table indicating an input / output relationship as a function approximating means. Robust controller using the method. 9. The method according to claim 6, wherein at least one of the state evaluator and the action generator comprises a linear model, a polynomial model or a multilayer neural network as a function approximating means. Robust controller using the robust control method of.