JP2966076B2 - Learning device self-learning method - Google Patents

Description

Detailed Description of the Invention [Table of Contents] Overview Industrial application field Conventional technology (FIGS. 10 to 13) Problems to be Solved by the Invention Means for Solving the Problems (FIG. 1) 1) Embodiment (a) One embodiment of the present invention a. Overall configuration (FIGS. 2 and 3) b. Overall operation (FIGS. 2 and 3) c. Determination of evaluation value (Fig. 4) d. Learning results (FIGS. 5 to 9) (b) Another embodiment of the present invention [Summary] A teacher pattern is created based on an external evaluation of the behavior of the learning device, Regarding the self-learning method of the learning device that learns based on patterns, there is an interface for external evaluation, and there is no need to create and replace self-evaluation criteria each time according to the application field of the learning device. To provide a self-learning method of a learning device that can externally evaluate the behavior of the learning device in a simple manner and that can perform external evaluation at appropriate time intervals, and to use past behavior as a teacher pattern. The purpose of the present invention is to provide a learning method of a learning device which can be used and can select an appropriate number of teacher patterns. When a value is given, a first evaluation value is calculated and given to a past action based on the external evaluation so that the absolute value becomes smaller as going back to the past. The second evaluation value is calculated by weakening the overall evaluation value of the second evaluation value by the predetermined forgetting rate, and the sum of the first evaluation value for the external evaluation and the second evaluation value weakened by the predetermined forgetting rate is calculated in the past in the present. For each action as a comprehensive evaluation value of the action,
A configuration in which an action whose overall evaluation value is equal to or greater than a set value is selected as a teacher pattern, learning is performed based on the teacher pattern, and the teacher pattern generation processing and the learning processing are performed on the behavior of the learning device after learning. I do.

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a self-learning method in a learning device such as a neural network. In particular, the present invention creates a teacher pattern based on an externally-applied evaluation of the behavior of the learning device. The present invention relates to a self-learning method of a learning device that learns based on a self learning.

2. Description of the Related Art In recent years, learning devices configured with neural networks and the like have been applied to various controls such as pattern recognition such as alphabet recognition and image recognition, adaptive filters, and robots. Such a learning device performs learning using teacher data, and outputs a desired result to an input signal based on the learning result.

However, in a learning method that requires teacher data, it is often difficult or impossible to create teacher data by hand, and a learning method that can automatically perform learning adapted to an external environment is desired. It has become.

[Prior Art] A learning device composed of a neural network or the like performs learning using a teacher pattern in order to output a correct output pattern (output data) for a certain input pattern (input data). Done. That is,
Weights in the neural network are determined by a back propagation method (BP method) using a teacher pattern prepared in advance so that a correct output pattern can be output for a certain input pattern.

Conventionally, such a teacher pattern has been prepared by a human in advance. However, a teacher pattern must be created each time according to the actual application of the learning device, and depending on the application, a time-series pattern is required as the teacher pattern or a pattern in which the teacher pattern itself changes is required. And the need for patterns for unpredictable states increases the types and number of teacher patterns, making it difficult to prepare all of them in advance, and creating a very long time. There was such a problem.

For this reason, the applicant of the present application has set the output result (action) of the learning device.
Is evaluated in accordance with a self-evaluation criterion held in the learning device, the correspondence of the output pattern to the input pattern when the evaluation is high (input / output pattern) is taken in as a teacher pattern, and self-learning is performed using the teacher pattern. Japanese Patent Application No. Hei 1-244409 (filing date: September 20, 1999, title of invention: self-learning processing method for learning device).

FIG. 10 is a block diagram of a learning device according to the self-learning method of the present application, and 21 is an input in a format capable of processing a signal (external information) Sa detected by a sensor (for example, a visual sensor of a robot) at a subsequent stage. An input unit for converting into a pattern (generally information of a plurality of bits) Sb, a processing unit 22 which is a neural network, learns by a BP method based on a teacher pattern TPN, and inputs an input pattern based on a learning result
It outputs predetermined output data Sc to Sb. Reference numeral 23 denotes a noise generation unit that generates noise Sd and adds it to the output data Sc of the processing unit 22 for the purpose of discovering the teacher pattern by trial and error.
3 'is an adder, 24 is an output unit for moving a control target (for example, a robot) by driving a motor (not shown) based on the adder output (output pattern) Se, and 25 is an input / output pattern corresponding unit. This input / output pattern corresponding unit 25 pairs the input pattern Sb and the corresponding output pattern Se,
Create and output the input / output pattern Sf. 26 is an evaluation unit which compares the immediately preceding input / output pattern with the current input / output pattern and evaluates whether or not the immediately preceding input / output pattern is taken in as a teacher pattern according to a built-in self-evaluation criterion (good or bad). Is output. Reference numeral 27 denotes a teacher pattern table unit which, when an evaluation signal of "good" is output from the evaluation unit 26, fetches and stores the immediately preceding input / output pattern as a teacher pattern.

As shown in FIG. 11, the evaluation unit 26 includes a short-term storage unit 31 that stores the immediately preceding input / output pattern Sf ′, and a current input / output pattern Sf ′.
An evaluation input unit 32 for extracting information to be evaluated from Sf and the immediately preceding input / output pattern Sf ′ and outputting an evaluation input pattern Sg
And, a self-evaluation unit 33 that evaluates the evaluation input pattern Sg according to a self-evaluation criterion prepared in advance, and outputs an evaluation signal Sh based on the evaluation result, and when the evaluation signal Sh is “good”,
An evaluation output unit 34 stores the immediately preceding input / output pattern Sf 'as a teacher pattern in the teacher pattern table unit 27.

In such a learning device, when activation of teacher pattern creation is started, the input unit 21 converts the external information Sa into the input pattern Sb and inputs it to the processing unit 22. The processing unit 22 performs the processing of the neural network in an unlearned state, and the adder 23 ′ adds the noise Sd generated from the noise generation unit 23 to the processing result Sc to form an output pattern Se, which is learned via the output unit 24. Output to the outside of the device. In parallel with this, inside the device, the input pattern Sb and the output pattern Se output from the adder 23 'are associated by the input / output pattern correspondence unit 25 and input to the evaluation unit 26 as the input / output pattern Sf.

Assuming that the above operation is performed at time t,
The external environment changes according to the external output, and the time (t + 1)
In, new external information is provided to the input unit 21. As a result, at time (t + 1), the input / output pattern is provided from the input / output pattern corresponding unit 25 to the evaluation unit 26 in the same manner as at time t.

On the other hand, the input / output pattern Sf 'at time t is temporarily stored in the short-term storage unit 31, and the input / output pattern Sf at time (t + 1) and the output of the short-term storage unit 31, that is, at time t
Is input to the evaluation input unit 32, and the input / output pattern Sf 'at time t is input from the evaluation input unit 32.
The evaluation input pattern Sg including the input pattern Sb at time (t + 1) and the input pattern Sb is provided to the self-evaluation unit 33. The self-evaluation unit 33 evaluates the input evaluation input pattern Sg according to a self-evaluation criterion, and outputs an evaluation signal Sh indicating good or bad to an evaluation output unit.
The evaluation output section 34 outputs the evaluation signal Sh if it is good.
The input / output pattern Sf 'at time t is input to the teacher pattern table unit 27 as a teacher pattern and stored.

For example, 11 visual sensors EYS (see Fig. 12 (a))
12 (a) shows a state in which the target is not captured, and FIG. 12 (b) shows two visual sensors.
If E1 and E2 are capturing the right edge of the target,
FIG. 12C shows a case where the visual sensors E10 and E11 capture the left end of the target, and FIG. 12D shows a case where the entirety of the target is captured. Then, as shown in FIG. 13, if the input patterns at time t and (t + 1) are 10000000000 and 11100000000, respectively, the robot moves in the direction of better capturing the target by the output pattern (command rotation angle of the motor) θt at time t. Since the rotation has been completed, the input / output pattern at the time t is given a good evaluation and is taken in as a teacher pattern. However, the input pattern at the time (t + 2) becomes 00000000000, and the output pattern θt + 1 at the time (t + 1) no longer catches the target until then, so a bad evaluation is given and the target pattern is not taken in.

Thereafter, if a large number of teacher patterns are generated by repeating the above processing for a predetermined time, the processing unit 22 performs the first learning by the BP method using the teacher patterns. After learning, the above-described process of generating the teacher pattern and learning based on the generated teacher pattern are repeated, and when the error is within an allowable range or when a predetermined time has elapsed, the learning is terminated, and the learning device is applied to an actual application. I do.

[Problem to be Solved by the Invention] As described above, the learning method of the present invention has an advantage that a teacher pattern can be automatically created. However, since such a learning method does not have an interface for receiving an external evaluation, it is necessary to incorporate a self-evaluation criterion in the apparatus as a fixed one in advance. For this reason, when the learning device is applied in various fields, it is necessary to create and replace self-evaluation criteria for each application field, and there is a problem of lack of flexibility.

Therefore, a first object of the present invention is to provide an interface corresponding to an external evaluation, create a teacher pattern according to the external evaluation of the behavior of the learning device, immediately self-learn based on the teacher pattern, and An object of the present invention is to provide a real-time self-learning method that repeats the above processing for the behavior of a learning device after learning.

A second object of the present invention is to provide a learning method of a learning device which has an interface for external evaluation and does not need to create and replace a self-evaluation criterion each time according to the application field of the learning device. .

By the way, in the learning method of the learning device provided with an interface for the external evaluation, the external evaluation is, for example, a human evaluation while observing the behavior of the learning device. For this reason, it is a serious task if all actions of the learning device are evaluated one by one and at several levels.

Also, in the learning method that learns in real time simultaneously with the action, it is necessary to use past actions as a teacher pattern, but in learning of neural networks including the BP method,
When learning a new teacher pattern, the past teacher pattern cannot be reproduced, which is a problem. For this reason, if it is simply considered, it is sufficient to repeatedly present past behaviors as a teacher pattern until the learning device grows. However, this would lead to too many teacher patterns, which would take a long time to learn, and would require real-time learning. Not suitable for learning.

In view of the above, a third object of the present invention is to provide a self-learning method of a learning device that can externally evaluate the behavior of the learning device by a simple method and can perform external evaluation at appropriate time intervals. .

A fourth object of the present invention is to provide a self-learning method of a learning device that can easily perform external evaluation with two kinds of signals of good and bad.

A fifth object of the present invention is to provide a learning method of a learning device that can use past actions as a teacher pattern and can select an appropriate number of teacher patterns.

A sixth object of the present invention is to reflect the external evaluation on past actions, reduce the evaluation value as it goes to the past, and attenuate the evaluation of past actions with the passage of time. It is an object of the present invention to provide a learning method of a learning device that can set a moderate number of actions of a teacher as a teacher pattern.

A seventh object of the present invention is to provide a learning method of a learning device that can make the condition for terminating the learning stricter as the evaluation is higher, that is, can reduce the maximum allowable error.

[Means for Solving the Problems] FIG. 1 is an explanatory view of the principle of the present invention.

3 is a processing unit such as a neural network, 8 is an external evaluation input unit for inputting external evaluations of “good” and “bad” with respect to the behavior of the learning device, and 9 is each step of the learning device based on the external evaluation. It is a teacher pattern creation unit that creates a teacher pattern by evaluating an action. In the teacher pattern creating unit 9, reference numeral 11 denotes an input / output pattern corresponding unit that expresses the behavior of the learning device as a pair of an input pattern and an output pattern, and 12 denotes a current step (s-th step) and a previous (s-th) step.
-1) An evaluation storage unit that stores an overall evaluation value for each past action in the step, and calculates and assigns an evaluation value to each action based on an external evaluation so that the absolute value becomes smaller as going back to the past. An external evaluation conversion unit 14 performs a forgetting processing unit that weakens the overall evaluation value of each past action one step before the current step by a predetermined forgetting rate, and 15 performs a first evaluation value A ( s, p) and the second evaluation value B (s, p) weakened by the predetermined forgetting rate are obtained as the total evaluation value R (s, p) of the past action, and stored in the evaluation storage unit 12. An operation unit 16 is a teacher pattern determination unit that selects past actions whose overall evaluation value is equal to or greater than a set value as a teacher pattern, and 18 is a teacher pattern storage unit that stores each teacher pattern together with a convergence condition (maximum allowable error). .

[Operation] When an external evaluation value C is given from the external evaluation input unit 8 to the behavior of the learning device, the external evaluation conversion unit 13 is based on the external evaluation value C (+1 for good and -1 for bad). For each action in the past, the evaluation value is calculated so that the absolute value becomes smaller as going back to the past. Also, the forgetting processing unit 14 weakens the total evaluation value R (s−1, p) of each past action one step before by a predetermined forgetting rate, and the adder 15 performs the first evaluation on the external evaluation.
Value A (s, p) and the second weakened by the predetermined forgetting rate
Of the past behavior at the present time is calculated as the total evaluation value R (s, p) of the past behavior for each past behavior and stored in the evaluation storage unit 12. Then, the teacher pattern determination unit 16 selects past actions whose overall evaluation value R (s, p) is equal to or greater than the set value as a teacher pattern and stores them in the teacher pattern storage unit 18, and the teacher pattern generation processing is completed. I do.

Thereafter, the processing unit 3 of the neural network performs learning by the BP method based on the teacher pattern TPN. Then, the learning device after the learning is caused to perform the action of the next step, and the learning pattern is subjected to the teacher pattern generation processing and the learning, and the output error for each teacher pattern is equal to or less than the maximum allowable error. At this time, the learning ends.

As described above, the learning apparatus of the present invention has an interface for external evaluation, creates a teacher pattern according to the external evaluation of its own behavior, immediately learns based on the teacher pattern, and performs the above-described learning for the behavior after learning. You can learn in real time by repeating the process. Therefore, it is not necessary to create and replace the self-evaluation criteria each time according to the application field of the learning device.

The external evaluation may be of two types, good and bad. In addition, it is not necessary to evaluate each step, and the external evaluation may be performed at appropriate time intervals. Therefore, the external evaluation can be easily performed.

Further, past behavior can be used as a teacher pattern. In this case, the external evaluation is reflected in the past behavior, the evaluation value is reduced as the past goes, and the evaluation of the past behavior is changed with time. By attenuating accordingly,
The number of past actions to be adopted as teacher patterns can be limited, and the number of teacher patterns can be moderated.

Also, the maximum allowable error of the output data is determined for each teacher pattern, and when the output data for all the teacher patterns becomes less than the maximum allowable error, the learning is terminated. The higher the value of the teacher pattern,
If the maximum allowable error is reduced and the convergence condition is strict, sufficient learning can be performed on the learning device, and processing with a small error can be performed in an actual application.

Examples (a) One embodiment of the present invention a. Overall Configuration FIG. 2 is a configuration diagram of a learning device according to an embodiment of the present invention, in which a robot tracks a target.

Reference numeral 1 denotes a robot, and a robot body 1a is provided with 36 visual sensors 1b for capturing a target. It should be noted that, of the portions where the visual sensor has reacted, the portion where the boundary of the target is captured is converted into an analog value in accordance with the ratio of the captured amount, and the other values are set to 1 and the portions which do not respond are set to 0. An input unit 2 converts an external signal Sa input from the visual sensor 1b into an input pattern Sb that can be processed by the learning device. Numeral 3 denotes a processing unit composed of a hierarchical neural network, which learns by the BP method based on the teacher pattern TPN, calculates an input pattern Sb in feedforward based on the learning result, and outputs output data Sc
4 is a noise generating unit that generates noise Sd of uniform random number and adds it to the output data Sc of the processing unit 3 for the purpose of finding a teacher pattern by trial and error, and 5 is an output Sc of the processing unit 3. An output unit for generating a moving distance command and a rotation angle command of the robot based on the adder output (output pattern) Se; a motor control unit for controlling various motors of the robot; 8 is an external evaluation input unit for observing the behavior of the learning device and inputting a good / bad external evaluation value C (C = 1 if good, C = -1 if bad), 9 is based on the external evaluation value A teacher pattern creating unit that creates a teacher pattern by evaluating past behaviors of the learning device.

Processing unit 3 The processing unit 3 is configured by, for example, a perceptron-type hierarchical neural network, has an input layer, an intermediate layer, and an output layer, and outputs an output pattern from the output layer in response to an input pattern for the input layer. Has become. Learning of such a hierarchical neural network is performed by the well-known BP method. The BP method changes the weight of the connection by the error between the actual output and the output indicated by the teacher pattern,
For example, the weight Wji of the connection from the unit j of the input layer to the intermediate layer i is updated by an amount determined by the following equation: ΔWij (n + 1) = β · δj · Oi + α · ΔWji (n) (4) Here, ΔWji (n) is the update amount at the time of the nth weight update, δj is the error of unit j, and Oi is the output of unit i. Further, β is a learning constant, and the larger the weight is, the larger the amount of weight update at one time is, and the learning speed can be increased, but vibration occurs. α is a learning rate constant, which determines the magnitude of the influence of the previous update of the weight on the update of the weight. Note that β = 0.
1, α = 0.05.

Teacher Pattern Creation Unit 9 FIG. 3 is a block diagram of an embodiment of the teacher pattern creation unit 9. In the following notation, P (s) means the action in the s-th step, and A (s, p) indicates the action that occurred in the p-th step when the external evaluation given in the s-th step is reflected. B (s, p) means the evaluation value of the s-th step obtained by attenuating the comprehensive evaluation in the (s-1) -th step of the action occurring in the p-th step in accordance with the forgetting rate. , R (s, p) mean the overall evaluation value of the action that occurred in the p-th step in the s-th step.

In FIG. 3, reference numeral 11 denotes an input / output pattern corresponding unit that expresses the behavior of the learning device as a pair of an input pattern and an output pattern, and 12 denotes at least past behaviors P (s−1), P at the current step (s-th step). (S-2), P (s-3),
... and past actions P (s-2), P (s-3),... In the previous (s-1) step.
An evaluation storage unit 13 for storing a comprehensive evaluation of the past actions P (s−1), P (n) from the previous (s−1) th step to the nth step before based on the external evaluation value C s-
2), P (s−3),... P (s−n), evaluation values A (s, p), A such that the absolute value decreases as going back in the past.
(S, p-1), A (s, p-2) ... A (s, p-n + 1)
(Where p = s−1) is an external evaluation conversion unit, and 14 is a past action P (s−2), P (s−3),... One step before the current step (s step). , R (s−1, p), R (s−1, p−1),... (Where p =
a forgetting processing unit for weakening s-2) by a predetermined forgetting rate b (= 0.8); and a first evaluation value A (s, p) for an external evaluation and a second evaluation value B ( s, p) is calculated for each action as a total evaluation value R (s, p) in the current step (s-th step), and is stored in the evaluation storage unit 12. s, p) (p ≦ s−
1) a teacher pattern determination unit that selects past actions that are equal to or more than a set value, for example, 0.2 or more, as teacher patterns,
Reference numeral 17 denotes a convergence condition determining unit that determines a convergence condition (maximum allowable error) of each teacher pattern, and 18 denotes a teacher pattern storage unit that stores the teacher pattern and the convergence condition.

External evaluation conversion unit 13 The external evaluation conversion unit 13 performs the past actions P (s−1), n steps before the previous (s−1) step based on the external evaluation value C, as described above. P (s-2), P (s-
3),... With respect to P (s−n), evaluation values A (s, p), A (s, p−1),
(S, p−2)... A (s, p−n + 1) (where p = s−
1) is calculated and assigned.

In the conventional algorithm, the self-evaluation unit in the learning device evaluates each action that the user has made one by one. However, in the present invention, the outside, for example, a human evaluates the behavior of the learning device while observing the behavior. Therefore, it is difficult to evaluate the behavior of each device one by one. Therefore, an algorithm that can be evaluated from outside at appropriate intervals is required. In the present invention, when the behavior of the learning device is evaluated at an appropriate timing while observing the behavior of the learning device from the outside, the external evaluation conversion unit 13 reflects the external evaluation in each of a series of past behaviors up to the present. I have. In other words, the external evaluation signal C (good is 1 and bad is -1) is given to the action immediately before the evaluation, but is given because the action immediately before was good or bad. Not something. External evaluation at an appropriate timing means that a series of actions performed by the learning device are observed and evaluated. Therefore, the external evaluation conversion unit 13 exponentially weakens the evaluation signal, and assigns an evaluation value to a series of actions retroactive from the immediately preceding action. In this way, the evaluation value having the highest absolute value is given to the immediately preceding action, and the evaluation value is weakened as going back to the past. The reason why the exponential function is used to weaken the evaluation value is that the memory of the human brain is exponentially weakened. When this is represented by an equation, dA (t) / dt = −α · A (t). Here, t represents the number of steps stepped back, and α is a count of an exponential function, for example, 0.3. Therefore, the p-th calculated from the evaluation signal C given in step s
The evaluation value A (s, p) (where p ≦ s−1) of the action P (p) that occurred in the step is represented by the following equation: A (s, p) = C · exp [−α (s−p−1) )] (1) is calculated by the external evaluation conversion unit 13. Note that the external evaluation value C is not always given for each step.

Forgetting processing unit 14 As described above, the forgetting processing unit 14 performs the past actions P (s-2) and P (s-3) in the (s-1) th step one step before the current step (sth step). ,... (Where p = s−2) is weakened by a predetermined forgetting rate b (= 0.8). Things.

Generally, when learning a new teacher pattern, including the BP method, in neural network learning, past teacher patterns often become unreproducible. Also, in real-time learning in which learning is performed simultaneously with behavior, the teacher pattern changes one after another. This makes it impossible to use past actions as teacher patterns in real-time learning, which is a major problem. Therefore, simply thinking, it is only necessary to repeatedly show past actions as a teacher pattern until the learning device grows by learning.
In this case, there are too many teacher patterns, so that it takes a long time to learn, and is not suitable for real-time learning. That is, it is necessary to select an appropriate past teacher pattern. Therefore, the present invention introduces the concept of forgetting in addition to the concept of evaluation storage.

That is, in the present invention, the evaluation values of past actions are stored. If the evaluation value is high, it is clearly stored as a good action, and if the evaluation value is positive and low, it is stored as a good action, but the memory is thin. Conversely, if the evaluation value is negative and high, it is clearly stored as a bad action, and if the evaluation value is low and negative, it is stored as a bad action, but the memory is thin. From the above, "weak the evaluation value with the passage of time" means whether it is a good behavior or a bad behavior,
It means "weakened the memory of behavior."

With such a method, a teacher pattern can be presented for a certain period of time, and too many past teacher patterns are naturally removed, so that a large number of teacher patterns cannot be formed. This can be expressed as follows. That is, the comprehensive evaluation values R (s−1, p), R (s) of the past actions P (s−2), P (s−3),... One step before the current step (s-th step). −1, p−1),... (Where p = s−2), they are obtained in step s by the following equation: B (s, p) = bR (s−1, p) -It is weakened according to (2) and is calculated by the forgetting processing unit 14.

Arithmetic Unit 15 The arithmetic unit 15 calculates the first evaluation value A () for the external evaluation by the following equation (p ≦ s−1) according to the following equation: R (s, p) = A (s, p) + B (s, p) (3) s, p) and the second evaluation value B (s, p) weakened by the predetermined forgetting rate b, and calculates the past actions P (s−1), P in the current step (s-th step). (S-2), P
(S-3),... Total evaluation value R (s, p) (p ≦ s−
1) and stored in the evaluation storage unit 12.

Convergence condition determination unit 17 The convergence condition determination unit 17 calculates the total evaluation value R (s, p) (p ≦ s
The convergence condition is changed according to the magnitude of -1). That is, the convergence condition (maximum permissible error) is made stricter for the teacher pattern of the behavior that has been highly evaluated, and the convergence condition is loosened for the one that is evaluated low. For example, when the total evaluation value is 1.0 (the maximum value), the maximum allowable error E (s, p) is set to 0.05, and the evaluation value is set to 0.2.
In the case of (lowest value), it is set to 0.2, and the maximum allowable error is calculated by linear interpolation during that time. By doing so, it is possible to easily satisfy the convergence condition and not to end the learning, and it is also possible to satisfy the convergence condition by a considerable number of learnings and end the learning.

B. Overall Operation When the external signal Sa is input in the current step (s-th step), the input unit 2 converts the external information Sa into an input pattern Sb and inputs the same to the processing unit 3. The processing unit 3 is a neural
Network processing is performed, and the adder 5 adds the noise Sd generated from the noise generating unit 4 to the processing result Sc and outputs an output pattern Se. The output pattern Se is input to the motor control unit 7 via the output unit 6 as a moving distance command and a rotation angle command. In parallel with this, inside the device, the input pattern Sb and the output pattern Se output from the adder 5 are correlated by the input / output pattern correspondence unit 11 (FIG. 3) to become the input / output pattern P (s).

In this state, assuming that the evaluation signal C is sent from the external evaluation input unit 8, the external evaluation conversion unit 13 performs (1)
According to the formula, the past actions P (s-1), P (s-2), P (s
-3), the evaluation value A (s, p) (where p ≦ s−1) is calculated and assigned so that the absolute value decreases as going back in the past.

In addition, the forgetting processing unit 14 calculates the total evaluation value R (s-1, p) of the past actions P (s-2), P (s-3), etc. in the (s-1) th step (where p ≦ s-2) is weakened by the predetermined forgetting rate b according to the equation (2).

The adder 15 calculates a first evaluation value A (s, p) for the external evaluation.
And the second evaluation value B (s, p) weakened by the predetermined forgetting rate
Is calculated according to equation (3), and the calculation result is obtained as a comprehensive evaluation value R (s, p) (P ≦ s−1) for the past action at the current step (the action that occurred at the p-th step), It is stored in the evaluation storage unit 12.

Next, the teacher pattern determination unit 16 determines the overall evaluation value R (s,
p) (p ≦ s−1), each of past actions whose set value (= 0.2) or more is selected as a teacher pattern, and a convergence condition determination unit
17 determines a convergence condition (maximum allowable error) E (s, p) based on the comprehensive evaluation value R (s, p), and stores the paired teacher pattern and convergence condition in the teacher pattern storage unit 18.

As described above, when the teacher pattern in the s-th step is generated, the processing unit 3 of the neural network thereafter performs learning by the BP method based on the teacher pattern TPN stored in the teacher pattern storage unit 18. Then, the learning device after the learning is caused to act in the next (s + 1) step, and the teacher pattern generation process and the learning process are executed for the action in the (s + 1) step.

Thereafter, the same processing is repeated, and the learning is terminated when the output error for each teacher pattern in a certain step becomes equal to or less than the maximum allowable error or when a set learning time has elapsed. It should be noted that since the convergence condition is set for each teacher pattern, learning is not performed on the pattern that satisfies the convergence condition.

C. Method for Determining Evaluation Value Next, a method for determining the overall evaluation value in the teacher pattern creating section 9 will be specifically described with reference to FIG. FIG. 4 shows the change of the evaluation value with respect to the past action P (s), and the first evaluation value A (s, p), the second evaluation value B (s, p), The total evaluation value R (s, p) is shown, and the past action P (s) is shown beside. First, step 0 represents an initial state. The information input from the environment in step 0 is processed to determine an action. This determined action is indicated by a black circle. Hereinafter, the black circles represent the actions at that step. According to this, it moves to the next state.

In step 1, the action in step 0 is represented by a white circle. In the following, white circles indicate past actions. For example, the robot moves due to the action performed in step 0 (indicated by a white circle in step 1), and the environment changes. This is observed externally and given an evaluation. In this example, it is assumed that there is no external evaluation in step 1. No external evaluation means 0 (C = 0) in the evaluation value. That is,
The external evaluation value (first evaluation value) of the action of step 0 in step 1 is 0.0. Regarding the forgetting of the memory of the past action, the second evaluation value B (s, p) is not calculated because the action has just been made and there is no memory of the past action (memory of the comprehensive evaluation value). The sum of these two is the overall evaluation value of the action P (0) that occurred in step 0 in step 1, and the value is 0.0.

Hereinafter, similarly, the process proceeds to Steps 2 and 3, during which no external evaluation is given. Therefore, the first evaluation value is assigned 0.0.

As a result of the action performed in step 3, the state of step 4 is set. At this time, an evaluation of “good (C = 1)” is given from outside. Then, past actions include (1)
A first evaluation value is assigned according to the equation. This is the fourth
This is the value written in the first row of the figure. "Good" in step 4
Was evaluated, so the action P that occurred in step 3
A first evaluation value of 1.0 is assigned to (3).
Hereinafter, 0 is set for the action P (2) performed in step 2.
74, 0.5 for action P (1) taken in step 1
5. 0.41 for the action P (0) taken in step 0
Becomes This is a value calculated from the equation (1). On the other hand, regarding the forgetting of the past evaluation, since the evaluation value of each action is 0.0 in Step 3, it is 0.0 even when calculated using the equation (2). After all, in step 4, the evaluation value from the outside is determined by each action P in this step.
The final comprehensive evaluation value of (0) to P (3) is obtained.

In step 5, since the external evaluation is not given here, the first evaluation value of the first stage is 0.0. But,
Now that the overall rating for each action has been given in step 4, the forgetting must be calculated. The result calculated using equation (2) is the value written in the second row of step 5.

 The value changes in this way.

In step 7, an external evaluation of “bad” (C = −
Although 1) is given, this is the same processing except that the calculation of the evaluation value from the external evaluation described above is made negative.

As described above, the learning device causes the unprocessed or learned processing unit 3 to perform an action of processing the external information. If an evaluation is given from the outside at that time, the learning device calculates the influence of the external evaluation signal on the past behavior in the external evaluation conversion unit 13 in order to reflect the evaluation in a series of actions, It is converted into an evaluation value A (s, p). In the forgetting processor 14, the forgetting degree B (s, s) of the comprehensive evaluation value for each pattern calculated one step before is calculated.
Calculate p). Then, the result of adding the first evaluation value A (s, p) obtained from the external evaluation conversion unit 13 and the second evaluation value B (s, p) obtained from the forgetting processing unit 14 is the current It becomes the comprehensive evaluation value R (s, p) for the pattern (past behavior). The total evaluation value R (s, p) thus obtained
Using (p ≦ s−1), a teacher pattern is selected in the teacher pattern determination unit 16. In other words, the teacher pattern determination unit 16 selects, as a teacher pattern, those whose overall evaluation value is equal to or greater than a certain reference value (= 0.2), and selects the convergence condition determination unit.
17 changes the convergence condition depending on the height of the total evaluation value. Highly evaluated behaviors (input / output patterns) have strict convergence conditions, while those with low evaluations have looser convergence conditions. When the teacher pattern and the convergence condition generated by such processing are presented to the processing unit 3, the processing unit 3 performs learning using these.

When the learning progresses to some extent, the processing unit 3 can perform an action according to the external evaluation criteria, and the evaluation from the outside increases the “good” evaluation.

As described above, if the learning method of the present invention is used, it is only necessary to externally observe only the behavior of the learning device and give an evaluation with a simple signal of “good” or “bad”, and it is necessary to change the self-evaluation standard of the device. Disappears. In addition, since real-time learning can be performed, learning according to the situation on the spot can be performed. In addition, external evaluation methods are "good" and "bad"
Since these two types of evaluation signals are used, the versatility is sufficiently provided.

D. Learning results Fig. 5 is an explanatory diagram of the movement trajectory of the target. TG is the target, Ps is the movement start point of the target, RT is the movement trajectory of the target, WL is the peripheral wall, and Fig. 5 (a) shows that the target is linear. In the case of moving, the target in FIG.
(C) shows the movement trajectory of the target when the target rotates counterclockwise, and FIG. (D) shows the movement trajectory of the target when the target rotates clockwise.

FIG. 6 is a diagram showing a robot following locus RR when the robot RB follows the target TG when the target TG moves linearly. FIG. 6A shows the case where noise is added, and FIG. b) is a case where noise is not added. Note that Ps' is a robot movement start point.

FIG. 7 is a diagram showing a robot following locus RR in the case where the robot RB follows the target TG when the target TG moves in an inverted four-character shape. FIG. 7 (a) shows the case where noise is added. FIG. 6B shows a case where noise is not added.

FIG. 8 is a diagram showing a robot following locus RR when the robot RB follows the target TG when the target TG moves in the counterclockwise direction. FIG. 8A shows the case where noise is added. (B) is a case where noise is not added.

FIG. 9 is a diagram showing a robot following locus RR when the robot RB follows the target TG when the target TG moves in the clockwise direction. FIG. b) is a case where noise is not added.

When learning, the robot RB starts from the start point Ps' and collides with the target TG, or when it collides with the wall WL, returns to the start point again and starts over again.
Also returns to the start point and completes the movement shown in FIG. And if these are defined as one try,
The results shown in FIGS. 6 to 9 are the results of 1000 trial learning. It can be seen that for all patterns, the target is successfully tracked. Among these, the straight line follows well, but if the target moves in a circle, it seems a little difficult to follow. The influence of the noise on the learning result was not so much, probably because the noise was small.

(B) Another embodiment of the present invention In the above, the external evaluation was reflected in the past behavior by the equation (1). However, the present invention is not limited to the equation (1), and another decay function is used. Is also good.

In addition, the past behavior evaluation is forgotten according to the equation (2). However, the present invention is not limited to the equation (2), and another function can be used.

Further, in the above description, the case where the robot is trained on the target tracking and the target is actually tracked after the learning has been described. However, the present invention is not limited to such a case and can be applied to another application.

As described above, the present invention has been described with reference to the embodiments. However, the present invention can be variously modified in accordance with the gist of the present invention described in the claims, and the present invention does not exclude these.

[Effects of the Invention] As described above, according to the present invention, an interface for external evaluation is provided, a teacher pattern is created in accordance with an external evaluation of one's own behavior, and self-learning is immediately performed based on the teacher pattern. This process can be repeated in real time to learn in real time. For this reason, there is no need to create and replace the self-evaluation criteria each time according to the application field of the learning device, and it is flexible.

The external evaluation may be of two types, good and bad. In addition, it is not necessary to evaluate each step, and the external evaluation may be performed at appropriate time intervals. Therefore, the external evaluation can be easily performed. There is.

Further, past behavior can be used as a teacher pattern. In this case, the external evaluation is reflected in the past behavior, the evaluation value is reduced as the past goes, and the evaluation of the past behavior is changed with time. By attenuating accordingly,
The number of past actions to be adopted as teacher patterns can be limited, and the number of teacher patterns can be moderated.

Also, the maximum allowable error of the output data is determined for each teacher pattern, and when the output data for all the teacher patterns becomes less than the maximum allowable error, the learning is terminated. The higher the value of the teacher pattern,
Since the maximum allowable error was reduced and the convergence condition was strict,
Sufficient learning can be performed on the learning device, and processing with few errors can be performed in actual applications.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating the principle of the present invention, FIG. 2 is a block diagram of an embodiment of the present invention, FIG. 3 is a block diagram of an embodiment of a teacher pattern creating unit, and FIG. FIG. 5 is an explanatory view of a moving path of a target, FIG. 6 is an explanatory view of a robot following path when the target moves linearly, and FIG. 7 is a figure in which the target moves in an inverted four-character shape. FIG. 8 is an explanatory view of a robot following locus when the target moves counterclockwise, FIG. 9 is an explanatory view of a robot following locus when the target moves clockwise, FIG. Fig. 11 is a block diagram of a learning device related to a self-learning method of which application has been filed, Fig. 11 is a block diagram of an evaluation unit in Fig. 10, Fig. 12 is an explanatory diagram of a visual sensor, and Fig. 13 is a conceptual explanatory diagram of evaluation. . 3 Processing unit (neural network) 8 External evaluation input unit 9 Teacher pattern creation unit 11 Input / output pattern correspondence unit 12 Evaluation storage unit 13 External evaluation conversion unit 14 Forgetting process Unit 15 Operation unit 16 Teacher pattern determination unit 17 Convergence condition determination unit 18 Teacher pattern storage unit

Continuation of the front page (72) Inventor Minoru Sekiguchi 1015 Uedanaka, Nakahara-ku, Kawasaki City, Kanagawa Prefecture Inside Fujitsu Limited (56) References Tamami Kousaka, Kazushige Saga, Minoru Sekiguchi, Shigemi Nagata, “Self-Teacher Signal Learning System with Generation Function ”, Proc. Of the IEICE Autumn National Convention 1990 (6) 6-455 to 6-456 (September 15, 1990) (58) Field surveyed (Int. Cl. ⁶ , DB name) G06F 15/18 JICST file (JOIS)

Claims (7)

(57) [Claims] 1. A self-learning method for a learning device that creates a teacher pattern based on an evaluation given externally to the behavior of the learning device and learns based on the teacher pattern. When an external evaluation is given from, a first evaluation value is calculated and given to a past action based on the external evaluation so that the absolute value becomes smaller as going back to the past, and the past action one step before is given. A second evaluation value is calculated by weakening an overall evaluation value of each action by a predetermined forgetting rate, and a sum of a first evaluation value for the external evaluation and a second evaluation value weakened by the predetermined forgetting rate is calculated at the present time. A learning device is obtained for each action as an overall evaluation value of past actions, an action having an overall evaluation value equal to or greater than a set value is selected as a teacher pattern, and learning is performed based on the teacher pattern. To act, self-learning method for learning device characterized by performing said teacher pattern generating process and the learning process. 2. The external evaluation is classified into a good evaluation and a bad evaluation. The external evaluation value of the former is +1 and the external evaluation value of the latter is -1.
The self-learning method of the learning device according to claim 1, wherein: 3. When the current step is the s step, the p step
The first evaluation value A (s, p) for the past action that occurred in the step (p ≦ s−1) is expressed by the following equation: A (s, p) = C · exp [−α (s−p−1) The self-learning method of the learning device according to claim 2, wherein the calculation is performed by: 4. The p-th step one step before the current step
The overall evaluation value of the action that occurred in the step (p ≦ s−1) is R
(S−1, p), where b is the forgetting rate, the second evaluation value B (s, p) is calculated by the following equation: B (s, p) = b · R (s−1, p) 4. The self-learning method for a learning device according to claim 3, wherein 5. The method according to claim 1, wherein the action is represented by a pair of an input pattern and an output pattern to the learning device.
Self-learning method of the described learning device. 6. An actual output pattern output from a learning device and an output pattern included in the teacher pattern at the time of learning based on the teacher pattern, determining a maximum allowable error of the output pattern corresponding to the teacher pattern. The learning is not performed on the teacher pattern when the error between them is equal to or less than a predetermined maximum allowable error, and the learning is terminated when the output pattern is equal to or less than the maximum allowable error for all the teacher patterns. The self-learning method of the learning device according to claim 5, wherein: 7. The self-learning method for a learning device according to claim 6, wherein the maximum allowable error is reduced as the overall evaluation value of the behavior serving as the teacher pattern increases.