JP6858724B2 - Machine learning system - Google Patents

Description

The present disclosure relates to a machine learning system.

Automation at water purification plants, power plants, factories, or facilities that operate any machine is usually programmed according to rules. Human experts usually predefine the rules. However, rule-based systems are often inadequate. This is because many decisions can arise from the experience of experts who are not predefined as rules. In addition, as the number of specialists is decreasing due to the aging of the population, smart automation and smart automation systems will be indispensable in the future.

Reinforcement learning methods allow the system to be trained to operate automatically and efficiently. The conventional method is well known as the Deep Reinforcement Learning Method (DRL). The DRL has a neural network that learns a Q value indicating an action value and operates based on the highest Q value. For example, in the case of waterworks, the action is how much the system should increase or decrease the amount of water intake. The water intake rate is ^{represented by +/- xxm 3} / h (xx is a positive number). The neural network learns the Q value of each operation. The higher the Q value, the better the behavior.

In the conventional DRL method, the experience experienced by the agent controlling the behavior of the system is sampled from the replay memory (database), and the experience is learned from the experience by the backpropagation method. These experiences are generated during learning and stored in replay memory. When the replay memory is full, the oldest experience is deleted (eg, US Application Publication No. 2015/0100530).

US Application Publication No. 2015/0100530

Experiences stored in replay memory include rare and unique experiences. For example, in the case of waterworks, storms occur only once every few years and do not occur frequently. With a limited size (maximum number of records) replay memory, rare and unique experiences cannot be kept in the replay memory and are erased before the neural network learns.

In other respects, when the replay memory is filled with random behaviors (early in training), it is difficult to obtain behaviors suitable for a series of trainings that lead to good or bad results. Therefore, it takes a long time for the neural network to actually be "exposed" to various experiences in training. On the other hand, if the replay memory is filled with only good behaviors (at the end of training), the neural network is exposed only to good behavioral experiences and may forget the bad behavioral experiences. Therefore, the reliability of the trained neural network may be reduced.

However, it is not possible to provide an infinite storage area to store all possible experiences. Therefore, a technique capable of maintaining an experience that enables effective learning in a limited storage area is desired.

The machine learning system of one aspect of the present disclosure is an experience sampled from a first memory database for storing experiences, a second memory database for storing experiences, the first memory database, and the second memory database. The learning department, which trains and updates the function that determines the value of behavior in the environment, determines whether the experience is important based on preset rules, and determines that the experience is important in the second memory database. The important experience evaluation unit and the learning agent unit, which are stored in the environment, and the learning agent unit, which determines the behavior in the environment by selecting either a decision based on the output of the function or an arbitrary decision during the training period of the function, and the environment It includes an environment unit provided to the learning agent unit and storing the experience of the action determined by the learning agent unit in the first memory database.

According to one aspect of the present disclosure, it is possible to maintain an experience in which learning can be effectively performed in a limited storage area.

It is explanatory drawing of the structure of the learning system which concerns on 1st Embodiment. It is a functional block diagram of the learning system which concerns on 1st Embodiment. It is explanatory drawing of the memory database which concerns on 1st Embodiment. It is explanatory drawing of the important experience memory database which concerns on 1st Embodiment. It is a flowchart of the environment part which concerns on 1st Embodiment. It is a flowchart of the important experience evaluation part which concerns on 1st Embodiment. It is a flowchart of the learning part which concerns on 1st Embodiment. It is a flowchart of the learning agent part which concerns on 1st Embodiment. It is explanatory drawing of the structure of the learning system which concerns on 2nd Embodiment. It is a functional block diagram of the learning system which concerns on 2nd Embodiment. It is a flowchart of the test agent part which concerns on 2nd Embodiment. It is a flowchart of the important experience evaluation part which concerns on 2nd Embodiment. It is explanatory drawing of the structure of the learning system which concerns on 3rd Embodiment. It is a functional block diagram of the learning system according to 3rd Embodiment. It is a flowchart of the artificial operation part which concerns on 3rd Embodiment. It is explanatory drawing of the structure of the learning system which concerns on 4th Embodiment. It is a functional block diagram of the learning system according to 4th Embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. It should be noted that the present embodiment is merely an example for realizing the present invention and does not limit the technical scope of the present invention. The same reference numerals are given to common configurations in each figure.

In the following, a stable learning system including a plurality of memory databases (also simply referred to as memory or database) is disclosed. The technique of the present disclosure can be applied to an automatic operation system not only in facilities such as water purification plants, power plants, and factories, but also in facilities that operate arbitrary machines.

The learning system includes a separate memory database for important experiences to improve training efficiency. Significant experiences can include experiences of good (appropriate) or bad (inappropriate) behaviors that rarely occur during training, or experiences collected from human manipulation. By storing important experiences in a separate memory database, important experiences can be maintained and reused even after the memory database that stores other experiences is full.

Deep reinforcement learning agents learn from multiple memory databases, including a memory database that stores important experiences. The number of memory databases used is two or more. Significant experience may be collected based on entropy or some specific method of collecting experience.

Agents that perform actions to generate significant experiences may be generated and used separately from agents that learn. The decisions of human experts may be used as an important experience. The learning process learns from, for example, a plurality of memory databases at the same time, or learns from different memory databases at different times.

<First Embodiment>
The first embodiment will be described with reference to FIGS. 1 to 8. The learning system not only learns from the experience of the first memory database, but also learns from the important experience memory database which is the second memory database. The critical experience assessment program determines which experiences are important to the learning process and stores them in the critical experience memory database.

FIG. 1 is an explanatory diagram of a configuration of a learning system according to the first embodiment. The server 100 includes a processor 110, a memory 120, and an auxiliary storage device 130. The memory 120 and / or the auxiliary storage device 130 is a storage device. The above components are connected to each other by a bus. The processor 110 refers to the memory 120 and executes various arithmetic processes.

The memory 120 includes an environmental program 121, an important experience evaluation program 122, a learning program 123, and a learning agent program 124. The program stored in the memory 120 and the data used by the program are loaded from, for example, the auxiliary storage device 130 or the outside via the network.

The environmental program 121 can observe the characteristics of the real world or simulate the real environment. For example, in the case of a real-world waterworks, environmental program 121 can observe all the characteristics of the waterworks, such as those observed by a sensor array that senses all the properties of water and the amount and level of water. In the case of a water purification plant in the simulation, the environmental program 121 can simulate changes in water characteristics, changes in water volume, and changes in water level after the operation is performed.

Based on these characteristics of the environment, the system can identify the current state of the environment, the next state of the environment after the action has been taken in the operation, and the action is good for the purpose. It holds the criteria for determining whether it is good or bad.

The critical experience evaluation program 122 evaluates the experience defined as a tuple of four information elements. The four information elements are the original state (current state), the agent's behavior, the reward received by the agent, and the next state. The critical experience evaluation program 122 evaluates whether the experience is important and determines whether to store the experience in a particular database called the critical experience memory database.

The learning program 123 includes a neural network for learning the action value. The neural network receives the input of the current state (set of sensor readings) and outputs the value of each action (action value). The higher the value, the better the behavior. The learning program 123 also updates and optimizes the parameters of the neural network. The learning program 123 may use a function different from that of the neural network, for example, a linear regression function may be used.

The learning program 123 samples the experience stored in the memory database 131 and the important experience memory database 132 from the auxiliary storage device 130, and optimizes the parameters of the neural network by backpropagation. By doing so, the neural network can more accurately estimate the action value (or Q value) of each action in a given state, and thus the system can perform the action more wisely.

The learning program 123 executes pack propagation to optimize the parameters of the neural network. By learning from the important experience memory database 132, the neural network can be learned faster and more stably.

The learning agent program 124 determines the behavior of the system. For example, in the case of a water purification plant, changing the water intake rate changes the amount of water in the reservoir. Therefore, as a result of this action, it is necessary to secure sufficient water in the reservoir. The server 100 also includes an auxiliary storage device 130 for storing the memory database 131 and the critical experience memory database 132.

The memory database 131 stores a set of tuples having four elements. The four elements are the original state, behavior, reward, and the next state (new state), and the tuple is the agent's experience during the training period. The memory database 131 has a limited capacity (eg, 100 experiences) predefined by the engineer. When the memory database 131 is full, the first experience of the memory database 131 is deleted, creating space for the new experience. The structure of the memory database 131 will be described later with reference to FIG.

The critical experience memory database 132 has a structure in which additional information on the importance of experience is added to the structure of the memory database 131. Importance is calculated by the Critical Experience Assessment Program 122 and indicates how important the experience is. The critical experience memory database 132 also has a limited capacity, and when it is full, less important experiences are deleted.

FIG. 2 is a functional block diagram of the learning system according to the first embodiment. The environment unit 1210 is realized by the processor 110 that executes the environment program 121. The environment unit 1210 can observe the current state of the real environment or simulate the real world environment. The environment unit 1210 receives an action from the learning agent unit 1240, executes the action, and shifts to the next state. Next, the environment unit 1210 determines the reward based on the next state, and stores the experience, which is a tuple including the information of the original state, the action, the reward, and the next state, in the memory database 131. The Environment Department 1210 also sends the experience to the Important Experience Evaluation Department 1220. The processing flow of the environment unit 1210 will be described in detail later with reference to FIG.

The Important Experience Evaluation Department 1220 evaluates the importance of experience. When the importance of experience is evaluated to be high, it is stored in the important experience memory database 132. The processing flow of the important experience evaluation unit 1220 will be described in detail later with reference to FIG.

The learning unit 1230 optimizes the weight and bias (parameter) of the neural network so that the neural network can more accurately estimate the Q value of each operation with respect to the given current state during the training time. .. The learning unit 1230 is realized by executing the learning program 123 by the processor 110.

The learning unit 1230 samples a certain number of experiences from the memory database 131 and performs backpropagation to optimize the weights and biases of the neural network. Further, at predetermined intervals, the learning unit 1230 samples a predetermined number of experiences of the important experience memory database 132 and performs backpropagation to optimize the weight and bias of the neural network. The processing flow of the learning unit 1230 will be described in detail later with reference to FIG. 7.

The learning agent unit 1240 determines the behavior of the learning system. The learning agent unit 1240 is realized by executing the learning agent program 124 by the processor 110. The learning agent unit 1240 receives sensor data (a set of sensor readings) as the current state from the environment unit 1210, and uses the feed forward of the neural network included in the learning unit 1230 to take Q for each action to be taken. Get the value.

However, the learning agent unit 1240 may or may not use the Q value information during the training period. The learning agent unit 1240 has a parameter called epsilon (ε) and determines whether to use the information from the Q value or to search for random behavior. The parameter ε is preset by the engineer with a fixed value (eg, ε = 0.5) or decreases linearly from 1 to 0 during the training period.

The learning agent unit 1240 randomizes the numbers and compares them with the ε value to determine exploitation (use) or exploitation (search). Finally, the learning agent unit 1240 sends the determined action to the environment unit 1210. The processing flow of the learning agent unit 1240 will be described in detail later with reference to FIG.

FIG. 3 is an explanatory diagram of the memory database 131 according to the first embodiment. The memory database 131 includes an index column 301, a status column 302, an action column 303, a reward column 304, and the next status column 305.

The index column 301 indicates an index, which is an integer indicating a sequence of experiences stored in the memory database 131. The index provides information on which oldest experience is stored in memory and should be deleted to store the new experience when the memory is full.

Each cell in the state column 302 stores a set of sensor readings (sensor data) from the real world or the simulation world that defines the current state (original state).

The action column 303 indicates a numerical value indicating an action that the system can take. The behavior is based on the specifications of the real world system. For example, in order to increase or decrease the rate of water intake from a water purification plant, the action can take any value between -1 and 1.

The reward column 304 indicates a number that defines the reward that can be obtained after the learning agent unit 1240 takes an action to transition to the next state. For example, in the case of a water purification plant, if the water intake rate is increased during high demand hours, the reservoir will be supplied with sufficient water and a positive reward such as +1 will be obtained. Or, if you do not increase the water intake rate during high demand times, the reservoir will not be supplied with enough water and the water level will drop to dangerous levels, with negative rewards such as -0.5. Or, when the water level reaches zero, you get a high negative reward such as -1.

The next state column 305 indicates a transition state after the system receives any action of the action column 303 from any state of the state column 302. The following states allow the system to define the reward shown in reward 304. Each cell in the next state column 305 stores a set of sensor readings from a real-world or simulated environment.

FIG. 4 is an explanatory diagram of the important experience memory database 132 according to the first embodiment. The important experience memory database 132 includes an index column 401, a status column 402, an action column 403, a reward column 404, the next status column 405, and an importance column 406. The first five columns 401-405 have a structure similar to the memory database 131. The sixth column, the importance column 406, stores the importance of the associated experience. In this example, the importance is normalized in the range 0 to 1. 0 means very low importance and 1 means very high importance.

FIG. 5 is a flowchart of the environment unit 1210 according to the first embodiment. First, the environment unit 1210 observes the current state of the environment in step 501. When the Environment Department 1210 acquires information from the real world, the current sensor array data indicates the current state of the environment. When the environment unit 1210 acquires information from the simulation, the current sensor array data is the current observed value of the sensor data in the simulation environment.

Next, in step 502, the environment unit 1210 receives an action from the learning agent unit 1240. Next, in step 503, the environment unit 1210 makes a transition from the current state in the environment to the next state by executing an action in the real world or in a simulation. In step 504, the Environment Department 1210 determines the reward for the next new state. In step 505, the Environment Department 1210 compiles information on the original state, behavior, reward, and next state into a tuple.

In step 506, the environment unit 1210 stores the information summarized in the tuple in step 505, including the original state, action, reward, and the next state, in the memory database 131. Finally, in step 507, the Environment Department 1210 sends the same tuple to the Important Experience Evaluation Department 1220. This completes the processing of the environment unit 1210.

FIG. 6 is a flowchart of the important experience evaluation unit 1220 according to the first embodiment. First, in step 601 the important experience evaluation unit 1220 receives the experience from the environment unit 1210 in the form of a tuple having four pieces of information: the current state, the action, the reward, and the next state. Next, in step 602, the important experience evaluation unit 1220 evaluates the importance of the experience. There are several ways to assess the importance of experience. An example of the evaluation method is described below, but other methods can also be used.

One method is evaluation based on the rewards received. High positive or low negative rewards are usually essential for the system to learn. Therefore, experiences involving high positive or low negative rewards are of high importance and vice versa.

Another method is an entropy-based evaluation of experience. Entropy (information theory) is usually the measurement of the amount of information missing prior to reception and can be used to assess the importance of experience. The entropy of experience is the importance of experience and is calculated according to the following entropy formula.

b is usually equal to 2, xi is each information element in the tuple of experience, and n is equal to 4.

After evaluating the importance of the experience, the critical experience evaluation unit 1220 compares the result with the threshold value in step 603. The threshold is usually preset by an engineer. If it is determined that the importance of the experience is greater than the threshold value (603: YES), the experience information is stored in the important experience memory database 132 in step 604, and the process of the important experience evaluation unit 1220 ends. When it is determined that the importance of experience is less than or equal to the threshold value (603: NO), the process of the important experience evaluation unit 1220 ends.

FIG. 7 is a flowchart of the learning unit 1230 according to the first embodiment. The learning unit 1230 starts processing of this flowchart every time a new experience is stored in the memory database 131, for example. First, the learning unit 1230 determines in step 700 whether to learn from the important experience memory database 132.

For example, learning from the critical experience memory database 132 is determined periodically. The learning unit 1230 determines that sampling is performed from the important experience memory database 132 once every specified number of times of execution of this process, for example. In one example, the sampling frequency from the memory database 131 is greater than the sampling frequency from the critical experience memory database 132. Important experience The experience stored in the memory database 132 is biased more than the experience stored in the memory database 131, which enables learning based on appropriate experience.

When it is determined not to learn from the important experience memory database 132 (700: NO), the learning unit 1230 determines to learn from the normal memory database 131. Next, in step 701, the learning unit 1230 checks whether sufficient experience is stored in the memory database 131 before learning. Usually, it is necessary to give some experience to the memory database 131 before sampling and learning. The number of experiences to be stored is predetermined by the system engineer.

When it is determined in step 701 that sufficient experience is not stored in the memory database 131 (701: NO), the process of the learning unit 1230 ends. If it is determined in step 701 that sufficient experience is stored in the memory database 131 (701: YES), the learning unit 1230 randomly samples N experiences from the memory database 131 in step 702. N is an integer predetermined by the engineer (eg, N = 32).

Each sampled experience is represented as <ss, aa, rr, ss'>. ss is the original state, aa is the action taken in the original state, rr is the reward in the next new state, and ss'is the next new state.

In step 703, the learning unit 1230 calculates the target value (desired output) tt of each experience for training the neural network based on each experience sampled from the memory database 131 according to the following rule. When ss'is the last state, tt = rr. In other cases, tt = rr + γmax aa'[Q (ss _{', aa')].} aa'is an action that can be taken in the following states. γ is the discount rate. γ is a value from 0 to 1 preset by the engineer, for example 0.9.

Next, in step 704, the learning unit 1230 uses the loss function (tt−Q (ss, aa)) ² to perform mini-batch backpropagation to calculate changes in neural network weights and biases. To do. Then, in step 705, the learning unit 1230 updates the weight and bias of the neural network.

When it is determined in step 700 to learn from the important experience memory database 132 (700: YES), the learning unit 1230 uses the memory database 131 and similarly to the process of steps 701 to 705 described above, the important experience memory database 132. Are used to perform steps 706-710. This completes the process of the learning unit 1230.

FIG. 8 is a flowchart of the learning agent unit 1240 according to the first embodiment. First, in step 801 the learning agent unit 1240 receives the sensor data from the environment unit 1210 as the current state. Next, in step 802, the learning agent unit 1240 transfers this state to the neural network of the learning unit 1230, and receives the Q value of each action from the neural network.

Next, the learning agent unit 1240 generates a random number from 0 to 1 in step 803. Then, in step 804, the learning agent unit 1240 compares the random number generated in step 803 with the parameter ε.

If it is determined in step 804 that the random number is not greater than ε (804: NO), the learning agent unit 1240 randomly selects an action in step 805. If it is determined in step 804 that the random number is greater than ε (804: YES), the learning agent unit 1240 selects the action with the highest Q value in step 806. Finally, in step 807, the learning agent unit 1240 sends the selected action to the environment unit 1210 and executes the action. This completes the process of the learning agent unit 1240.

As described above, according to the first embodiment, efficient and stable learning is realized by preparing a plurality of memory databases including an important experience memory database and sampling from each of the plurality of memory databases for learning. Will be done.

The above example samples experience for learning from two memory databases, but the system may sample experience for learning from three or more memory data. For example, the system may include two critical experience memory databases. The first important experience memory database stores experiences of importance of the first threshold and the second threshold, and the second important experience memory database stores experiences of higher importance than the second threshold. These sampling periods may be the same or different.

In the above example, the experience of high importance is stored in both of the two memory databases, but the experience of high importance may be stored only in the important experience memory database.

<Second Embodiment>
Next, the second embodiment will be described with reference to FIGS. 9 to 12. In the first embodiment, only the experience during the training period has the opportunity to be evaluated and stored in the critical experience memory database 132. However, during the training period, actions are taken randomly when ε is high. Therefore, the experience when ε is 0 and ε does not affect the behavior of the neural network can be considered as an important experience.

From this point of view, test agents are created at regular intervals during the training period to gather important experience. The experience collected by the test agent at a particular point in time can be interpreted as an indication of how well a neural network can behave if it is used at that point in time during the training period. Those experiences from the test agent can be considered to be of high importance and are stored in the critical experience memory database 132.

FIG. 9 is an explanatory diagram of the configuration of the learning system according to the second embodiment. In this configuration, the components other than the added test agent program 125 in the memory 120 are the same as the components of the learning system according to the first embodiment. The test agent program 125 operates only according to the exploitation rules. That is, the test agent program 125 only selects the action with the highest Q value from the neural network. The processing flow of the test agent program 125 will be described later with reference to FIG.

FIG. 10 is a functional block diagram of the learning system according to the second embodiment. The components other than the newly added test agent unit 1250 are the same as the components of the functional block diagram of the learning system according to the first embodiment. The test agent unit 1250 is realized by the processor 110 that executes the test agent program 125.

The test agent unit 1250 is created periodically to test the ability of the neural network in a particular snapshot of training. The test agent unit 1250 determines the behavior in the same manner as the learning agent unit 1240, except that it acts only using the exploitation rule (ε = 0). The experience obtained by the action of the test agent unit 1250 is sent to the important experience evaluation unit 1220, determined to be important experience, and stored in the important experience memory database 132.

FIG. 11 is a flowchart of the test agent unit 1250 according to the second embodiment. First, in step 1101, the test agent unit 1250 receives the sensor data from the environment unit 1210 as the current state. Next, in step 1102, the test agent unit 1250 transfers this state to the neural network of the learning unit 1230, and receives the Q value of each action from the neural network.

In step 1103, the test agent unit 1250 selects the action having the highest Q value. In step 1104, the test agent unit 1250 sends the selected action to the environment unit 1210 and executes the action. Finally, the test agent unit 1250 obtains experience from the environment unit 1210 and sends this experience to the important experience evaluation unit 1220. As a result, the processing of the test agent unit 1250 is completed.

FIG. 12 is a flowchart of the important experience evaluation unit 1220 according to the second embodiment. First, in step 1201, the important experience evaluation unit 1220 receives the experience from the environment unit 1210 or the test agent unit 1250 in the form of a tuple having four pieces of information: the current state, the action, the reward, and the next state.

Next, the critical experience evaluation unit 1220 checks in step 1202 whether the experience came from the test agent unit 1250. If it is determined that the experience is from the test agent unit 1250 (1202: YES), in step 1205, the important experience evaluation unit 1220 sends and stores the information of the experience directly to the important experience memory database 132. If it is determined that the experience is not from the test agent (1202: NO), the critical experience evaluation unit 1220 evaluates the importance of the experience, as described in step 1203 in the first embodiment.

After evaluating the importance of the experience, the critical experience evaluation unit 1220 compares the importance with the threshold value in step 1204. The threshold is usually preset by an engineer. When it is determined that the importance of the experience is greater than the threshold value (1204: YES), in step 1205, the important experience evaluation unit 1220 transmits and stores the information of the experience in the important experience memory database 132, and evaluates the important experience. The processing of unit 1220 ends. When it is determined that the importance of experience is less than or equal to the threshold value (1204: NO), the process of the important experience evaluation unit 1220 ends.

<Third Embodiment>
Next, the third embodiment will be described with reference to FIGS. 13 and 14. In the second embodiment, the test agent program 125 generates important experiences stored in the critical experience memory database 132. This embodiment illustrates that instead of using a test agent, the manipulative program 126 can also be a source of significant experience. Both the test agent program 125 and the artificial operation program 126 may be executed.

FIG. 13 is an explanatory diagram of the configuration of the learning system according to the third embodiment. In this configuration, the test agent 125 of the configuration according to the second embodiment is replaced with the artificial operation program 126. Further, the input device 142 and the display device 143 are connected to the bus via the interface 141. The display device 143 is an output device. Other components are the same as the components of the learning system of the second embodiment.

The artificial operation program 126 is an interface between the system and a human being. The artificial operation program 126 provides the user with a graphical user interface, for example, together with the input device 142 and the display device 143. The main purpose of the user interface, including the manipulative program 126, is to collect the decisions of human experts in a given state. These experiences are also treated as important experiences and are stored in the important experience memory database 132.

FIG. 14 is a functional block diagram of the learning system according to the third embodiment. The configuration is the same as the functional block diagram of the learning system according to the second embodiment, except that the test agent unit 1250 is replaced with the artificial operation unit 1260. The artificial operation unit 1260 is realized by the processor 110 that executes the artificial operation program 126.

The artificial operation unit 1260 can function before or during the training period. The artificial operation unit 1260 functions as a data collection unit and collects the decisions of human experts as an experience for the reinforcement learning system to learn. The collected experience is sent to the important experience evaluation unit 1220 and stored in the important experience memory database 132.

FIG. 15 is a flowchart of the artificial operation unit 1260 according to the third embodiment. First, in step 1501, the artificial operation unit 1260 receives the sensor data as the current state from the environment unit 1210. Next, in step 1502, the artificial operation unit 1260 displays this current state on the display device 143, for example, an image that is only a chart, a figure, or a number. The point is to enable humans to recognize the current state of the environment and make decisions about that state.

Next, in step 1503, the artificial operation unit 1260 waits for the input of the action from the human expert, and receives the action input by the expert from the input device 142. In step 1504, the artificial operation unit 1260 sends the selected action to the environment unit 1210 and executes the action. Finally, the artificial operation unit 1260 obtains experience from the environment unit 1210 in step 1505 and sends this experience to the important experience evaluation unit 1220. With the above, the processing of the artificial operation unit 1260 is completed. The important experience evaluation unit 1220 determines that the experience acquired from the artificial operation unit 1260 is important, and stores it in the important experience memory database 132.

<Fourth Embodiment>
Next, the fourth embodiment will be described with reference to FIGS. 16 and 17. In the third embodiment, it is described that the artificial operation program 126 collects important experiences stored in the important experience memory database 132. In this embodiment, instead of collecting data directly using the manipulative operation program 126, the critical experience evaluation program 122 receives pre-collected experiences from past operations by humans and evaluates their importance. It is stored in the important experience memory database 132. The present embodiment and the artificial operation program 126 may be executed at the same time.

FIG. 16 is an explanatory diagram of the configuration of the learning system according to the fourth embodiment. The configuration other than the added past artificial operation database 133 is the same as the configuration of the learning system according to the first embodiment. The past man-made operation database 133 has exactly the same structure as the memory database 131, except that all experiences in the database are based on past human operations and have already been collected.

FIG. 17 is a functional block diagram of the learning system according to the fourth embodiment. The configuration other than the added past artificial operation database 133 is the same as the configuration of the functional block diagram of the learning system according to the first embodiment.

The important experience evaluation unit 1220 is executed before the training period is started, and evaluates all the experiences of the past artificial operation database 133 as described in the first embodiment (importance exceeds a predetermined threshold). Import the high-priority experience into the critical experience memory database 132. As a result, the experience of high importance can be more appropriately stored in the important experience memory database 132.

The important experience evaluation unit 1220 may determine that all the experiences stored in the past artificial operation database 133 are important experiences to be stored in the important experience memory database 132. The important experience evaluation unit 1220 may evaluate that the experience stored in the past artificial operation database 133 is relatively more important than the experience stored in the memory database 131. For example, in the method of evaluating the importance of the experience of the past artificial operation database 133, a constant may be added or multiplied by the method of evaluating the experience of the memory database 131.

The learning system may sample the experience for learning directly from the past manipulative database 133 without importing the experience from the past manipulative database 133 into the critical experience memory database 132.

The present invention is not limited to the above-described examples, and includes various modifications. For example, the above-described embodiment has been described in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to the one including all the configurations described. Further, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Further, it is possible to add / delete / replace a part of the configuration of each embodiment with another configuration.

Further, each of the above-mentioned configurations, functions, processing units and the like may be realized by hardware by designing a part or all of them by, for example, an integrated circuit. Further, each of the above configurations, functions, and the like may be realized by software by the processor interpreting and executing a program that realizes each function. Information such as programs, tables, and files that realize each function can be placed in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), or a recording medium such as an IC card or an SD card. In addition, control lines and information lines are shown as necessary for explanation, and not all control lines and information lines are shown in the product. In practice, it can be considered that almost all configurations are interconnected.

100 server, 110 processor, 120 memory, 121 environment program, 122 important experience evaluation program, 123 learning program, 124 learning agent program, 125 test agent program, 126 human operation program, 130 auxiliary storage device, 131 memory database, 132 important experience Memory database, 133 Past human operation database, 1210 Environment department, 1220 Important experience evaluation department, 1230 Learning department, 1240 Learning agent department, 1250 Test agent department, 1260 Human operation department

Claims (8)

A first memory database to store experience,
A second memory database to store experiences,
A learning unit that trains and updates functions that determine the value of behavior in an environment based on the experience sampled from the first memory database and the second memory database.
An important experience evaluation unit that determines whether experience is important based on preset rules and stores the experience determined to be important in the second memory database.
During the training period of the function, a learning agent unit that determines the behavior in the environment by selecting either a decision based on the output of the function or an arbitrary decision.
An environment unit that provides the environment to the learning agent unit and stores the experience of the action determined by the learning agent unit in the first memory database.
Machine learning system including. The machine learning system according to claim 1.
The important experience evaluation unit is a machine learning system that acquires experience from the environment unit and stores the acquired experience in the second memory database when the importance of the acquired experience exceeds a threshold value. The machine learning system according to claim 1 or 2.
In addition, it includes a test agent section that determines actions based on the output of a particular snapshot of the function during the training period of the function.
The important experience evaluation unit is a machine learning system that stores the experience of the behavior determined by the test agent unit in the second memory database. The machine learning system according to claim 1, 2 or 3.
It further includes an artificial operation unit that collects the experience of human-operated behavior in the environment during the training period.
The important experience evaluation unit is a machine learning system that stores the experience of behavior by the artificial operation in the second memory database. The machine learning system according to any one of claims 1 to 4.
The important experience evaluation unit is a machine learning system that stores the experience of actions determined in the past by human operation in the second memory database. The machine learning system according to claim 5.
It further includes a third database that stores the experience of previously determined behaviors by human manipulation in the environment.
The important experience evaluation unit is a machine learning system that stores the experience acquired from the third database in the second memory database. The machine learning system according to claim 6.
The important experience evaluation unit is a machine learning system that stores in the second memory database the experience selected based on the importance from the experience acquired from the third database. A method in which a computer system including a storage device and a processor that operates according to a program stored in the storage device performs machine learning.
The storage device is
A first memory database that stores experience,
Includes a second memory database that stores experience,
In the method, the processor
Based on the experience sampled from the first memory database and the second memory database, the functions that determine the value of behavior in the environment are trained and updated.
Whether or not the experience is important is determined based on a preset rule, and the experience determined to be important is stored in the second memory database.
During the training period of the function, one of the decisions based on the output of the function or the arbitrary decision is selected to determine the behavior in the environment.
The environment is provided and the experience of the determined behavior is stored in the first memory database.
How to include that.

Images