JP7635849B2 - Reinforcement learning system, reinforcement learning device, reinforcement learning method and program - Google Patents

Description

The present invention relates to a reinforcement learning system, a reinforcement learning device, and a reinforcement learning method.

Research is currently being conducted into reinforcement learning, which learns actions that maximize the reward obtained when the next action is performed in a certain state. Patent Document 1 describes a technology that uses reinforcement learning to learn images of concave and convex parts and the control amount when combining parts in assembly work using a robot arm. Patent Document 2 describes a technology that uses reinforcement learning to learn the accelerator operation amount and selects an action consisting of a throttle opening command value and retard amount according to the state. Patent Document 2 also describes that a function approximator may be used for the action value function Q.

International Publication No. WO 2018/146770 Japanese Patent Application Publication No. 2021-67193

However, the techniques described in Patent Documents 1 and 2 have room for improvement in terms of selecting more suitable actions. If the action value function can be accurately estimated in reinforcement learning, an appropriate action can be selected, but the action value function estimated in the techniques described in Patent Documents 1 and 2 contains errors. It is difficult to accurately estimate the action value function, especially when the state-action space is huge.

One aspect of the present invention has been made in consideration of the above problems, and one example of its objective is to provide technology that enables the selection of more appropriate actions.

A reinforcement learning system according to one aspect of the present invention comprises an acquisition means for acquiring a first state in an environment that is the subject of reinforcement learning, a generation means for generating a second state by adding noise to the first state, a calculation means for calculating a first action value function according to the second state, and a selection means for selecting an action according to the first action value function.

A reinforcement learning device according to one aspect of the present invention comprises an acquisition means for acquiring a first state in an environment that is the subject of reinforcement learning, a generation means for generating a second state by adding noise to the first state, a calculation means for calculating a first action value function according to the second state, and a selection means for selecting an action according to the first action value function.

A reinforcement learning method according to one aspect of the present invention includes generating a second state by adding noise to the first state in an environment that is the subject of reinforcement learning, calculating a first action value function according to the second state, and selecting an action according to the first action value function.
A program according to one aspect of the present invention is a program for causing a computer to function as a reinforcement learning device, causing the computer to function as: acquisition means for acquiring a first state in an environment that is the subject of reinforcement learning; generation means for generating a second state by adding noise to the first state; calculation means for calculating a first action value function in accordance with the second state; and selection means for selecting an action in accordance with the first action value function.

According to one aspect of the present invention, a more suitable action can be selected.

1 is a block diagram showing a configuration of a reinforcement learning system according to an exemplary embodiment 1 of the present invention. 1 is a flow diagram showing a flow of a reinforcement learning method according to an exemplary embodiment 1 of the present invention. 1 is a block diagram illustrating a configuration of a reinforcement learning system according to an exemplary embodiment 1 of the present invention. 1 is a block diagram showing an example of a device configuration for realizing an exemplary embodiment 1 of the present invention. FIG. 11 is a block diagram showing a configuration of a reinforcement learning system according to an exemplary embodiment 2 of the present invention. FIG. 11 is a flow chart showing the flow of a reinforcement learning method according to an exemplary embodiment 2 of the present invention. FIG. 11 is a diagram showing an example of a game screen according to an exemplary embodiment 3 of the present invention. FIG. 11 is a diagram illustrating a first state according to an exemplary embodiment 3 of the present invention. FIG. 13 is a diagram showing an example of an evaluation result according to an application example of the exemplary embodiment 3 of the present invention. FIG. 13 is a diagram showing an example of an evaluation result according to an application example of the exemplary embodiment 3 of the present invention. FIG. 13 is a diagram showing an example of an evaluation result according to an application example of the exemplary embodiment 3 of the present invention. FIG. 13 is a diagram showing an example of an evaluation result according to an application example of the exemplary embodiment 3 of the present invention. FIG. 1 is a block diagram showing the configuration of a reinforcement learning device, a terminal 20, and a computer functioning as a server 30 according to exemplary embodiments 1 to 7 of the present invention.

[Example embodiment 1]
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A first exemplary embodiment of the present invention will be described in detail with reference to the accompanying drawings. This exemplary embodiment is a basic form of the exemplary embodiments described below.

<Configuration of Reinforcement Learning System>
The configuration of a reinforcement learning system 1 according to this exemplary embodiment will be described with reference to FIG. 1. FIG. 1 is a block diagram showing the configuration of the reinforcement learning system 1. The reinforcement learning system 1 is a system that selects an action by reinforcement learning. The reinforcement learning system 1 is, for example, a system that controls the construction operation of a construction machine such as an excavator, a system that controls transportation by a transport device, or a system for autonomous play of a computer game. However, the reinforcement learning of the reinforcement learning system 1 is not limited to the above examples, and the reinforcement learning performed by the reinforcement learning system 1 can be applied to various systems. The action is the action of an agent in reinforcement learning, and for example, the excavation operation control of an excavator, the transport operation control of a transport device, or the autonomous play control of a computer game. However, the action is not limited to these examples, and may be other than the above.

As shown in FIG. 1, the reinforcement learning system 1 includes an acquisition unit 11, a generation unit 12, a calculation unit 13, and a selection unit 14. The acquisition unit 11 is configured to realize the acquisition means in this exemplary embodiment. The generation unit 12 is configured to realize the generation means in this exemplary embodiment. The calculation unit 13 is configured to realize the calculation means in this exemplary embodiment. The selection unit 14 is configured to realize the selection means in this exemplary embodiment.

The acquisition unit 11 acquires a first state. The first state is a state in an environment that is the target of reinforcement learning. For example, when the reinforcement learning system 1 is a system for selecting an excavation operation of an excavator, the first state includes, as an example, a part or all of the posture and position of the excavator excavating the soil, the shape of the soil to be excavated, and the amount of soil in the bucket of the excavator. When the reinforcement learning system 1 is a system for selecting a transport operation of a transport device, the first state includes, as an example, a part or all of the position, movement direction, speed, and each angle of the transport device, the position of the passage, and the position and speed of a static obstacle or a dynamic obstacle. When the reinforcement learning system 1 is a system for autonomous play of a computer game, the first state includes, as an example, a state of an object that affects the progress of the game in the computer game. However, the first state is not limited to the above-mentioned ones, and may be other states. The first state may include, for example, an environmental state such as temperature or weather.

The generation unit 12 generates the second state by adding noise to the first state. An example of the noise is a random number such as a normal random number or a uniform random number. However, the noise added to the first state by the generation unit 12 is not limited to these, and may be noise other than the above. The generation unit 12 may add noise to all of the elements included in the first state, or may add noise to some of the elements included in the first state.

The calculation unit 13 calculates the first action value function according to the second state. As an example, the calculation unit 13 calculates the first action value function using a state sequence including a plurality of second states. The calculation unit 13 may also calculate the first action value function using a state sequence including the first state and one or more second states. In other words, the state sequence used by the calculation unit 13 to calculate the first action value function includes one or more second states, and the states included in the state sequence are the first state or the second state. In the following description, when it is not necessary to distinguish between the first state and the second state, they are also simply referred to as "states".

The first action value function is a function for evaluating an action in a state. As an example, the first action value function is an action value function used in Q-learning, and is updated, for example, by the following formula (1). However, the first action value function is not limited to that given by formula (1), and may be another function.

In formula (1), s _t ⁽ⁱ⁾ (1≦i≦n; i and n are natural numbers) is a state included in the state sequence (i.e., the first state or the second state), a is an action, and Q(s _t ⁽ⁱ⁾ , a) is the first action-value function. α is the learning rate, s _t+1 ⁽ⁱ⁾ is the state after the transition, r _t+1 is the reward the agent obtains when it transitions to state s _t+1 ⁽ⁱ⁾ , and γ (0≦γ≦1) is the discount rate. Also, a′∈A, and set A is a set of actions the agent can take in state s _t ⁽ⁱ⁾ .

The reward is a reward that the agent obtains from the environment by taking action. As an example, the reward is a value that is added or subtracted depending on the amount of excavation by the excavator, the time required for excavation, the time required for transportation, whether or not an obstacle is encountered during transportation, the outcome of the game, or the game score. However, the reward is not limited to these examples and may be something other than the above.

When formula (1) is used, the calculation unit 13 calculates the first action-value function for each state included in the state sequence. In other words, the calculation unit 13 calculates the first action-value function for the number of states included in the state sequence.

The selection unit 14 selects an action according to the first action value function. As an example, the selection unit 14 selects an action that maximizes the first action value function. The selection unit 14 may select an action by an ε-greedy method, a roulette selection used in a genetic algorithm, a softmax method using a Boltzmann distribution, or the like.

In addition, when a plurality of first action value functions are used, the selection unit 14 may select an action using one of the plurality of first action value functions, or may calculate a second action value function using the plurality of first action value functions calculated by the calculation unit 13, and select an action using the calculated second action value function. The second action value function is a function for evaluating an action in a state. The second action value function may be, for example, an expected value of the plurality of first action value functions, or may be, for example, a function that becomes smaller than the expected value as the variation of the plurality of first action value functions increases. The second action value function is, for example, given by the following formula (2) or formula (3). However, the second action value function is not limited to that given by formula (2) or (3), and may be a function other than these.

In formula (2) and formula (3), J(s _t , a) is the second action value function, s _t is the first state, a is an action, θ is a hyperparameter, Q(s _t ⁽ⁱ⁾ , a) is the first action value function, s _t ⁽ⁱ⁾ is a state included in the state sequence, and E is an expected value. Note that formula (3) is obtained by Taylor expansion of formula (2), adopting up to second-order terms and discarding third-order and subsequent terms.

When the selection unit 14 calculates the second action value function, the selection unit 14 selects an action that maximizes the second action value function using a measure given by, for example, formula (4). Note that the measure for selecting an action is not limited to the measure given by formula (4) and may be another measure. The selection unit 14 may select an action using, for example, the ε-greedy method, roulette selection used in genetic algorithms, or a softmax method using the Boltzmann distribution. When the ε-greedy method is used, the measure is given by the following formula (5), for example.

式（４）及び式（５）において、πは選択する次の行動、ａ´は第１の状態ｓ_ｔにおいてエージェントが可能な行動である。また、式（５）において、ε（０＜ε＜１）は定数、ｖは、０≦ｖ≦１を満たす乱数である。

In formulas (4) and (5), π is the next action to be selected, a' is an action that the agent can take in the first state s _t , and in formula (5), ε (0<ε<1) is a constant, and v is a random number that satisfies 0≦v≦1.

<Effects of reinforcement learning system>
According to the reinforcement learning system 1 of this exemplary embodiment, the first action value function that takes into account the variability of the states can be calculated by calculating the action value function using the second state obtained by adding noise to the first state. By selecting an action using this first action value function, the reinforcement learning system 1 can select a more suitable action.

<Reinforcement learning method flow>
2 is a flow diagram showing the flow of the reinforcement learning method S1 executed by the reinforcement learning system 1. The reinforcement learning system 1 repeatedly selects an action by repeating the reinforcement learning method S1. Note that the contents already explained will not be explained again.

The reinforcement learning method S1 includes steps S11 to S14. In step S11, the acquisition unit 11 acquires a first state. In step S12, the generation unit 12 generates a second state by adding noise to the first state.

In step S13, the calculation unit 13 calculates a first action value function according to the second state. Here, in the nth (n is a natural number) repetition, the data that the calculation unit 13 refers to in order to calculate the first action value function are, for example, the states, actions, and rewards accumulated up to the (n-1)th repetition. In step S14, the selection unit 14 selects an action according to the first action value function.

<Effects of reinforcement learning methods>
According to the reinforcement learning method S1 of this exemplary embodiment, an action value function that takes into account the variability of states can be calculated by calculating an action value function using a second state obtained by adding noise to a first state. By selecting an action using this action value function, a more suitable action can be selected.

<Example of device configuration for reinforcement learning system>
Next, an example of the device configuration of the reinforcement learning system 1 according to this exemplary embodiment will be described with reference to the drawings. FIG. 3 is a block diagram showing an example of the configuration of the reinforcement learning system 1. In the example of FIG. 3, the reinforcement learning system 1 includes a reinforcement learning device 10. The reinforcement learning device 10 includes an acquisition unit 11, a generation unit 12, a calculation unit 13, and a selection unit 14. The reinforcement learning device 10 is, for example, a server device, a personal computer, or a game device, but is not limited to these, and may be a device other than the above. For example, the reinforcement learning device 10 may acquire the first state by receiving the first state via a communication interface.

Figure 4 is a block diagram showing another example of the configuration of the reinforcement learning system 1. In the example of Figure 4, the reinforcement learning system 1 includes a terminal 20 and a server 30. The terminal 20 is, by way of example, a personal computer or a game device, but is not limited to these and may be a device other than those mentioned above. The terminal 20 includes an acquisition unit 11. The server 30 includes a generation unit 12, a calculation unit 13, and a selection unit 14. The terminal 20 acquires a first state and supplies the acquired first state to the server 30.

In this exemplary embodiment, Figures 3 and 4 are shown as examples of the configuration of the reinforcement learning system 1, but the configuration of the reinforcement learning system 1 is not limited to that shown in Figures 3 and 4, and various other configurations can be applied.

Exemplary embodiment 2
A second exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first exemplary embodiment are given the same reference numerals and will not be described repeatedly.

<Configuration of Reinforcement Learning System>
Fig. 5 is a block diagram showing a configuration of the reinforcement learning system 2. As shown in Fig. 5, the reinforcement learning system 2 includes a terminal 40 and a reinforcement learning device 50. The terminal 40 and the reinforcement learning device 50 are configured to be able to communicate with each other via a communication line N. Although the specific configuration of the communication line N does not limit the present exemplary embodiment, as an example, a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public line network, a mobile data communication network, or a combination of these networks can be used.

The terminal 40 is, for example, a general-purpose computer, and more specifically, for example, a control device that controls construction machinery such as an excavator, a management device that manages transportation by a transportation device, or a game device for playing computer games. Note that the terminal 40 is not limited to these, and may be a device other than the above. The reinforcement learning device 50 is, for example, a server device.

<Device configuration>
The terminal 40 includes a communication unit 41, a control unit 42, and an input receiving unit 43. The communication unit 41 transmits and receives information to and from the reinforcement learning device 50 via a communication line N under the control of the control unit 42. Hereinafter, the transmission and reception of information between the control unit 42 and the reinforcement learning device 50 via the communication unit 41 will also be simply described as the control unit 42 transmitting and receiving information to and from the reinforcement learning device 50.

The control unit 42 includes a state providing unit 421, an action executing unit 422, and a reward providing unit 423. The state providing unit 421 acquires a first state and provides the acquired first state to the reinforcement learning device 50. In this exemplary embodiment, the first state acquired by the state providing unit 421 includes a plurality of elements to which attributes are attached. The attribute is information indicating the characteristics and/or type of the element, and includes information indicating, for example, whether the element is a dynamic element that moves within the environment or a static element that does not move within the environment. The attribute may also be information indicating the type of element, such as a person, a car, a bicycle, or a building. However, the attribute is not limited to the above examples, and may be other information other than the above.

As one example, the status providing unit 421 may acquire sensor information output by a sensor that detects the operation of a construction machine, a transport device, or the like, as the first state. Also, as another example, the status providing unit 421 may acquire the first state of an object that affects the progress of a computer game. However, the first state acquired by the status providing unit 421 is not limited to the above-mentioned examples, and may be a state other than the above.

As an example, the state providing unit 421 accepts an input of a first state via the input accepting unit 43, and provides the accepted first state to the reinforcement learning device 50. As an example, the state providing unit 421 may also receive a first state from another device connected via the communication unit 41, and provide the received first state to the reinforcement learning device 50.

The action execution unit 422 executes the action determined by the reinforcement learning device 50. As an example, the action execution unit 422 outputs control information for causing a construction machine, a transport device, or the like to execute the action determined by the reinforcement learning device 50. As another example, the action execution unit 422 controls the action of an object that is the subject of a user operation in a computer game. However, the action executed by the reinforcement learning device 50 is not limited to the above example, and may be an action other than the above.

The reward providing unit 423 provides the reinforcement learning device 50 with a reward obtained when the agent executes the action determined by the reinforcement learning device 50. As an example, the reward providing unit 423 provides the reinforcement learning device 50 with information indicating the amount of excavation by the excavator, the time required for excavation, the time required for transportation by the transportation device, whether or not an obstacle was contacted during transportation, the outcome of the game, or the game score as a reward. However, the reward provided by the reward providing unit 423 is not limited to the above-mentioned example, and may be a reward other than the above.

As an example, the reward providing unit 423 provides the reinforcement learning device 50 with a reward acquired via the input receiving unit 43. The reward providing unit 423 may also receive a reward from another device connected via the communication unit 41 and provide the received reward to the reinforcement learning device 50.

The input reception unit 43 receives various inputs to the terminal 40. Although the specific configuration of the input reception unit 43 does not limit this exemplary embodiment, as an example, the input reception unit 43 may be configured to include input devices such as a keyboard and a touchpad. The input reception unit 43 may also be configured to include a data scanner that reads data via electromagnetic waves such as infrared rays and radio waves, and a sensor that senses the state of the environment. As an example, the reward provision unit 423 measures the time required for transportation by the transportation device based on the sensing result acquired by the input reception unit 43, and provides a reward indicating the measurement result to the reinforcement learning device 50.

The input reception unit 43 supplies the received input information to the control unit 42 via the above-mentioned input devices, data scanners, sensors, etc. The input reception unit 43, as an example, acquires the above-mentioned state and the above-mentioned reward, and supplies the acquired state and reward to the control unit 42.

<Configuration of Reinforcement Learning Device>
The reinforcement learning device 50 includes a communication unit 51, a control unit 52, and a storage unit 53. The communication unit 51 transmits and receives information to and from the reinforcement learning device 50 via a communication line N under the control of the control unit 52. Hereinafter, the transmission and reception of information between the control unit 52 and the terminal 40 via the communication unit 51 will also be simply described as the control unit 52 transmitting and receiving information to and from the terminal 40.

The control unit 52 includes a reward acquisition unit 521, a state observation unit 522, a state randomization unit 523, a learning unit 524, an estimation unit 525, and a selection unit 526. The state observation unit 522 is a configuration that realizes an acquisition means in this exemplary embodiment. The state randomization unit 523 is a configuration that realizes a generation means in this exemplary embodiment. The estimation unit 525 is a configuration that realizes a calculation means in this exemplary embodiment. The selection unit 526 is a configuration that realizes a selection means in this exemplary embodiment.

The reward acquisition unit 521 acquires the reward provided by the terminal 40 via the communication unit 51. The state observation unit 522 acquires the first state provided by the terminal 40 via the communication unit 51. The state randomization unit 523 generates one or more second states by adding noise to the first state acquired by the state observation unit 522. The learning unit 524 trains an action value function model 531 for updating the first action value function. The action value function model 531 is used to estimate the first action value function.

The estimation unit 525 calculates a first action value function according to a state sequence including a first state and one or more second states, or a state sequence including multiple second states. The estimation unit 525 also calculates a second action value function using the first action value function.

The selection unit 526 selects an action using the second action value function, stores information indicating the selected action in the memory unit 53, and transmits the information indicating the selected action to the terminal 40.

The memory unit 53 stores various data referenced by the control unit 52. As an example, the memory unit 53 stores an action value function model 531 and learning data 532. The action value function model 531 is a learning model for updating the first action value function. The learning data 532 is data used in the reinforcement learning performed by the reinforcement learning device 50. As an example, the learning data 532 includes a first state, a second state, an action, and a reward.

<Reinforcement learning method flow>
6 is a flow diagram showing the flow of the reinforcement learning method S2 executed by the reinforcement learning system 2. The reinforcement learning system 2 repeatedly selects an action by repeating steps S21 to S29. Note that some steps may be executed in parallel or in a different order.

In step S21, the state providing unit 421 acquires a first state s _t , and provides the acquired first state s _t to the reinforcement learning apparatus 50. In step S22, the state observing unit 522 acquires the first state s _t from the terminal 40.

In step S23, the state randomizer 523 generates one or more second states by adding noise to the first state s _t . The noise added to the first state s _t by the state randomizer 523 is, for example, a normal random number or a uniform random number. However, the noise added to the first state s _t by the state randomizer 523 is not limited to these, and may be noise other than the above. The second state to which noise has been added represents a state in which the first state s _t is slightly blurred.

In this operation example, the state randomizer 523 generates a second state by selectively adding noise to a plurality of elements included in the first state s _t according to the attribute. As an example, the state randomizer 523 adds noise to an element associated with an attribute that satisfies a predetermined condition. For example, the predetermined condition is an attribute indicating a dynamic element, or an attribute indicating a static element. However, the predetermined condition is not limited to the above example, and may be another condition.

Furthermore, the state randomization unit 523 generates a state sequence {s _t ⁽ⁱ⁾ } (1≦i≦n; i is a natural number, n is a natural number equal to or greater than 2) including the generated second state. The state sequence {s _t ⁽ⁱ⁾ } is a state sequence including a first state s _t and one or more second states, or a state sequence including multiple second states. In other words, the state sequence {s _t ⁽ⁱ⁾ } includes at least the second state, and may or may not include the first state s _t .

In step S24, the estimation unit 525 calculates a first action value function Q(s _t ⁽ⁱ⁾ , a) according to the state sequence {s _t ^{( i) }. As an example, the estimation unit 525 calculates the first action value function Q(s t (i), a) for each of the multiple states s t (i) included in the state sequence {s t (i)} _{} .} ^More _specifically ^, as an example, the estimation unit 525 updates the first action value function Q(s _t ⁽ⁱ⁾ , a) for the state s _t ⁽ⁱ⁾ using the above formula (1). In this operation example, the first action value function (s _t ⁽ⁱ⁾ , a) is an m-dimensional (m is an integer equal to or greater than 2) vector, and m is the number of elements in set A (i.e., the number of types of actions a).

In step S25, the estimation unit 525 calculates a second action value function J(s t , a) based on the calculated multiple first action value functions _Q (s _t ⁽ⁱ⁾ , a). The second action value function J(s _t , a) is given by the above formula (2) or formula (3), as an example. In other words, the estimation unit 525 calculates the second action value function given by the above formula (2) or formula (3). The second action value function given by the above formula (2) or formula (3) is a function whose value is lower than the expected value of the first action value function Q(s _t ⁽ⁱ⁾ , a) as the variation of the multiple first action value functions Q(s _t ⁽ⁱ⁾ , a) increases.

In step S26, the selection unit 526 selects action a according to the second action value function J(s _{t , a) calculated based on the first action value function Q(s t (} ⁱ⁾ , a). As an example, the selection unit 526 selects action a using the strategy given by the above formula (4). Note that the strategy for selecting action a is not limited to the strategy given by the above formula (4), and other strategies such as the ε-greedy strategy and the softmax method may be used. The selection unit 526 notifies the terminal 40 of the selected action a.

In step S27, the action performing unit 422 performs the action a notified by the reinforcement learning device 50. In step S28, the reward providing unit 423 provides the reinforcement learning device 50 with the reward r _t obtained by performing the action selected by the reinforcement learning device 50. In step S29, the reward acquiring unit 521 accumulates learning data including the state sequence {s _t ⁽ⁱ⁾ } and the reward r _t .

<Effects of reinforcement learning system>
In reinforcement learning, the value of the action value function may differ greatly even if the state is only slightly different. In other words, a slight difference in the state may have a large effect on the value of the action value function. In this exemplary embodiment, the first action value function Q is calculated using a second state in which a slight noise is intentionally added to the first state s _t , thereby allowing the first action value function Q to be calculated taking into account the variation in the state. By selecting the action a using this first action value function Q, the action a can be more appropriately selected according to this exemplary embodiment.

Furthermore, the reinforcement learning system 2 according to this exemplary embodiment employs a configuration in which the first action-value function Q is calculated according to a plurality of states s _t ⁽ⁱ⁾ including the second state to which noise is added. Therefore, the reinforcement learning system 2 according to this exemplary embodiment has the effect of being able to select a more appropriate action a in addition to the effect of the reinforcement learning system 1 according to the exemplary embodiment 1.

In addition, in this exemplary embodiment, when the reinforcement learning system 2 calculates the second action value function J using the above-mentioned formula (2), the second action value function J becomes an index that is sensitive to risk (variability) including higher-order influences. By the reinforcement learning system 2 selecting an action a using the second action value function J, it is possible to select an action a that is more sensitive to risk.

In addition, in this exemplary embodiment, when the reinforcement learning system 2 calculates the second action value function J using the above-mentioned formula (3), no exponential operation is included in formula (3), so that no overflow occurs in the calculation process. By the reinforcement learning system 2 selecting action a using the second action value function J, action a can be selected more preferably, and the processing load related to the selection of action a is reduced.

Exemplary embodiment 3
A third exemplary embodiment of the present invention will be described with reference to the drawings. Note that components having the same functions as those described in the first and second exemplary embodiments are denoted by the same reference numerals, and the description thereof will not be repeated.

<Configuration of Reinforcement Learning System>
A reinforcement learning system according to this exemplary embodiment (hereinafter referred to as "reinforcement learning system 3") is obtained by applying the reinforcement learning system 2 according to the above-mentioned exemplary embodiment 2 to autonomous play of a computer game. The reinforcement learning system 3 has a similar configuration to the reinforcement learning system 2 shown in Fig. 5 in the above-mentioned exemplary embodiment 2. The components of the reinforcement learning system 3 are similar to the components of the reinforcement learning system 2, and the description thereof will not be repeated here.

In this exemplary embodiment, the first state s _t includes, for example, a state of an object that affects the progress of the computer game. The action a includes, for example, an action of an object operated by a player of the computer game. The reward r _t includes, for example, a reward related to winning or losing the game or a score of the game.

Figure 7 is a diagram showing a screen SC1, which is an example of a game screen of a computer game related to the reinforcement learning system 3. The screen SC1 includes a first dynamic object C11, second dynamic objects C21-C23, first static objects C31-C34, and a second static object C4. The first dynamic object C11, the second dynamic objects C21-C23, the first static objects C31-C34, and the second static object C4 are examples of objects that affect the progress of the game.

The computer game of Figure 7 is a game in which the player of the game specifies the direction of movement of a first dynamic object C11 moving within a maze, and the round is cleared when the player collects first static objects C31 to C34 placed within the maze while evading pursuit by second dynamic objects C21 to C23.

The first dynamic object C11 and the second dynamic objects C21 to C23 are objects that move on the screen during the game, and are an example of a dynamic element that moves within the environment. On the other hand, the first static objects C31 to C34 and the second static object C4 are objects that do not move on the screen during the game, and are an example of a static element that does not move within the environment. The first dynamic object C11 is an object that is operated by the player. The first dynamic object C11 moves at a constant speed within the maze during the game, and changes the direction of movement according to the player's operation. The second dynamic objects C21 to C23 are objects that move following the first dynamic object C11 during the game. Although three second dynamic objects C21 to C23 are illustrated in FIG. 7, the number of second dynamic objects is not limited to three, and may be more or less than this.

The first static objects C31-C34 are objects that are placed in the maze and are collected by the first dynamic object C11. When the first dynamic object C11 collides with the first static objects C31-C34, the first static objects C31-C34 are collected by the first dynamic object C11. Although four first static objects C31-C34 are illustrated in FIG. 7, the number of first static objects is not limited to four and may be more or less than this. The second static object C4 is a wall that constitutes the maze.

In the example of Fig. 7, the first state s _t includes states related to the first dynamic object C11, the second dynamic objects C21 to C23, the first static objects C31 to C34, and the second static object C4. In other words, the first state includes states related to dynamic elements that move within the environment, and states related to static elements that do not move within the environment. More specifically, the first state s _t includes the position of the first dynamic object C11, the positions of the second dynamic objects C21 to C23, the positions of the first static objects C31 to C34, and the position of the second static object C4.

In this exemplary embodiment, the first state s _t is an image representing a game play screen.
8 is a diagram showing an image Img11 which is an example of the first state s _t . The image Img11 is a grayscale image in which elements included in the game screen are represented by pixel values of 0 to 255. The image Img11 is divided into a predetermined number of squares, and each square is represented by a pixel value according to the attribute of the element located in each square. As an example, the position of the first dynamic object C11 is represented by a pixel value of 255, the positions of the second dynamic objects C21 to C23 by a pixel value of 160, the positions of the first static objects C31 to C34 by a pixel value of 128, the position of the passage formed by the second static object C4 by a pixel value of 64, and the positions where movement is not possible by a pixel value of 0.

In this exemplary embodiment, the action a is the movement of the first dynamic object C11, and there are four types: moving up, moving down, moving right, and moving left. The reward r _t is, for example, a predetermined added value (e.g., +1) obtained when the score is increased, and a predetermined subtracted value (e.g., -10) obtained when the object is captured by the second dynamic objects C21 to C23. Regardless of the degree of increase in the score in one action, a predetermined added value (e.g., +1) may be obtained as the reward r _t when the score is increased by an action.

<Reinforcement learning method flow>
The reinforcement learning system 3 executes the reinforcement learning method S2 of Fig. 6 according to the above-mentioned exemplary embodiment 2. In the following, characteristic operations in this exemplary embodiment will be mainly described, and the contents described in the above-mentioned exemplary embodiment 2 will not be described repeatedly.

In this exemplary embodiment, in step S23, the state randomizer 523 generates the second state by adding noise to the states of the dynamic elements included in the first state s _t . As an example, the state randomizer 523 generates the second state by randomizing the position of the first dynamic object C11 and the positions of the second dynamic objects C21 to C23 by random walk.

More specifically, as an example, the state randomization unit 523 divides the game screen into a predetermined number of squares (e.g., 33 x 33 squares) and selects equal probability for moving forward/backward/left/right and one square in each possible direction (direction of a path). The state randomization unit 523 performs σ ² random walks (σ is an integer equal to or greater than 1) for the position of the first dynamic object C11 and the second dynamic objects C21 to C23. By performing σ ² random walks, the dynamic elements move by σ squares on average.

The state sequence {s _t ⁽ⁱ⁾ } generated by the state randomization unit 523 in step S23 includes a total of n states, including the first state s _t and (n-1) second states obtained by randomizing the first state s _t . In addition, since there are four types of action a, namely, moving up, moving down, moving right, and moving left, the first action value function (s _t ⁽ⁱ⁾ , a) calculated by the estimation unit 525 in step S24 is a four-dimensional vector.

In step S26, the selection unit 526 selects one of four types of movement direction of the first dynamic object C11, up, down, left, or right, as action a at an intersection or corner (i.e., a point where the movement direction can be changed). However, the selection unit 526 excludes directions in which the first dynamic object C11 cannot move.

Evaluation of this exemplary embodiment
9 to 12 are diagrams showing examples of evaluation results of autonomous play of a computer game involving the reinforcement learning system 3. In the computer game according to this exemplary embodiment, the first dynamic object has one life, and the game is over when the first dynamic object is captured by the second dynamic object. In addition, there is one stage, and the game ends when the game is cleared, that is, when all the first static objects are collected.

In the examples of Figures 9 to 12, the results of the reinforcement learning system 3 autonomously playing a computer game under multiple conditions in which the values of σ and θ in the reinforcement learning of the reinforcement learning system 3 were changed were evaluated. The results of autonomous play using a conventional reinforcement learning method other than the reinforcement learning system 3 were also used for comparison. The conventional reinforcement learning method used was an improved action selection policy in the DQN (deep Q-network) method.

Figure 9 is a graph showing scores from autonomous play when σ = 2. As mentioned above, σ is the average number of moves in a random walk. In Figure 9, the vertical axis shows the score. Graph g91 shows the average score from autonomous play using conventional reinforcement learning. Graphs g11 to g14 show the average score from autonomous play using the reinforcement learning system 3. Graphs g11 to g14 each have a different value for the hyperparameter θ in the equation (equation (2) or equation (3) above) representing the second action value function J. Graphs g11 to g14 show the average score when the hyperparameter θ is set to "0", "0.001", "0.01", and "0.1", respectively.

Comparing graph g91 with graphs g11 to g14, the score of the reinforcement learning system 3 according to this exemplary embodiment is higher than the score obtained by conventional reinforcement learning, and the score is particularly high when the value of the hyperparameter θ is set to "0.01."

Figure 10 is a graph showing the recovery rate of the first static object through autonomous play when σ = 2. In Figure 10, the vertical axis shows the recovery rate. Graph g92 shows the average recovery rate of autonomous play through conventional reinforcement learning. Graphs g21 to g24 show the average recovery rate of autonomous play through reinforcement learning system 3. Graphs g21 to g24 each have a different value for the hyperparameter θ of the equation (equation (2) or equation (3) above) representing the second action value function J. Graphs g21 to g24 are graphs showing the average recovery rate when the hyperparameter θ is set to "0", "0.001", "0.01", and "0.1", respectively.

Comparing graph g92 with graphs g21 to g24, the recovery rate of the reinforcement learning system 3 according to this exemplary embodiment tends to be higher than the recovery rate of conventional reinforcement learning, and the score is particularly high when the value of the hyperparameter θ is set to "0.01."

Figure 11 is a graph showing the relationship between score and σ in autonomous play. In Figure 11, the horizontal axis shows σ, and the vertical axis shows score. Graphs g31 to g34 show the average score when σ is 1 to 5, and the hyperparameter θ is "0", "0.001", "0.01", and "0.1", respectively. Note that the average score in autonomous play using conventional reinforcement learning is "2009".

In the example of Figure 11, the scores for σ values between 1 and 3 are often higher than those obtained with conventional reinforcement learning. In particular, the score for θ = 0.01 and σ = 2 is higher than the others.

Figure 12 is a graph showing the relationship between recovery rate and σ due to autonomous play. In Figure 12, the horizontal axis shows σ, and the vertical axis shows recovery rate. Graphs g41 to g44 show the average recovery rate for each value of σ when the hyperparameter θ is "0", "0.001", "0.01", and "0.1", respectively. The average recovery rate for autonomous play using conventional reinforcement learning is 67.5%.

In the example of Figure 12, the recovery rate for σ values between 1 and 3 is often higher than that achieved by conventional reinforcement learning. In particular, the recovery rate for θ = 0.01 and σ = 2 is higher than the others.

As described above, according to this exemplary embodiment, the reinforcement learning system 3 is able to more appropriately select actions in autonomous play of a computer game by calculating a first action value function using a second state in which noise is added to the first state.

Exemplary embodiment 4
A fourth exemplary embodiment of the present invention will be described. Note that components having the same functions as those described in the first to third exemplary embodiments will be designated by the same reference numerals and will not be described repeatedly.

The reinforcement learning system according to this exemplary embodiment (hereinafter referred to as "reinforcement learning system 4") is the reinforcement learning system 2 according to the exemplary embodiment 2 described above applied to the control of construction machinery such as an excavator that excavates soil. The reinforcement learning system 3 has a similar configuration to the reinforcement learning system 2 shown in FIG. 5 in the exemplary embodiment 2 described above. The components of the reinforcement learning system 4 are similar to the components of the reinforcement learning system 2, and the description thereof will not be repeated here.

The reinforcement learning system 4 selects the operation of the construction machine, such as the excavation operation of a hydraulic excavator when excavating soil, by using reinforcement learning. As an example, the objective of the action in the reinforcement learning is to excavate a bucket full of soil without tilting or dragging the vehicle body during excavation.

In this exemplary embodiment, the first state s _t includes, for example, some or all of the following: the attitude and position of a construction machine such as a hydraulic excavator, the shape (3D data, etc.) of the soil to be excavated, and the amount of soil in the bucket of the excavator. The attitude of the construction machine includes, for example, the angles of the bucket, arm, boom, and rotating body of the construction machine. The position of the construction machine includes, for example, the position and direction of the crawler of the construction machine.

The action a includes, for example, posture control of the construction machine (angle control of the bucket, arm, boom, rotating body, etc.). The reward r _t includes, for example, a part or all of a positive reward whose absolute value increases as the amount of excavation increases, and a negative reward whose absolute value increases as the degree of inclination of the body of the construction machine, the degree of dragging, or the time taken for excavation increases.

The state randomizer 523 may add noise to all of the elements included in the first state s _t , or may add noise to some of the elements. When adding noise to some of the elements, the elements to which noise is added may include, for example, the attitude of the hydraulic excavator and 3D data of the observed soil and sand.

According to this exemplary embodiment, the reinforcement learning system 4 can more appropriately select the operation of the construction machine by calculating the first action-value function using the second state obtained by adding noise to the first state s _t .

Exemplary embodiment 5
A fifth exemplary embodiment of the present invention will be described. Note that components having the same functions as those described in the first to fourth exemplary embodiments will be designated by the same reference numerals and will not be described repeatedly.

The reinforcement learning system according to this exemplary embodiment (hereinafter referred to as "reinforcement learning system 5") applies the reinforcement learning system 2 according to the exemplary embodiment 2 to the control of a transport device that transports luggage. As an example, the transport device is an automated guided vehicle (AGV) that runs automatically. The reinforcement learning system 5 has a similar configuration to the reinforcement learning system 2 shown in FIG. 5 in the exemplary embodiment 2 described above. The components of the reinforcement learning system 5 are similar to the components of the reinforcement learning system 2, and the description thereof will not be repeated here.

When transporting luggage from a given location to another, the reinforcement learning system 5 selects actions that minimize the transport time (increase the transport speed) and avoid contact with static obstacles (shelves, luggage, etc.) and dynamic obstacles (people, other robots, etc.) along the way.

In this exemplary embodiment, the first state s _t includes, for example, some or all of the following: the position, moving direction, speed, and angular speed of the conveying device that conveys the conveyed object, the position of the passage, the position of a static obstacle, and the position and moving speed of a dynamic obstacle. The action a includes, for example, speed control and angular velocity control of the conveying device. The reward r _t includes, for example, some or all of the following: a positive reward obtained when conveying is completed, a negative reward obtained when contacting an obstacle, or a negative reward whose absolute value increases as the conveying time is longer.

The state randomization unit 523 may add noise to all of the elements included in s _t in the first state, or may add noise to some of the elements. When adding noise to some of the elements, the elements to which noise is added may include, for example, the position, direction, speed, and angular velocity of the transport device, and may also include the position of a static obstacle, or the position and speed of a dynamic obstacle. In addition, the state randomization unit 523 may add noise to obstacles located in the traveling direction or on the traveling path of the transport device, and not add noise to obstacles located outside the traveling direction or outside the traveling path.

According to this exemplary embodiment, the reinforcement learning system 5 can more appropriately perform transportation control of the transportation device by calculating the first action value function using the second state in which noise is added to the first state s _t .

Exemplary embodiment 6
A sixth exemplary embodiment of the present invention will be described. Note that components having the same functions as those described in the first to fifth exemplary embodiments will be designated by the same reference numerals and will not be described repeatedly.

The reinforcement learning system according to this exemplary embodiment (hereinafter referred to as "reinforcement learning system 6") applies the reinforcement learning system 2 according to the above exemplary embodiment 2 to the control of a forklift. The reinforcement learning system 6 has a similar configuration to the reinforcement learning system 2 shown in FIG. 5 in the above exemplary embodiment 2. The components of the reinforcement learning system 6 are similar to the components of the reinforcement learning system 2, and the description thereof will not be repeated here.

When transporting a pallet from a given position to another, the reinforcement learning system 6 selects actions that minimize the transport time (increase the transport speed) and avoid contact with static obstacles (shelves, luggage, etc.) and dynamic obstacles (people, other debris, etc.) along the way.

In this exemplary embodiment, the first state s _t includes, for example, some or all of the position, moving direction, speed, and angular speed of the forklift, the position of the passage, the position of a static obstacle, and the position and speed of a dynamic obstacle. The action a includes, for example, speed control and angular speed control of the forklift. The reward r _t includes, for example, some or all of a positive reward obtained when the transport is completed, a negative reward obtained when the obstacle is contacted, or a negative reward whose absolute value increases as the transport time is longer.

The state randomization unit 523 may add noise to all of the elements included in s _t in the first state, or may add noise to some of the elements. When adding noise to some of the elements, the elements to which noise is added may include, for example, the position, direction, speed, and angular velocity of the forklift, and may also include the position of a static obstacle, or the position and speed of a dynamic obstacle. In addition, the state randomization unit 523 may add noise to obstacles located in the traveling direction or on the traveling path of the forklift, and not add noise to obstacles located outside the traveling direction or on the traveling path.

According to this exemplary embodiment, the reinforcement learning system 5 can more appropriately control the forklift by calculating the first action-value function using the second state obtained by adding noise to the first state s _t .

Exemplary embodiment 7
A seventh exemplary embodiment of the present invention will be described. Note that components having the same functions as those described in the first to sixth exemplary embodiments will be designated by the same reference numerals and will not be described repeatedly.

The reinforcement learning system according to this exemplary embodiment (hereinafter referred to as "reinforcement learning system 7") has a configuration similar to that of the reinforcement learning system 2 shown in FIG. 5 in the above-described exemplary embodiment 2. The components of the reinforcement learning system 6 are similar to those of the reinforcement learning system 2, and the description thereof will not be repeated here.

In this exemplary embodiment, the first state s _t includes a plurality of elements with attributes. When adding noise to the first state s _t , the state randomizer 523 weights the addition of noise differently depending on the attributes. As an example, the state randomizer 523 may increase the weight of dynamic elements that move within the environment, while decreasing the weight of static elements that do not move within the environment. As an example, the state randomizer 523 may increase the weight of the position of a person among the dynamic elements that move within the environment, more than the weight of other dynamic elements.

According to this exemplary embodiment, the first action value function is calculated using the second state in which noise is added with weighting according to the attribute of the element, so that the first action value function can be calculated taking into account the variability of data according to the attribute. By selecting an action using this first action value function, it is possible to select an action taking into account the variability of data according to the attribute.

In addition, in this exemplary embodiment, the state randomizer 523 may change the weighting of the noise addition during reinforcement learning. As an example, the state randomizer 523 may perform control such that the weighting is increased when a dynamic element is moving in the environment, and the weighting is decreased for a dynamic element that is not moving in the environment.

[Software implementation example]
Some or all of the functions of the reinforcement learning device 10, the terminal 20, the server 30, the terminal 40, and the reinforcement learning device 50 (hereinafter referred to as "the reinforcement learning device 10, etc.") may be realized by hardware such as an integrated circuit (IC chip), or by software.

In the latter case, the reinforcement learning device 10 etc. is realized, for example, by a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 13. Computer C has at least one processor C1 and at least one memory C2. Memory C2 stores program P for operating computer C as reinforcement learning device 10 etc. In computer C, processor C1 reads and executes program P from memory C2, thereby realizing each function of reinforcement learning device 10 etc.

The processor C1 may be, for example, a central processing unit (CPU), a graphic processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a microcontroller, or a combination of these. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination of these.

The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may further include a communications interface for transmitting and receiving data to and from other devices. The computer C may further include an input/output interface for connecting input/output devices such as a keyboard, mouse, display, and printer.

The program P can also be recorded on a non-transitory, tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communications network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[Additional Note 1]
The present invention is not limited to the above-described embodiment, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the above-described embodiment are also included in the technical scope of the present invention.

[Additional Note 2]
Some or all of the above-described embodiments can be described as follows. However, the present invention is not limited to the aspects described below.

(Appendix 1)
An acquisition means for acquiring a first state in an environment that is a target of reinforcement learning;
generating means for generating a second state by adding noise to the first state;
A calculation means for calculating a first action value function according to the second state;
a selection means for selecting an action in accordance with the first action-value function;
A reinforcement learning system comprising:

According to the above configuration, a more suitable action can be selected by calculating the first action value function using a second state in which noise is added to the first state.

(Appendix 2)
the calculation means calculates the first action value function according to the first state and the second state.
2. The reinforcement learning system of claim 1.

According to the above configuration, a more suitable action can be selected by calculating the first action value function using multiple states including the second state to which noise has been added.

(Appendix 3)
The calculation means calculates the first action-value function for each of the first state and the second state;
the selection means selects the action in accordance with a second action-value function calculated based on a plurality of the first action-value functions.
3. The reinforcement learning system of claim 2.

According to the above configuration, a more suitable action can be selected by using a second action value function calculated using multiple first action value functions.

(Appendix 4)
The first state includes at least one of a position, a moving direction, a speed, and an angular velocity of a conveying device that conveys an object, a position of a passage, and a position and a speed of a static or dynamic obstacle;
4. The reinforcement learning system of claim 1.

According to the above configuration, the transport operation of the transport device can be more appropriately selected through reinforcement learning.

(Appendix 5)
The first state includes at least one of the following: an attitude and a position of a construction machine, a shape of soil to be excavated, and an amount of soil in a bucket of an excavator.
4. The reinforcement learning system of claim 1.

According to the above configuration, the construction operations of the construction machine can be more appropriately selected through reinforcement learning.

(Appendix 6)
the first state includes a plurality of elements having attributes associated therewith;
the generating means generates the second state by selectively adding noise to a plurality of elements included in the first state according to the attribute.
6. The reinforcement learning system of claim 1.

According to the above configuration, a first action value function can be calculated that takes into account the variability in data for elements associated with attributes that satisfy specified conditions.

(Appendix 7)
the first state includes a state regarding a dynamic element moving within an environment;
the generating means generates the second state by adding noise to a state of the dynamic element included in the first state.
7. The reinforcement learning system of claim 6.

According to the above configuration, a first action value function can be calculated that takes into account the variability of data for dynamic elements.

(Appendix 8)
An acquisition means for acquiring a first state in an environment that is a target of reinforcement learning;
generating means for generating a second state by adding noise to the first state;
A calculation means for calculating a first action value function according to the second state;
a selection means for selecting an action in accordance with the first action-value function;
A reinforcement learning device comprising:

According to the above configuration, a more suitable action can be selected by calculating the first action value function using a second state in which noise is added to the first state.

(Appendix 9)
the calculation means calculates the first action value function according to the first state and the second state.
9. The reinforcement learning device according to claim 8.

According to the above configuration, a more suitable action can be selected by calculating the first action value function using multiple states including the second state to which noise has been added.

(Appendix 10)
the calculation means calculates the first action-value function for each of a plurality of states included in the state sequence;
the selection means selects the action in accordance with a second action-value function calculated based on a plurality of the first action-value functions.
10. The reinforcement learning device according to claim 9.

According to the above configuration, a more suitable action can be selected by using a second action value function calculated using multiple first action value functions.

(Appendix 11)
The first state includes at least one of a position, a moving direction, a speed, and an angular velocity of a conveying device that conveys an object, a position of a passage, and a position and a speed of a static or dynamic obstacle;
11. The reinforcement learning device according to any one of appendixes 8 to 10.

According to the above configuration, the transport operation of the transport device can be more appropriately selected through reinforcement learning.

(Appendix 12)
The first state includes at least one of the following: an attitude and a position of a construction machine, a shape of the soil to be excavated, and an amount of soil in a bucket of an excavator.
11. The reinforcement learning device according to any one of claims 8 to 10.

According to the above configuration, the construction operations of the construction machine can be more appropriately selected through reinforcement learning.

(Appendix 13)
the first state includes a plurality of elements having attributes associated therewith;
the generating means generates the second state by selectively adding noise to a plurality of elements included in the first state according to the attribute.
13. A reinforcement learning device according to any one of appendixes 8 to 12.

According to the above configuration, a first action value function can be calculated that takes into account the variability in data for elements associated with attributes that satisfy specified conditions.

(Appendix 14)
the first state includes a state regarding a dynamic element moving within an environment;
the generating means generates the second state by adding noise to a state of the dynamic element included in the first state.
14. The reinforcement learning device according to claim 13.

According to the above configuration, a first action value function can be calculated that takes into account the variability of data for dynamic elements.

(Appendix 15)
Obtaining a first state of an environment that is the subject of reinforcement learning;
generating a second state by adding noise to the first state;
calculating a first action-value function in response to the second state;
selecting an action in response to said first action-value function;
Reinforcement learning methods including

According to the above configuration, a more suitable action can be selected by calculating the first action value function using a second state in which noise is added to the first state.

(Appendix 16)
In calculating the first action value function,
calculating the first action-value function according to the first state and the second state;
16. The reinforcement learning method of claim 15.

According to the above configuration, a more suitable action can be selected by calculating the first action value function using multiple states including the second state to which noise has been added.

(Appendix 17)
In calculating the first action-value function, the first action-value function is calculated for each of a plurality of states included in the state sequence;
In selecting the action, the action is selected according to a second action value function calculated based on a plurality of the first action value functions.
17. The reinforcement learning method of claim 16.

According to the above configuration, a more suitable action can be selected by using a second action value function calculated using multiple first action value functions.

(Appendix 18)
The first state includes at least one of a position, a moving direction, a speed, and an angular velocity of a conveying device that conveys an object, a position of a passage, and a position and a speed of a static or dynamic obstacle;
18. The reinforcement learning method of any one of appendix 15 to 17.

According to the above configuration, the transport operation of the transport device can be more appropriately selected through reinforcement learning.

(Appendix 19)
The first state includes at least one of the following: an attitude and a position of a construction machine, a shape of soil to be excavated, and an amount of soil in a bucket of an excavator.
18. The reinforcement learning method of any one of appendix 15 to 17.

According to the above configuration, the construction operations of the construction machine can be more appropriately selected through reinforcement learning.

(Appendix 20)
the first state includes a plurality of elements having attributes associated therewith;
generating the second state by selectively adding noise to a plurality of elements included in the first state according to the attribute;
20. The reinforcement learning method of any one of appendix 15 to 19.

According to the above configuration, a first action value function can be calculated that takes into account the variability in data for elements associated with attributes that satisfy specified conditions.

(Appendix 21)
the first state includes a state regarding a dynamic element moving within an environment;
generating the second state by adding noise to the state of the dynamic element included in the first state;
21. The reinforcement learning method of claim 20.

According to the above configuration, a first action value function can be calculated that takes into account the variability of data for dynamic elements.

(Appendix 22)
the first state includes a plurality of elements having attributes associated therewith;
The generating means varies a weight of the noise addition depending on the attribute.
6. The reinforcement learning system of claim 1.

According to the above configuration, by using the second state in which noise is added with weighting according to the attributes of the element, it is possible to calculate a first action value function that takes into account the variability of the data according to the attributes.

(Appendix 23)
The first state includes some or all of the following: the attitude and position of the construction machine, the shape of the soil to be excavated, and the amount of soil in the bucket of the excavator;
The action includes attitude control of the construction machine.
19. The reinforcement learning system of claim 1.

According to the above configuration, by calculating the first action value function using a second state in which noise is added to the first state, it is possible to more appropriately select the excavation operation of the excavator through reinforcement learning.

(Appendix 24)
The first state includes some or all of the following: a position, a moving direction, a speed, and an angular velocity of a conveying device that conveys an object, a position of a passage, and a position and a speed of a static or dynamic obstacle;
The action includes speed control and angular velocity control of the transport device.
19. The reinforcement learning system of claim 1.

According to the above configuration, by calculating the first action value function using a second state in which noise is added to the first state, it is possible to more appropriately select the transportation operation of the transportation device through reinforcement learning.

(Appendix 25)
the first state includes a state of an object that affects progress of the computer game;
the action includes a movement of an object controlled by a player of the computer game;
19. The reinforcement learning system of claim 1.

According to the above configuration, by calculating the first action value function using a second state in which noise is added to the first state, it is possible to more appropriately select the action of an object in autonomous play of a computer game.

(Appendix 26)
A program for causing a computer to function as a reinforcement learning device,
The program causes the computer to
An acquisition means for acquiring a first state in an environment that is a target of reinforcement learning;
generating means for generating a second state by adding noise to the first state;
A calculation means for calculating a first action value function according to the second state;
a selection means for selecting an action in accordance with the first action-value function;
A program characterized by causing the program to function as a

(Appendix 27)
the calculation means calculates the first action value function according to the first state and the second state.
27. The program according to claim 26,

(Appendix 28)
The calculation means calculates the first action-value function for each of the first state and the second state;
the selection means selects the action in accordance with a second action-value function calculated based on a plurality of the first action-value functions.
28. The program according to claim 27,

(Appendix 29)
The first state includes at least one of a position, a moving direction, a speed, and an angular velocity of a conveying device that conveys an object, a position of a passage, and a position and a speed of a static or dynamic obstacle;
29. The program of any one of appendices 26 to 28.

(Appendix 30)
The first state includes at least one of the following: an attitude and a position of a construction machine, a shape of soil to be excavated, and an amount of soil in a bucket of an excavator.
29. The program of any one of appendices 26 to 28.

(Appendix 31)
the first state includes a plurality of elements having attributes associated therewith;
the generating means generates the second state by selectively adding noise to a plurality of elements included in the first state according to the attribute.
31. The program according to any one of appendices 26 to 30.

(Appendix 32)
the first state includes a state regarding a dynamic element moving within an environment;
the generating means generates the second state by adding noise to a state of the dynamic element included in the first state.
28. The program according to claim 27,

[Additional Note 3]
A part or all of the above-described embodiments can be further expressed as follows.

At least one processor, the processor comprising:
An acquisition process for acquiring a first state in an environment that is a target of reinforcement learning;
a generation process for generating a second state by adding noise to the first state;
A calculation process of calculating a first action value function according to the second state;
a selection process for selecting an action according to the first action-value function;
A reinforcement learning device that executes.

The reinforcement learning device may further include a memory, and the memory may store a program for causing the processor to execute the acquisition process, the generation process, the calculation process, and the selection process. The program may also be recorded on a computer-readable, non-transitory, tangible recording medium.

1, 2, 3, 4, 5, 6, 7 Reinforcement learning system 10, 50 Reinforcement learning device 11 Acquisition unit 12 Generation unit 13 Calculation unit 14, 526 Selection unit 20, 40 Terminal 30 Server 41, 51 Communication unit 42, 52 Control unit 43 Input reception unit 53 Memory unit 421 State provision unit 422 Action execution unit 423 Reward provision unit 521 Reward acquisition unit 522 State observation unit 523 State randomization unit 524 Learning unit 525 Estimation unit

Claims (12)

An acquisition means for acquiring a first state in an environment that is a target of reinforcement learning;
generating means for generating a second state by adding noise to the first state;
A calculation means for calculating a first action value function according to the second state;
a selection means for selecting an action in accordance with the first action-value function;
Equipped with
The calculation means calculates the first action-value function for each of the first state and the second state;
the selection means selects the action in accordance with a second action-value function calculated based on a plurality of the first action-value functions .
A reinforcement learning system characterized by: The first state includes at least one of a position, a moving direction, a speed, and an angular velocity of a conveying device that conveys an object, a position of a passage, and a position and a speed of a static or dynamic obstacle;
The reinforcement learning system of claim 1 . The first state includes at least one of the following: an attitude and a position of a construction machine, a shape of soil to be excavated, and an amount of soil in a bucket of an excavator.
The reinforcement learning system according to claim 1 or 2 . the first state includes a plurality of elements having attributes associated therewith;
the generating means generates the second state by selectively adding noise to a plurality of elements included in the first state according to the attribute.
The reinforcement learning system according to any one of claims 1 to 3 . the first state includes a state regarding a dynamic element moving within an environment;
the generating means generates the second state by adding noise to a state of the dynamic element included in the first state.
The reinforcement learning system according to claim 4 . An acquisition means for acquiring a first state in an environment that is a target of reinforcement learning;
generating means for generating a second state by adding noise to the first state;
A calculation means for calculating a first action value function according to the second state;
a selection means for selecting an action in accordance with the first action-value function;
Equipped with
The calculation means calculates the first action-value function for each of the first state and the second state;
the selection means selects the action in accordance with a second action-value function calculated based on a plurality of the first action-value functions .
A reinforcement learning device characterized by: Obtaining a first state of an environment that is the subject of reinforcement learning;
generating a second state by adding noise to the first state;
calculating a first action-value function in response to the second state;
selecting an action in response to said first action-value function;
Including,
In the calculating step, the first action-value function is calculated for each of the first state and the second state;
In the selecting step, the action is selected according to a second action-value function calculated based on a plurality of the first action-value functions.
Reinforcement learning methods. A program for causing a computer to function as a reinforcement learning device, the program comprising:
An acquisition means for acquiring a first state in an environment that is a target of reinforcement learning;
generating means for generating a second state by adding noise to the first state;
A calculation means for calculating a first action value function according to the second state;
a selection means for selecting an action in accordance with the first action-value function;
Function as a
The calculation means calculates the first action-value function for each of the first state and the second state;
the selection means selects the action in accordance with a second action-value function calculated based on a plurality of the first action-value functions .
program . An acquisition means for acquiring a first state in an environment that is a target of reinforcement learning;
generating means for generating a second state by adding noise to the first state;
A calculation means for calculating a first action value function according to the second state;
a selection means for selecting an action in accordance with the first action-value function;
Equipped with
the first state includes a plurality of elements having attributes associated therewith;
the generating means generates the second state by selectively adding noise to a plurality of elements included in the first state according to the attribute .
A reinforcement learning system characterized by: An acquisition means for acquiring a first state in an environment that is a target of reinforcement learning;
generating means for generating a second state by adding noise to the first state;
A calculation means for calculating a first action value function according to the second state;
a selection means for selecting an action in accordance with the first action-value function;
Equipped with
the first state includes a plurality of elements having attributes associated therewith;
the generating means generates the second state by selectively adding noise to a plurality of elements included in the first state according to the attribute.
A reinforcement learning device characterized by: Obtaining a first state of an environment that is the subject of reinforcement learning;
generating a second state by adding noise to the first state;
calculating a first action-value function in response to the second state;
selecting an action in response to said first action-value function;
Including,
the first state includes a plurality of elements having attributes associated therewith;
In the generating step, the second state is generated by selectively adding noise to a plurality of elements included in the first state according to the attribute.
Reinforcement learning methods. A program for causing a computer to function as a reinforcement learning device, the program comprising:
An acquisition means for acquiring a first state in an environment that is a target of reinforcement learning;
generating means for generating a second state by adding noise to the first state;
A calculation means for calculating a first action value function according to the second state;
a selection means for selecting an action in accordance with the first action-value function;
Function as a
the first state includes a plurality of elements having attributes associated therewith;
the generating means generates the second state by selectively adding noise to a plurality of elements included in the first state according to the attribute.
program.

Images