JP7142605B2 - LEARNING DEVICE, LEARNING METHOD AND LEARNING PROGRAM - Google Patents

Description

The present invention relates to a learning device, a learning method, and a learning program.

Conventionally, techniques for optimizing the behavior of reinforcement learning agents have been proposed. In one technique, a model of behavior is optimized by substituting the average reward for the episode for the reward for actions and state changes during the episode [1].

RONALD J. WILLIAMS “Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning” [online], [searched May 20, 2019], Internet (https://link.springer.com/content/pdf/10.1007%2FBF00992696. pdf)

However, the above conventional techniques have room for improvement. For example, in the prior art described above, good behavior during a failed episode may be considered a bad example as well as bad behavior during a failed episode. In such cases, the above conventional techniques cannot always improve the performance of reinforcement learning.

The present application is made in view of the above and aims at improving the performance of reinforcement learning.

In the learning device according to the present application, processing steps including an input process of inputting state information indicating a state to a model and an update process of updating the state using output information output by the model are performed when the state is a predetermined state. In the case of repeated execution until a final state that satisfies a condition, an evaluation unit that evaluates each executed processing step according to the final state, output information output by the model in the processing step, and the processing A learning unit that learns the model based on the updated state of the step and the evaluation result of the processing step.

According to one aspect of the embodiment, it is possible to improve the performance of reinforcement learning.

FIG. 1 is a diagram illustrating an example of learning processing according to the embodiment. FIG. 2 is a diagram illustrating an example of processing step evaluation processing. FIG. 3 is a diagram illustrating a configuration example of a learning device according to the embodiment; FIG. 4 is a flowchart showing a learning processing procedure by the learning device according to the embodiment. FIG. 5 is a diagram illustrating an example of a hardware configuration;

Embodiments for implementing a learning device, a learning method, and a learning program according to the present application (hereinafter referred to as "embodiments") will be described in detail below with reference to the drawings. Note that the learning device, the learning method, and the learning program according to the present application are not limited to this embodiment. Further, each embodiment can be appropriately combined within a range that does not contradict the processing contents. Further, in each of the following embodiments, the same parts are denoted by the same reference numerals, and overlapping descriptions are omitted.

[1. Learning process]
First, an example of learning processing according to the embodiment will be described with reference to FIG. FIG. 1 is a diagram illustrating an example of learning processing according to the embodiment. In the example of FIG. 1, the learning process is performed by the learning system 1 according to the embodiment.

As shown in FIG. 1 , learning system 1 includes user device 10 and learning device 100 . Although not shown in FIG. 1 , the learning system 1 may include multiple user devices 10 and multiple learning devices 100 .

The user device 10 is an information processing device used by a user. The user device 10 may be any type of information processing device, including smart phones, desktop PCs (Personal Computers), notebook PCs, and tablet PCs.

In the example of FIG. 1, user device 10 is a client device requesting content on which advertisements are placed.

The learning device 100 is an information processing device that learns a model for determining values and policies in reinforcement learning. Learning device 100 may be any type of information processing device, including a server. Although not shown in FIG. 1, the learning device 100 communicates with the user device 10 by wire or wirelessly via a network (for example, the Internet).

In the example of FIG. 1, when the behavior of the user of user device 10 is input as a state, learning device 100 determines whether or not advertisements are to be provided (that is, whether or not predetermined advertisements are to be provided). , as the action taken by the agent.

In the example of FIG. 1, the above states are user behavior on the web (ie, user behavior on the network). User behavior on the web includes behavior such as viewing top pages, viewing news, searching for travel, and viewing travel services.

In the example of FIG. 1, the action of the agent described above is an advertisement for accommodation. Publishing an accommodation-related advertisement is an action of determining whether or not to place a predetermined accommodation-related advertisement in the content. In this example, the agent described above decides whether advertisement Y1 or advertisement N1 will be placed in the content, with the goal of lodging reservations.

By the way, in reinforcement learning, the behavior of the agent is optimized so that the reward obtained from the environment is maximized. An agent can be represented by a model represented by a neural network that outputs an action when a state is input, for example. In such optimization, the agent repeatedly performs a large number of trials. Regarding the relationship between agents, environments, and rewards, in games such as shogi and chess, it is sometimes difficult to evaluate actions and episodes in the middle of episodes. In such cases, the model of behavior may be optimized, for example, by substituting the average reward for the episode for the reward for actions and state changes during the episode.

In the optimization techniques described above, bad behavior during a failed episode is legitimately considered a bad example when the model is optimized with a failed episode. However, in this case good behavior during a failed episode is considered a bad example as well as bad behavior during a failed episode. In this regard, by trialing a large number of episodes, it is believed that individual behaviors included in successful episodes are gradually used as good examples in optimizing the model. However, trials of such a large number of episodes are considered inefficient.

Also, conventionally, it has been proposed to use the Q-value (also called action-value function Q, or state-action-value) to learn a policy to generate an agent's action. For example, policy gradient methods such as the REINFORCE algorithm have been proposed. In the REINFORCE algorithm, the Q value is substituted with the time average of rewards. From the point of view of the credit assignment problem, it is proposed to assign this time-averaged reward to actions during an episode.

For example, initially the reward is determined by the final state of the episode. The reward determined by the final state is then averaged over the time it took to obtain this reward. Averaged rewards are then assigned to states and behaviors during the episode. This defines the time-averaged reward as the reward assigned to a combination of state and action at a given time. In such techniques, the agent's strategy is optimized with a reward assigned to the combination of state and action at a given time.

However, with the techniques described above, the agent's strategy may be optimized such that good behavior is denied during the episode. For illustration purposes, assume that in some circumstances the reward is determined only at the end state of the episode. Examples of such an environment include Go (eg, winning or losing a game), advertising placement (eg, success or failure of conversion), and the like. In this example, the agent may select the correct action in the middle of the episode and then the wrong action at the end of the episode. As a result, the agent may fail this episode. In such cases, the agent cannot obtain the reward from the environment even though the agent has chosen the correct action in the middle of the episode. Thus, rewards are not always assigned to correct actions. An agent's strategy may be optimized by using a large number of episodes (eg, a large number of successful episodes) as training data. However, policy learning efficiency is poor.

Thus, in the exemplary embodiment of FIG. 1, learning device 100 optimizes value assessments (eg, Q-networks) and agent strategies (eg, neural nets) such that good behavior during episodes is evaluated. For this purpose, the learning process described below is performed.

First, the learning device 100 acquires user behavior on the web (step S11).

For example, learning device 100 acquires user behavior on the web as a "state". An "environment" is, for example, a content delivery network. For example, the learning device 100 acquires the state 'user behavior on the web' from the environment 'content distribution network'. As described above, user behavior on the web includes, for example, top page browsing, news browsing, travel search, travel service browsing, and the like.

Next, learning device 100 inputs the user's behavior as a state into the model, and acquires the presence or absence of advertisement provision as behavior (step S12).

For example, the learning device 100 inputs a feature vector (also called feature vector or instance) corresponding to user behavior into the model to determine whether or not an advertisement is provided (that is, whether or not a predetermined advertisement is provided). Output as an action taken by the agent.

For example, if the above model is a model based on behavior or state values (i.e., policy-on algorithm), learning device 100 inputs feature vectors corresponding to user behavior into this model (e.g., neural network). outputs the probability that a given ad will be served. Then, learning device 100 determines whether or not a predetermined advertisement is provided based on the output probability. On the other hand, if the model is a policy-based model (that is, a policy-off algorithm), learning device 100 inputs a feature vector corresponding to user behavior to this model (for example, a Q network) to obtain a predetermined state (eg, a given user action) and a Q value corresponding to a given action (eg, serving a given advertisement). Then, learning device 100 determines whether or not to provide a predetermined advertisement based on the output Q value. For example, the learning device 100 uses the epsilon greedy method to determine the action corresponding to the maximum Q value as the action taken by the agent with a predetermined probability (ie, epsilon).

Next, study device 100 provides content according to whether or not advertisements are provided (step S13).

For example, when the action taken by the agent is to provide a predetermined advertisement, the learning device 100 provides the user device 10 with content in which this predetermined advertisement is posted. As described above, the predetermined advertisement is, for example, an accommodation-related advertisement such as advertisement Y1 or advertisement N1.

Next, when the user behavior satisfies a predetermined condition, the learning device 100 determines that the episode has reached the final state, and evaluates the processing steps up to that point according to the success or failure of the final state (step S14).

For example, if the state (that is, the user behavior) is not the final state in step S11, the learning device 100 repeats the series of processing steps of steps S11, S12, and S13 until the state reaches the final state in step S11. repeat. The above-mentioned predetermined condition is, for example, the condition that "the user makes a reservation for accommodation" (that is, conversion). The predetermined condition described above may be a condition that "the user leaves" (that is, leave). More specifically, this condition may be a condition that "the user terminates a predetermined page". The predetermined page is, for example, a travel service page. The learning device 100 evaluates a series of processing steps of steps S11, S12, and S13, for example, depending on whether the accommodation reservation is successful or not.

FIG. 2 is a diagram illustrating an example of processing step evaluation processing. In the example of FIG. 2, the learning device 100 executes a plurality of episodes EP11 corresponding to the series of processing steps of steps S11, S12 and S13, as described above. In the example of FIG. 2, states correspond to user actions. Also, the placement of advertisements corresponds to the actions of agents. Note that "time" corresponds to a discrete time when the agent acquires the state (eg, user behavior) from the environment and takes action (eg, placing an advertisement). In the example of FIG. 2, the agent's goal is to book a hotel. When the state reaches the final state, learning device 100 determines a reward according to the outcome of the episode. For example, when the user makes an accommodation reservation in step S11 described above, the agent acquires the reward "100" from the environment. In other words, learning device 100 determines the value “100” as the reward based on the episode result “accommodation reservation”.

For illustrative purposes, in the example of FIG. 2, it is assumed that learning device 100 receives a request to view a travel service page from user device 10 at time "6". Further assume that learning device 100 provides advertisement Y1 to user device 10 along with the travel service page in response to receiving a viewing request. In this example, when the user of the user device 10 selects the advertisement Y1 displayed in the travel service page and then makes an accommodation reservation, the agent obtains the reward "100" from the environment.

Regarding the evaluation of the above-described processing steps, the learning device 100 determines the reservation rate RR11 (i.e., conversion rate). For example, the learning device 100 calculates the reservation rate of each processing step. More specifically, learning device 100 calculates a reservation rate corresponding to a combination of a state and an advertisement placement. In the example of FIG. 2, for example, the reservation rate corresponding to the combination of the state "top page viewed" and advertisement placement "Y1" is "1%". This means that the probability that the user of the user device 10 selects the advertisement Y1 displayed in the TOP page and then makes an accommodation reservation is "1%". Then, the learning device 100 reflects the total reservation rate in the remuneration. For example, the learning device 100 redefines the product of the calculated reservation rate corresponding to the combination of the state and the advertisement and the determined reward as the reward corresponding to the combination of the state and the advertisement. . For example, the learning device 100 sets the product "1" of the reservation rate "1%" corresponding to the combination of the state "TOP page viewed" and the advertisement placement "Y1" and the determined reward "100" to this state. defined as a reward corresponding to a combination of In the example of FIG. 2, the learning device 100 generates a plurality of episodes EP12 with redefined rewards from the plurality of episodes EP11.

As described above, in the example of FIG. 2, the learning device 100 can evaluate a series of processing steps of steps S11, S12, and S13 according to success or failure of the final state of the episode. That is, learning device 100 can evaluate intermediate processing steps using the final state. For example, the learning device 100 can give rewards to intermediate processing steps according to the success or failure of the final state by executing a plurality of episodes corresponding to this series of processing steps.

Returning to FIG. 1, learning device 100 then updates the model using information on each evaluated processing step (step S15).

For example, the learning device 100 updates the model using multiple episodes EP12 as shown in FIG. For example, if the model described above is a model based on behavior or state values, the learning device 100 performs Q-learning using a plurality of episodes EP12. On the other hand, when the model is a policy-based model, the learning device 100 performs policy learning using a plurality of episode EP12. For example, the learning device 100 can train the policy of a model (eg, neural net) using the policy gradient method.

Next, learning device 100 acquires user behavior on the web (step S16).

For example, as in the case of step S11, the learning device 100 acquires actions such as top page viewing, news viewing, travel search, and travel service viewing.

After that, learning device 100 provides content according to the presence or absence of advertisement provision based on the updated model (step S17).

As shown in FIG. 2, the learning device 100 redefines rewards for multiple episodes based on reservation rates corresponding to combinations of states and advertisement placements. Then, learning device 100 updates the model using a plurality of episodes with redefined rewards. In the example of FIG. 2, the reward "2" corresponding to the combination of the state "news viewing" and the advertisement placement "Y1" is the reward "0. 5" taller. Therefore, the probability that the advertisement Y1 is displayed in the content according to the presence/absence of advertisement provision according to the model after the update is higher than the probability that the advertisement N1 is displayed in the content according to the presence/absence of the advertisement provision according to the updated model. In the example of FIG. 2, the reservation rate "2%" corresponding to the combination of the state "news viewing" and advertisement placement "Y1" is the reservation rate corresponding to the combination of the state "news viewing" and advertisement placement "N1". Higher than "0.5%". Therefore, the learning device 100 can improve the reservation rate of accommodation reservations.

As described above, the learning device 100 according to the embodiment rewards intermediate processing steps based on the final state of the episode. In this regard, learning device 100 can quantify the evaluation of intermediate processing steps, which is known by evaluating the final state. In this way, learning device 100 can redesign rewards using knowledge obtained from the final state. Then, the learning device 100 can give the quantified evaluation as a reward to the intermediate processing steps. As a result, the learning device 100 can apply the reward information redefined by the final state of the episode to the reinforcement learning optimization problem. As a result, learning device 100 can optimize the evaluation of value and the strategy of the agent so that good behavior during the episode is evaluated. Therefore, the learning device 100 can improve the accuracy of reinforcement learning. Furthermore, the learning device 100 can shorten the learning time of reinforcement learning. The learning device 100 that implements such learning processing will be described in detail below.

[2. Configuration of learning device]
Next, a configuration example of the learning device 100 according to the embodiment will be described with reference to FIG. FIG. 3 is a diagram showing a configuration example of the learning device 100 according to the embodiment. As shown in FIG. 3 , learning device 100 includes communication unit 110 , storage unit 120 , and control unit 130 . The learning device 100 has an input unit (for example, a keyboard, a mouse, etc.) for receiving various operations from an administrator or the like who uses the learning device 100, and a display unit (for example, a liquid crystal display) for displaying various information. may

(Communication unit 110)
The communication unit 110 is realized by, for example, a NIC (Network Interface Card) or the like. The communication unit 110 is connected to a network by wire or wirelessly, and transmits and receives information to and from the user device 10 via the network.

(storage unit 120)
The storage unit 120 is realized by, for example, a semiconductor memory device such as a RAM (Random Access Memory) or flash memory, or a storage device such as a hard disk or an optical disk. As shown in FIG. 3 , the storage section 120 has a model storage section 121 and a learning data storage section 122 .

(Model storage unit 121)
The model storage unit 121 stores models (that is, model data) for determining values and policies in reinforcement learning. The model storage unit 121 stores, for example, models received by the reception unit 132 . The model storage unit 121 may store a model (for example, a neural network) for determining value in reinforcement learning. The model storage unit 121 may store a model (for example, Q network) for determining policies in reinforcement learning.

(Learning data storage unit 122)
The learning data storage unit 122 stores sets of learning data (that is, data sets). For example, the learning data storage unit 122 stores a history of processing steps. The learning data storage unit 122 may store a series of processing steps as experience. For example, the learning data storage unit 122 may store a series of processing steps executed by the second acquisition unit 133, the third acquisition unit 134, and the provision unit 138 as experience. More specifically, the learning data storage unit 122 may store an agent's experience indicating states, actions, rewards, transition destination states, and episode end flags.

(control unit 130)
The control unit 130 is a controller, and for example, a CPU (Central Processing Unit), MPU (Micro Processing Unit), or other processor executes various programs (learning programs) stored in a storage device inside the learning apparatus 100. (equivalent to one example) is implemented by executing the RAM or the like as a work area. Also, the control unit 130 is a controller, and may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

As shown in FIG. 3, the control unit 130 includes a first acquisition unit 131, a reception unit 132, a second acquisition unit 133, a third acquisition unit 134, a determination unit 135, an evaluation unit 136, and a learning unit. 137 and a providing unit 138, and implements or executes the information processing functions and actions described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 3, and may be another configuration as long as it performs information processing described later.

(First acquisition unit 131)
The first acquisition unit 131 acquires a model (that is, model data) for determining values and policies in reinforcement learning. The first acquisition unit 131 acquires the model received by the reception unit 132, for example. The first acquisition unit 131 may acquire the model from a predetermined storage device. For example, the first acquisition unit 131 may acquire the model from the model storage unit 121 described above. The first acquisition unit 131 may store the acquired model in the model storage unit 121 described above.

(Receiver 132)
The receiving unit 132 receives a model (that is, model data) for determining values and policies in reinforcement learning. For example, the receiving unit 132 receives a model from a predetermined information processing device. For example, the receiving unit 132 receives a model from a device of an entity (for example, a company) related to the learning device 100 . The receiving unit 132 may store the received model in the model storage unit 121 .

The receiving unit 132 receives state information indicating a state. As will be described later, the state is the state of the agent in reinforcement learning. The receiving unit 132 receives state information from a predetermined information processing device. For example, the receiving unit 132 receives status information from devices of entities (eg, users, companies) related to the learning device 100 . The receiving unit 132 may store the received state information in the learning data storage unit 122 .

For example, the receiving unit 132 receives behavior information indicating user behavior as the above-described state information. The receiving unit 132 may store the received behavior information in a behavior information storage unit (not shown), which is a predetermined storage area within the storage unit 120 .

The receiving unit 132 may receive user behavior information via content provided by the providing unit 138 . The content provided by the providing unit 138 includes search, portal, shopping, auction, news, finance, travel, restaurant introduction, sports, television, video, game, map, route information, cooking, real estate, automobile, blog, marriage. , comics, fortune-telling, and local information.

The receiving unit 132 receives content information of content (that is, content data). For example, the receiving unit 132 receives content information from a predetermined information processing device. For example, the receiving unit 132 receives content information from devices of entities (for example, companies) related to the learning device 100 . The receiving section 132 may store the received content information in a content information storage section (not shown), which is a predetermined storage area within the storage section 120 .

(Second acquisition unit 133)
The second acquisition unit 133 acquires state information indicating a state. A state is the state of an agent in reinforcement learning. More specifically, the environment returns states (ie, state observations) to the agent in response to the agent's actions. That is, states correspond to observations.

For example, the second acquisition unit 133 acquires behavior information indicating user behavior as the above-described state information. The second acquisition unit 133 acquires the action information received by the reception unit 132, for example. The second acquisition unit 133 may acquire behavior information from a predetermined storage device. For example, the second acquisition unit 133 may acquire behavior information from the behavior information storage unit (not shown) described above. The second acquisition unit 133 may store the acquired behavior information in the behavior information storage unit.

In one example, the second acquisition unit 133 acquires user behavior on the web as a “state” (that is, observation). An "environment" is, for example, a content delivery network. In this example, the environment returns to the agent the state "user behavior on the web" in response to the agent's action "content delivery". In other words, the second acquisition unit 133 acquires the state “user behavior on the web” from the environment “content distribution network”. User behavior on the web includes, for example, browsing top pages, browsing news, searching for travel, browsing travel services, and the like.

(Third Acquisition Unit 134)
The third acquisition unit 134 acquires output information output by a model for determining values and policies in reinforcement learning. For example, the third acquisition unit 134 acquires output information indicating actions taken by the agent.

In one example, the third acquisition unit 134 inputs the user behavior as a state into the model, and acquires the presence or absence of advertisement provision as the behavior.

For example, the third acquisition unit 134 inputs a feature vector (also called feature vector or instance) corresponding to user behavior into the model to determine whether or not an advertisement is to be provided (that is, whether or not a predetermined advertisement is to be provided). as the action taken by the agent.

For example, if the above-mentioned model is a model based on behavior or state values (i.e. policy-on algorithm), the third acquisition unit 134 extracts feature vectors corresponding to user behavior from this model (e.g. neural network) to output the probability that a given ad will be served. Then, the third obtaining unit 134 determines whether or not a predetermined advertisement is provided based on the output probability. On the other hand, if the model is a policy-based model (i.e., off-policy algorithm), the third acquisition unit 134 inputs feature vectors corresponding to user behavior into this model (e.g., Q network) to obtain , outputs a Q-value corresponding to a given state (eg, a given user action) and a given action (eg, the serving of a given advertisement). Then, the third obtaining unit 134 determines whether or not a predetermined advertisement is provided based on the output Q value. For example, the third acquisition unit 134 uses the epsilon greedy method to determine the action corresponding to the largest Q value as the action taken by the agent with a predetermined probability (that is, epsilon).

(Determination unit 135)
The determining unit 135 determines a reward according to the final state when the state is repeatedly executed until the final state is reached.

In one example, the second acquiring unit 133, the third acquiring unit 134, and the providing unit 138 of the learning device 100 perform the “first step of acquiring user behavior on the web”, “user behavior as a state”, as described above. Input to the model and execute multiple episodes corresponding to the series of processing steps of the second step of acquiring the presence or absence of advertisement provision as behavior and the third step of providing content according to the presence or absence of advertisement provision do. In this example, states correspond to user actions. Also, the placement of advertisements corresponds to the actions of agents. Note that "time" corresponds to a discrete time when the agent acquires the state (eg, user behavior) from the environment and takes action (eg, placing an advertisement). In this example, the agent's goal is to book a hotel room. When the state reaches the final state, the determination unit 135 determines a reward according to the outcome of the episode. For example, when the user makes an accommodation reservation in the first step described above, the agent obtains the reward "100" from the environment. In other words, the determining unit 135 determines the value “100” as the reward based on the episode result “accommodation reservation”.

The determination unit 135 may store the history of processing steps in the learning data storage unit 122 . For example, the decision unit 135 may accumulate a series of processing steps as experience. For example, a series of processing steps executed by the second acquisition unit 133, the third acquisition unit 134, and the provision unit 138 may be stored in the learning data storage unit 122 as experience. More specifically, the determination unit 135 may store in the learning data storage unit 122 the experience of the agent indicating the state, action, reward, transition destination state, and episode end flag.

(Evaluation unit 136)
The evaluation unit 136 performs an input process of inputting state information indicating a state into the model and an update process of updating the state using the output information output by the model. Each executed processing step is evaluated according to this final state when it is repeatedly executed until a state is reached. For example, the evaluation unit 136 evaluates each executed processing step according to the reward determined by the determination unit 135 .

For example, the evaluation unit 136 evaluates each processing step according to whether the final state satisfies a predetermined objective. For example, the evaluation unit 136 evaluates each processing step performed when the final state satisfies the predetermined objective higher than each processing step performed when the final state does not satisfy the predetermined objective. Also, for example, the evaluation unit 136 highly evaluates the processing step as the state updated in the processing step is closer to a predetermined target. In one example, the evaluation unit 136 determines that the reward given to the agent's action when the state updated according to the agent's action is browsing travel services is determined to be travel when the state updated according to the agent's other action is traveling. Determine the reward given to an action of the agent to be higher than the reward given to other actions of this agent when searching.

The evaluation unit 136 obtains a plurality of episodes including a plurality of processing steps repeatedly executed until the state satisfies a predetermined condition, and evaluates the processing steps included in each episode according to the final state of each obtained episode. may For example, the evaluation unit 136 identifies a set of the pre-update state in the processing step included in each episode and the output information output by the model in this processing step, and sets each set for all processing steps included in each episode. Each processing step may be evaluated based on the appearance ratio of . For example, the evaluation unit 136 may evaluate a processing step with a high appearance frequency of the corresponding pair higher than a processing step with a low appearance frequency of the corresponding pair.

For example, if the user behavior satisfies a predetermined condition, the evaluation unit 136 evaluates the processing steps up to that point according to the success or failure of the final state, assuming that the episode has reached the final state.

For example, in the “first step of acquiring user behavior on the web”, if the state (that is, user behavior) is not the final state, the second acquiring unit 133, the third acquiring unit 134, and the providing unit of the learning device 100 138, "First step of acquiring user behavior on the web", "Input user behavior as a state into the model Then, a series of processing steps of "the second step of acquiring the presence or absence of advertisement provision as an action" and "the third step of providing content according to the presence or absence of advertisement provision" are repeated. The above-mentioned predetermined condition is, for example, the condition that "the user makes a reservation for accommodation" (that is, conversion). The predetermined condition described above may be a condition that "the user leaves" (that is, leave). More specifically, this condition may be a condition that "the user terminates a predetermined page". The predetermined page is, for example, a travel service page. The evaluation unit 136 evaluates a series of processing steps of the first step, the second step, and the third step, for example, according to the success or failure of the accommodation reservation.

In one example, the evaluator 136 aggregates booking rates (ie, conversion rates) corresponding to combinations of predetermined states (eg, predetermined user actions) and predetermined actions (eg, provision of predetermined advertisements). For example, the evaluation unit 136 calculates the reservation rate of each processing step. More specifically, the evaluation unit 136 calculates the reservation rate corresponding to the combination of the state and the placement of the advertisement. In this example, for example, the reservation rate corresponding to the combination of the state "top page viewed" and advertisement placement "Y1" is "1%". This means that the probability that the user of the user device 10 selects the advertisement Y1 displayed in the TOP page and then makes an accommodation reservation is "1%". Then, the evaluation unit 136 reflects the total reservation rate in the remuneration. For example, the evaluation unit 136 redefines the product of the calculated reservation rate corresponding to the combination of the state and the advertisement and the determined remuneration as the remuneration corresponding to the combination of the state and the advertisement. . For example, the evaluation unit 136 calculates the product of the reservation rate “1%” corresponding to the combination of the state “top page viewed” and the advertisement placement “Y1” and the reward “100” determined by the determination unit 135, “1”. is defined as the reward corresponding to this combination of state and ad placement. In this example, the evaluation unit 136 generates a plurality of episodes with redefined rewards from the plurality of episodes.

As described above, the evaluation unit 136 performs the “first step of acquiring user behavior on the web” and the “second step of inputting user behavior into the model as a state and acquiring the presence or absence of advertisement provision as an action”. and "the third step of providing content according to the presence or absence of advertisements" can be evaluated according to the success or failure of the final state of the episode. That is, the evaluation unit 136 can use the final state to evaluate intermediate processing steps. For example, by executing a plurality of episodes corresponding to this series of processing steps, the evaluation unit 136 can give rewards to intermediate processing steps according to the success or failure of the final state.

(Learning unit 137)
The learning unit 137 learns the model based on the output information output by the model in the processing step, the updated state in this processing step, and the evaluation result for this processing step.

In one example, the learning unit 137 updates the model using information on each processing step evaluated by the evaluation unit 136 .

For example, the learning unit 137 updates the model using multiple episodes stored in the learning data storage unit 122 . For example, if the above-described model is a model based on behavior or state values, the learning unit 137 uses a plurality of episodes stored in the learning data storage unit 122 to perform Q-learning. On the other hand, when the model is a policy-based model, the learning unit 137 performs policy learning using a plurality of episodes stored in the learning data storage unit 122 . For example, the learner 137 can use the policy gradient method to train the policy of the model (eg, neural net).

(Providing unit 138)
The providing unit 138 provides content to the user device 10 . The providing unit 138 provides the user device 10 with the content received by the receiving unit 132, for example. The providing unit 138 may acquire content from the content information storage unit (not shown) described above. The providing unit 138 may provide the acquired content to the user device 10 .

In one example, the providing unit 138 provides content depending on whether advertisements are provided or not.

For example, when the action taken by the agent is to provide a predetermined advertisement, the providing unit 138 provides the user device 10 with content in which the predetermined advertisement is posted. As described above, the predetermined advertisement is, for example, an accommodation advertisement.

[3. Flow of learning process]
Next, a procedure of learning processing by the learning device 100 according to the embodiment will be described. FIG. 4 is a flowchart showing a learning processing procedure by the learning device 100 according to the embodiment.

As shown in FIG. 4, first, the first acquisition unit 131 initializes a model that determines agent behavior (step S101).

Next, the second acquisition unit 133 acquires the state from the environment (step S102). For example, the second acquisition unit 133 acquires the first state from the environment.

Next, the third acquisition unit 134 inputs the state acquired by the second acquisition unit 133 to the model (that is, the agent) initialized by the first acquisition unit 131, thereby acquiring behavior from this model. (Step S103). For example, the third acquisition unit 134 inputs the first state acquired by the second acquisition unit 133 to the model initialized by the first acquisition unit 131, thereby acquiring the first behavior from the model. do.

Next, the second acquisition unit 133 acquires a new state from the environment based on the acquired state and the action acquired by the third acquisition unit 134 (step S104). For example, the second acquisition unit 133 acquires the second state from the environment based on the acquired first state and the first action acquired by the third acquisition unit 134 .

Next, the second acquisition unit 133 accumulates the acquired state and the action acquired by the third acquisition unit 134 as the state and action at a certain time (step S105). For example, the second acquisition unit 133 accumulates the acquired first state and the first action acquired by the third acquisition unit 134 as the state and action at the first time.

Next, the determination unit 135 updates the state to a new state, and determines whether the new state is the final state (step S106). For example, the determination unit 135 updates the first state to the second state and determines whether the second state is the final state.

If it is determined that the new state is not the final state (step S106; No), the learning process procedure returns to step S103. In this case, as described above, the third acquisition unit 134 acquires a new action from the model by inputting this new state into the model. For example, the third acquisition unit 134 acquires the second behavior from the model by inputting the second state into the model.

On the other hand, when it is determined that the new state is the final state (step S106; Yes), the determination unit 135 determines a reward for the episode (step S107). For example, when the second state is determined to be the final state, the determining unit 135 determines the reward for the episode ending at the first time.

Next, the evaluation unit 136 redefines the rewards for the intermediate processing steps determined by knowing the final state as the accumulated state and action rewards at each time (step S108). For example, if the second state is determined to be the final state, the evaluator 136 may replace the reward for the intermediate processing step, determined by knowing the final state, with the accumulated state at the first time. and redefine as a reward for action.

Next, the learning unit 137 updates the model using the rewards for the intermediate processing steps redefined by the evaluation unit 136 (step S109).

Next, the determination unit 135 calculates the sum of rewards obtained in episodes (that is, new episodes) in which the agent has determined actions using the model updated by the learning unit 137 (step S110).

After that, the determination unit 135 determines whether or not the sum of the calculated rewards is greater than the threshold (step S111).

When it is determined that the calculated sum of rewards is not greater than the threshold (step S111; No), the learning processing procedure returns to step S102.

On the other hand, when it is determined that the calculated sum of rewards is greater than the threshold (step S111; Yes), the learning processing procedure ends.

[4. Modification]
The learning device 100 according to the above embodiments may be implemented in various different forms other than the above embodiments. Therefore, another embodiment of the learning device 100 will be described below.

[4-1. Evaluation of processing steps according to intermediate state and final state]
The evaluation unit 136 may further evaluate each processing step according to a comparison result between the state updated in each processing step and the state when a predetermined condition is satisfied. For purposes of illustration, assume that a first state (eg, user action) transitions to a second state in response to action of a first agent. Further assume that the second state transitions to a final state (eg, conversion) in response to the action of the second agent. In this example, the first agent's actions indirectly contribute to the transition from the second state to the final state. Therefore, the evaluator 136 may reward actions of the first agent based on the final state.

[4-2. Evaluation of processing steps according to order]
The evaluation unit 136 may evaluate each processing step repeatedly executed until the state satisfies a predetermined condition, according to the order in which each processing step was executed. For example, the evaluator 136 may determine a discount rate for each processing step based on the order in which each processing step was performed. Then, the evaluation unit 136 may evaluate each processing step repeatedly performed until the state satisfies a predetermined condition based on the determined discount rate. For example, based on the determined discount rate, the evaluation unit 136 may determine a reward corresponding to each processing step repeatedly performed until the state satisfies a predetermined condition.

[4-3. Estimation of Rewards Leading to Successful Episodes]
The evaluation unit 136 performs an input process of inputting state information indicating a state into the model and an update process of updating the state using the output information output by the model. If performed repeatedly until a state is reached, each processing step performed may be evaluated by estimating the reward leading to this final state. For example, the evaluation unit 136 may determine the action of the agent corresponding to the successful episode as the action of an expert (that is, an excellent agent). Additionally, the evaluator 136 may train the model using a predetermined reward function. Then, the evaluation unit 136 may use a method of inverse reinforcement learning to compare the output of the model and the action of the expert, and update the reward function based on the comparison result. This allows the evaluator 136 to estimate the rewards leading to the final state described above.

[4-4. Applicable domain]
In the exemplary embodiment described above, the learning device 100 evaluates processing steps relating to advertisement placement, and uses the evaluation results to perform model learning, but is not limited to this. The learning device 100 according to the embodiment can be applied to domains other than advertising. Regarding the evaluation of processing steps, for example, the evaluation unit 136 can evaluate the state at each processing step based on the difference between the state at each processing step and the final state. The evaluator 136 can employ any setting for evaluating the state at each processing step.

For example, the learning device 100 according to the embodiment may be applied to route prediction. In one example, the path prediction is a baseball outfield fly. In this example, the state and action in each processing step correspond to the position of the ball and the action of the fielder (eg, the position of the fielder and the height of the glove), respectively. The final state corresponds to success or failure to catch the ball. For example, the evaluation unit 136 evaluates each state based on the difference between the state "position of the ball" in each processing step and the final state "position where the fielder caught the ball" or final state "position where the fielder failed to catch the ball". The status "fielder position" in the processing step may be evaluated. In another example, the path prediction is projectile interception. In this example, the state and action in each processing step correspond to the position of the projectile and the position of the interceptor, respectively. The final state corresponds to success or failure of the interception. For example, the evaluation unit 136 determines the state "position of the flying object" in each processing step and the final state "position where the intercepting system intercepted the flying pair" or the final state "position where the intercepting system failed to intercept the flying pair". Based on the difference in , the state "position of projectile" at each processing step may be evaluated.

Also, for example, the learning device 100 according to the embodiment may be applied to search for a route. In one example, the path finding is maze searching. In this example, the state and action in each processing step correspond to the position of the wall in the maze and the agent's movement action, respectively. The final state corresponds to the success or failure of reaching the goal within a given time. For example, the evaluation unit 136 determines the state of the agent in each processing step based on the difference between the state of the wall in the maze in each processing step and the final state of the time taken to reach the goal. "Moving motion" may be evaluated.

[5. others〕
Also, among the processes described in the above embodiments, some of the processes described as being automatically performed can also be performed manually. Alternatively, all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, information including processing procedures, specific names, various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

Also, each component of each device illustrated is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution and integration of each device is not limited to the one shown in the figure, and all or part of them can be functionally or physically distributed and integrated in arbitrary units according to various loads and usage conditions. Can be integrated and configured.

For example, part or all of the storage unit 120 shown in FIG. In this case, the learning device 100 acquires various information such as models and learning data by accessing the storage server.

[6. Hardware configuration]
Also, the learning device 100 according to the above-described embodiments is implemented by a computer 1000 configured as shown in FIG. 5, for example. FIG. 5 is a diagram illustrating an example of a hardware configuration; A computer 1000 is connected to an output device 1010 and an input device 1020, and an arithmetic device 1030, a primary storage device 1040, a secondary storage device 1050, an output IF (Interface) 1060, an input IF 1070, and a network IF 1080 are connected via a bus 1090. have

The arithmetic device 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, programs read from the input device 1020, and the like, and executes various processes. The primary storage device 1040 is a memory device such as a RAM that temporarily stores data used by the arithmetic device 1030 for various calculations. The secondary storage device 1050 is a storage device in which data used for various calculations by the arithmetic device 1030 and various databases are registered, and is implemented by a ROM (Read Only Memory), HDD, flash memory, or the like.

The output IF 1060 is an interface for transmitting information to be output to the output device 1010 that outputs various types of information such as a monitor and a printer. It is realized by a connector conforming to a standard such as HDMI (registered trademark) (High Definition Multimedia Interface). Also, the input IF 1070 is an interface for receiving information from various input devices 1020 such as a mouse, keyboard, scanner, etc., and is realized by, for example, USB.

Note that the input device 1020 includes, for example, optical recording media such as CDs (Compact Discs), DVDs (Digital Versatile Discs), PDs (Phase change rewritable discs), magneto-optical recording media such as MOs (Magneto-Optical discs), and tapes. It may be a device that reads information from a medium, a magnetic recording medium, a semiconductor memory, or the like. Also, the input device 1020 may be an external storage medium such as a USB memory.

Network IF 1080 receives data from other devices via network N and sends the data to arithmetic device 1030, and also transmits data generated by arithmetic device 1030 via network N to other devices.

The arithmetic device 1030 controls the output device 1010 and the input device 1020 via the output IF 1060 and the input IF 1070 . For example, arithmetic device 1030 loads a program from input device 1020 or secondary storage device 1050 onto primary storage device 1040 and executes the loaded program.

For example, when computer 1000 functions as learning device 100 , arithmetic device 1030 of computer 1000 implements the function of control unit 130 by executing a program loaded on primary storage device 1040 .

[7. effect〕
As described above, the learning device 100 according to the embodiment has the evaluation section 136 and the learning section 137 . The evaluation unit 136 performs an input process of inputting state information indicating a state into the model and an update process of updating the state using the output information output by the model. Each executed processing step is evaluated according to this final state when it is repeatedly executed until a state is reached. The learning unit 137 learns the model based on the output information output by the model in the processing step, the updated state in this processing step, and the evaluation result for this processing step.

Further, the learning device 100 according to the embodiment has a determination unit 135 that determines a reward according to the final state when the state is repeatedly executed until the final state is reached. Also, in the learning device 100 according to the embodiment, the evaluation unit 136 evaluates each executed processing step according to the reward determined by the determination unit 135 .

Also, in the learning device 100 according to the embodiment, the evaluation unit 136 evaluates each processing step depending on whether the final state satisfies a predetermined objective.

In addition, in the learning device 100 according to the embodiment, the evaluation unit 136 compares each processing step executed when the final state satisfies the predetermined purpose to each processing step executed when the final state does not satisfy the predetermined purpose. Rate it higher than the processing step.

In addition, in the learning device 100 according to the embodiment, the evaluation unit 136 further performs each processing step according to the comparison result between the state updated in each processing step and the state when a predetermined condition is satisfied. evaluate.

In addition, in the learning device 100 according to the embodiment, the evaluation unit 136 highly evaluates the processing step as the state updated in the processing step is closer to a predetermined target.

Moreover, in the learning device 100 according to the embodiment, the evaluation unit 136 acquires a plurality of episodes including a plurality of processing steps repeatedly executed until the state satisfies a predetermined condition, and determines the final state of each acquired episode. , to evaluate the processing steps involved in each episode.

Further, in the learning device 100 according to the embodiment, the evaluation unit 136 identifies a set of the pre-update state in the processing step included in each episode and the output information output by the model in this processing step. Each processing step is evaluated based on the percentage of occurrence of each set relative to all processing steps involved.

In addition, in the learning device 100 according to the embodiment, the evaluation unit 136 evaluates a processing step with a high appearance frequency of the corresponding pair higher than a processing step with a low appearance frequency of the corresponding pair.

Also, in the learning device 100 according to the embodiment, the evaluation unit 136 evaluates each processing step repeatedly executed until the state satisfies a predetermined condition, according to the order in which each processing step was executed.

Through the above-described processes, the learning device 100 can improve the performance of reinforcement learning.

As described above, some of the embodiments of the present application have been described in detail based on the drawings. It is possible to carry out the invention in other forms with modifications.

Further, the learning device 100 described above may be implemented by a plurality of server computers, and some functions may be implemented by calling an external platform or the like using an API (Application Programming Interface) or network computing. Flexible to change.

Also, the above-mentioned "section, module, unit" can be read as "means" or "circuit". For example, the first acquisition unit can be read as a first acquisition means or a first acquisition circuit.

1 learning system 10 user device 100 learning device 110 communication unit 120 storage unit 121 model storage unit 122 learning data storage unit 130 control unit 131 first acquisition unit 132 reception unit 133 second acquisition unit 134 third acquisition unit 135 determination unit 136 evaluation Section 137 Learning Section 138 Providing Section

Claims (14)

A processing step including an input process of inputting state information indicating a state into a model and an update process of updating the state using the output information output by the model is a final state in which the state satisfies a predetermined condition. an evaluator that rewards each executed processing step according to its final state if repeatedly executed to
a learning unit that learns the model based on the output information output by the model in the processing step, the updated state in the processing step , and the reward given to the processing step; and learning device. a determination unit that determines a reward according to the final state when the state is repeatedly executed until the final state is reached;
The evaluator rewards each performed processing step according to the reward determined by the determiner.
2. The learning device according to claim 1, wherein: The evaluator rewards each processing step according to whether the final state satisfies a predetermined objective.
3. The learning device according to claim 1 or 2, characterized in that: The evaluator rewards each processing step performed when the final state satisfies a predetermined objective higher than each processing step performed when the final state fails to satisfy a predetermined objective.
4. The learning device according to claim 3, characterized in that: The evaluation unit further rewards each processing step according to a comparison result between the state updated in each processing step and the state when the predetermined condition is satisfied.
4. The learning device according to any one of claims 1 to 3, characterized in that: The evaluation unit gives a higher reward to the processing step as the state updated in the processing step is closer to a predetermined target.
5. The learning device according to claim 4, characterized in that: The evaluation unit obtains a plurality of episodes including a plurality of processing steps repeatedly executed until the state satisfies a predetermined condition , and rewards the processing steps included in each episode according to the final state of each obtained episode. give
The learning device according to any one of claims 1 to 6, characterized in that: The evaluator rewards each processing step repeatedly executed until the state satisfies a predetermined condition, according to the order in which the processing steps were executed .
The learning device according to any one of claims 1 to 7 , characterized in that: A processing step including an input process of inputting state information indicating a state into a model and an update process of updating the state using the output information output by the model is a final state in which the state satisfies a predetermined condition. an evaluation unit that evaluates each executed processing step according to the final state when repeatedly executed until
a learning unit that learns the model based on the output information output by the model in the processing step, the updated state in the processing step, and the evaluation result for the processing step;
has
The evaluation unit acquires a plurality of episodes including a plurality of processing steps that are repeatedly executed until the state satisfies a predetermined condition, and acquires a state before updating in the processing steps included in each episode and the model in the processing steps. Each processing step is evaluated based on the obtained final state in each episode and the appearance ratio of each set to all processing steps included in each episode.
A learning device characterized by: The evaluation unit identifies a set of a pre-update state in a processing step included in each episode and output information output by the model in the processing step, and determines each set for all processing steps included in each episode. 10. The learning device according to claim 9 , wherein each processing step is evaluated based on the appearance ratio. A learning method executed by a learning device,
A processing step including an input process of inputting state information indicating a state into a model and an update process of updating the state using the output information output by the model is a final state in which the state satisfies a predetermined condition. an evaluation step that rewards each processing step performed according to its final state if iteratively performed to
a learning step of learning the model based on the output information output by the model in the processing step, the updated state in the processing step , and the reward given to the processing step. and learning method. A learning method executed by a learning device,
A processing step including an input process of inputting state information indicating a state into a model and an update process of updating the state using the output information output by the model is a final state in which the state satisfies a predetermined condition. an evaluation step of evaluating each executed processing step according to the final state when repeatedly executed until
a learning step of learning the model based on the output information output by the model in the processing step, the updated state in the processing step, and the evaluation result for the processing step;
including,
The evaluation step acquires a plurality of episodes including a plurality of processing steps that are repeatedly executed until the state satisfies a predetermined condition, and obtains a state before updating in the processing steps included in each episode and the model in the processing steps. Each processing step is evaluated based on the obtained final state in each episode and the appearance ratio of each set to all processing steps included in each episode.
A learning method characterized by: A processing step including an input process of inputting state information indicating a state into a model and an update process of updating the state using the output information output by the model is a final state in which the state satisfies a predetermined condition. an evaluation procedure that rewards each executed processing step according to its final state if repeatedly executed to
causing a computer to execute a learning procedure for learning the model based on the output information output by the model in the processing step, the updated state in the processing step , and the reward given to the processing step study program for A processing step including an input process of inputting state information indicating a state into a model and an update process of updating the state using the output information output by the model is a final state in which the state satisfies a predetermined condition. an evaluation procedure for evaluating each executed processing step according to the final state when repeatedly executed until
a learning procedure for learning the model based on the output information output by the model in the processing step, the updated state in the processing step, and the evaluation result for the processing step;
on the computer, and
The evaluation procedure obtains a plurality of episodes including a plurality of processing steps that are repeatedly executed until the state satisfies a predetermined condition, and acquires a state before updating in the processing steps included in each episode and the model in the processing step. Each processing step is evaluated based on the obtained final state in each episode and the appearance ratio of each set to all processing steps included in each episode.
A learning program characterized by:

Images