JP7308073B2 - Logistics management system - Google Patents

Description

The present invention relates to a physical distribution management system.

In logistics management in the supply chain, the so-called Bullwhip Effect (below BE
) is known. BE is a phenomenon in which, as a result of demand forecasting and decision-making in the downstream of the supply chain, demand expands and propagates from the downstream to the upstream of the supply chain. Since this phenomenon leads to excess inventory and shortages, the mechanism of BE generation and methods for suppressing BE have been the subject of research for many years. Factors that cause BE include the price list, order frequency, return policy, frequency and depth of price sales measures, degree of information sharing, demand forecasting method, and distribution rules for shortages.

Also, Beer Game (hereinafter referred to as “BG”) is known as one technique for verifying BE. BG is a tandem beer supply chain simulation game in which four players (retailers, wholesalers, distributors, and manufacturers) compete to minimize costs within a defined period of time.

The process of BG is known as the Markov Decision Process (MDP), and in BG, the information that each player can observe is only orders with neighboring players, product exchanges, and own inventory levels, so it is so-called Partially Observable Markov Decision Process (POMDP). In BG, each player chooses the action that minimizes the cost from observable information. However, BG has a large observation space and action space, and it is a complicated problem because it deals with non-stationary time series. Therefore, it has been shown that by applying deep reinforcement learning to BG, it is possible to achieve overall optimization of logistics processes in the supply chain (Non-Patent Documents 1 to 3).

V. Mnih et al. Human-level control through deep reinforcement learning, doi 10.1038/nature14236 (2015). Afshin Oroojlooyjadid et al. A Deep Q-Network for the Beer Game: A Reinforcement Learning Algorithm to Solve Inventory Optimization Problems, arXiv:1708.05924v2 (2018). Taiki Fuji et al. Deep Multi-Agent Reinforcement Learning using DNN-Weight Evolution to Optimize Supply Chain Performance. doi 10.24251/HICSS.2018.157 (2018).

In mail-order sales using the Internet (online shopping), the difference between the efficiency of information flow in cyberspace and the efficiency of product flow in physical space is becoming more pronounced, and this difference is a new cause of BE. may become. For example, Electronic Commerce (
As a result of highly efficient sales measures on EC) sites amplifying fluctuations in demand, delivery delays, shortages, excess inventory, etc. may occur. For this reason, even if the above conventional technology is used, it may not be possible to achieve total optimization of the logistics process in the supply chain for online shopping. No method was proposed.

Therefore, the technique of the present disclosure has been made in view of the above circumstances, and its purpose is to provide a technique for supporting overall optimization of physical distribution management.

The prediction device of the present disclosure includes an acquisition unit that acquires product information of electronic commerce products, and the acquired product information and the state of the environment for multiple values of behavioral variables that are allowed in the environment related to product distribution. Generate a third behavior model based on reinforcement learning of a first behavior model and a second behavior model that calculate mutually different rewards using variables, and use the third behavior model to perform optimal behavior in the environment and a controller for predicting the value of the variable. As a result, with this prediction device, it is possible to generate a well-balanced behavioral model that achieves overall optimization from two behavioral models in which contradictory goals are set in product logistics management, and to predict the optimum behavioral variables. can.

In the prediction device described above, the behavioral variable is the number of products ordered by the product distributor, and the product information acquired by the acquisition unit reflects fluctuations in the number of orders, shipments, and receipts of products over a certain period of time. The state variables of the environment are the sales of goods, the purchase price of goods, the stockout costs associated with stockouts of goods, the promotional costs associated with promoting goods, and the overstock of goods. The associated inventory cost and the delivery cost associated with the delivery of the product. The control unit uses the acquired product information and the environmental state variables for each of the order numbers corresponding to the multiple values of the behavior variable. A third behavior model may be generated based on reinforcement learning of a first behavior model and a second behavior model that calculate different rewards, and an optimal number of orders may be predicted using the third behavior model. . This controls the BEI so that it does not exceed the threshold, and the shipments are made to meet demand as much as possible, balancing the conflicting goals of minimizing inventory costs and maximizing sales. Product inventory can be optimized.

Further, in the prediction device described above, the behavioral variable is man-hours related to the storage of the product in the warehouse, the product information acquired by the acquisition unit is information indicating the implementation date of the sales promotion of the product, and the environmental The state variable is a sales promotion cost determined according to the day of the week on which the sales promotion of the product is implemented, and the control unit stores the acquired product information and Generate a third behavior model based on reinforcement learning of the first behavior model and the second behavior model that calculate mutually different rewards using environmental state variables, and use the third behavior model to find the optimal You can predict man hours. This balances the conflicting goals of maximizing sales and minimizing shipping costs by shipping to meet demand that fluctuates due to sales promotions and keeping man hours low in warehouses where products are stored. , it is possible to optimize the man-hours allocated to shipping the product on the day of the sales promotion set in this simulation.

Further, in the prediction device described above, the behavioral variable is a value that indicates the ease with which a product is recommended on the EC site, and the product information acquired by the acquisition unit is the number of clicks on the product over a certain period of time on the EC site and displayed. The environmental state variables are product size, product sales, product purchase price, and inventory cost determined according to product size. A first behavior model and a second behavior that calculate mutually different rewards using acquired product information and environmental state variables for each value that indicates the ease of recommending a product corresponding to a plurality of values. A third behavior model may be generated based on model reinforcement learning, and the third behavior model may be used to predict an optimal value indicating the likelihood of a product being recommended. This optimizes product recommendations on EC sites by identifying products that do not sell because they are not recommended and products that do not sell even if they are recommended, and recommending products that are more likely to sell. can be achieved.

In the prediction device described above, the behavior variable is an incentive given to the user by changing the designated delivery date of the product on the EC site, and the product information acquired by the acquisition unit includes the delivery destination and delivery date of the product. Information indicating the date and time, the delivery area of the product delivery company, the number of deliverable destinations, the time required for product delivery, and the delivery distance for product delivery. A third behavior model based on reinforcement learning of the first behavior model and the second behavior model that calculates mutually different rewards using the acquired product information and the environmental state variables for each of the incentives corresponding to , and a third behavioral model may be used to predict the optimal incentive. By doing so, we aim to balance the two goals of minimizing delivery costs and minimizing incentive costs, while optimizing the size of incentives given when prompting users to change the delivery date of the relevant product on the e-commerce site. be able to.

According to the technology of the present disclosure, it is possible to provide a technology that supports overall optimization of physical distribution management.

FIG. 1 shows an example of a prediction device according to the first embodiment. FIG. 2A shows the relationship between players and supply chain in BG, and FIG. 2B shows the relationship between players and supply chain in Netshop Game executed in the first embodiment. FIG. 3 shows an example of a simulation algorithm executed in the first embodiment. FIG. 4 shows an example of environmental parameters set in the first embodiment. FIG. 5 shows an example of Netshop Game rewards set in the first embodiment. FIG. 6 shows an example of updating of the above state variables executed in the first embodiment. FIG. 7 shows an example of goal conditions for meta players in the first embodiment. FIG. 8 is a graph showing an example of changes in time steps and rewards for meta players in the first embodiment. FIG. 9 is a graph showing an example of changes in rewards earned by cyber players in the first embodiment. FIG. 10 is a graph showing an example of changes in rewards earned by physical players according to the first embodiment. FIG. 11 is a graph showing an example of changes in rewards earned by meta players in the first embodiment. FIG. 12 shows an example of simulation results for each player in the first embodiment. 13A shows an example of simulation rewards in Modification 1, FIG. 13B shows an example of updating the simulation state variables in Modification 1, and FIG. 13C shows an example of simulation goal conditions in Modification 1. show. 14A shows an example of simulation rewards in Modification 2, FIG. 14B shows an example of updating the simulation state variables in Modification 2, and FIG. 14C shows an example of simulation goal conditions in Modification 2. show. 15A shows an example of simulation rewards in Modification 3, FIG. 15B shows an example of updating the simulation state variables in Modification 3, and FIG. 15C shows an example of simulation goal conditions in Modification 3. show.

Preferred embodiments of the technology disclosed herein will be described below with reference to the drawings. However, the configuration of the components described below should be appropriately changed according to the configuration of the device to which the technique of the present disclosure is applied and various conditions. Therefore, the technical scope of the technology disclosed herein is not intended to be limited to the following description.

(First embodiment)
FIG. 1 is a diagram showing a schematic configuration of a prediction device according to the first embodiment. As shown in FIG. 1 , the prediction device 1 has a control unit 11 , a storage unit 12 , an operation unit 13 , a display unit 14 and a communication unit 15 .

In online shopping, there are differences in characteristics between processes that are completed in cyberspace, such as e-commerce sites and online transactions, and processes that are performed in physical space, such as distribution warehouses and delivery centers. For example, in cyber space, 100 products are just numerical data. For this reason, it can be said that increasing the number of sales to 1000 units, which is 10 times higher, through aggressive sales measures is less subject to physical restrictions. On the other hand, in the physical space, 100 products are entities with volume and weight. Therefore, it can be said that increasing the sales volume to 1000 is greatly affected by physical restrictions such as warehouse capacity and shipping capability. Also, in the above BG, there is a lead time for orders to upstream players in the supply chain, but the lead time for orders can be ignored in online shopping transactions. For this reason, when dealing with the problem of the physical distribution process of online shopping, there is a possibility that a solution to the problem cannot be found using the BG model described above.

Here, FIG. 2 shows the relationship between players and the supply chain in each of BG (FIG. 2A) and Netshop Game (FIG. 2B) executed in the first embodiment. In the first embodiment, the prediction device 1 executes a simulation called Netshop Game in which the environment illustrated in FIG. 2A is set. As shown in FIG. 2B, in the above BG, the four players of the retailer, wholesaler, distributor, and manufacturer are connected in series, but in the Netshop Game, the player is only the retailer, and the retailer Wholesalers, distributors, and manufacturers positioned upstream in the supply chain are grouped into one player as a supplier. In the first embodiment, it is assumed that there is one type of electronic commerce product handled by the Netshop Game. Furthermore, in the Netshop Game, retailers are divided into cyber players who manage processes occurring in cyber space and physical players who manage processes occurring in physical space.

A cyber player acts so that a fill rate (also referred to as FR), which indicates the extent to which products can be supplied without shortage in response to customer demand, is greater than a preset threshold. Cyber players aim to maximize rewards in cyberspace. Cyber players actively carry out sales promotions such as sales on e-commerce sites, for example. In addition, cyber players ignore the costs caused by increased inventory and act to minimize opportunity losses due to shortages.

Also, the physical player acts so that the Bullwhip Effect Index (also referred to as BEI) is less than a preset threshold. Physical players aim to minimize logistics costs. The physical player ignores the opportunity loss due to stockouts and acts to minimize the costs associated with warehousing and shipping by keeping inventories as low as possible and avoiding large BEs.

In the first embodiment, the environment shown in FIG. 2B is implemented by OpenAI Gym in the prediction device 1, and deep Q-Network (DQN) is used to learn the optimal actions for achieving the above objectives for each of the cyber player and the physical player. and evaluation. In addition, the
In one embodiment, a meta player is introduced that acts to balance the rewards of the cyber player and the physical player and the goal conditions of the Netshop Game. The cyber player and the physical player are examples of the first behavior model and the second behavior model that calculate mutually different rewards using product information and environmental state variables. Also, the metaplayer is an example of a third behavior model generated based on reinforcement learning of the first behavior model and the second behavior model.

ここで、ＩＬ_ｔは、タイムステップｔにおける在庫数、ＯＯ_ｔは供給業者に対して発注済みであるが未入荷の状態である商品数、ｄ_ｔは顧客からの商品の需要数、ＲＳ_ｔは、供給業者から入荷した商品数、ＳＳ_ｔは、顧客に出荷済みの商品数、ａ_ｔは、タイムステップｔにおいて発生するアクション、すなわち供給業者への商品の発注数である。このように、第１実施形態では、これら６変数の回帰分析を用いる。 Next, details of settings for each player in Netshop Game will be described. First, in Netshop Game, an observable state variable o _t at time step t is defined by the following equation (1).

where IL _t is the number of items in stock at time step t, OO _t is the number of items that have been ordered from suppliers but have not yet arrived, d _t is the number of items demanded by customers, and RS _t is , the number of items received from the supplier, SS _t is the number of items shipped to the customer, and a _t is the action that occurs at time step t, ie, the number of items ordered from the supplier. Thus, in the first embodiment, regression analysis of these six variables is used.

Also, in the Netshop Game, one episode is defined as the period from the start of the game to the achievement of the goal condition, and the state variable _hot of all time steps in one episode is stored as shown in the following equation (2).

Next, the action space in Netshop Game will be explained. Actions can also be said to be behavioral variables that are permissible in an environment related to the distribution of goods. Also, each value of the action becomes each value of the action variable. As shown in the action above, the action in this embodiment is the number of orders for the product to the supplier. If the action space, that is, the degree of freedom of the number of orders allowed for the player, that is, the range from the lower limit to the upper limit of the number of orders is too wide, the processing efficiency of the prediction device 1 may decrease. Therefore, in the present embodiment, as an example, the Netshop Game is implemented with the action space being a set of discrete values [0, 1, 2, . . . , 20] from 0 to 20.

Next, the reward in the Netshop Game will be explained. In the above BG, the reward is determined by the inventory quantity at time step t, as shown in the following equation (3).

ここで、変数ｘについて（ｘ）^＋：ｍａｘ（０，ｘ）、（ｘ）^－：ｍａｘ（０，－ｘ）である。また、在庫数が正の場合は在庫数分の在庫コストｃｈを乗算し、在庫数が負の場合は欠品による機会損失コストｃｐを乗算する。また、ｉは１～４までの整数であり各値
が各プレーヤーに対応する。したがって、式（３）によって全プレーヤーの報酬の総計が算出される。

Here, for the variable x, (x) ⁺ : max(0, x), (x) ⁻ : max(0, −x). If the inventory quantity is positive, it is multiplied by the inventory cost ch for the inventory quantity, and if the inventory quantity is negative, it is multiplied by the opportunity loss cost cp due to shortage. Also, i is an integer from 1 to 4, and each value corresponds to each player. Therefore, equation (3) calculates the total reward for all players.

ここで、ｓ_ｐは売値、ｃ_ｒは仕入れ値、ｃ_ｓは販売促進費、ｃ_ｄは配送費である。Netshop Gameでは各プレーヤーはこれらの値を観測できないものとする。 In Netshop Game, selling price, purchase price, sales promotion cost, and delivery cost are added to the calculation of reward. Specifically, the reward for the cyber player is calculated by equation (4), the reward for the physical player by equation (5), and the reward for the meta player by equation (6).

Here, s _p is the selling price, _cr is the purchase price, _cs is the sales promotion cost, and _cd is the delivery cost. In Netshop Game, each player shall not be able to observe these values.

As Equation (4) shows, the cyber player adds the difference between the number of orders to the supplier and the demand as sales promotion costs in addition to the lost opportunity cost of the shortage. Also, as shown in Equation (5), the physical player adds the inventory cost for the number of inventory and the delivery cost for the number of shipments to the customer to the remuneration. Rewards are set so that cyber players don't care about excess inventory and physical players don't care about shortages. is. On the other hand, as shown in Equation (6), the metaplayer adds each of the above costs to the reward so as to obtain a global optimal solution that balances the above orientations.

Next, the goal conditions for each player in the Netshop Game will be described. In the above BG, each player competes according to the total value of rewards within a predetermined time step period, but in Netshop Game, the following two indices are set as goal conditions. Each player finishes one episode when he/she achieves one of the indicators.

ここで、ｄｅｍａｎｄは、タイムステップｔにおける顧客からの需要数の直近ｐ期間分の配列である（ｐは正の整数）。また、ｓｈｉｐｐｅｄは、タイムステップｔにおける商品の出荷数の直近ｐ期間分の配列である。また、Ｖａｒ（ｘ）は、変数ｘの分散、Ｍｅａｎ（ｘ）は、変数ｘの平均である。さらに、Netshop Gameでは、以下の式（９）および式（１０）によって各プレーヤーのゴール条件の達成を判断する。

ここで、式（９）がフィジカルプレーヤーのゴール条件に用いられ、式（１０）がサイバープレーヤーのゴール条件に用いられる。また、式（９）および式（１０）がメタプレーヤーのゴール条件に用いられる。 One index of the goal condition in the Netshop Game is BEI indicated by the following formula (7), and another index is FR indicated by the following formula (8).

Here, demand is an array of the number of demands from customers at time step t for the most recent p periods (p is a positive integer). Also, shipped is an array of the number of products shipped at time step t for the most recent p periods. Also, Var(x) is the variance of the variable x, and Mean(x) is the mean of the variable x. Furthermore, in Netshop Game, each player's achievement of the goal condition is determined by the following equations (9) and (10).

Here, equation (9) is used for the physical player's goal condition, and equation (10) is used for the cyber player's goal condition. Also, equations (9) and (10) are used for the goal condition of the meta player.

なお、初期値は、１から１０までの自然数からランダムに選択された値が採用される。 Next, the number of demands from customers generated in the Netshop Game will be explained. In the Netshop Game, the number of demands Dt from customers at time step t, as shown in the following equation (11), is a random variable x that follows a normal distribution to the average value of the number of demands for the last p period of time step t. additionally generated.

A value randomly selected from natural numbers from 1 to 10 is adopted as the initial value.

FIG. 3 shows an example of a simulation algorithm performed by OpenAI Gym in this embodiment. As shown in Fig. 3, Netshop Game uses the so-called epsilon greedy algorithm as a method of balancing the accumulation and utilization of experience obtained from action results, and performs state evaluation using a DNN (Deep Neural Network). . Since the algorithm in FIG. 3 is well-known and is also described in the above non-patent document, detailed description is omitted here.

Next, the definition of each player's parameters and reward calculation in this embodiment will be described. Figure 4 shows an example of environmental parameters set by OpenAI Gym. As shown in FIG. 4, in the simulation performed in this embodiment, the BEI threshold (“bei_threshold”) used for the goal condition is 0.95, and the FR threshold (“fillrate_threshold”) is 0.95.
95, Inventory cost ("stock_cost") is 0.5, Shortage cost ("shortage_cost") is 1
. 0, promotion cost ("promotion_cost") is 0.01, delivery cost ("delivery_cost")
) is 0.01. In addition, to generate the number of demand from customers, "demand_volatility" in the figure
(demand fluctuation), "demand_min" (minimum value) and "demand_max" (maximum value) values are used. The sales price ("sales_price") of the product is 2.0, and the purchase price ("purchase_price") of the product is 1.2.

FIG. 5 shows an example of Netshop Game rewards set by OpenAI Gym. As shown in FIG. 5, rewards are set for each of the cyber player, physical player, and meta player.

To explain the meta player's reward more specifically, if the product is out of stock, the meta player will be charged the cost of the stock shortage ("cost += abs(IL) * self. shortage_cost if IL < 0 else 0"). Also, if the number of orders is greater than the number of demands from customers, the sales promotion cost is added ("cost += (action - d) * self.promotion_cost if action > d else 0" in the figure). Also, the inventory cost for the number of inventory is added to the current inventory (“cost += IL * self.stock_cost if IL > 0 else 0” in the figure). Also, the delivery cost is added to the number of shipments (“cost += SS * self.delivery_cost if SS > 0 else 0” in the figure). Also, the selling price (sales) is a value obtained by multiplying the number of shipments by the selling price ("sales_price = SS * self.sales_price" in the figure). Also, the purchase price is the value obtained by multiplying the number of arrivals by the purchase unit price ("purchase_price = RS * self.purchase_price" in the figure). and the reward is
The value obtained by dividing the purchase price and each of the above costs from the sales ("reward = abs(sales_price)-abs(purchase_price)-abs(cost)" in the figure). The meta player aims to achieve the goal conditions described later based on the rewards set in this way.

Next, FIG. 6 shows an example of updating each of the above state variables performed in OpenAI Gym. As shown in Figure 6, in one time step, the current number of inventory ("IL = self.state[0]" in the figure), the number of unarrived orders ("OO = self.state[1]" in the figure) ), and the number of orders placed in the previous time step (“a = self.state[5]” in the figure) is used. In addition, by adding the number of fluctuations following a normal distribution to the moving average of demand, the number of demand from customers is generated using random numbers ("d = max (self.demand_min,int(dave + np.random .randn() * self.demand_volatility))”). In addition, it is assumed that the goods corresponding to the number of orders are received with the lead time of the number of shipments being 1 ("RS = a # received shipment = last order (shipping lead time = 1
)”).

First, the number of products received from the supplier is added to the number of inventories (“IL += RS” in the figure). After that, if the number of demand from the customer is smaller than the number of inventory ("if d <IL:" in the figure), the number of products for the number of demand is shipped to the customer ("SS = d" in the figure), If the quantity of demand is greater than the quantity of inventory (“else: # d >= IL” in the figure), the quantity of inventory is set as the number of shipments to the customer (“SS = IL” in the figure), and the quantity of inventory is changed ( “IL -= SS” in the figure). Then, the value obtained by dividing the received quantity from the current ordered quantity is added to the unarrived ordered quantity ("OO += action - RS" in the figure). Each state variable is changed in this way, and the changed state variables are used to order, receive, and ship products in the next time step.

FIG. 7 shows an example of goal conditions for metaplayers. As shown in Fig. 7, the distribution of the number of shipments ("vars4 = np.var(shipped)" in the figure) and the distribution of the number of demand ("vars2 =np.var(demand)”), and BEI is calculated from the calculated variance (“bei=vars4/vars2 if vars2 !=0 else bei_threshold” in the figure). If the calculated BEI is smaller than the BEI threshold ("if bei <bei_threshold:" in the figure), it is assumed that the calculated BEI satisfies the BEI condition ("bei_flag=True" in the figure). Also, from the current timestep to the most recent fixed period (
For example, 100 days), the average number of shipments to the number of demand is calculated (“fillrate=np.mean(shipped/demand)” in the figure). If the calculated average is greater than the above FR threshold ("if fillrate >self.fillrate_threshold:" in the figure), then the calculated average satisfies the Fillrate condition ("fill_flag = True" in the figure). . Then, when both of the above two conditions are satisfied, it is considered that the goal condition has been achieved ("done = fill_flag &sum_flag" in the figure).

Each player repeatedly updates the state variables shown in FIG. 6 at each time step, aiming to achieve the above goal conditions based on rewards set as shown in FIG. FIG. 8 is a graph showing an example of changes in time steps and rewards for meta players when reinforcement learning is performed with the upper limit of time steps set to 50000 steps in Netshop Game. The horizontal axis of the graph represents time steps, and the vertical axis of the graph represents the rewards that metaplayers earn as described above. As shown in the graph of FIG. 8, it can be seen that the meta player's reward rises toward a constant value as the time step progresses. Therefore, it can be said that the meta player's learning is accumulated each time the time step progresses.

Figures 9 to 11 show an example of changes in demand, orders, and inventory in a test in which 100 episodes of the Netshop Game were run again using the behavior models of each player learned by the Netshop Game above. It is a graph showing. The horizontal axis of the graph represents the time step ("step" in the figure), and the vertical axis of the graph represents the number of products ("items" in the figure). FIG. 9 shows changes in rewards earned by cyber players, FIG. 10 shows changes in rewards earned by physical players, and FIG. 11 shows changes in rewards earned by meta players. Here, it is assumed that the player has already learned, as shown in FIG. 9, in which the player's learning has been accumulated to the point where the reward obtained by the player remains substantially constant. Also, the upper limit of the number of time steps in one episode is set to 1000 steps, and the episode ends when the player completes the actions of the 1000th time step without achieving the above goal conditions. It should be noted that the graph in each figure shows the transition of the acquired reward for the last 100 steps of each episode, that is, 100 steps back from the end of the episode. In each figure, "stock" represents the number of stocks, "demand" represents the number of demand, and "shipped" represents the number of shipments.

As shown in FIG. 9 for changes in the number of inventories, it is thought that in Netshop Games, excess inventories tend to increase and remain high due to the actions of cyber players. Further, in FIG. 10, since the state in which the inventory quantity is 0 continues over a plurality of steps, it is conceivable that there is a tendency for shortages to occur frequently due to the behavior of the physical player. On the other hand, in FIG. 11, the graph showing changes in the number of stocks is generally sawtooth-shaped. This means that even if the inventory count becomes 0, the inventory count will increase in the next step or within a few steps, and even if the inventory count is too high, demand and shipments will work to push down the inventory count. be able to. Therefore, it is thought that there is a possibility that a balanced number of inventories can be achieved through the actions of meta players.

FIG. 12 shows the average values of 100 episodes of the required number of steps, the BEI value, the FR value, and the obtained reward value for each player to achieve the game conditions in the above test. In the graphs of FIGS. 9 to 11, the smaller the number of steps (“steps” in the figures) required to achieve the goal condition, the better. Also, when the value of BEI is less than 1.0, it is considered that the influence of BE is suppressed. Also, the value of FR indicates the ratio of the products that can be shipped to the demand from customers, and the closer to 1.0 the better. As can be seen from Figure 12, if the goal is only to maximize rewards, Cyber Player is the best choice. There is a continuation of the state of affairs. One of the reasons why this phenomenon occurs is that the setting of rewards for cyber players does not provide an element to control excess inventory.

As shown in FIG. 12, in the case of the physical player, the number of inventories is maintained at a small value, so the BEI is also lower than that of the cyber player. The state in which the inventory quantity is 0 continues for a plurality of steps, that is, the state of out-of-stock may continue, and as a result, the FR value is also a low value.

In addition, as shown in FIG. 12, in the case of the meta player, the acquisition reward is lower than that of the cyber player and the physical player, but the BEI value is smaller than that of the cyber player and the physical player, and the FR value is The value lies between cyber players, who have more excess inventory, and physical players, who have more shortages. In addition, in the graph of FIG. 11 showing changes in the number of inventories, there is a portion where the cycle and fluctuation range of changes in the number of inventories are almost constant over multiple steps, so the number of stocks, the number of arrivals, and the number of shipments are well balanced. In other words, it is considered that the learning effect of appropriate inventory management for stabilizing inventory is obtained.

In the first embodiment, the control unit 11 of the prediction device 1 controls the communication unit 15 as an acquisition unit to communicate with the product information DB (database) 2 . The communication unit 15 acquires information on the number of ordered products, the number of shipments, and the number of arrivals of products for a certain period of time, which is stored in the product information DB 2 . The information acquired from the product information DB 2 here is an example of product information indicating changes in the number of orders, the number of shipments, and the number of arrivals of products over a certain period of time. The control unit 11 generates the behavior model by reinforcement learning that calculates rewards determined based on each state variable based on the information acquired by the communication unit 15 while changing the number of orders. Then, the control unit 11 uses this behavior model to simulate the behavior of the metaplayer in the Netshop Game.

Specifically, the control unit 11 repeats the simulation while changing the number of products ordered by the meta player from the upper limit to the lower limit, for example, increasing the number of orders by one from 0 to 20 in the above example. As an example, the control unit 11 sets the upper limit of the number of time steps in one episode to 1000 steps as described above, and when the metaplayer completes the action of the 1000th time step without achieving the above goal condition, the episode starts. Based on the results of executing 100 episodes for each order quantity, the number of steps to achieve the goal conditions for each order quantity, BEI value, FR value, and the average value of 100 episodes for the acquisition reward Calculation results are output by storing the calculation results in the storage unit 12, displaying them on the display unit 14, or transmitting them from the communication unit 15 to an external device (not shown). The calculation result by the prediction device 1 is an example of the optimal number of orders predicted by the prediction device.

The user can confirm the calculation result by the prediction device 1 and determine the actual number of orders for the simulation target product. Therefore, according to the first embodiment, the number of products ordered by the seller is adjusted based on the calculation result of the simulation performed by the prediction device 1 . This controls the BEI so that it does not exceed the threshold, and the shipments are made to meet demand as much as possible, balancing the conflicting goals of minimizing inventory costs and maximizing sales. Product inventory can be optimized.

The above is the description of the present embodiment, but the configuration and processing of the prediction device 1 of the present embodiment are not limited to the contents of the above description, and are within the scope of the technical idea and identity of the present invention. Various changes are possible in Modifications of the above embodiment will be described below. In the following description, the same reference numerals are given to the same configurations and processes as those described above, and detailed description thereof will be omitted.

(Modification 1)
In Modified Example 1, the number of man-hours required for one day's shipment is determined for a warehouse worker who stocks merchandise according to a sales promotion strategy for one merchandise sold on an EC site. In the simulation executed in this modified example, it is assumed that one time step is one day, the day of the week is set for each day, and the day on which the sales promotion of the product is to be implemented is set in advance. An example of a sales promotion implementation date is a day including 5 (5th, 15th, 25th), fifty days, and the like. Information indicating the date of implementation of the sales promotion may be stored in the product information DB 2 and acquired by the prediction device 1 from the product information DB 2, for example, or may be input by the user by operating the operation unit 13. . FIG. 13 shows an example of rewards for this simulation set by OpenAI Gym in prediction device 1 (FIG. 13A) and an example of update of each state variable executed in OpenAI Gym (FIG. 13B). Note that the action (behavior variable) in this simulation is the man-hours required to ship the product in one day, and a value from 0 to a predetermined upper limit is selected.

As shown in FIG. 13A, in the simulation executed in this modified example, the cumulative absolute value of the difference between the quantity of demand and the quantity of inventory that can be shipped is calculated for the metaplayer ("cost + =abs(b)”). It also determines whether the day of the week at the current step is Saturday or Sunday ("if self.sat_sun_day_flag(y):" in the figure). If it is a Saturday or Sunday, the difference between the required man-hours and the man-hours available for shipping is multiplied by the coefficient for Saturdays and Sundays (Saturday and Sunday operation coefficient) to determine the cost ("cost + = abs(f) * self.manpower_cost_holiday”). On the other hand, if the day of the week is a weekday, the difference between the required man-hours and the man-hours available for shipping is multiplied by the coefficient for weekdays (weekday operation coefficient) to determine the cost ("cost += abs (f) *self.manpower_cost_weekday”). It should be noted that these costs are an example of sales promotion costs determined according to the day of the week on which the sales promotion of the product is carried out. Then, the negative value of the calculated cost is used as a reward. The meta player aims to achieve the goal conditions described later based on the rewards set in this way.

Also, as shown in FIG. 13B, in one time step, first, the date is advanced by one day (“self.step_date += datetime.timedelta(days=1)” in the figure). Next, it is determined whether or not the current date is the date of the sales promotion ("t = self.five_six_day_flag(self.step_date.day)" in the figure). Next, the current day of the week is specified ("y = self.step_date.weekday()" in the figure). Then, the sales promotion date (t), the day of the week (y), and the moving average (dave) are used to determine the demand for the product (“d = self.demand_by_date (t, y, dave)” in the figure). . Next, the number of shipments per hour is determined ("r = self.avg_productivity" in the figure). Then, the man-hours required to ship the products for the demand are calculated ("n = int(d/r)" in the figure). Next, identify the man-time selected by the action ("m = action" in the figure). Then, the number of inventory that can be shipped is calculated according to the man-hours selected in the action ("p = m * r" in the figure). Also, calculate the difference (f) between the man-hours (n) required to ship the number of products required and the man-hours (m) available for shipping ("f = n - m" in the figure). . As can be seen from FIGS. 13A and 13B, the difference (f) is used to calculate the reward. Then, calculate the accumulated difference between the number of demand and the number of available inventory ("b = self.state[B] + (d - p)" in the figure). Each state variable is changed in this way, and the product is shipped based on the generated demand at the next time step using the changed state variable.

FIG. 13C shows an example of goal conditions in simulation in this modified example. As shown in FIG. 13C, the average of the number of shipments to the number of demands for a fixed period (for example, 100 days) from the current time step is calculated (“fillrate=np.mean(shipped/demand)” in the figure). If the calculated average is greater than the above FR threshold ("if fillrate >self.fillrate_threshold:" in the figure), then the calculated average satisfies the Fillrate condition ("fill_flag = True" in the figure). . In addition, the sum of man hours n and m for each day in a fixed period (for example, 100 days) from the current time step is calculated (“ssum=np.sum(history,axis=0)” in the figure). Also, calculate the total number of man-hours n per month required to ship the products for the number of demands at the current time step ("nsum = int(ssum[N]/self.months)" in the figure) . Also, the sum of n man-hours m selected in the action per month is calculated (“msum=int(ssum[M]/self.months)” in the figure). If the total calculated man-hours m per month is less than or equal to the calculated man-hours n per month (“if nsum >= msum:” in the figure), it is assumed that cost minimization has been achieved. Assume that the time condition is satisfied ("sum_flag = True" in the figure). Then, when both of the above two conditions are satisfied, it is considered that the goal condition has been achieved ("done = fill_flag &sum_flag" in the figure).

In this modification, the control unit 11 generates a behavior model by reinforcement learning that calculates the reward for each of a plurality of man-hours, and performs a simulation using the generated behavior model. Specifically, for example, the control unit 11 increases man-hours necessary for shipping from 0 to an upper limit value that can be selected by an action by a predetermined number of man-hours, while changing the man-hours required for shipping in one day (one step) from the lower limit to the upper limit. while repeating the simulation. As an example, the control unit 11 sets the upper limit of the number of time steps in one episode to 1000 steps as described above, and when the metaplayer completes the action of the 1000th time step without achieving the above goal condition, , and based on the results of executing 100 episodes for each man-hour, the average value of 100 episodes is calculated for the number of time steps until the goal condition is achieved for each man-hour and the value of FR. Then, the control unit 11 outputs the calculation result by storing the calculation result in the storage unit 12, displaying the calculation result on the display unit 14, or transmitting the calculation result from the communication unit 15 to an external device.

The user can confirm the calculation result by the prediction device 1 and determine the man hours for shipment on the day of the sales promotion for the simulated product. Therefore, according to the first embodiment, the man-hours assigned to shipping products are adjusted based on the calculation result of the simulation performed by the prediction device 1 . This balances the conflicting goals of maximizing sales and minimizing shipping costs by shipping to meet demand that fluctuates due to sales promotions and keeping man hours low in warehouses where products are stored. , it is possible to optimize the man-hours allocated to shipping the product on the day of the sales promotion set in this simulation.

(Modification 2)
In Modified Example 2, immovable inventory whose stock quantity does not fluctuate is specified for products sold on an EC site, and it is determined whether or not the product should be recommended. In addition, in the simulation executed in this modification, one time step is set to one day, and the user uses an information processing terminal to search for information indicating the product page of the EC site on the Internet, and from the search result page It is assumed that the user moves to the product page of the EC site. In addition, the number of times the information of the product page is displayed in the product information search results is defined as the display count of the product, and the number of times of movement to the product page is defined as the number of clicks. It is also assumed that a size, which is an index of the space occupied by one product in the warehouse, is determined in advance.

14A shows an example of simulation rewards set by OpenAI Gym in prediction device 1, and FIG. 14B shows an example of update of each state variable executed in OpenAI Gym. Note that the action (behavior variable) in this simulation is a threshold value for determining whether product recommendation is necessary. be done. Note that the threshold is an example of a value indicating the ease with which a product is recommended on an EC site.

As shown in FIG. 14A, in the simulation executed in this modified example, the number of clicks (“click_num” in the figure) on the product page and the number of product page Based on the number of views (“view_num” in the figure), the click-through rate of the product page during the period is calculated (“click_through_rate = click_num / view_num” in the figure). Then, the click-through rate multiplied by profit and size is used as a reward ("reward = click_through_rate * profit * size" in the figure). That is, the greater the profit or size, the greater the reward. The meta player aims to achieve the goal conditions described later based on the rewards set in this way.

Also, as shown in FIG. 14B, in one time step, the profit per product (as an example, the amount obtained by subtracting the purchase price and the inventory cost from the selling price) (“profit” in the figure) and the product The number of page clicks (“click_num_one_day” in the figure), the number of times the product page is displayed per day (“view_num_one_day” in the figure), and the number of days the product is stored in the warehouse (“stock_days” in the figure) , the stock quantity ("stock_num" in the figure) and the threshold value ("boost_value" in the figure) selected by the above action are the state variables. Further, in this modified example, information indicating values other than "boost_value" in the drawing is stored in the product information DB2. In addition, it is assumed that each of the above values is stored in the product information DB 2 by day over a certain period of time in the past, such as the past 30 days. The prediction device 1 acquires information on each value over a certain period stored in the product information DB 2, and each time the time step advances by one, specifies each value for the next day based on the acquired information, and determines the specified value. to update each state variable. Each state variable is changed in this manner, and the necessity of product recommendation at the next time step is determined using the changed state variable.

FIG. 14C shows an example of goal conditions in simulation in this modified example. As shown in FIG. 14C , if the clickthrough rate in the above constant period is greater than a preset threshold (“if click_through_rate >= click_through_rate _threshold:” in the figure), the clickthrough rate condition is satisfied ( "click_through_rate_flag
= True"). Also, if the stock days average "stock_days" of the product for a certain period (e.g. 100 days) from the current time step is greater than the preset threshold ("if stock_days >= stock_days_threshold:" in the figure), Assume that the conditions for the average number of days in stock are satisfied ("stock_days_flag = True" in the figure). Then, when both of the above two conditions are satisfied, the goal condition is considered to have been achieved ("done = click_through_rate_flag
&stock_days_flag").

In this modification, the control unit 11 provides a remuneration determined based on the acquired product information, product size, product sales, product purchase price, and inventory cost determined according to product size. is calculated for each of a plurality of values indicating the ease of recommending a product, a behavior model is generated by reinforcement learning, and a simulation is performed using the generated behavior model. Specifically, the control unit 11 repeats the simulation while changing the threshold for determining the necessity of product recommendation from the lower limit to the upper limit, increasing the threshold by 0.1 from 0.0 to 1.0, for example. Here, the control unit 11 sets the number of time steps for one episode as the number of days in the target period of the information of each value obtained from the product information DB 2, and based on the result of executing 100 episodes for each threshold value, each Calculate the average value of 100 episodes for the number of time steps to achieve the goal condition for the threshold, the click-through rate, and the average number of days in stock. For example, when the information of each of the above values over the past 30 days is acquired from the product information DB2, the number of time steps for one episode is 30. The control unit 11 outputs the calculation result by storing the calculation result in the storage unit 12, displaying the calculation result on the display unit 14, or transmitting the calculation result from the communication unit 15 to an external device.

The user can confirm the calculation result by the prediction device 1 and decide whether or not to recommend the product targeted for the simulation. Therefore, according to Modification 1, the necessity of product recommendation on the EC site is determined based on the calculation result of the simulation by the prediction device 1 . As a result, on the EC site, products whose number of days in stock exceeds the threshold are recommended, that is, displayed on the user's information processing terminal. Then, if the user does not click on the displayed product, or if the user does not operate the recommended product, the product is not recommended. As a result, products that do not sell because they are not recommended and products that do not sell even if they are recommended are determined, and products that are more likely to sell are recommended, thereby optimizing product recommendations on EC sites. can be achieved.

(Modification 3)
In Modified Example 3, the incentive given to the user is determined by changing the designated delivery date when the user orders the product for sale on the EC site. Here, the incentive is a point or the like that can be used on the EC site that is returned to the user for ordering a product. Since incentives are well known, detailed description thereof will be omitted. Note that the action (behavior variable) in this modified example is an incentive, and any value is selected from 0.0 to 1.0 in increments of 0.1, for example. Also, in the simulation executed in this modified example, it is assumed that one time step is one day, and that the user can change the designated delivery date when ordering the product on the EC site.

15A shows an example of simulation rewards set by OpenAI Gym in prediction device 1, and FIG. 15B shows an example of update of each state variable executed in OpenAI Gym.

As shown in FIG. 15A, in the simulation executed in this modified example, the reward is the value of man-hours required to deliver the simulation target product (“reward=click_through_rate*profit*size” in the figure). That is, the greater the profit or size, the greater the reward. The meta player aims to achieve the goal conditions based on the rewards set in this way.

In addition, as shown in FIG. 15B, in one time step, product order information (order ID, latitude and longitude of delivery destination, specified delivery date and delivery time zone that is currently specified delivery date and time, delivery unit number) ("order_info: (order_id, latitude, longitude,
date, time, parcel_num)"), spatio-temporal cluster information for product delivery (current delivery date and delivery time zone, latitude and longitude of the central position of the cluster that delivers the product, radius of the cluster, cluster Number of delivery destinations that can be delivered, number of delivery destinations currently scheduled within the cluster) ("time_space_cluster_info: [(date,time,latitude,longitude,radius,max,num), ….]" in the figure) ), total man-hours required for delivery (“man_hour” in the figure), total number of delivery workers required for delivery (“deivers_num” in the figure), total number of clusters (“cluster_num” in the figure), and time required for delivery. A certain total waiting time ("idle_time" in the figure), total delivery distance ("distance" in the figure), the difference between the specified delivery date and time and the optimized date and time ("difference" in the figure), and when the specified delivery date is changed The incentive given to the user (as an example, the number of points. In the figure, "incentive: basic_point * action (0.0 to 1.0)") is a state variable. In addition, in this modification, information indicating values other than "incentive: basic_point*action (0.0 to 1.0)" in the drawing is stored in the product information DB2. A cluster is an example of a delivery area of a delivery company. The prediction device 1 acquires information on each value over a certain period stored in the product information DB 2, and each time the time step advances by one, specifies each value for the next day based on the acquired information, and determines the specified value. to update each state variable. Each state variable is changed in this manner, and the changed state variables are used to calculate the total man-hours required for delivery in the next time step.

FIG. 15C shows an example of goal conditions in simulation in this modified example. As shown in FIG. 15C, the total man-hours required for delivery (“man_hour_sum” in the figure) and the maximum man-hours allocatable to the delivery (“ max_man_hour_sum”). Then, based on the calculated man-hours, the operating rate of the delivery staff in the relevant period is calculated (“man_hour_rate=man_hour/max_man_hour_sum” in the figure). If the calculated operating rate of the delivery staff is equal to or less than a preset threshold ("if man_hour_rate <= man_hour_rate_threshold:" in the figure), it is assumed that the conditions for the operating rate of the delivery staff are satisfied ("man_hour_rate_flag = True"). Also, if the value of the incentive ("incentive") for the period is less than the preset threshold ("if incentive <= incentive_threshold:" in the figure), the incentive conditions are assumed to be met (
"incentive_flag = True" in the figure). And when both said two conditions are satisfy|filled, it considers that goal conditions were achieved ("done=man_hour_rate_flag&incentive_flag" in the figure).

In this modification, the control unit 11 generates a behavior model through reinforcement learning that calculates rewards determined based on total man-hours required for delivery calculated from acquired product information for each of a plurality of incentives. , a simulation is performed using the generated behavioral model. Specifically, the control unit 11 repeats the simulation while changing the value indicating the incentive ratio from the lower limit to the upper limit, increasing the value by 0.1 from 0.0 to 1.0, for example. Here, the control unit 11 sets the number of time steps in one episode to the number of days in the target period of the information of each value obtained from the product information DB 2, and based on the result of executing 100 episodes for each incentive ratio , the number of time steps to achieve the goal condition for each incentive ratio, and the operating rate of the delivery staff, the average value of 100 episodes is calculated. For example, when the information of each of the above values over the past 30 days is acquired from the product information DB2, the number of time steps for one episode is 30. The control unit 11 outputs the calculation result by storing the calculation result in the storage unit 12, displaying the calculation result on the display unit 14, or transmitting the calculation result from the communication unit 15 to an external device.

The user confirms the calculation result by the prediction device 1, and when the user orders the simulated product on the EC site, changes the specified delivery date of the product, thereby changing the incentive given to the user. size can be determined. Therefore, according to Modification 1, the magnitude of the incentive is adjusted at the EC site based on the calculation result of the simulation performed by the prediction device 1 . By doing so, we aim to balance the two goals of minimizing delivery costs and minimizing incentive costs, while optimizing the size of incentives given when prompting users to change the delivery date of the relevant product on the e-commerce site. be able to.

1 prediction device 11 control unit 2 product information DB

Claims (4)

an acquisition unit that acquires product information of products for electronic commerce;
A first behavioral model and a first behavioral model for calculating mutually different rewards using the obtained product information and the state variables of the environment for a plurality of values of behavioral variables allowed in the environment related to the distribution of the product; a control unit that generates a third behavior model based on reinforcement learning of the behavior model of 2, and uses the third behavior model to predict an optimal behavior variable value in the environment;
the behavioral variable is the number of orders for the product by the seller of the product;
The product information acquired by the acquisition unit is information indicating changes in the number of orders, the number of shipments, and the number of arrivals of the product over a certain period of time,
The state variables of the environment include sales of the item, stock prices of the item, stockout costs associated with stockouts of the item, promotional costs associated with promoting the item, and excess inventory of the item. inventory costs associated with and shipping costs associated with delivering said goods;
The control unit includes the first behavior model that calculates different rewards using the acquired product information and the environmental state variables for each order quantity corresponding to a plurality of values of the behavior variables; generating the third behavior model based on reinforcement learning of the second behavior model, and predicting the optimal order quantity using the third behavior model;
A prediction device characterized by : an acquisition unit that acquires product information of products for electronic commerce;
A first behavioral model and a first behavioral model for calculating mutually different rewards using the obtained product information and the state variables of the environment for a plurality of values of behavioral variables allowed in the environment related to the distribution of the product; a control unit that generates a third behavior model based on reinforcement learning of the behavior model of 2, and uses the third behavior model to predict the values of the optimal behavior variables in the environment;
has
wherein the behavioral variable is man-hours associated with storing the goods in a warehouse;
The product information acquired by the acquisition unit is information indicating the implementation date of the sales promotion of the product,
The state variable of the environment is a sales promotion cost determined according to the day of the week when the sales promotion of the product is implemented,
The control unit includes the first behavior model that calculates different rewards using the acquired product information and the environmental state variables for each person-time corresponding to the plurality of values of the behavior variables; A prediction device, wherein the third behavior model is generated based on reinforcement learning of the second behavior model, and the optimum human time is predicted using the third behavior model. an acquisition unit that acquires product information of products for electronic commerce;
A first behavioral model and a first behavioral model for calculating mutually different rewards using the obtained product information and the state variables of the environment for a plurality of values of behavioral variables allowed in the environment related to the distribution of the product; a control unit that generates a third behavior model based on reinforcement learning of the behavior model of 2, and uses the third behavior model to predict the values of the optimal behavior variables in the environment;
has
The behavioral variable is a value that indicates the ease with which the product is recommended on an EC site,
The product information acquired by the acquisition unit is information indicating the number of clicks and the number of displays of the product over a certain period of time on the EC site,
the state variables of the environment are the size of the product, the sales of the product, the purchase price of the product, and an inventory cost determined according to the size of the product;
The control unit uses the obtained product information and the environment state variables to provide different rewards for each of the values indicating the ease of recommendation of the product corresponding to the plurality of values of the behavior variables. The third behavior model is generated based on the reinforcement learning of the calculated first behavior model and the second behavior model, and the ease with which the product is recommended is indicated using the third behavior model. A prediction device characterized by predicting an optimum value. an acquisition unit that acquires product information of products for electronic commerce;
A first behavioral model and a first behavioral model for calculating mutually different rewards using the obtained product information and the state variables of the environment for a plurality of values of behavioral variables allowed in the environment related to the distribution of the product; a control unit that generates a third behavior model based on reinforcement learning of the behavior model of 2, and uses the third behavior model to predict the values of the optimal behavior variables in the environment;
has
The action variable is an incentive given to the user by changing the designated delivery date of the product on the EC site,
The product information acquired by the acquisition unit includes the delivery destination and delivery date and time of the product, the delivery area of the delivery company of the product, the number of deliverable delivery destinations, the time required for delivery of the product, and the delivery time of the product. is information indicating the delivery distance in the delivery of
The control unit includes the first behavioral model and the A prediction device that generates the third behavior model based on reinforcement learning of the second behavior model and predicts an optimal incentive using the third behavior model.

Images