JP7663030B2 - Delivery plan creation device and delivery plan creation system - Google Patents

Description

This invention relates to a delivery plan creation device and a delivery plan creation system, and in particular to a delivery plan creation device and a delivery plan creation system that create a delivery plan that shows product delivery routes to multiple vending machines.

Conventionally, vending machine management systems are known (see, for example, Patent Document 1).

The above-mentioned Patent Document 1 discloses a vending machine management system for vending machines that sell products in response to electronic money. In this vending machine management system, electronic money data is collected from the vending machines using a terminal carried by a route man who visits multiple vending machines. In addition, in conventional vending machines such as those described in the above-mentioned Patent Document 1, an operator (route man) performs tasks such as replenishing products and collecting money.

JP 2009-301402 A

In vending machines, products may sell out, resulting in lost sales opportunities and a decline in sales. For this reason, workers (routemen) who deliver (refill) products to vending machines by visiting multiple vending machines as needed or traveling along a pre-defined delivery route are tasked with replenishing products. However, there is a problem in that the patrol route and the pre-defined delivery route are not necessarily efficient delivery routes that can prevent an increase in delivery time during actual delivery while also preventing a decline in sales of vending machine products caused by products being sold out.

This invention has been made to solve the above problems, and one objective of the invention is to provide a delivery plan creation device and delivery plan creation system that can create efficient delivery routes that suppress increases in delivery time while suppressing declines in sales of products in vending machines due to sold-out items.

In order to achieve the above object, a delivery plan creation device according to a first aspect of the present invention includes a delivery plan creation unit that creates a delivery plan indicating a delivery route for products to multiple vending machines by machine learning, and an output unit that outputs the delivery plan created by the delivery plan creation unit to a mobile device carried by an operator who performs work to replenish products in the vending machines via a network. The delivery plan is configured to create a delivery plan using a trained model that represents a value obtained by the delivery plan as an action for the forecasted demand and inventory information as a state of the delivery plan creation unit, based on a forecasted demand that is a forecast value of sales of products from the multiple vending machines and inventory information of products for each of the multiple vending machines. The delivery plan creation device is configured to update the trained model of the delivery plan creation unit by performing reinforcement learning of the trained model regarding the value of the delivery plan represented by the trained model of the delivery plan creation unit using delivery performance information including sales of products from the multiple vending machines , the total travel time required to deliver the products, and inventory information of products for each of the multiple vending machines when the delivery plan is actually implemented, and the output unit is configured to output the delivery plan created by the trained model of the delivery plan creation unit after the update to the mobile device via the network.

In the delivery plan creation device according to the first aspect, as described above, a delivery plan is created by the trained model of the delivery plan creation unit based on the predicted demand, which is a predicted value of the sales of products from multiple vending machines, and the inventory information of each product from the multiple vending machines. This makes it possible to create a delivery plan that can suppress a decrease in sales of products due to sellouts. In addition, the trained model of the delivery plan creation unit is updated by performing machine learning using the trained model of the delivery plan creation unit based on delivery performance information when the delivery plan is actually implemented. This makes it possible to update the trained model of the delivery plan creation unit to a trained model that reflects delivery performance information including the sales of products from multiple vending machines and the total travel time required to deliver the products when the delivery plan is actually implemented. As a result, the delivery plan creation unit can create an efficient delivery plan that reflects delivery performance information including the sales of products from multiple vending machines and the total travel time required to deliver the products when the delivery plan is actually implemented at the time of delivery after the trained model is updated. As a result, it is possible to create an efficient delivery route that suppresses an increase in delivery time while suppressing a decrease in sales of products from vending machines due to sellouts. In addition, the output unit outputs the delivery plan created by the updated trained model of the delivery plan creation unit to the mobile device via the network, so the worker can check, on the mobile device, an efficient delivery plan that reflects the delivery performance information and suppresses an increase in delivery time while suppressing a decrease in sales of vending machine products due to sold-outs. As a result, the worker can deliver products efficiently, reflecting the delivery performance information and suppressing an increase in delivery time while suppressing a decrease in sales of vending machine products due to sold-outs.

In the delivery plan creation device according to the first aspect, preferably, the trained model of the delivery plan creation unit includes a state-action value function that represents the value obtained by the delivery plan as an action for the predicted demand and inventory information as a state, and the delivery plan creation unit is configured to create a delivery plan based on the state-action value function, and is configured to update the state-action value function of the trained model by reinforcement learning using the trained model of the delivery plan creation unit based on delivery performance information. With this configuration, a delivery plan that has a short delivery time and high sales of products from vending machines, in which the delivery performance information is reflected, can be created by the state-action value function updated based on the delivery performance information. As a result, the delivery plan creation unit can create an efficient delivery route that can suppress an increase in delivery time when delivering products to multiple vending machines, while suppressing a decrease in sales of products from vending machines due to products being sold out.

In the delivery plan creation device according to the first aspect, the trained model of the delivery plan creation unit is preferably configured to update the trained model of the delivery plan creation unit by machine learning using the trained model of the delivery plan creation unit based on delivery performance information after delivery of products to the multiple vending machines is completed. With this configuration, the trained model of the delivery plan creation unit can be updated to reflect the delivery performance information from the start of delivery of products to the multiple vending machines to the completion of delivery. As a result, the delivery plan creation unit after updating the trained model can create a more efficient delivery plan (delivery route) that can further suppress an increase in delivery time when delivering products to the multiple vending machines and further suppress a decrease in sales of products in the vending machines due to products being sold out.

In the delivery plan creation device according to the first aspect, delivery record information including the location information of the worker when the worker delivers the product is preferably acquired, and the trained model of the delivery plan creation unit is configured to be updated by machine learning using the trained model based on the delivery record information including the acquired location information. With this configuration, the travel time between multiple vending machines when the worker delivers the product can be acquired from the delivery record information including the location information of the worker. As a result, if the travel time between multiple vending machines in the actual delivery of the product is longer than the travel time assumed when the delivery plan was created, the trained model of the delivery plan creation unit can be updated based on the travel time between multiple vending machines acquired from the delivery record information including the location information of the worker. This makes it possible to create a more optimal delivery route by changing the route between multiple vending machines or the order of the vending machines to be delivered, and therefore to create a more efficient delivery plan.

In the delivery plan creation device according to the first aspect, the delivery plan creation unit is preferably configured to use predicted demand, which is a predicted value of product sales of multiple vending machines over multiple days, when creating the delivery plan. With this configuration, it is possible to create a delivery route that has a short total travel time or total travel distance required to deliver products over multiple days. As a result, the delivery plan creation unit can create an efficient delivery route for products to multiple vending machines when delivering over multiple days.

In the delivery plan creation device according to the first aspect, preferably, when the worker delivers the goods, delivery performance information when the delivery plan is actually implemented is acquired, and the delivery plan creation unit is configured to output a delivery plan using delivery performance information including inventory information acquired when the worker delivers the goods and forecast demand, which is a forecast value of sales of the goods of the multiple vending machines, as input, and is configured to correct the delivery plan based on the acquired delivery performance information without updating the trained model of the delivery plan creation unit. With this configuration, the delivery plan creation unit can correct the delivery plan based on actual inventory information before updating the trained model. As a result, the delivery plan can be revised to a more efficient delivery plan by reflecting actual inventory information before updating the trained model. In addition, by correcting the delivery plan without updating the trained model of the delivery plan creation unit, it is easier to revise the delivery plan compared to the case where the trained model of the delivery plan creation unit is updated from time to time.

In the delivery plan creation device according to the first aspect, preferably, when a worker delivers goods, delivery performance information when the delivery plan is actually implemented is acquired via a network, and the delivery plan is modified while the worker is delivering the goods based on the acquired delivery performance information without updating the trained model of the delivery plan creation unit, and the modified delivery plan is transmitted via a network to each mobile device of the multiple workers. With this configuration, while the worker is delivering the goods, a delivery plan modified based on the acquired delivery performance information is transmitted to each mobile device of the multiple workers, so that the worker can deliver the goods efficiently while the goods are being delivered, reflecting the situation during delivery in real time. In addition, by modifying the delivery plan without updating the trained model of the delivery plan creation unit, the delivery plan can be modified in real time while the goods are being delivered more quickly than when the trained model of the delivery plan creation unit is updated from time to time.

In the delivery plan creation device according to the first aspect, preferably, when a plurality of workers deliver goods, the product transport volume, which is the number of goods that each of the plurality of workers can deliver, is acquired via a network, and the delivery plan is modified during the delivery of goods by the workers based on the acquired product transport volume of each of the plurality of workers without updating the trained model of the delivery plan creation unit, and the modified delivery plan is transmitted via a network to each mobile terminal of the plurality of workers. With this configuration, the delivery plan can be modified based on the acquired product transport volume of each of the plurality of workers while the workers are delivering goods, so that even if a certain worker's product transport volume is insufficient, the delivery route can be modified so that another worker who has a surplus in product transport volume replenishes the goods instead. Then, since the modified delivery plan is transmitted to each mobile terminal of the plurality of workers, the workers can deliver goods efficiently reflecting the product transport volume of each worker. In addition, by modifying the delivery plan without updating the trained model of the delivery plan creation unit, the delivery plan based on the acquired product transport volume of each of the plurality of workers can be modified during the delivery of goods, compared to the case where the trained model of the delivery plan creation unit is updated from time to time.

In the delivery plan creation device according to the first aspect, preferably, when creating the trained model of the delivery plan creation unit, the delivery plan creation unit creates a tentative delivery plan based on at least one of the predicted values or past actual values of sales of products of the multiple vending machines and at least one of the predicted values or past actual values of the inventory quantity of each product of the multiple vending machines, obtains predicted delivery performance information including product sales in the simulation result of product delivery to the multiple vending machines based on the tentative delivery plan and the total travel time required for product delivery in the simulation result of product delivery to the multiple vending machines, and performs reinforcement learning based on the predicted delivery performance information as machine learning in the delivery plan creation unit to create the trained model. With this configuration, learning by reinforcement learning can be performed using the predicted delivery performance information obtained by the simulation result based on the tentative delivery plan before actually delivering the products. As a result, a trained model of the delivery plan creation unit that can create a more efficient delivery plan before actually delivering the products can be created compared to the case where reinforcement learning is not performed using the predicted delivery performance information obtained by the simulation result based on the tentative delivery plan.

In order to achieve the above object, a delivery plan creation system according to a second aspect of the present invention includes a mobile terminal carried by an operator who replenishes products in vending machines, and a delivery plan creation device that creates a delivery plan indicating a delivery route for products to a plurality of vending machines, the delivery plan creation device including a delivery plan creation unit that creates a delivery plan by machine learning, and an output unit that outputs the delivery plan created by the delivery plan creation unit to the mobile terminal via a network, and based on a predicted demand that is a predicted value of sales of products in the plurality of vending machines and inventory information of each of the plurality of vending machines, a running forecast of the delivery plan creation unit's status with respect to the predicted demand and inventory information. The delivery plan is configured to create a delivery plan using a trained model that represents the value obtained by the delivery plan as an action , and is configured to update the trained model of the delivery plan creation unit by performing reinforcement learning of the trained model regarding the value of the delivery plan represented by the trained model of the delivery plan creation unit using delivery performance information including sales of products from multiple vending machines , total travel time required to deliver the products, and inventory information of each product in the multiple vending machines when the delivery plan is actually implemented, and the output unit is configured to output the delivery plan created by the updated trained model of the delivery plan creation unit to a mobile terminal via a network.

In the delivery plan creation system according to the second aspect, as described above, the delivery plan creation device creates a delivery plan using the trained model of the delivery plan creation unit based on the predicted demand, which is a predicted value of the sales of products from multiple vending machines, and the inventory information of each product from the multiple vending machines. This makes it possible to create a delivery plan that can suppress a decrease in sales of products due to sellouts. In addition, the trained model of the delivery plan creation unit is updated by performing machine learning using the trained model of the delivery plan creation unit based on delivery performance information when the delivery plan is actually implemented. This makes it possible to update the trained model of the delivery plan creation unit to a trained model that reflects delivery performance information including the sales of products from multiple vending machines and the total travel time required to deliver the products when the delivery plan is actually implemented. As a result, the delivery plan creation unit can create an efficient delivery plan that reflects delivery performance information including the sales of products from multiple vending machines and the total travel time required to deliver the products when the delivery plan is actually implemented, at the time of delivery after the trained model is updated. As a result, it is possible to create an efficient delivery route that suppresses an increase in delivery time while suppressing a decrease in sales of products from vending machines due to sellouts. In addition, the output unit outputs the delivery plan created by the updated trained model of the delivery plan creation unit to the mobile device via the network, so the worker can check, on the mobile device, an efficient delivery plan that reflects the delivery performance information and suppresses an increase in delivery time while suppressing a decrease in sales of vending machine products due to sold-outs. As a result, the worker can deliver products efficiently, reflecting the delivery performance information and suppressing an increase in delivery time while suppressing a decrease in sales of vending machine products due to sold-outs.

As described above, the present invention makes it possible to create an efficient delivery route that prevents increases in delivery time while also preventing declines in sales of products from vending machines due to sold-out items.

FIG. 1 is a diagram illustrating an example of a delivery plan creation system according to an embodiment of the present invention. FIG. 2 is a functional block diagram of a delivery plan creation device according to an embodiment of the present invention. FIG. 13 is a diagram showing data input and output when creating a trained model using simulation results. FIG. 13 is a diagram showing an example of creating a tentative delivery plan. FIG. 2 illustrates the generation of a Q function. FIG. 13 is a diagram showing an example of a simulation result. FIG. 11 is a diagram showing another example of a simulation result. FIG. 13 shows an example of simulation results of a tentative distribution plan created by a trained model after reinforcement learning using the simulation results. FIG. 1 is a first diagram showing an example of a simulation result based on demand information for a plurality of days. FIG. 2 is a second diagram showing an example of a simulation result based on demand information for a plurality of days. FIG. 13 is a diagram showing an example of creating a delivery plan. FIG. 1 is a first diagram showing an example of a correction of a delivery plan during delivery of goods. FIG. 2 is a second diagram showing an example of a correction of a delivery plan during delivery of goods. FIG. 1 is a first diagram showing another example of correction of a delivery plan during delivery of goods. FIG. 2 is a second diagram showing another example of correction of a delivery plan during delivery of goods. This figure shows the input and output of data when updating a trained model after delivery is completed. FIG. 13 is a diagram showing an update of the Q function after delivery is completed. FIG. 1 is a diagram showing an example of a change in a delivery plan due to an update of a trained model after delivery is completed. FIG. 2 shows an example of a change in the delivery plan due to an update of the trained model after delivery is completed.

The following describes an embodiment of the present invention with reference to the drawings.

(Overall configuration of the delivery plan creation system)
The overall configuration of a delivery plan creation system 100 according to this embodiment will be described with reference to FIG.

As shown in FIG. 1, the delivery plan creation system 100 includes a delivery plan creation device 101 that creates a delivery plan showing product delivery routes to multiple vending machines V, and a mobile terminal 102 carried by a worker M (route man) who refills the vending machines V with products.

The delivery plan creation device 101 notifies (displays) the delivery route on a mobile terminal 102 carried (held) by a worker M (routeman) via a network 200. In addition, delivery performance information, which will be described later, is periodically transmitted to the delivery plan creation device 101. The transmission of the delivery performance information may be performed by an operation (input) by the worker M, or may be performed automatically by the mobile terminal 102. The delivery plan creation device 101 then modifies the delivery plan to reflect the received delivery performance information, and notifies the mobile terminal 102 of the worker M.

The delivery plan creation device 101 is a server connected to be able to communicate with the mobile terminal 102 and the vending machine V via the network 200. The delivery plan creation device 101 transmits and receives data to and from the mobile terminal 102 via the network 200. The delivery plan creation device 101 acquires delivery performance information (described below) via the network 200. Then, the delivery plan creation device 101 transmits the created delivery plan to the mobile terminal 102 via the network 200.

The delivery plan creation device 101 also transmits and receives data to and from the vending machine V via the network 200. For example, the delivery plan creation device 101 acquires data such as sales (number of items sold and sales amount) and inventory of products of the vending machine V via the network 200. The delivery plan creation system 100 may also be configured to notify the delivery plan creation device 101 via the network 200 that refilling or maintenance work is being performed on the vending machine V, and provide the delivery plan creation device 101 with location information of the worker M based on the state of the vending machine V. In addition, while FIG. 1 shows an example in which data from the vending machine V is transmitted to the delivery plan creation device 101 via the network 200, data from the vending machine V may be transmitted to the delivery plan creation device 101 via a management server (vending machine management server) for the vending machine V. Data from the vending machine V may also be transmitted to the delivery plan creation device 101 via the mobile terminal 102.

The mobile terminal 102 is a terminal that can be connected to the network 200. For example, the mobile terminal 102 is a terminal such as a smartphone or a tablet PC. An application is installed in the mobile terminal 102 for performing data communication with the delivery plan creation device 101 and notifying (displaying) the delivery plan. The mobile terminal 102 is also configured to be able to communicate with the vending machine V, and is capable of transmitting and receiving data to and from the vending machine V.

Then, the worker M delivers and replenishes products to the vending machine V according to the delivery plan (delivery route) created by the delivery plan creation device 101 and notified to the mobile terminal 102. The delivery plan created by the delivery plan creation device 101 (delivery plan creation system 100) includes the delivery routes of each of the multiple workers M in charge of delivery (replenishment) work for the multiple vending machines V managed by the user who uses the delivery plan creation device 101. The delivery plan created by the delivery plan creation device 101 (delivery plan creation system 100) may also include information on the type and quantity of products that each of the multiple workers M delivers (replenishes) to the vending machine V (type and quantity of products to be transported from the delivery base).

(Configuration of the delivery plan creation device)
As shown in FIG. 2, the delivery plan creation device 101 includes a database unit 1, a processing unit 2, an input unit 3, and an output unit 4.

The database unit 1 stores data (such as inter-base movement information, initial inventory information, delivery plans, forecast demand, location information, demand information, inventory information, total travel time, sales, and product transport volume) used by the delivery plan creation unit 23 described below to create a delivery plan. The database unit 1 includes a storage device (internal storage) such as a hard disk drive (HDD) or solid state drive (SSD).

The processing unit 2 includes a calculation unit 21, a simulator 22, a delivery plan creation unit 23, and a prediction unit 24. In this embodiment, the calculation unit 21, the simulator 22, the delivery plan creation unit 23, and the prediction unit 24 are configured as functional blocks within the delivery plan creation device 101, but each may be an independent device. The processing unit 2 (calculation unit 21) includes a processor such as a CPU (Central Processing Unit), a memory, and the like.

The calculation unit 21 is configured to process the functions and data of the database unit 1, the simulator 22, the delivery plan creation unit 23, and the prediction unit 24. The simulator 22 is configured to perform a simulation of the delivery of products to multiple vending machines V according to a tentative delivery plan described below.

The delivery plan creation unit 23 is also configured to create a delivery plan showing product delivery routes to multiple vending machines V through machine learning.

In addition, in this embodiment, the delivery plan creation unit 23 performs reinforcement learning (offline learning) using the simulation results, and reinforcement learning (online learning) using delivery performance information (total travel time, sales, product transport volume, location information, and inventory information) obtained when actually carrying out delivery.

The trained model of the delivery plan creation unit 23 also includes a state-action value function (Q function) that represents the value obtained by a delivery plan as an action with respect to forecasted demand and inventory information as a state, as described below. The delivery plan creation unit 23 is configured to create a delivery plan based on the state-action value function (Q function).

The prediction unit 24 is configured to predict the demand for products sold by the vending machine V (prediction of sales volume). The prediction unit 24 is also configured to predict the future inventory volume of products in the vending machine V (changes in the inventory volume of products in the vending machine V). The sales volume and inventory volume predictions made by the prediction unit 24 are calculated using existing prediction techniques such as neural networks.

The delivery plan creation device 101 is also configured to create a delivery plan using the learned model of the delivery plan creation unit 23 based on the predicted demand, which is the predicted sales value of products from multiple vending machines V, and the inventory information of each product from multiple vending machines V.

The delivery plan creation device 101 is also configured to update the trained model of the delivery plan creation unit 23 by performing machine learning using the trained model of the delivery plan creation unit 23 based on delivery performance information when the delivery plan is actually implemented, as described below. The delivery performance information includes the sales of products from multiple vending machines V and the total travel time required to deliver the products. In this embodiment, the delivery performance information includes the total travel time required to deliver the products, the sales of products from multiple vending machines V (number of sales), the amount of products transported, location information of the worker M, and inventory information of products from multiple vending machines V (see FIG. 16). The delivery performance information may include the sales amount in addition to the number of sales as the sales of products from multiple vending machines V. The delivery performance information is information (data) obtained from the mobile terminal 102 and the vending machine V.

The input unit 3 is configured to input delivery record information (receive delivery record information) via the network 200.

The output unit 4 of the delivery plan creation device 101 is configured to output the delivery plan created by the delivery plan creation unit 23 to the mobile terminal 102 via the network 200.

The output unit 4 is also configured to output the delivery plan created by the trained model of the delivery plan creation unit 23 after the update (machine learning using delivery performance information has been performed) to the mobile terminal 102 via the network 200.

(Machine learning using simulation results)
Next, machine learning using simulation results for creating a trained model used in creating a delivery plan (delivery plan calculation) will be described with reference to Figures 3 to 10. Machine learning using simulation results is performed before actual delivery is performed by the worker M (before the delivery plan creation system 100 is put into actual operation).

When creating a trained model for the delivery plan creation unit 23, the delivery plan creation device 101 creates a provisional delivery plan based on the predicted sales values of products in multiple vending machines V and the predicted inventory values of each product in multiple vending machines V (see FIG. 3) using the delivery plan creation unit 23.

In this embodiment, the delivery plan creation unit 23 creates a provisional delivery plan using the predicted sales values of the products of the multiple vending machines V over multiple days predicted by the prediction unit 24 and the predicted stock quantity of each product of the multiple vending machines V over multiple days predicted by the prediction unit 24. Note that the period (range of multiple days) used when creating the provisional delivery plan is the same for the predicted sales values of the products of the multiple vending machines V and the predicted stock quantity of each product of the multiple vending machines V.

As shown in FIG. 3, the delivery plan creation device 101 performs a simulation based on the tentative delivery plan. In addition, in machine learning using the simulation results, a delivery plan (tenant delivery plan) to be used in the simulation is created (calculated) from demand information and inventory information.

The delivery plan creation device 101 acquires predicted delivery performance information including product sales in the simulation results of product delivery to multiple vending machines V based on a tentative delivery plan, and the total travel time required to deliver the products in the simulation results of product delivery to multiple vending machines V. In this embodiment, the predicted delivery performance information includes the total travel time required to deliver the products, the product sales (number of units sold) of the multiple vending machines V, the product transport volume, location information of the worker M, and product inventory information of the multiple vending machines V (see FIG. 3). Furthermore, the predicted delivery performance information may include the sales amount in addition to the number of units sold as the sales of the products of the multiple vending machines V.

Then, through a simulation by the simulator 22, inventory information, total travel time, sales volume of products, the amount of products (product transport volume) being transported by the worker M (routeman), and the location information of the worker M are calculated for each time.

In machine learning using the simulation results, learning is performed based on demand information, inventory information (initial inventory information), a tentative delivery plan, and predicted delivery performance information (total travel time required for product delivery, sales of products from multiple vending machines V, product transport volume, location information of worker M, and product inventory information from multiple vending machines V). In machine learning using the simulation results, machine learning is repeated multiple times for a certain continuous period of time using simulator 22. Note that the demand information can use past demand performance and predicted demand calculated using existing prediction technology such as neural networks.

The delivery plan creation device 101 creates a trained model by reinforcement learning. The delivery plan creation device 101 creates a trained model by performing reinforcement learning based on predicted delivery performance information as machine learning in the delivery plan creation unit 23.

As shown in FIG. 4, demand information, inventory information, product transportation volume, and location information of worker M are used to calculate the delivery plan (provisional delivery plan). At the beginning of a continuous simulation period, a preset inventory quantity (initial inventory information) is used as inventory information, and in subsequent simulations, inventory information output by simulator 22 is used.

In addition, the demand information uses predicted values or past actual values, and demand information for future periods is used. The delivery destination uses the learned results. Depending on the calculation results, there are some delivery destinations (vending machines V) to which delivery is not made, and there are also cases where delivery is not made at all. If the delivery plan calculated by the delivery plan creation unit 23 is A, the delivery plan (A) is planned according to the following formula (1).
A=π(S)...(1)
π is a function (policy function) that calculates a delivery plan from the state (S) and represents calculations using a neural network or optimization calculations. The state (S) represents demand information, inventory information, location information of each worker M, and the amount of goods transported.

In a simulation in simulator 22, for example, the number of products at the delivery destinations indicated in the delivery plan is replenished to the maximum value. Then, based on the demand information, inventory information and sales for the next time are calculated. If the demand for products is greater than the stock quantity (number of products) in vending machine V, the existing stock quantity (number of products) becomes sales, and inventory becomes 0. Also, if the stock quantity (number of products) is greater than demand, the stock quantity (number of products) minus the demand quantity becomes inventory for the next time, and sales become equal to the demand quantity.

The total travel time is the sum of the travel times of each worker M (multiple workers M). The travel time of each worker M is calculated based on the inter-base travel time included in the inter-base travel information. The inter-base travel time is set (input) as the travel time between each delivery destination (between each vending machine V) and the travel time between the delivery base and the delivery destination (vending machine V).

As shown in Fig. 5, the delivery plan creation unit 23 of the delivery plan creation device 101 performs learning using the simulation results. Learning using the simulation results is performed by an existing reinforcement learning method. The reward r required for reinforcement learning is calculated, for example, by adding up the total travel time (R1) and sales (R2) with a weighting (α) as shown in the following formula (2).
r=α×R1+R2...(2)
The remuneration r may be learned by taking into account the total delivery completion time (the maximum total travel time of each routeman), the sales amount of the products, and the profit from the sales (profit margin), etc.

In this embodiment, the Deep Q Network is used as the reinforcement learning method. The state-action value function (Q function: Q(S, A)) for the delivery plan (A), which is an action according to the state (S) of the inventory information and demand information, is learned by the neural network.

Note that other existing reinforcement learning methods may be used for machine learning. For example, DoubleDQN, A3C, Actor-Critic, policy gradient method, Rainbow, Q-learning, SARSA, Ape-X, R2D2, NGU, Agent57, SAC, TD3, DDPG, TRPO, PPO, etc. may be used.

As shown in FIG. 5, when the Q function is learned, in calculating the delivery plan (A), for example, a delivery plan (A ^* ) that optimizes the Q function (Q function becomes a minimum value) is calculated by the following formula (3).
A ^* = π (S) = argmin _A Q (S, A)... (3)
The delivery plan created by the trained model learned using the simulation results includes, in addition to the optimal delivery plan (A ^* ), a non-optimal delivery plan (e.g., a random delivery plan) with probability λ, and a better delivery plan (smaller minimum value) is searched for.

Alternatively, the optimum delivery plan (A ^* ) can be calculated using a neural network as shown in the following formula (4).
A ^* =π(S)=NSθ(S)...(4)
In addition, θ is a parameter of the neural network. In this case, learning of NSθ(S) that calculates the delivery plan is also performed in machine learning using the simulation results and machine learning using delivery record information.

An example of learning using the results of a simulation will be described with reference to Figures 6 to 8. In the example shown in Figures 6 to 8, worker M loads products from a delivery base and replenishes them in vending machine V, which is the delivery destination. When delivery is made, all products at the delivery destination are replenished to their maximum. There are three types of products to be replenished (products A1 to A3), and the maximum number of products that can be replenished in each vending machine V is 10. Furthermore, the worker M who makes the delivery is one worker M1, and the delivery destinations are three bases (vending machines V1 to V3).

Figure 6 shows the results of a simulation in which travel time is prioritized and products are delivered only to vending machines V1 and V2. As shown on the left side of Figure 6, the demand information indicates that the demand for each of products A to C in vending machine V3 is eight, while the inventory information indicates that the inventory for each of products A to C in vending machine V3 is one. In this case, the inventory (number of products) of vending machine V3 is insufficient compared to the demand, and if products are delivered only to vending machines V1 and V2, the sales of each of products A to C in vending machine V3 will be seven less than the demand. Also, Figure 7 shows the results of a simulation in which sales (number of sales) are taken into consideration and all vending machines (vending machines V1 to V3) are delivered. In this case, by delivering to vending machine V3 as well, products are replenished to meet the demand, and sales (number of sales) are higher than when products are not delivered to vending machine V3 (see Figure 6). Specifically, when no products are delivered to vending machine V3 (see FIG. 6), the sales (number of bottles sold) of vending machine V3 increase from 3 bottles (product A: 1 bottle, product B: 1 bottle, product C: 1 bottle) to 21 bottles (product A: 7 bottles, product B: 7 bottles, product C: 7 bottles) (see FIG. 7). Meanwhile, the total travel time also increases from 3.5 hours (see the bottom right of FIG. 6) to 4 hours (see the bottom right of FIG. 7). In this way, the creation of a tentative delivery plan and reinforcement learning using the simulation results based on the tentative delivery plan are repeated.

Figure 8 shows the results of a simulation using the trained model of the delivery plan creation unit 23, which has learned that vending machine V2 does not need to be delivered to due to low demand. By setting delivery destinations to vending machines V1 and V3, a delivery plan can be created that ensures a sufficient number of products to meet demand and has a short total travel time. Specifically, without reducing the sales (number of products sold) of vending machines V1 and V2, the sales (number of products sold) of vending machine V3 are increased to 21 (product A: 7 products, product B: 7 products, and product C: 7 products) (see right of Figure 8), and the total travel time is kept to 3.5 hours (see bottom of Figure 8).

Figures 9 and 10 also show an example of a simulation result based on demand information for multiple days. In the simulation shown in Figures 9 and 10, demand information for two days is used. This simulation shows that by using future (two-day) demand information, a delivery plan can be made so that travel time over two days (total travel time) is shorter, even if travel time for one day is longer.

The example shown in FIG. 9 does not take into account the demand for the product on the second day, and delivers to vending machines V1 and V3 on the first day, and delivers only to vending machine V2 on the second day, which is inefficient. In this case, the total travel time over the two days is 6.5 hours (see bottom of FIG. 9). On the other hand, in the example shown in FIG. 10, the demand for the product on the second day is taken into account, and deliveries are made to vending machine V2 in advance on the first day in anticipation of the need for replenishment on the second day. In this case, the total travel time over the two days is 4 hours (see bottom of FIG. 10). By delivering in this way, it is no longer necessary to deliver only to vending machine V2 on the second day, and the total travel time over the two days is reduced from 6.5 hours (see bottom of FIG. 9) to 4 hours (see bottom of FIG. 10), shortening the total travel time over the two days.

Note that although Figs. 9 and 10 show an example in which demand information for two days is used, demand information for more than two days may be used. For example, demand information for a long period such as one week, one month, or one year may be used.

Machine learning using the simulation results is performed again if there are changes in the number of vending machines V, the locations of the vending machines V, the number of workers M, road conditions, etc. Machine learning using the simulation results may also be performed again when there is a change in the products sold by the vending machines V, or when the seasons change.

When worker M actually delivers products (when the delivery plan creation system 100 is in actual operation), a delivery plan is first created using the results of machine learning using the simulation results (trained model). Then, worker M actually carries out delivery according to the created delivery plan (delivery route). Then, the delivery plan creation system 100 (delivery plan creation device 101) performs reinforcement learning using delivery performance information (total travel time required to deliver products, sales of products from multiple vending machines V, product transport volume, location information of worker M, and product inventory information from multiple vending machines V), which is the actual delivery performance.

(Creating delivery plans and amending delivery plans during delivery)
The delivery plan creation unit 23 is configured to use predicted demand (see FIG. 11), which is a predicted value of sales of products from multiple vending machines V over multiple days, when creating a delivery plan.

In addition, the delivery plan creation device 101 acquires delivery performance information when the delivery plan is actually implemented when the worker M delivers the product. Specifically, the delivery plan creation device 101 acquires delivery performance information when the delivery plan is actually implemented when the worker M delivers the product via the network 200.

The delivery plan creation unit 23 is configured to output a delivery plan using delivery performance information, including inventory information acquired when the worker M delivers the products, and forecast demand, which is a forecast value of sales of products from multiple vending machines V, as input (see FIG. 11).

The delivery plan creation device 101 is also configured to revise the delivery plan based on the acquired delivery performance information, without updating the trained model of the delivery plan creation unit 23. Specifically, the delivery plan creation device 101 revise the delivery plan while the worker M is delivering the goods, based on the acquired delivery performance information, without updating the trained model of the delivery plan creation unit 23. Then, the delivery plan creation device 101 is configured to transmit the revised delivery plan to the mobile terminal 102 of each of the multiple workers M via the network 200.

More specifically, the delivery plan creation device 101 acquires, via the network 200, the product transport volume, which is the number of products that each of the multiple workers M can deliver, when the multiple workers M delivers the products. Then, the delivery plan creation device 101 modifies the delivery plan while the worker M is delivering the products, based on the acquired product transport volume of each of the multiple workers M, without updating the trained model of the delivery plan creation unit 23. Furthermore, the delivery plan creation device 101 is configured to transmit the modified delivery plan, which is modified based on the acquired product transport volume of each of the multiple workers M, to the mobile terminal 102 of each of the multiple workers M via the network 200.

In this embodiment, as described above, the delivery plan creation device 101 modifies the delivery plan based on the amount of goods transported by the worker M with respect to the delivery plan at the start of delivery (see FIG. 12). For example, if the sales (number of sales) of the vending machine V4 in charge of the worker M2 are greater than expected, and the amount of goods to be replenished to the vending machine V4 assumed (predicted) in the delivery plan creation (predicted replenishment amount) was three (see FIG. 12), but six goods actually needed to be replenished (see FIG. 13), the number of goods to be delivered (goods transport amount) will be insufficient, and the worker M2 will not be able to replenish goods to the vending machine V7. Therefore, the sales (number of sales) of the vending machine V1 are less than expected, and the predicted replenishment amount of the vending machine V1 was five (see FIG. 12), but only one goods actually needed to be replenished. Therefore, the delivery plan is modified (see FIG. 13) so that the worker M1, who has a surplus in the amount of goods transported, replenishes goods to the vending machine V7 instead of the worker M2. Also, another worker M, who was previously set with a larger amount of product to be transported in the delivery plan, may replenish products in vending machine V7. As a result, even if the amount of product to be replenished in a particular vending machine (vending machine V4) becomes greater than the replenishment amount for the predicted number of sales (expected replenishment amount), and the amount of product transported by worker M (worker M2) is insufficient, it is possible to prevent a case in which the amount of product to be replenished in vending machine V is insufficient by cooperating with another worker M (worker M1) to revise the delivery plan (see FIG. 13).

The delivery plan creation device 101 is also configured to acquire delivery performance information including the location information of the worker M at the time of delivery of the goods by the worker M. The delivery plan creation device 101 then modifies the delivery plan while the worker M is delivering the goods based on the acquired location information of the worker M, without updating the trained model of the delivery plan creation unit 23. Furthermore, the delivery plan creation device 101 is configured to transmit the modified delivery plan, modified based on the acquired location information of the worker M, to the mobile terminals 102 of the multiple workers M via the network 200.

The location information of the worker M may be transmitted (notified) from the mobile terminal 102 to the delivery plan creation device 101 by an input operation by the worker M or by a GPS (Global Positioning System), or may be transmitted (notified) from the vending machine V to the delivery plan creation device 101 when the mobile terminal 102 communicates with the vending machine V and when the vending machine V is opened or closed when the vending machine V is refilled with products (during maintenance).

In addition, in the delivery plan (delivery route) shown in FIG. 14, when a specific worker M (worker M2) is delayed in delivery due to traffic jams or the like, the delivery plan creation device 101 modifies the delivery plan (see FIG. 15) so that another worker M (worker M1) delivers the product to the vending machine (vending machine V7) in charge of the worker M2 who is delayed in delivery, instead of the worker M2 whose delivery is delayed. For example, when delivering products according to the delivery plan (delivery route) shown in FIG. 14, if the delivery by worker M2 is delayed due to traffic jams or the like, and the delivery by worker M1 has progressed to vending machine V3, but the delivery by worker M2 has only progressed to vending machine V4 (see FIG. 15), the delivery plan is modified. In such a case, the delivery plan is revised from a delivery plan (see FIG. 14) in which worker M1 delivers to vending machines V1, V2, and V3 in that order, and worker M2 delivers to vending machines V4, V5, V6, and V7 in that order, to a delivery plan (see FIG. 15) in which worker M1 delivers to vending machines V1, V2, V3, and V7 in that order, and worker M2 delivers to vending machines V4, V5, and V6 in that order. This allows multiple workers M to cooperate with each other and speed up the time when all workers M complete their deliveries, even if the deliveries are delayed.

(Machine learning using delivery record information)
As shown in Fig. 16, the delivery plan creation device 101 is configured to update the trained model of the delivery plan creation unit 23 by machine learning using the trained model of the delivery plan creation unit 23 based on delivery performance information (total travel time, sales, product transportation volume, location information, and inventory information) after the delivery of products to multiple vending machines V is completed. Note that the trained model of the delivery plan creation unit 23 may be updated daily or every few days. Furthermore, the trained model of the delivery plan creation unit 23 may be updated at any period or timing of the user's choice.

The delivery plan creation device 101 updates the state-action value function (Q function) of the learned model by reinforcement learning using the learned model of the delivery plan creation unit 23 based on delivery performance information. Specifically, the delivery plan creation device 101 is configured to update the state-action value function (Q function) of the learned model by reinforcement learning using the learned model of the delivery plan creation unit 23 based on delivery performance information after the delivery of products to multiple vending machines V is completed.

Figure 17 shows an example of learning using delivery performance information, and shows an example of using a reinforcement learning method that uses a state-action value function (Q function). In learning using delivery performance information, as in the case of learning using simulation results, reward r is calculated from the actual total travel time and actual sales performance of worker M (routeman) when he actually makes a delivery.

Then, using the delivery plan for the actual delivery, delivery performance information (total travel time, sales, product transport volume, location information, and inventory information), and predicted demand, the state-action value function (Q function) learned using the simulation results is updated (modified) before the actual delivery. In this embodiment, by re-learning using the performance associated with the actual delivery (delivery performance information), it becomes possible to calculate a more optimal delivery route (delivery plan). Note that, like learning using the delivery performance information, existing reinforcement learning techniques can be used for learning using the simulation results. In this embodiment, the learned model (Q function) is updated by reinforcement learning similar to learning using the simulation results.

As described above, the delivery plan creation device 101 also acquires delivery performance information including the location information of the worker M at the time of delivery of the worker M's goods. The trained model of the delivery plan creation unit 23 is configured to be updated by machine learning using the trained model based on the delivery performance information including the acquired location information.

Figures 18 and 19 show the results of reinforcement learning using delivery record information including the location information of worker M. In reinforcement learning using delivery record information including the location information of worker M, it is possible to learn a delivery plan that avoids traffic jams, for example. In learning using delivery record information, the delivery route (see Figure 18), which delivered to vending machine V1 first, then vending machine V3, is changed to vending machine V3 first, then vending machine V1 (see Figure 19) in learning that takes traffic jams into account (by learning past travel time results). As a result, the total travel time is reduced from 4.5 hours (see bottom right of Figure 18) to 3.5 hours (see bottom right of Figure 19), making it possible to create a delivery plan in which the total travel time is shorter (shortened by one hour).

(Effects of this embodiment)
In this embodiment, the following effects can be obtained.

In the delivery plan creation device 101 (delivery plan creation system 100) according to this embodiment, a delivery plan is created by the trained model of the delivery plan creation unit 23 based on the predicted demand, which is a predicted value of the sales of products of multiple vending machines V, and the inventory information of each product of multiple vending machines V. This makes it possible to create a delivery plan that can suppress a decrease in sales of products due to sell-out. In addition, the trained model of the delivery plan creation unit 23 is updated by performing machine learning using the trained model of the delivery plan creation unit 23 based on the delivery performance information when the delivery plan is actually implemented. This makes it possible to update the trained model of the delivery plan creation unit 23 to a trained model that reflects delivery performance information including the sales of products of multiple vending machines V and the total travel time required for delivery of the products when the delivery plan is actually implemented. As a result, the delivery plan creation unit 23 can create an efficient delivery plan that reflects delivery performance information including the sales of products of multiple vending machines V and the total travel time required for delivery of the products when the delivery plan is actually implemented, at the time of delivery after the trained model is updated. As a result, it is possible to create a delivery route that suppresses an increase in delivery time while suppressing a decrease in sales of products from the vending machine V due to selling out. In addition, the output unit 4 outputs the delivery plan created by the updated trained model of the delivery plan creation unit 23 to the mobile terminal 102 via the network 200, so that the worker M can check, by the mobile terminal 102, an efficient delivery plan that reflects the delivery performance information and suppresses an increase in delivery time while suppressing a decrease in sales of products from the vending machine V due to selling out. As a result, the worker M can perform efficient product delivery that reflects the delivery performance information and suppresses an increase in delivery time while suppressing a decrease in sales of products from the vending machine V due to selling out.

In addition, in this embodiment, as described above, the trained model of the delivery plan creation unit 23 includes a state-action value function (Q function) that represents value, and the delivery plan creation unit 23 is configured to create a delivery plan based on the state-action value function. The delivery plan creation device 101 then updates the state-action value function of the trained model by reinforcement learning using the trained model of the delivery plan creation unit 23 based on delivery performance information. This makes it possible to create a delivery plan that reflects the delivery performance information, has a short delivery time, and has high sales of products from the vending machine V, using the state-action value function updated based on the delivery performance information. As a result, the delivery plan creation unit 23 can create an efficient delivery route that can suppress an increase in delivery time when delivering products to multiple vending machines V, while suppressing a decrease in sales of products from the vending machine V due to selling out.

In addition, in this embodiment, as described above, after the delivery of products to multiple vending machines V is completed, the trained model of the delivery plan creation unit 23 is updated by machine learning using the trained model of the delivery plan creation unit 23 based on the delivery performance information. This makes it possible to update the trained model of the delivery plan creation unit 23 so as to reflect the above-mentioned delivery performance information from the start of delivery of products to multiple vending machines V to the completion of delivery. As a result, the delivery plan creation unit 23 after updating the trained model can create a more efficient delivery plan (delivery route) that can further suppress an increase in delivery time when delivering products to multiple vending machines V and further suppress a decrease in sales of products from vending machines V due to products being sold out.

In addition, in this embodiment, as described above, the delivery plan creation device 101 acquires delivery record information including the location information of the worker M at the time of delivery of the worker M's goods. Then, the trained model of the delivery plan creation unit 23 is updated by machine learning using the trained model based on the delivery record information including the acquired location information. As a result, it is possible to acquire the travel time between multiple vending machines V at the time of delivery of the worker M's goods from the delivery record information including the location information of the worker M. As a result, if the travel time between multiple vending machines V in the actual delivery of goods is longer than the travel time assumed when the delivery plan was created, it is possible to update the trained model of the delivery plan creation unit 23 based on the travel time between multiple vending machines V acquired from the delivery record information including the location information of the worker M. As a result, it is possible to create a more optimal delivery route by changing the route between multiple vending machines V or the order of the vending machines V to be delivered, and therefore it is possible to create a more efficient delivery plan.

In addition, in this embodiment, as described above, the delivery plan creation unit 23 uses predicted demand, which is a predicted value of sales of products from multiple vending machines V over multiple days, when creating a delivery plan. This makes it possible to create a delivery route that has a short total travel time or total travel distance required to deliver products over multiple days. As a result, the delivery plan creation unit 23 can create an efficient delivery route for products to multiple vending machines V when delivering over multiple days.

In addition, in this embodiment, as described above, the delivery plan creation device 101 acquires delivery performance information when the delivery plan is actually implemented when the worker M delivers the product. Then, the delivery plan creation unit 23 is configured to output a delivery plan using delivery performance information including inventory information acquired when the worker M delivers the product and predicted demand, which is a predicted value of sales of products from multiple vending machines V, as inputs. Furthermore, the delivery plan creation device 101 corrects the delivery plan based on the acquired delivery performance information without updating the trained model of the delivery plan creation unit 23. This allows the delivery plan creation unit 23 to correct the delivery plan before updating the trained model based on actual inventory information. As a result, the delivery plan can be corrected to a more efficient delivery plan by reflecting actual inventory information before updating the trained model. In addition, by correcting the delivery plan without updating the trained model of the delivery plan creation unit 23, it is easier to correct the delivery plan compared to the case where the trained model of the delivery plan creation unit 23 is updated from time to time.

In addition, in this embodiment, as described above, the delivery plan creation device 101 acquires delivery performance information when the delivery plan is actually implemented when the worker M delivers the product via the network 200. Then, the delivery plan creation device 101 modifies the delivery plan during the delivery of the product by the worker M based on the acquired delivery performance information without updating the trained model of the delivery plan creation unit 23, and transmits the modified delivery plan via the network 200 to each of the mobile terminals 102 of the multiple workers M. As a result, while the worker M is delivering the product, the delivery plan modified based on the acquired delivery performance information is transmitted to each of the mobile terminals 102 of the multiple workers M, so that the worker M can deliver the product efficiently while the product is being delivered, reflecting the delivery situation in real time. Also, by modifying the delivery plan without updating the trained model of the delivery plan creation unit 23, the delivery plan can be modified in real time during the delivery of the product more quickly than when the trained model of the delivery plan creation unit 23 is updated from time to time.

In the present embodiment, as described above, the delivery plan creation device 101 acquires the product transport volume, which is the number of products that each of the multiple workers M can deliver, via the network 200 when the multiple workers M deliver the products. The delivery plan creation device 101 then modifies the delivery plan during the delivery of the products by the workers M based on the acquired product transport volume of each of the multiple workers M without updating the trained model of the delivery plan creation unit 23, and transmits the modified delivery plan to each mobile terminal 102 of the multiple workers M via the network 200. As a result, the delivery plan can be modified based on the acquired product transport volume of each of the multiple workers M during the delivery of the products by the workers M, so that even if the product transport volume of a certain worker M is insufficient, the delivery route can be modified so that another worker M who has a surplus in product transport volume replenishes the products instead. Then, the modified delivery plan is transmitted to each mobile terminal 102 of the multiple workers M, so that the workers M can efficiently deliver the products reflecting the product transport volume of each worker M. In addition, by revising the delivery plan without updating the trained model of the delivery plan creation unit 23, the delivery plan can be revised quickly while the goods are being delivered, based on the acquired product transportation volume for each of the multiple workers M, compared to when the trained model of the delivery plan creation unit 23 is updated from time to time.

In addition, in this embodiment, as described above, when creating the trained model of the delivery plan creation unit 23, the delivery plan creation device 101 creates a tentative delivery plan based on the predicted sales value of the products of the multiple vending machines V and the predicted inventory value of each product of the multiple vending machines V by the delivery plan creation unit 23. Then, the delivery plan creation device 101 acquires predicted delivery performance information including the product sales in the simulation result of the delivery of products to the multiple vending machines V by the tentative delivery plan and the total travel time required for the delivery of products in the simulation result of the delivery of products to the multiple vending machines V. Then, the delivery plan creation device 101 creates a trained model by performing reinforcement learning based on the predicted delivery performance information as machine learning in the delivery plan creation unit 23. As a result, learning by reinforcement learning can be performed before actually delivering products by using the predicted delivery performance information obtained by the simulation result based on the tentative delivery plan. As a result, it is possible to create a trained model of the delivery plan creation unit 23 that can create a more efficient delivery plan before delivering actual products, compared to a case where reinforcement learning is not performed using predicted delivery performance information obtained by simulation results based on a tentative delivery plan.

[Modification]
The embodiments disclosed herein should be considered to be illustrative and not restrictive in all respects. The scope of the present invention is indicated by the claims rather than the description of the above embodiments, and further includes all modifications (variations) within the meaning and scope of the claims.

For example, in the above embodiment, the delivery performance information includes the total travel time required to deliver the products, the sales (number of units sold) of the products of multiple vending machines V, the amount of products transported, the location information of the worker M, and inventory information of the products of multiple vending machines V, but the present invention is not limited to this. In the present invention, the delivery performance information may include the total travel distance required to deliver the products, and may include the sales amount and profit (profit margin) of the products sold by the vending machine V.

In the above embodiment, the predicted delivery performance information includes the total travel time required to deliver the products, the sales (number of units sold) of the products of the multiple vending machines V, the amount of products transported, the location information of the worker M, and the inventory information of the products of the multiple vending machines V, but the present invention is not limited to this. In the present invention, the predicted delivery performance information may include the total travel distance required to deliver the products, and may include the sales amount and profit (profit margin) of the products sold by the vending machine V.

In the above embodiment, the delivery plan creation device 101 updates the state-action value function of the learned model by reinforcement learning using the learned model of the delivery plan creation unit 23 based on delivery performance information, but the present invention is not limited to this. In the present invention, the delivery plan creation device 101 may be configured to update the learned model by machine learning other than reinforcement learning, such as learning using a neural network, unsupervised learning, and supervised learning.

In addition, in the above embodiment, an example was shown in which after the delivery of products to multiple vending machines V is completed, the trained model of the delivery plan creation unit 23 is updated by machine learning using the trained model of the delivery plan creation unit 23 based on delivery performance information, but the present invention is not limited to this. In the present invention, the delivery plan creation device 101 may update the trained model of the delivery plan creation unit 23 while the worker M is delivering the products.

In addition, in the above embodiment, an example was shown in which the trained model of the delivery plan creation unit 23 is updated by machine learning using the trained model based on the delivery performance information including the acquired location information, but the present invention is not limited to this. In the present invention, the trained model of the delivery plan creation unit may be updated by machine learning using the trained model based on the acquired product transportation volume of each of the multiple workers.

In addition, in the above embodiment, an example is shown in which the delivery plan is revised while the worker M is delivering the goods based on the acquired product transport volume of each of the multiple workers M without updating the trained model of the delivery plan creation unit 23, and the delivery plan is revised while the worker M is delivering the goods based on the acquired location information without updating the trained model of the delivery plan creation unit 23, but the present invention is not limited to this. In the present invention, a configuration may be used in which the delivery plan is revised while the worker is delivering the goods based on either the acquired product transport volume or location information of each of the multiple workers without updating the trained model of the delivery plan creation unit.

In the above embodiment, the delivery plan creation unit 23 uses predicted demand, which is a predicted value of product sales of multiple vending machines V over multiple days, when creating the delivery plan, but the present invention is not limited to this. In the present invention, the delivery plan creation unit 23 may use predicted demand, which is a predicted value of product sales of multiple vending machines V for one day, when creating the delivery plan and the tentative delivery plan.

In addition, in the above embodiment, an example was shown in which the delivery plan creation device 101 does not update the trained model of the delivery plan creation unit 23, but instead modifies the delivery plan based on the acquired delivery performance information while the worker M is delivering the goods, but the present invention is not limited to this. In the present invention, it is possible to only update the trained model after delivery is completed, and not to modify the delivery plan while the worker M is delivering the goods.

In the above embodiment, the delivery plan creation unit 23 creates a provisional delivery plan using the predicted sales values of products from multiple vending machines V and the predicted stock quantities of each product from multiple vending machines V, but the present invention is not limited to this. In the present invention, the provisional delivery plan may be created using past actual values of the sales and stock quantities of products from multiple vending machines, or both the predicted values and past actual values of the sales and stock quantities of products from multiple vending machines. In the present invention, the provisional delivery plan may be created using both the predicted value and past actual value of each of the sales and stock quantities of products from multiple vending machines, and either the predicted value or the past actual value may be used for the other of the sales and stock quantities of products from multiple vending machines.

4 Output unit 23 Delivery plan creation unit 100 Delivery plan creation system 101 Delivery plan creation device 102 Mobile terminal 200 Network M, M1, M2 Worker V, V1, V2, V3, V4, V5, V6, V7 Vending machine

Claims (10)

a delivery plan creation unit that creates a delivery plan showing product delivery routes to multiple vending machines using machine learning;
an output unit for outputting the delivery plan created by the delivery plan creation unit via a network to a mobile terminal carried by an operator who replenishes products in the vending machines;
The delivery plan is created based on a predicted demand, which is a predicted value of sales of products of the plurality of vending machines, and inventory information of each of the products of the plurality of vending machines, using a trained model that represents a value obtained by the delivery plan as an action with respect to the predicted demand and the inventory information as a state of the delivery plan creation unit,
The trained model of the delivery plan creation unit is updated by performing reinforcement learning of the trained model regarding the value of the delivery plan represented by the trained model of the delivery plan creation unit using delivery record information including sales of products of the multiple vending machines , a total travel time required to deliver the products, and inventory information of the products of each of the multiple vending machines when the delivery plan is actually implemented,
The output unit is configured to output the delivery plan created by the trained model of the delivery plan creation unit after updating to the mobile terminal via a network. The trained model of the delivery plan creation unit includes a state-action value function that represents a value obtained by the delivery plan as an action with respect to the forecasted demand and the inventory information as a state,
The delivery plan creation unit is configured to create the delivery plan based on the state-action value function,
The delivery plan creation device according to claim 1, configured to update the state-action value function of the trained model by reinforcement learning using the trained model of the delivery plan creation unit based on the delivery performance information. The delivery plan creation device according to claim 1 or 2, configured to update the trained model of the delivery plan creation unit by machine learning using the trained model of the delivery plan creation unit based on the delivery performance information after the delivery of products to the multiple vending machines is completed. Acquire the delivery record information including location information of the worker at the time of delivery of the goods by the worker;
The delivery plan creation device according to claim 3, wherein the trained model of the delivery plan creation unit is configured to be updated by machine learning using the trained model based on the delivery performance information including the acquired location information. The delivery plan creation device according to any one of claims 1 to 4, wherein the delivery plan creation unit is configured to use the predicted demand, which is a predicted value of sales of products from the multiple vending machines over multiple days, in creating the delivery plan. When the worker delivers the goods, the delivery performance information is acquired when the delivery plan is actually carried out,
the delivery plan creation unit is configured to output the delivery plan using as input the delivery record information including the inventory information acquired when the worker delivers the products and the predicted demand, which is a predicted value of sales of the products of the plurality of vending machines;
The delivery plan creation device according to any one of claims 1 to 5, configured to correct the delivery plan based on the acquired delivery record information without updating the trained model of the delivery plan creation unit. When the worker delivers the goods, the delivery performance information when the delivery plan is actually executed is acquired via a network;
The delivery plan is modified during delivery of the goods by the worker based on the acquired delivery performance information without updating the trained model of the delivery plan creation unit;
The delivery plan creation device according to any one of claims 1 to 6, configured to transmit the modified delivery plan to the mobile terminals of each of the plurality of workers via a network. When the plurality of workers delivers the goods, a product transportation volume, which is the number of goods that each of the plurality of workers can deliver, is acquired via a network;
The delivery plan is modified during delivery of the goods by the workers based on the acquired product transportation volume of each of the multiple workers without updating the trained model of the delivery plan creation unit;
The delivery plan creation device according to any one of claims 1 to 7, configured to transmit the revised delivery plan to the mobile terminals of each of the multiple workers via a network. When creating the trained model of the delivery plan creation unit,
creating a provisional distribution plan based on at least one of a predicted value or a past actual value of sales of the products of the plurality of vending machines and at least one of a predicted value or a past actual value of inventory quantity of the products of each of the plurality of vending machines by the distribution plan creation unit;
Acquire predicted delivery performance information including sales of products in a simulation result of product delivery to the multiple vending machines according to the tentative delivery plan and a total travel time required for product delivery in a simulation result of product delivery to the multiple vending machines;
The delivery plan creation device according to any one of claims 1 to 8, wherein the learned model is created by performing reinforcement learning based on the predicted delivery performance information as machine learning in the delivery plan creation unit. A mobile terminal carried by an operator who replenishes products in a vending machine;
a delivery plan creation device that creates a delivery plan indicating product delivery routes to a plurality of vending machines;
The delivery plan creation device includes a delivery plan creation unit that creates the delivery plan by machine learning, and an output unit that outputs the delivery plan created by the delivery plan creation unit to the mobile terminal via a network,
The delivery plan is created based on a predicted demand, which is a predicted value of sales of products of the plurality of vending machines, and inventory information of each of the products of the plurality of vending machines, using a trained model that represents a value obtained by the delivery plan as an action with respect to the predicted demand and the inventory information as a state of the delivery plan creation unit,
The trained model of the delivery plan creation unit is updated by performing reinforcement learning of the trained model regarding the value of the delivery plan represented by the trained model of the delivery plan creation unit using delivery record information including sales of products of the multiple vending machines , a total travel time required for delivering the products, and inventory information of the products of each of the multiple vending machines when the delivery plan is actually implemented,
The output unit is configured to output the delivery plan created by the trained model of the delivery plan creation unit after updating to the mobile terminal via a network.

Images