JP7664121B2 - Logistics transport speed control agent learning device, logistics transport speed control agent learning method, and logistics transport speed control agent learning program - Google Patents

Description

The present invention relates to a logistics transport speed control agent learning device, a logistics transport speed control agent learning method, and a logistics transport speed control agent learning program.

For example, as shown in Patent Document 1, a logistics facility has multiple transport paths for transporting workpieces. These transport paths are formed by various transport units, such as conveyors that transport the workpieces horizontally and transfer robots that transfer the workpieces.

JP 2020-11817 A

When the transport of workpieces is delayed on the transport path, a so-called traffic jam occurs. For example, while the entire logistics facility is in operation, some of the transport units that form the transport path may stop for maintenance. In such a case, the transport of workpieces is delayed on the transport path that includes the transport unit undergoing maintenance, causing a traffic jam. If the workpiece congestion is not resolved, over time the transport path will no longer have space to temporarily store the workpieces, and the transport path will no longer be able to accept the workpieces. This inability to accept workpieces supplied from upstream on the transport path is referred to as a drop. In other words, when a large number of workpieces gather on the transport path in excess of the transport capacity, the transport path will no longer be able to accept the workpieces, and a drop will occur.

One way to prevent such drops from occurring is to keep the transport speed of the transport unit at its maximum. By always keeping the transport speed of the transport unit at its maximum, the number of workpieces on the transport path can be minimized. This reduces the number of workpieces on the transport path when workpieces are concentrated or when transport unit maintenance occurs, making it possible to maximize the number of workpieces that can be temporarily accepted on the transport path during congestion. This therefore makes it possible to prevent drops from occurring.

However, if the conveying speed is always set to the maximum, the conveying speed of the conveying unit will be unnecessarily set to the maximum even when there is no congestion. This is expected to result in high energy consumption.

The present invention was made in consideration of the above-mentioned problems, and aims to reduce energy consumption while preventing workpieces from being unable to be accepted due to congestion on the transport route in logistics facilities.

The present invention adopts the following configuration as a means for solving the above problems.

The first aspect of the present invention is a logistics transport speed control agent learning device that performs reinforcement learning on a learning agent that controls the transport speeds of multiple transport units that form a logistics transport route, and includes a transport route simulator that uses a model of the logistics transport route to calculate the state of the logistics transport route and a reward based on the state, and the learning agent that learns about the transport speed so that the evaluation based on the reward increases, and the transport route simulator calculates the reward based on a transport completion reward obtained when the transport of a workpiece is completed on the logistics transport route, an acceptance refusal penalty that is given when the logistics transport route cannot accept the workpiece, and an energy consumption increase penalty that increases as the energy consumption related to the transport speed increases.

The second aspect of the present invention is the first aspect, in which the modeled logistics transport route has multiple control target parts, each of which includes a single or multiple transport units, and the learning agent controls the transport speed for each of the control target parts.

The third aspect of the present invention is the second aspect, in which the transport path simulator calculates the energy consumption increase penalty based on the speed command value for each transport unit.

The fourth aspect of the present invention is the third aspect, in which the transport path simulator calculates the energy consumption increase penalty based on the sum of the squared speed command values for each transport unit.

The fifth aspect of the present invention is the second aspect, in which the transport path simulator calculates the energy consumption increase penalty based on the acceleration of each transport unit.

The sixth aspect of the present invention is any one of the second to fifth aspects, in which a maintenance-incurring transport unit that may experience a downtime period due to maintenance during the transport of workpieces on the logistics transport route is included in the multiple transport units that form the logistics transport route, and a single controlled part includes one or less maintenance-incurring transport units.

The seventh aspect of the present invention is a logistics transport speed control agent learning method for performing reinforcement learning on a learning agent that controls the transport speeds of multiple transport units that form a logistics transport route, and employs a transport route simulator to calculate the state of the logistics transport route and a reward based on the state using the modeled logistics transport route, the learning agent learns about the transport speed so that the evaluation based on the reward increases, and the transport route simulator calculates the reward based on a transport completion reward obtained when the transport of a workpiece is completed on the logistics transport route, an acceptance refusal penalty given when the logistics transport route cannot accept the workpiece, and an energy consumption increase penalty that increases as the energy consumption related to the transport speed increases.

The eighth aspect of the present invention is a logistics transport speed control agent learning program that causes a computer to function as a logistics transport speed control agent learning device that performs reinforcement learning on a learning agent that controls the transport speeds of multiple transport units that form a logistics transport route, and causes the computer to function as a transport route simulator that uses a model of the logistics transport route to calculate the state of the logistics transport route and a reward based on the state, causes the computer to function as the learning agent that learns about the transport speed so that the evaluation based on the reward increases, and when the computer is caused to function as the transport route simulator, calculates the reward based on a transport completion reward obtained when the transport of a workpiece is completed on the logistics transport route, an acceptance refusal penalty that is given when the logistics transport route cannot accept the workpiece, and an energy consumption increase penalty that increases due to an increase in energy consumption related to the transport speed.

According to the present invention, the reward calculated by the transport route simulator is input to the learning agent, and the learning agent learns based on the reward. The reward is calculated based on a transport completion reward obtained when the transport of the workpiece is completed on the logistics transport route, an acceptance refusal penalty given when the logistics transport route cannot accept the workpiece, and an energy consumption increase penalty that increases as the energy consumption related to the transport speed increases. In other words, the reward input to the learning agent decreases as the energy consumption related to the transport speed increases. For this reason, in the present invention, the learning agent learns to reduce the energy consumption related to the transport speed. Also, the reward input to the learning agent decreases when the logistics transport route cannot accept the workpiece. Therefore, according to the present invention, the learning agent learns to avoid cases where the logistics transport route cannot accept the workpiece. Therefore, according to the present invention, it is possible to reduce energy consumption while suppressing the inability to accept the workpiece due to congestion on the transport route in the logistics facility.

1 is a block diagram showing an outline of a hardware configuration of a logistics transport speed control agent learning device in one embodiment of the present invention. 1 is a block diagram showing an outline of the functional configuration of a logistics transportation speed control agent learning device in one embodiment of the present invention. FIG. 1 is an image diagram of a modeled logistics transportation route. FIG. 13 is a schematic diagram illustrating dropping of a workpiece. 5 is a flowchart for explaining the operation of a logistics transportation speed control agent learning device in one embodiment of the present invention.

Below, an embodiment of a logistics transport speed control agent learning device, a logistics transport speed control agent learning method, and a logistics transport speed control agent learning program according to the present invention will be described with reference to the drawings.

Figure 1 is a block diagram showing an outline of the hardware configuration of the logistics transport speed control agent learning device 1 of this embodiment. Figure 1 is also a block diagram showing an outline of the functional configuration of the logistics transport speed control agent learning device 1 of this embodiment.

The logistics transport speed control agent learning device 1 of this embodiment is a device that causes the learning agent 3 (see FIG. 2) to perform reinforcement learning. The learning agent 3 is installed in the control device of the logistics facility after reinforcement learning. The learning agent 3 installed in the control device of the logistics facility controls the transport speed of the logistics transport route provided in the logistics facility. The logistics transport route provided in the logistics facility is formed by various transport units such as conveyors and transfer devices. The learning agent 3 controls, for example, the transport speed of each transport unit.

As shown in FIG. 1, the logistics transport speed control agent learning device 1 includes a memory unit 10, an operation unit 11, a communication unit 12, a calculation unit 13, and a display unit 14, and is formed by a computer.

The storage unit 10 is made up of memories such as ROM (Read Only Memory) and RAM (Random Access Memory), and storage such as HDD (Hard Disk Drive) and SSD (Solid State Drive). This storage unit 10 stores a learning program P (logistics transport speed control agent learning program). The storage unit 10 also stores various data D. This data D includes initial data used by the calculation unit 13 and the calculation results of the calculation unit 13.

The operation unit 11 is an input device that accepts operation instructions from an operator who operates the logistics transport speed control agent learning device 1, and more specifically, is a pointing device such as a keyboard or a mouse. This operation unit 11 outputs operation signals corresponding to the operator's operation instructions to the calculation unit 13. The communication unit 12 is a communication device that transmits and receives data to and from external devices via a specified communication line, and communicates with external devices using, for example, a communication protocol that complies with a LAN (Local Area Network) or the Internet.

The calculation unit 13 is a calculation device that performs calculations for making the learning agent 3 perform reinforcement learning based on the above-mentioned learning program P, data D, operation signals, etc. This calculation unit 13 is composed of hardware such as an interface circuit and a CPU (Central Processing Unit). The interface circuit is an electronic circuit that sends and receives various signals to and from the memory unit 10, operation unit 11, communication unit 12, and display unit 14. The CPU is a central processing unit that executes the above-mentioned learning program P.

The display unit 14 is a display device that displays the learning state of the learning agent 3 based on the learning program P, which is based on the image data generated by the calculation unit 13. Note that the logistics transport speed control agent learning device 1 does not necessarily need to include a display unit 14. In other words, an external device may display the data based on the data output from the logistics transport speed control agent learning device 1 via the communication unit 12.

The logistics transport speed control agent learning device 1 having the hardware configuration shown in FIG. 1 has multiple functional units shown in FIG. 2. These functional units are realized by the cooperation of the various hardware shown in FIG. 1 and the learning program P stored in the memory unit 10.

As shown in FIG. 2, the logistics transport speed control agent learning device 1 has the following functional units: a transport route simulator 2, a learning agent 3, a performance evaluation unit 4, and a hyperparameter setting unit 5.

The transport route simulator 2 calculates and outputs the "state" and "reward" from among the "state," "action," and "reward" used in the reinforcement learning of the learning agent 3. The transport route simulator 2 calculates the "state" and "reward" based on the "action" output from the learning agent 3 using a modeled logistics transport route. The modeled logistics transport route is a data group that models, for simulation purposes, a logistics transport route installed in a logistics facility in which the learning program P is installed after learning. The learning agent 3 is an agent that undergoes reinforcement learning by the logistics transport speed control agent learning device 1, and controls the transport speed of workpieces in the transport units that form the logistics transport route. This learning agent 3 calculates and outputs the "action" from among the "state," "action," and "reward."

In other words, in the logistics transport speed control agent learning device 1 of this embodiment, the "state" and "reward" calculated by the transport route simulator 2 are input to the learning agent, and the learning agent learns about the transport speed. Furthermore, the learning agent 3 selects (calculates) and outputs an "action", and the transport route simulator 2 again calculates the "state" and "reward" based on this "action". By repeating these operations until the learning of the learning agent 3 progresses, the logistics transport speed control agent learning device 1 causes the learning agent 3 to perform reinforcement learning.

As shown in FIG. 2, the transport path simulator 2 has a cell definition unit 2a, a transport path control unit 2b, and a remuneration calculation unit 2c. The cell definition unit 2a divides the logistics transport path into multiple parts (cells) and defines the operation of the cells, which are the unit elements.

Figure 3 is an image diagram of a modeled logistics transport route H. As shown in this figure, in this embodiment, the logistics transport route H is defined by dividing it into multiple cells S. Each cell S is provided, for example, for each transport unit (conveyor, transfer robot, etc.) that forms the logistics transport route H. In such a logistics transport route H, the work W is transferred between adjacent cells S, and the work W is transported from one end of the logistics transport route H to the other end.

The cell definition unit 2a defines operating conditions such as the settable range of the transport speed and acceleration for each cell S. In addition, in this embodiment, the cell definition unit 2a groups multiple cells S (i.e., transport units) into multiple control target parts (first control target part T1, second control target part T2, and third control target part T3).

Among the multiple transport units that form the logistics transport route H, there are transport units that require maintenance work while the logistics equipment is in operation. When maintenance work occurs on such a transport unit that requires maintenance work, work cannot pass through the transport unit undergoing maintenance work during the maintenance period. A cell S corresponding to a transport unit where maintenance work may occur is defined as a maintenance occurrence cell Sa (maintenance occurrence transport unit). In this embodiment, such maintenance occurrence cells Sa are set to one or less in the controlled part. In other words, the number of maintenance occurrence cells Sa included in a single controlled part is one or less. In this way, by setting the number of maintenance occurrence cells Sa included in a single controlled part to one or less, it is possible to prevent the transport speed of a single controlled part from being affected by multiple maintenance occurrence cells Sa.

In this embodiment, as shown in FIG. 3, for example, 14 cells S are arranged in series to form a logistics transport route H. These 14 cells S are grouped into three control target parts (first control target part T1, second control target part T2, and third control target part T3). The first control target part T1 includes three cells S. The second control target part T2 includes eight cells S. The third control target part T3 includes two cells S. Note that the last cell S is not included in the control target parts because it only receives the workpieces W and does not transport the workpieces W.

In addition, in this embodiment, of the cells S included in the second controlled section T2, the cell S located most upstream in the transport direction is a maintenance occurrence cell Sa, which is periodically subjected to maintenance. In addition, in this embodiment, of the cells S included in the third controlled section T3, the cell S located most upstream in the transport direction is a maintenance occurrence cell Sa, which is periodically subjected to maintenance.

In this way, when the controlled part includes a maintenance occurrence cell Sa, the maintenance occurrence cell Sa can be placed at the most upstream side of the controlled part. By placing the maintenance occurrence cell Sa at the most upstream side of the controlled part, when maintenance occurs in the maintenance occurrence cell Sa, it becomes possible to transport the work W to the cell S placed closest to the upstream end of the maintenance occurrence cell Sa.

Returning to FIG. 2, the transport path control unit 2b manages each cell S and calculates the overall "state" of the logistics transport path H based on the input "actions." Specifically, the learning agent 3 inputs a speed command value for each cell S to the transport path control unit 2b as an "action." The transport path control unit 2b calculates the "state" based on the speed command value using the modeled logistics transport path H under the conditions defined in the cell definition unit 2a. The transport path control unit 2b sets the transport speed of each cell S according to these speed command values.

The remuneration calculation unit 2c calculates the "remuneration" based on the "state" calculated by the transport path control unit 2b. In this embodiment, the remuneration calculation unit 2c calculates the "remuneration" based on, for example, the following formula (1).

In formula (1), r _t indicates the reward at time t. Also, x _t,catch is a variable that takes a value of 1 or 0 depending on whether the work W has been transported to the most downstream part of the logistics transport route H (the 14th cell S from the upstream side) at time t. If the work W has been transported to the most downstream part of the logistics transport route H, x _t,catch takes a value of 1. On the other hand, if the work W has not been transported to the most downstream part of the logistics transport route H, x _t,catch takes a value of 0.

In addition, in formula (1), x _t,drop is a variable that takes a value of 1 or 0 depending on whether or not a drop of the work W has occurred at time t. If a drop of the work W has occurred, x _t,drop takes a value of 1. On the other hand, if a drop of the work W has not occurred, x _t,drop takes a value of 0.

A drop means that an event has occurred in which the logistics transport route H is no longer able to accept the workpiece W that is being supplied from upstream to the logistics transport route H. Figure 4 is a schematic diagram that explains the dropping of the workpiece W.

For example, as shown in FIG. 4(a), when maintenance occurs in the maintenance occurrence cell Sa, the work W cannot pass through the maintenance occurrence cell Sa. However, as shown in FIG. 4(a), if there is a cell S upstream of the maintenance occurrence cell Sa where the work W can be stored, the work W can be received on the logistics transport route H, so a drop does not occur.

On the other hand, as shown in FIG. 4(b), if there is no cell S upstream of the maintenance occurrence cell Sa where the workpiece W can be stored, the workpiece W cannot be received by the logistics transport route H. Therefore, if there is no cell S upstream of the maintenance occurrence cell Sa where the workpiece W can be stored, the logistics transport route H cannot accept the workpiece W that is being supplied to the logistics transport route H from upstream, and the workpiece W is dropped.

In addition, in formula (1), v _{(t, i)} is a speed command value for the i-th cell S at time t. N is the total number of cells S. Furthermore, A, B, and C are hyperparameters.

In equation (1), the first term on the right hand side represents the transportation completion reward obtained when the transportation of the workpiece W is completed on the logistics transportation route H. In addition, the second term on the right hand side of equation (1) represents the acceptance refusal penalty given when the logistics transportation route H cannot accept the workpiece W. In addition, the third term on the right hand side of equation (1) represents the energy consumption increase penalty that increases as the amount of energy consumption related to the transportation speed increases.

In other words, in this embodiment, the reward calculation unit 2c calculates a "reward" based on a transport completion reward obtained when the transport of the work W is completed on the logistics transport route H, an acceptance refusal penalty given when the work W cannot be accepted on the logistics transport route H, and an energy consumption increase penalty that increases as the amount of energy consumption related to the transport speed increases.

In the logistics transport speed control agent learning device 1 of this embodiment, the first term on the right side of equation (1) can train the learning agent 3 to complete the transport of the workpiece W on the logistics transport route H. In addition, in the logistics transport speed control agent learning device 1 of this embodiment, the second term on the right side of equation (1) can train the learning agent 3 to prevent the workpiece W from being dropped. In addition, in the logistics transport speed control agent learning device 1 of this embodiment, the third term on the right side of equation (1) can train the learning agent 3 to suppress an increase in energy consumption related to the transport speed.

Therefore, in the logistics transport speed control agent learning device 1 of this embodiment, the learning agent 3 can be trained to prioritize completing the transport of the work W on the logistics transport route H and not causing the work W to be dropped, and to suppress an increase in energy consumption related to the transport speed.

Also, as shown in the third term on the right side of equation (1), in this embodiment, the reward calculation unit 2c calculates the energy consumption increase penalty based on the sum of the squared values of the speed command values for each cell S (transport unit). In other words, the reward calculation unit 2c calculates the energy consumption increase penalty based on the speed command value for each cell S (transport unit). The larger the speed command value, the more energy is consumed by the cell S (transport unit). Therefore, by calculating the energy consumption increase penalty based on the speed command value for each cell S (transport unit), the penalty can be changed according to the speed of the transport unit.

In this embodiment, the penalty for increased energy consumption is calculated based on the sum of the squared values of the speed command values for each cell S (transport unit). By adopting such a configuration, the penalty for attempting to further increase the transport speed when the transport unit's transport speed is high is greater than the penalty for attempting to increase the transport speed when the transport unit's transport speed is zero or low. For this reason, for example, in the transport unit, it is possible to quickly increase the transport speed from a state in which the workpiece W is stopped or the transport speed of the workpiece W is low to a certain speed, while the learning agent 3 can be trained to prevent an action that attempts to further increase the transport speed of the workpiece W when it is already high.

In this embodiment, the learning agent 3 performs reinforcement learning based on the PPO (Proximal Policy Optimization) algorithm. Note that the method of reinforcement learning in the learning agent 3 is not limited to the PPO algorithm. For example, the learning agent 3 may perform reinforcement learning based on the Q-learning method, the Soft Actor-Critic method, the Temporal Difference Learning method, etc.

In this embodiment, the learning agent 3 includes a value estimation network 3a, a strategy network 3b, and a network update unit 3c. The value estimation network 3a estimates the state value using the "state" calculated by the transport path simulator 2. The strategy network 3b uses the state value estimated by the value estimation network 3a and the "state" calculated by the transport path simulator 2 to determine the next "action" (speed command value for each cell S).

The network update unit 3c updates the value estimation network 3a and the policy network 3b based on the "reward" calculated by the transport path simulator 2. For example, the network update unit 3c updates the parameters of the value estimation network 3a and the policy network 3b so as to maximize the obtained "reward" according to the update formula of the PPO algorithm. For example, the network update unit 3c updates the value estimation network 3a and the policy network 3b by obtaining a series of accumulated rewards from the series of "reward" calculated by the transport path simulator and obtaining a series of advantages using the series of state values.

The performance evaluation unit 4 judges whether the learning agent 3 satisfies a predetermined performance. For example, the performance evaluation unit 4 judges that the learning agent 3 satisfies a predetermined performance when the energy consumption related to the transport speed falls below a predetermined threshold, assuming that the transport of the workpiece W on the logistics transport route H is completed and that the workpiece W is not dropped.

If the performance evaluation unit 4 determines that the learning agent 3 meets the predetermined performance, the learning agent 3 is marked as having learned, and the reinforcement learning of the learning agent 3 is terminated. On the other hand, if the performance evaluation unit 4 determines that the learning agent 3 does not meet the predetermined performance, the learning agent 3 is not marked as having learned, and the reinforcement learning of the learning agent 3 continues.

The hyperparameter setting unit 5 sets hyperparameters in the reward calculation formula (e.g., the above-mentioned formula (1)) used by the reward calculation unit 2c. For example, the learning rate of the learning agent 3 and the coefficients used in the reward calculation formula are set as hyperparameters by the hyperparameter setting unit 5.

The learning rate of the learning agent 3 is a parameter that determines the degree of updating in the network update unit 3c. In addition, when the above formula (1) is used as the reward calculation formula, the coefficients A, B, and C in formula (1) are hyperparameters.

Furthermore, the hyperparameter setting unit 5 updates the hyperparameters in the reward calculation formula when the performance evaluation unit 4 determines that the learning agent 3 does not satisfy the predetermined performance. In other words, when the above-mentioned formula (1) is used as the reward calculation formula, the hyperparameter setting unit 5 updates the coefficients A, B, and C in formula (1) when the performance evaluation unit 4 determines that the learning agent 3 does not satisfy the predetermined performance.

The hyperparameter setting unit 5 determines the hyperparameters based on a pre-stored algorithm. For example, a Bayesian optimization algorithm can be used as this algorithm.

Next, the operation of the logistics transport speed control agent learning device 1 of this embodiment (logistics transport speed control agent learning method) will be described with reference to the flowchart in Figure 5.

In the following description, it is assumed that modeling of the logistics transport route H and definition of the cell S by the cell definition unit 2a have already been completed.
5, first, hyperparameters are determined (step S1). Here, the hyperparameter setting unit 5 determines hyperparameters (the learning rate of the learning agent 3 and the coefficients used in the reward calculation formula) used in the reinforcement learning of the learning agent 3.

Next, the transport path simulator 2 and the learning agent 3 are initialized (step S2). Here, if the transport path simulator 2 or the learning agent 3 has changed from its initial settings due to the previous processing, the transport path simulator 2 or the learning agent 3 is returned to its initial setting state.

In other words, when the hyperparameters are determined in step S1, the transport path simulator 2 and the learning agent 3 are initialized in step S2. Therefore, in this embodiment, when the hyperparameters are updated (when returning to step S1 from step S8 described later), the transport path simulator 2 and the learning agent 3 are returned to their initial settings.

When the initialization of the transport route simulator 2 and the learning agent 3 is completed, the time step t is set to 0 (step S3), and a data collection step (step S4) is performed. In the data collection step, the transport route control unit 2b calculates the overall "state" of the logistics transport route H based on the "action" input from the learning agent 3. Note that, for example, if there is no "action" input from the learning agent 3 in the initial state, the transport route control unit 2b outputs the "state" stored as the initial value.

In addition, in the data collection step, the "action" (speed command value of each cell S) that the learning agent 3 will take in the next time step is calculated based on the "state" input from the transport path simulator 2. Here, the learning agent 3 estimates the state value for the input "state" using the value estimation network 3a. Furthermore, the learning agent 3 selects the "action" to be taken after a unit time based on the state value and the "state".

In the data collection step, when an "action" is input from the learning agent 3 to the transport path simulator 2, the transport path simulator 2 calculates the "state" after a unit of time according to the operation defined in the cell definition section 2a, and calculates the "reward" for the current time step based on the transition of the "state" between the present and after the unit of time.

In this embodiment, the transport path simulator 2 calculates a "reward" in the reward calculation unit 2c based on a transport completion reward obtained when the transport of the work W is completed on the logistics transport path H, an acceptance refusal penalty given when the work W cannot be accepted on the logistics transport path H, and an energy consumption increase penalty that increases as the amount of energy consumption related to the transport speed increases.

In the data collection step, the calculation of the "state" and "reward" in the transport path simulator 2 and the calculation of the "action" in the learning agent 3 are performed a predetermined number of times. In the data collection step, the "reward" obtained in the above process is collected as data in the learning agent 3.

Next, a parameter update step (step S5) is performed. In the parameter update step, the network update unit 3c updates the parameters of the value estimation network 3a and the policy network 3b of the learning agent 3 based on the "reward" collected in the data collection step. The network update unit 3c updates the parameters of the value estimation network 3a and the policy network 3b in accordance with the update formula of the PPO algorithm so as to maximize the obtained "reward". Here, for example, the network update unit 3c updates the value estimation network 3a and the policy network 3b by obtaining a series of accumulated rewards from the series of collected "reward" and obtaining a series of advantages using the series of state values.

When the parameter update step is completed, the time step is updated (step S6). Here, the current time step t is incremented by one. Then, it is determined whether the time step updated in step S6 is equal to or greater than a predetermined maximum number of time steps T (step S7).

If the updated time step is not equal to or greater than the maximum number of time steps T, the process returns to step S4. On the other hand, if the updated time step is equal to or greater than the maximum number of time steps T, a determination is made as to whether the learning agent 3 has sufficient performance (step S8). Here, the performance evaluation unit 4 evaluates and judges the performance of the learning agent 3.

If it is determined in step S8 that the performance of the learning agent 3 is sufficient, the learning agent 3 is deemed to have completed learning, and the reinforcement learning of the learning agent 3 is terminated. On the other hand, if it is determined in step S8 that the performance of the learning agent 3 is not sufficient, the process returns to step S1 again to determine the hyperparameters. At this time, the hyperparameters are updated by an algorithm such as the Bayesian optimization method.

The determination of the hyperparameters in step S1 may be performed, for example, by an operator or an external device. In such a case, it is also possible not to provide the hyperparameter setting unit 5 in the logistics transport speed control agent learning device 1.

In step S8, the learning agent 3 that is determined to have learned is installed in the logistics facility and controls the speed of the logistics transport route installed in the logistics facility. The learned learning agent 3 performs speed control to prevent the energy consumption related to the transport speed from increasing, with priority given to completing the transport of the workpiece on the logistics transport route and not causing the workpiece to be dropped. Therefore, by using the learning agent 3, it is possible to reliably transport the workpiece in the logistics facility and reduce energy consumption.

For example, when there is a period during which no maintenance is performed on the transport unit or there is sufficient time before maintenance is performed, such a trained learning agent 3 controls the speed of the transport unit so that no drops occur and energy consumption is reduced, without exceeding the specified transport time.

Furthermore, for example, when maintenance of a transport unit is approaching, the trained learning agent 3 increases the transport speed on the upstream side before the transport unit undergoes maintenance, thereby reducing the number of workpieces located upstream of the transport unit that is the subject of maintenance. For example, when a transport unit is undergoing maintenance, the trained learning agent 3 slows down the transport speed of the workpieces upstream of the transport unit, thereby controlling the speed of the transport unit so that energy consumption is reduced without causing drops.

The logistics transport speed control agent learning device 1 of this embodiment as described above performs reinforcement learning on the learning agent 3 that controls the transport speed of multiple transport units that form a logistics transport route. The logistics transport speed control agent learning device 1 of this embodiment includes a transport route simulator 2 that uses a modeled logistics transport route H to calculate the state of the logistics transport route H and a reward based on the state. The logistics transport speed control agent learning device 1 of this embodiment also includes a learning agent 3 that learns about the transport speed so that the evaluation based on the reward becomes larger. The transport route simulator 2 calculates the reward based on the transport completion reward obtained when the transport of the work W is completed on the logistics transport route H, the acceptance refusal penalty given when the logistics transport route H cannot accept the work W, and the energy consumption increase penalty that increases as the energy consumption related to the transport speed increases.

According to the logistics transport speed control agent learning device 1 of this embodiment, the reward calculated by the transport route simulator 2 is input to the learning agent 3, and the learning agent 3 learns based on the reward. The reward is calculated based on a transport completion reward obtained when the transport of a workpiece is completed on the logistics transport route, an acceptance refusal penalty given when the workpiece cannot be accepted on the logistics transport route, and an energy consumption increase penalty that increases as the amount of energy consumption related to the transport speed increases. In other words, the reward input to the learning agent 3 decreases as the amount of energy consumption related to the transport speed increases.

Therefore, according to the logistics transport speed control agent learning device 1 of this embodiment, the learning agent 3 learns to reduce energy consumption related to the transport speed. Furthermore, the reward input to the learning agent 3 decreases when the logistics transport route cannot accept work. For this reason, in the logistics transport speed control agent learning device 1 of this embodiment, the learning agent 3 learns to avoid cases where the logistics transport route cannot accept work. Therefore, according to the logistics transport speed control agent learning device 1 of this embodiment, it is possible to reduce energy consumption while preventing the logistics facility from being unable to accept work due to congestion on the transport route.

In addition, in the logistics transport speed control agent learning device 1 of this embodiment, the modeled logistics transport route H has multiple control target parts (first control target part T1, second control target part T2, and third control target part T3) that include a single or multiple transport units. Furthermore, the learning agent 3 controls the transport speed for each control target part. This makes it possible to make control easier than controlling all the transport units individually.

In addition, in the logistics transport speed control agent learning device 1 of this embodiment, the transport route simulator 2 calculates the energy consumption increase penalty based on the speed command value for each transport unit. Therefore, the larger the speed command value, the more energy the transport unit consumes. Therefore, by calculating the energy consumption increase penalty based on the speed command value for each transport unit, the penalty can be changed according to the speed of the transport unit.

In addition, in the logistics transport speed control agent learning device 1 of this embodiment, the transport path simulator 2 calculates the energy consumption increase penalty based on the sum of the squared values of the speed command values for each transport unit. Therefore, the penalty for attempting to further increase the transport speed when the transport unit's transport speed is high is greater than the penalty for attempting to increase the transport speed when the transport unit's transport speed is zero or low. Therefore, for example, in the transport unit, it is possible to quickly increase the transport speed from a state in which the workpiece W is stopped or the transport speed of the workpiece W is low to a certain speed, while the learning agent 3 can be trained to prevent an action that attempts to further increase the transport speed of the workpiece W from a state in which the transport speed is already high.

In addition, in the logistics transport speed control agent learning device 1 of this embodiment, a maintenance occurrence transport unit (maintenance occurrence cell Sa) that may experience a downtime period due to maintenance during the transport of work W on the logistics transport route H is included in multiple transport units that form the logistics transport route, and a single controlled object part contains one or less maintenance occurrence transport units (maintenance occurrence cell Sa). In this way, by limiting the number of maintenance occurrence cells Sa included in a single controlled object part to one or less, it is possible to prevent the transport speed of a single controlled object part from being affected by multiple maintenance occurrence cells Sa.

The logistics transport speed control agent learning method of this embodiment as described above causes reinforcement learning of the learning agent 3 that controls the transport speed of multiple transport units that form a logistics transport route. In the logistics transport speed control agent learning method of this embodiment, the transport route simulator 2 calculates the state of the logistics transport route H and the reward based on the state using the modeled logistics transport route H. In addition, in the logistics transport speed control agent learning method of this embodiment, the learning agent 3 learns about the transport speed so that the evaluation based on the reward is increased, and the transport route simulator 2 calculates the reward. Furthermore, the transport route simulator 2 calculates the reward based on the transport completion reward obtained when the transport of the work is completed on the logistics transport route, the acceptance refusal penalty given when the logistics transport route cannot accept the work, and the energy consumption increase penalty that increases due to an increase in the energy consumption related to the transport speed.

According to the logistics transport speed control agent learning method of this embodiment, the reward calculated by the transport route simulator 2 is input to the learning agent 3, and the learning agent 3 learns based on the reward. The reward is calculated based on a transport completion reward obtained when the transport of a workpiece is completed on the logistics transport route, an acceptance refusal penalty given when the workpiece cannot be accepted on the logistics transport route, and an energy consumption increase penalty that increases as the amount of energy consumption related to the transport speed increases. In other words, the reward input to the learning agent 3 decreases as the amount of energy consumption related to the transport speed increases.

Therefore, according to the logistics transport speed control agent learning method of this embodiment, the learning agent 3 learns to reduce energy consumption related to the transport speed. Furthermore, the reward input to the learning agent 3 decreases when the logistics transport route cannot accept work. For this reason, in the logistics transport speed control agent learning method of this embodiment, the learning agent 3 learns to avoid cases where the logistics transport route cannot accept work. Therefore, according to the logistics transport speed control agent learning method of this embodiment, it is possible to reduce energy consumption while preventing the logistics facility from being unable to accept work due to congestion on the transport route.

In addition, in this embodiment, as described above, the learning program P causes the computer to function as a logistics transport speed control agent learning device 1. This learning program P causes the computer to function as a transport route simulator 2 and a learning agent 3. In other words, the learning program P causes the computer to function as a logistics transport speed control agent learning device 1 that performs reinforcement learning on the learning agent 3 that controls the transport speed in multiple transport units that form a logistics transport route. The learning program P also causes the computer to function as a transport route simulator 2 that uses a modeled logistics transport route H to calculate the state of the logistics transport route H and a reward based on the state. The learning program P also causes the computer to function as a learning agent 3 that learns about the transport speed so that the evaluation based on the reward becomes large. Furthermore, when the learning program P causes the computer to function as the transport route simulator 2, the reward is calculated based on the transport completion reward obtained when the transport of the work W is completed on the logistics transport route H, the acceptance refusal penalty given when the work W cannot be accepted on the logistics transport route H, and the energy consumption increase penalty that increases due to the increase in energy consumption related to the transport speed.

According to the learning program P of this embodiment, the reward calculated by the transport path simulator 2 is input to the learning agent 3, and the learning agent 3 learns based on the reward. The reward is calculated based on a transport completion reward obtained when the transport of a workpiece is completed on the logistics transport path, an acceptance refusal penalty given when the workpiece cannot be accepted on the logistics transport path, and an energy consumption increase penalty that increases as the amount of energy consumption related to the transport speed increases. In other words, the reward input to the learning agent 3 decreases as the amount of energy consumption related to the transport speed increases.

Therefore, according to the learning program P of this embodiment, the learning agent 3 learns to reduce energy consumption related to the transport speed. Furthermore, the reward input to the learning agent 3 decreases when the logistics transport route cannot accept the work. For this reason, in the learning program P of this embodiment, the learning agent 3 learns to avoid cases where the logistics transport route cannot accept the work. Therefore, according to the learning program P of this embodiment, it is possible to reduce energy consumption while preventing the logistics facility from being unable to accept the work due to congestion on the transport route.

The above describes a preferred embodiment of the present invention with reference to the attached drawings, but it goes without saying that the present invention is not limited to the above embodiment. The shapes and combinations of the components shown in the above embodiment are merely examples, and various modifications can be made based on design requirements, etc., without departing from the spirit of the present invention.

For example, in the above embodiment, a configuration was adopted in which the reward calculation formula calculates the energy consumption increase penalty based on the speed command value for each transport unit. However, the present invention is not limited to this. For example, the amount of energy consumed on a logistics transport route also changes depending on the acceleration of the transport unit. In other words, when the acceleration of the transport unit is large, the period until the target speed is reached is shorter, but the energy consumption is greater. On the other hand, when the acceleration of the transport unit is small, the period until the target speed is reached is longer, but the energy consumption is smaller. For this reason, it is also possible to adopt a configuration in which the transport route simulator 2 calculates the energy consumption increase penalty based on the acceleration of each transport unit.

In the above embodiment, the energy consumption increase penalty is calculated based on the sum of the values obtained by squaring the speed command values for each transport unit. However, the present invention is not limited to this. For example, the energy consumption increase penalty may be calculated based on the sum of the values obtained by raising the speed command value for each transport unit to a power of three or more. The energy consumption increase penalty may also be calculated for each transport unit. The energy consumption increase penalty may also be calculated using the transport unit with the largest speed command value among multiple transport units.

In the above embodiment, a configuration has been described in which the transport completion reward obtained when the transport of a workpiece is completed on the logistics transport route in the reward calculation formula, and the acceptance refusal penalty given when the workpiece cannot be accepted on the logistics transport route, are given as 0 or 1. However, the present invention is not limited to this. For example, it is also possible to adopt a configuration in which both or either one of the transport completion reward obtained when the transport of a workpiece is completed on the logistics transport route and the acceptance refusal penalty given when the workpiece cannot be accepted on the logistics transport route take an intermediate value between 0 and 1.

In the above embodiment, the reward calculation formula had three terms: a first term indicating a transport completion reward obtained when the transport of a workpiece is completed on the logistics transport route; a second term indicating an acceptance refusal penalty given when the workpiece cannot be accepted on the logistics transport route; and a third term indicating an energy consumption increase penalty that increases as the amount of energy consumed related to the transport speed increases. However, the present invention is not limited to this, and it is also possible to use a reward calculation formula having four or more terms.

1...Logistics transport speed control agent learning device, 2...Transport route simulator, 2a...Cell definition unit, 2b...Transport route control unit, 2c...Reward calculation unit, 3...Learning agent, 3a...Value estimation network, 3b...Policy network, 3c...Network update unit, 4...Performance evaluation unit, 5...Hyper parameter setting unit, H...Logistics transport route, P...Learning program (logistics transport speed control agent learning program), S...Cell (transport unit), Sa...Maintenance occurrence cell (maintenance occurrence transport unit), T1...First control target unit (control target unit), T2...Second control target unit (control target unit), T3...Third control target unit (control target unit), W...Work

Claims (8)

A logistics transport speed control agent learning device that performs reinforcement learning on a learning agent that controls transport speeds of a plurality of transport units that form a logistics transport route, comprising:
A transport route simulator that uses the modeled logistics transport route to calculate a state of the logistics transport route and a reward based on the state;
The learning agent learns about the conveying speed so as to increase an evaluation based on the reward,
The transport route simulator calculates the reward based on a transport completion reward obtained when the transport of a workpiece is completed on the logistics transport route, an acceptance refusal penalty imposed when the workpiece cannot be accepted on the logistics transport route, and an energy consumption increase penalty that increases as the amount of energy consumption related to the transport speed increases. The modeled logistics transport route has a plurality of control targets including one or more transport units;
2. The logistics transport speed control agent learning device according to claim 1, wherein the learning agent controls the transport speed for each of the control target parts. 3. The logistics transportation speed control agent learning device according to claim 2, wherein the transportation route simulator calculates the energy consumption increase penalty based on a speed command value for each of the transportation units. The logistics transport speed control agent learning device according to claim 3, characterized in that the transport path simulator calculates the energy consumption increase penalty based on the sum of the squared values of the speed command values for each transport unit. 3. The logistics and transportation speed control agent learning device according to claim 2, wherein the transportation route simulator calculates the energy consumption increase penalty based on the acceleration of each of the transportation units. a maintenance occurrence transport unit that may be stopped for maintenance during the transport of a workpiece on the logistics transport path is included in the plurality of transport units that form the logistics transport path;
6. The logistics transport speed control agent learning device according to claim 2, wherein the number of the maintenance occurrence transport units included in a single controlled object is one or less. A logistics transport speed control agent learning method for performing reinforcement learning on a learning agent that controls transport speeds of a plurality of transport units that form a logistics transport route, comprising:
Calculating a state of the logistics transportation route and a reward based on the state using the modeled logistics transportation route by a transportation route simulator;
the learning agent learns about the transport speed so as to increase an evaluation based on the reward;
a transport route simulator that calculates the reward based on a transport completion reward obtained when the transport of a workpiece is completed on the logistics transport route, an acceptance refusal penalty that is imposed when the workpiece cannot be accepted on the logistics transport route, and an energy consumption increase penalty that increases as the amount of energy consumption related to the transport speed increases. A logistics transport speed control agent learning program that causes a computer to function as a logistics transport speed control agent learning device that performs reinforcement learning on a learning agent that controls transport speeds in a plurality of transport units that form a logistics transport route, the program comprising:
The computer is caused to function as a transport route simulator that calculates a state of the logistics transport route and a remuneration based on the state by using the modeled logistics transport route;
causing the computer to function as the learning agent to learn about the conveying speed so as to increase an evaluation based on the reward;
A logistics transport speed control agent learning program characterized in that, when the computer is made to function as the transport route simulator, the reward is calculated based on a transport completion reward obtained when the transport of a workpiece is completed on the logistics transport route, an acceptance refusal penalty given when the workpiece cannot be accepted on the logistics transport route, and an energy consumption increase penalty that increases as the amount of energy consumption related to the transport speed increases.

Images