JP7084520B2 - Simulation equipment, simulation method and simulation program - Google Patents

Description

The present invention relates to a simulation apparatus, a simulation method and a simulation program.

In recent years, it has been proposed to use machine learning such as a reinforcement learning device for device control in various environments such as factories, plants, buildings and data centers. Such machine learning is strengthened by modeling the input and output of various devices such as control devices and air conditioners and building a simulation environment for various environments such as factories, plants, buildings and data centers. The control parameters of the learner are being searched. The search for control parameters is performed by a skilled worker through trial and error.

Kazuya Sato, 2 authors, "First Control Engineering", Kodansha Co., Ltd., October 2010 Richard S. Sutton, 1 author, "Reinforcement Learning", Morikita Publishing Co., Ltd., December 2000 Volodymyr Mnih 6 people outside, "Playing Atari with Deep Reinforcement Learning", [Online], December 2013, NIPS Deep Learning Workshop 2013, [Search January 29, 2019], Internet <https://arxiv.org /pdf/1312.5602.pdf>

However, in the past, there is a problem that it has been difficult to easily construct a high-precision simulation environment. For example, the construction of the simulation environment and the search for control parameters were performed separately, so when searching for control parameters, if the simulation environment is to be changed, the operator is required to instruct how to change it. Be done. Therefore, when the construction of the simulation environment and the search for the control parameters are repeated, a lot of labor and time are required, and it is difficult to easily construct a highly accurate simulation environment.

In order to solve the above-mentioned problems and achieve the object, the simulation apparatus of the present invention learns using the first reception unit that accepts the input of the training data and the received training data, and generates a prediction model. A second reception unit that accepts one or more of the generation unit, the evaluation data used for the simulation, the generated prediction model, and the reinforcement learning device that performs reinforcement learning in the simulation. Based on the received state of the arrangement, the execution unit that executes the simulation using the evaluation data, the prediction model, and the reinforcement learning device, and the learning result of the reinforcement learning device in the simulation. Based on this, it is determined whether or not to regenerate the prediction model, and when it is determined that the prediction model is to be regenerated, a determination unit that instructs the generation unit to regenerate the prediction model. It is characterized by having.

Further, the simulation method of the present invention includes a first reception step of accepting input of training data, a generation step of learning using the received training data to generate a prediction model, and evaluation data used for simulation. Based on the second reception step of accepting one or more arrangements of the generated prediction model and the reinforcement learning device for performing reinforcement learning in the simulation, and the state of the received arrangements. Whether to regenerate the prediction model based on the evaluation data, the execution step of executing the simulation using the prediction model, the reinforcement learning device, and the learning result of the reinforcement learning device in the simulation. When it is determined whether or not the prediction model is to be regenerated, the simulation device is characterized in that the simulation apparatus executes a determination step of instructing the generation step to regenerate the prediction model. ..

Further, the simulation program of the present invention includes a first reception step that accepts input of training data, a generation step that learns using the received training data and generates a prediction model, and evaluation data used for simulation. , The second reception step that accepts one or more arrangements of the generated prediction model and the reinforcement learning device that performs reinforcement learning in the simulation, and the state of the received arrangement. Whether to regenerate the prediction model based on the evaluation data, the execution step of executing the simulation using the prediction model, the reinforcement learning device, and the learning result of the reinforcement learning device in the simulation. When it is determined whether or not the prediction model is to be regenerated, the computer is made to execute a determination step instructing the generation step to regenerate the prediction model.

According to the present invention, there is an effect that a highly accurate simulation environment can be easily constructed.

FIG. 1 is a block diagram showing an example of the configuration of the simulation apparatus according to the first embodiment. FIG. 2 is a diagram showing an example of an arrangement screen. FIG. 3 is a diagram showing another example of the arrangement screen. FIG. 4 is a diagram illustrating an example of learning in the predicted image model. FIG. 5 is a flowchart showing an example of the simulation process in the first embodiment. FIG. 6 is a diagram showing an example of a computer that executes a simulation program.

Hereinafter, embodiments of the simulation apparatus, simulation method, and simulation program disclosed in the present application will be described in detail with reference to the drawings. The simulation apparatus, simulation method, and simulation program according to the present application are not limited to this embodiment.

[First Embodiment]
In the following embodiments, the configuration of the simulation apparatus 100 and the processing flow of the simulation apparatus 100 according to the first embodiment will be described in order, and finally, the effects of the first embodiment will be described.

[Simulation device configuration]
First, the configuration of the simulation device 100 will be described with reference to FIG. FIG. 1 is a block diagram showing an example of the configuration of the simulation apparatus according to the first embodiment. The simulation device 100 receives input of learning data from, for example, another information processing device. The simulation device 100 learns using the received learning data by the generation unit, and generates a prediction model. The simulation device 100 is one or more of the evaluation data used for the simulation, the generated prediction model, the reinforcement learning device for performing reinforcement learning in the simulation, and the imitation learning device for performing imitation learning in the simulation. Accept the placement of. The simulation device 100 executes a simulation using the evaluation data, the prediction model, the reinforcement learner, and the imitation learner based on the received arrangement state. The simulation device 100 determines whether or not to regenerate the prediction model based on the learning result of the reinforcement learning device in the simulation. When the simulation device 100 determines that the prediction model is to be regenerated, the simulation device 100 instructs the generation unit to regenerate the prediction model. As a result, the simulation device 100 can easily construct a highly accurate simulation environment.

As shown in FIG. 1, the simulation device 100 includes a communication unit 110, a display unit 111, an operation unit 112, a storage unit 120, and a control unit 130. In addition to the functional units shown in FIG. 1, the simulation device 100 may have various functional units of known computers, such as various input devices and voice output devices.

The communication unit 110 is realized by, for example, a NIC (Network Interface Card) or the like. The communication unit 110 is a communication interface that is connected to another information processing device by wire or wirelessly via a network (not shown) and controls information communication with the other information processing device. The communication unit 110 receives learning data and evaluation data from, for example, another information processing device. The communication unit 110 outputs the received learning data and evaluation data to the control unit 130. The communication unit 110 may receive, for example, a prediction model, a first trained model, a second trained model, and the like, which will be described later, from another information processing device.

The display unit 111 is a display device for displaying various information. The display unit 111 is realized by, for example, a liquid crystal display or the like as a display device. The display unit 111 displays various screens such as a display screen input from the control unit 130.

The operation unit 112 is an input device that receives various operations from the user of the simulation device 100. The operation unit 112 is realized by, for example, a keyboard, a mouse, or the like as an input device. The operation unit 112 outputs the operation input by the user to the control unit 130 as operation information. The operation unit 112 may be realized by a touch panel or the like as an input device, or the display device of the display unit 111 and the input device of the operation unit 112 may be integrated.

The storage unit 120 is realized by, for example, a RAM (Random Access Memory), a semiconductor memory element such as a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 120 includes a learning data storage unit 121, a prediction model storage unit 122, an evaluation data storage unit 123, an arrangement information storage unit 124, and a learned model storage unit 125. Further, the storage unit 120 stores information used for processing in the control unit 130.

The learning data storage unit 121 stores, for example, sensor information output by various sensors such as temperature and pressure provided in each unit of the plant as learning data. The training data is used for machine learning of the prediction model after the prediction target is preprocessed.

The prediction model storage unit 122 stores a prediction model in which preprocessed learning data is trained by machine learning so that output values of various parameters to be predicted can be obtained. The prediction model storage unit 122 outputs various parameters to be predicted based on the evaluation data in the simulation. Further, in the prediction model, for example, deep learning is performed using a neural network using sensor information as a feature quantity. As the neural network, for example, CNN (Convolutional Neural Network) can be used. That is, the prediction model stores, for example, various parameters (weighting coefficients) of the neural network as learning parameters. When a prediction model is acquired from another information processing device via the communication unit 110, the prediction model may not perform new machine learning in the simulation.

The evaluation data storage unit 123 stores, for example, sensor information output by various sensors such as temperature and pressure provided in each part of the plant as evaluation data. The evaluation data may be obtained by dividing the acquired sensor information data into learning data and evaluation data.

The placement information storage unit 124 stores the placement information representing the placement state of any one or more of the evaluation data received from the user, the prediction model, the reinforcement learner, and the imitation learner. do. The placement information also includes connection information between each placed element.

The trained model storage unit 125 stores the first trained model, which is the learning result of the reinforcement learner in the simulation, and the second trained model, which is the learning result of the imitation learner. The first trained model and the second trained model learn, for example, the opening degree of the valve with respect to the prediction model that outputs the output flow rate according to the open / closed state of the valve to be controlled. The trained model storage unit 125 stores, for example, various parameters of Q-learning as the first trained model. Further, the trained model storage unit 125 stores various parameters as the second trained model, for example, imitating the result when the valve is controlled by PID (Proportional Integral Differential). When the first trained model and the second trained model are acquired from another information processing device via the communication unit 110, the first trained model and the second trained model are subjected to new reinforcement learning and new training in the simulation. It is possible not to perform imitation learning.

The control unit 130 is realized by, for example, using a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like to execute a program stored in an internal storage device using the RAM as a work area. Further, the control unit 130 may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

The control unit 130 includes a first reception unit 131, a setting unit 132, a generation unit 133, a second reception unit 134, an execution unit 135, a determination unit 136, and an output control unit 137. Realize or execute the information processing functions and actions described. The internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 1, and may be any other configuration as long as it is configured to perform information processing described later.

The first reception unit 131 receives input of learning data from another information processing device, for example, via the communication unit 110, based on the user's operation. The first reception unit 131 stores the received learning data in the learning data storage unit 121. Further, the first reception unit 131 receives input of evaluation data from another information processing device, for example, via the communication unit 110, based on the user's operation. The first reception unit 131 stores the received evaluation data in the evaluation data storage unit 123. Further, when the first reception unit 131 receives the start of the setting for the prediction target from the user, the first reception unit 131 outputs a setting instruction to the setting unit 132. The first reception unit 131 may receive a plurality of prediction targets to be set by the user.

A setting instruction is input to the setting unit 132 from the first reception unit 131 or the generation unit 133. Further, a resetting instruction is input to the setting unit 132 from the determination unit 136. When the setting instruction or the resetting instruction is input, the setting unit 132 sets the prediction target. For example, if the prediction target is a valve, the setting unit 132 sets the output flow rate as the output value and sets the opening degree of the valve as the control target. When the resetting instruction is input, the setting unit 132 sets the prediction target under a condition different from the already tried setting condition. The setting conditions are changed, for example, by trying a list of parameters in order or changing the prediction model itself. When the prediction target is set, the setting unit 132 executes preprocessing corresponding to the set prediction target on the learning data of the learning data storage unit 121. The pretreatment includes, for example, replenishment of missing values, processing of abnormal values, standardization, and the like. When the preprocessing is completed, the setting unit 132 outputs a generation instruction to the generation unit 133.

When the generation instruction is input from the setting unit 132, the generation unit 133 reads the learning data from the learning data storage unit 121, performs machine learning, and generates a prediction model. The generation unit 133 performs deep learning such as CNN and generates a prediction model. The generation unit 133 stores the generated prediction model in the prediction model storage unit 122. The generation unit 133 determines whether or not the prediction model has an ungenerated prediction target. When the generation unit 133 determines that the prediction model has an ungenerated prediction target, the generation unit 133 outputs a setting instruction to the setting unit 132 to set the remaining prediction targets. When the generation unit 133 determines that there is no ungenerated prediction target in the prediction model, the generation unit 133 outputs a reception instruction to the second reception unit 134. The setting unit 132 and the generation unit 133 may be integrated. Further, the resetting instruction input from the determination unit 136 to the setting unit 132 indicates the regeneration of the prediction model, and when the setting unit 132 and the generation unit 133 are integrated, they are input to the generation unit after integration. Will be done.

When the reception instruction is input from the generation unit 133, the second reception unit 134 causes the display unit 111 to display the arrangement screen, and receives the arrangement of each element in the simulation environment from the user. The second reception unit 134 receives the arrangement of the generated prediction model as an element to be arranged. Further, the second reception unit 134 receives the arrangement of the evaluation data, the reinforcement learning device, and the imitation learning device as the elements to be arranged. Further, the second reception unit 134 receives, for example, settings such as control targets of the arranged reinforcement learner and imitation learner, that is, connection information between each element. When the second reception unit 134 receives the completion of the arrangement from the user, the second reception unit 134 stores the arrangement information including the arrangement and connection information of each element in the arrangement information storage unit 124.

Here, the arrangement screen will be described with reference to FIGS. 2 and 3. FIG. 2 is a diagram showing an example of an arrangement screen. The arrangement screen 10 shown in FIG. 2 has a selection area 11 for displaying each element to be arranged in the simulation environment, and a construction area 12 for constructing the simulation environment to be simulated. The elements to be arranged include, for example, data 13 corresponding to evaluation data, prediction models C1 to CX corresponding to each prediction model, function processing D1, D2, PID control P1, and reinforcement learning corresponding to a reinforcement learner. 14 and imitation learning 15 corresponding to the imitation learner. In FIG. 2, the PID parameter 16 of the selection area 11 is omitted.

In the construction area 12, for example, each element is arranged by the user performing a drag-and-drop operation for each element from the selection area 11. In the example of FIG. 2, data 13, prediction models C1 to C6, function processing D1, reinforcement learning 14, imitation learning 15, PID control P1 and PID parameter 16 are arranged. The function processing D1 is an element that performs some function processing on the input and outputs the output, and is, for example, a function for obtaining a moving average. Further, the PID control P1 corresponds to the PID control that performs feedback control. The PID parameter 16 is a parameter such as various gains, times, target values, and operation amounts of the PID control P1.

Further, in the example of FIG. 2, the data 13 is connected to the prediction models C1 to C6. The output of the prediction model C1 is connected to the PID control P1 and the PID parameter 16. The output of the PID parameter 16 is connected to the PID control P1. The output of the PID control P1 is connected to the reinforcement learning 14 and the imitation learning 15. Further, the outputs of the prediction models C2 to C6 and the function processing D1 are connected to the reinforcement learning 14 and the imitation learning 15. On the other hand, the control outputs of the reinforcement learning 14 and the imitation learning 15 are connected to the prediction models C1 to C6 and the function processing D1.

FIG. 3 is a diagram showing another example of the arrangement screen. The arrangement screen 20 shown in FIG. 3 is an arrangement in which a state prediction image is generated and input to the arrangement screen 10 of FIG. In the example of FIG. 3, the predicted image 21 and the predicted image model C7 are added to the selection area 11 as elements, and the predicted image 21 and the predicted image model C7 are added to the construction area 12 as compared with the arrangement screen 10. Is arranged.

In the example of FIG. 3, the data 13 is connected to the prediction models C1 to C6. The output of the prediction model C1 is connected to the PID control P1, the PID parameter 16, and the prediction image model C7. The output of the PID parameter 16 is connected to the PID control P1. The output of the PID control P1 is connected to the reinforcement learning 14 and the imitation learning 15. Further, the outputs of the prediction models C2, C3, C5, C6 and the function processing D1 are connected to the reinforcement learning 14 and the imitation learning 15. Further, the outputs of the prediction models C2 and C3 and the function processing D1 are connected to the prediction image model C7. Further, the output of the prediction model C4 is connected to the function process D1.

The prediction image model C7 is trained in the generation unit 133 using GAN (Generative Adversarial Networks) with the input as a latent variable (feature amount). The prediction image model C7 generates the prediction image 21 based on the outputs of the prediction models C1, C2, C3 and the function processing D1 in the simulation. That is, in the construction region 12, the output of the predicted image model C7 is connected to the predicted image 21. The predicted image 21 is image data and is connected to the reinforcement learning 14 and the imitation learning 15. That is, in the example of FIG. 3, the reinforcement learning 14 and the imitation learning 15 also learn the prediction image generated based on the output of the prediction model controlled by its own prediction value. The learning of the predicted image model C7 is not limited to GAN, and for example, CNN, MLP (MultiLayer Perceptron) or the like is used, and the output results of various sensors or the like are used as images for supervised learning. May be good.

Here, the generation of the predicted image will be described with reference to FIG. FIG. 4 is a diagram illustrating an example of learning in the predicted image model. In the learning example 30 shown in FIG. 4, when the latent variable (feature amount) 32 is input to the generator 31 which is a generator, the generator 31 generates a fake image 33 and causes the discriminator 34 which is a discriminator. Output. The classifier 34 outputs the probability that the fake image 33 is a real image to the sigmoid function 35. Further, the real image 36 is alternately input to the classifier 34 with the fake image 33. The sigmoid function 35 outputs to the determination device 37 whether the fake image 33 is genuine (1) or fake (0). The determination device 37 determines whether the input genuine (1) or fake (0) is the correct answer, and trains the generator 31 and the classifier 34 by error back propagation.

Next, in the learning example 30 shown in FIG. 4, the trained generator 31 is used as the image generation model 31a. That is, the image generation model 31a is a predictive image model in which machine learning is performed using image data as training data. The image generation model 31a corresponds to the prediction image model C7 in FIG. 3, and generates and outputs the prediction image 40 by inputting the simulator output value 39 output from the prediction model 38. The generated prediction image 40 can be used to control the prediction model 38, for example, through learning with a reinforcement learning device. Further, the predicted image 40 may be displayed for confirmation of the output result.

Returning to the description of FIG. When the execution unit 135 receives the simulation start instruction from the user, the execution unit 135 refers to the arrangement information storage unit 124 and executes the simulation based on the arrangement information. That is, the execution unit 135 executes reinforcement learning of the reinforcement learner and imitation learning of the imitation learner, and stores the first trained model and the second trained model in the trained model storage unit 125. Further, when the arrangement information includes the predicted image model, the execution unit 135 outputs the predicted image data to the output control unit 137 during the execution of the simulation. That is, the execution unit 135 outputs the predicted value of the prediction model according to the control output of the reinforcement learner, that is, the prediction image data corresponding to the prediction value of the prediction model based on the learning result of the reinforcement learner to the output control unit 137. do. For example, when the simulation of a predetermined number of times is completed, the execution unit 135 outputs a determination instruction to the determination unit 136. Further, when the arrangement information includes the predicted image model, the execution unit 135 outputs the predicted image data at the time of completion of the simulation to the output control unit 137. As the predetermined number of simulations, for example, a preset number of times or a number of times specified by the user can be used.

When the predicted image model is included in the arrangement information, the determination unit 136 inputs the evaluation received for the displayed predicted image data from the output control unit 137. When the determination instruction is input from the execution unit 135, the determination unit 136 determines whether or not the evaluation of the predicted image data has been accepted from the user. That is, when the arrangement information includes the predicted image model, the determination unit 136 determines whether or not the predicted image data displayed on the display unit 111 by the output control unit 137 has been evaluated by the user. If the determination unit 136 determines that the evaluation of the predicted image data is not accepted, or if the arrangement information does not include the predicted image model, the determination unit 136 regenerates the predicted model based on the learning result of the reinforcement learner. Judge whether or not.

For example, the determination unit 136 is based on the evaluation value of the error between the predicted value of the prediction model based on the learning result of the reinforcement learning device in the real environment and the predicted value of the prediction model based on the learning result of the reinforcement learning device in the simulation. Determine whether to regenerate the prediction model. As the evaluation value of the error, a mean squared error (RMSE: Root Mean Square Error) or a mean squared error (MSE: Mean Squared Error) can be used.

Further, for example, the determination unit 136 determines whether or not to regenerate the prediction model based on the degree of correlation between the prediction value of the prediction model based on the learning result of the reinforcement learning device in the simulation and the measured value. As the degree of correlation, for example, a correlation coefficient may be used, or a loss function may be used. That is, the determination unit 136 determines whether or not to generate a new prediction model when the result of reinforcement learning does not improve for a certain period of time based on the learning result.

On the other hand, when it is determined that the evaluation of the predicted image data has been accepted, the determination unit 136 determines whether or not to regenerate the prediction model based on the accepted evaluation. The evaluation of the predicted image data can be performed, for example, by a skilled worker evaluating a predicted image corresponding to a real image taken by thermography inside the furnace.

When the determination unit 136 determines that the prediction model is to be regenerated, the determination unit 136 outputs a reset instruction to the setting unit 132. The resetting instruction instructs the setting unit 132 and the generation unit 133 to regenerate the prediction model. If the determination unit 136 determines that the prediction model is not regenerated, the determination unit 136 ends the simulation process. That is, the determination unit 136 completes the construction of the simulation environment.

When the predicted image data is input from the execution unit 135, the output control unit 137 outputs the predicted image data to the display unit 111 and displays it. For example, the output control unit 137 may sequentially display the predicted image data during simulation execution, or may display the predicted image data extracted at predetermined time intervals side by side. Further, the output control unit 137 may display the predicted image data when the simulation is completed. Further, when the output control unit 137 accepts the evaluation of the displayed predicted image data, the output control unit 137 outputs the accepted evaluation to the determination unit 136.

[Processing procedure of simulation device]
Next, the operation of the simulation device 100 according to the first embodiment will be described. FIG. 5 is a flowchart showing an example of the simulation process in the first embodiment.

The first reception unit 131 receives, for example, input of learning data from another information processing device (step S1). The first reception unit 131 stores the received learning data in the learning data storage unit 121. Further, the first reception unit 131 receives input of evaluation data from, for example, another information processing device. The first reception unit 131 stores the received evaluation data in the evaluation data storage unit 123. Further, when the first reception unit 131 receives the start of the setting for the prediction target from the user, the first reception unit 131 outputs a setting instruction to the setting unit 132.

When the setting instruction or the resetting instruction is input, the setting unit 132 sets the prediction target (step S2). When the prediction target is set, the setting unit 132 executes preprocessing corresponding to the set prediction target for the learning data of the learning data storage unit 121 (step S3). When the preprocessing is completed, the setting unit 132 outputs a generation instruction to the generation unit 133.

When the generation instruction is input from the setting unit 132, the generation unit 133 reads the learning data from the learning data storage unit 121, performs machine learning, and generates a prediction model (step S4). The generation unit 133 stores the generated prediction model in the prediction model storage unit 122. The generation unit 133 determines whether or not there is an ungenerated prediction target in the prediction model (step S5). When the generation unit 133 determines that the prediction model has an ungenerated prediction target (step S5: Yes), the generation unit 133 outputs a setting instruction to the setting unit 132 to set the remaining prediction target, and steps. Return to S2. When the generation unit 133 determines that there is no ungenerated prediction target in the prediction model (step S5: No), the generation unit 133 outputs a reception instruction to the second reception unit 134.

When the reception instruction is input from the generation unit 133, the second reception unit 134 causes the display unit 111 to display the arrangement screen, and receives the arrangement of each element in the simulation environment from the user. The second reception unit 134 receives the arrangement of the generated prediction model (step S6). In addition, the second reception unit 134 receives the evaluation data, the reinforcement learning device, and the arrangement of the imitation learning device (step S7). Further, the second reception unit 134 receives settings such as control targets of the arranged reinforcement learner and imitation learner (step S8). When the second reception unit 134 receives the completion of the arrangement from the user, the second reception unit 134 stores the arrangement information including the arrangement and connection information of each element in the arrangement information storage unit 124.

When the execution unit 135 receives the simulation start instruction from the user, the execution unit 135 refers to the arrangement information storage unit 124, executes the simulation based on the arrangement information, and executes reinforcement learning (step S9). The execution unit 135 stores the first trained model of the reinforcement learner and the second trained model of the imitation learner in the trained model storage unit 125. Further, when the arrangement information includes the predicted image model, the execution unit 135 outputs the predicted image data to the output control unit 137 during the execution of the simulation. When the predicted image data is input from the execution unit 135, the output control unit 137 outputs the predicted image data to the display unit 111 and displays it (step S10). When the output control unit 137 receives the evaluation for the displayed predicted image data, the output control unit 137 outputs the received evaluation to the determination unit 136. For example, when the simulation of a predetermined number of times is completed, the execution unit 135 outputs a determination instruction to the determination unit 136.

When the determination instruction is input from the execution unit 135, the determination unit 136 determines whether or not the evaluation of the predicted image data has been accepted from the user (step S11). If the determination unit 136 determines that the evaluation of the predicted image data is not accepted (step S11: No), or if the arrangement information does not include the predicted image model, the determination unit 136 makes a prediction based on the learning result of the reinforcement learner. It is set when it is determined whether or not to regenerate the model (step S12). On the other hand, when it is determined that the evaluation of the predicted image data has been accepted (step S11: Yes), the determination unit 136 sets to determine whether or not to regenerate the prediction model based on the accepted evaluation (step S13). ).

The determination unit 136 determines whether or not to regenerate the prediction model (step S14). When the determination unit 136 determines that the prediction model is to be regenerated (step S14: Yes), the determination unit 136 outputs a reset instruction to the setting unit 132 and returns to step S2. When the determination unit 136 determines that the prediction model is not regenerated (step S14: No), the determination unit 136 ends the simulation process. As a result, the simulation device 100 can easily construct a highly accurate simulation environment. Further, the simulation device 100 can construct the simulation environment in a shorter time than the manual construction.

In the first embodiment, the predicted image model that generates the predicted image data is given as an example of the predicted model, but the predicted voice model that generates the predicted voice data may be used instead of the predicted image data. .. The predictive voice model can be arranged in the same manner as other predictive models on the arrangement screen shown in FIG. 3, for example.

[Effect of the first embodiment]
In this way, the simulation device 100 accepts the input of learning data. Further, the simulation device 100 learns using the received learning data by the generation unit, and generates a prediction model. Further, the simulation device 100 is one of an evaluation data used for the simulation, a generated prediction model, a reinforcement learning device for performing reinforcement learning in the simulation, and an imitation learning device for performing imitation learning in the simulation. Or accept multiple arrangements. Further, the simulation device 100 executes a simulation using the evaluation data, the prediction model, the reinforcement learner, and the imitation learner based on the received arrangement state. Further, the simulation device 100 determines whether or not to regenerate the prediction model based on the learning result of the reinforcement learning device in the simulation. Further, when the simulation device 100 determines that the prediction model is to be regenerated, the simulation device 100 instructs the generation unit to regenerate the prediction model. As a result, the simulation device 100 can easily construct a highly accurate simulation environment. Further, since the simulation device 100 can automate the process from the construction of the simulation environment to the execution of reinforcement learning, it is possible to construct a highly accurate simulation environment in a shorter time than the manual construction. The simulation environment to be constructed is also called a digital twin environment.

Further, the simulation device 100 is based on an evaluation value of an error between the predicted value of the prediction model based on the learning result of the reinforcement learner in the real environment and the prediction value of the prediction model based on the learning result of the reinforcement learner in the simulation. Determine whether to regenerate the prediction model. As a result, the simulation device 100 can improve the accuracy of the simulation environment.

Further, the simulation device 100 determines whether or not to regenerate the prediction model based on the degree of correlation between the predicted value of the prediction model based on the learning result of the reinforcement learning device in the simulation and the measured value. As a result, the simulation device 100 can improve the accuracy of the simulation environment.

Further, the learning data includes image data. Further, the simulation device 100 further generates a predictive image model that generates predictive image data based on the image data as a predictive model. Further, the simulation device 100 accepts the arrangement of the predicted image model. Further, the simulation device 100 executes a simulation including a predicted image model. Further, the simulation device 100 further outputs predicted image data corresponding to the predicted value of the prediction model according to the control output of the reinforcement learning device in the simulation, that is, the predicted value of the prediction model based on the learning result of the reinforcement learning device. As a result, the simulation device 100 can present the simulation situation in an easy-to-understand manner using the predicted image.

Further, the learning data includes voice data. Further, the simulation device 100 further generates a predicted voice model that generates predicted voice data based on the voice data as a prediction model. Further, the simulation device 100 accepts the arrangement of the predicted voice model. Further, the simulation device 100 executes a simulation including a predicted speech model. Further, the simulation device 100 further outputs the predicted value of the prediction model according to the control output of the reinforcement learning device in the simulation, that is, the predicted voice data corresponding to the predicted value of the prediction model based on the learning result of the reinforcement learning device. As a result, the simulation device 100 can present the simulation situation in an easy-to-understand manner using the predicted voice.

Further, the simulation device 100 further accepts evaluations of the output predicted image data or predicted voice data, and determines whether or not to regenerate the prediction model based on the received evaluations. As a result, the simulation device 100 can reflect the experience of a skilled worker in the simulation.

In the first embodiment, machine learning using CNN was performed in the generation of the prediction model, but the present invention is not limited to this. For example, machine learning using RNN (Recurrent Neural Network), SVM (Support Vector Machine), or the like may be performed.

Further, in the first embodiment, the training data is machine-learned to generate a prediction model, and then the reinforcement learning of the reinforcement learning device is performed, but the present invention is not limited to this. For example, the simulation device 100 machine-learns training data to generate a prediction model, and based on the result of executing a simulation using the generated prediction model (result when reinforcement learning is not performed), the prediction model It may be determined whether or not to perform regeneration. Further, for example, when the simulation device 100 acquires a prediction model from another information processing device, the simulation device 100 does not perform machine learning of the prediction model, but regenerates the prediction model based on the result of executing reinforcement learning in the simulation. It may be determined whether or not.

[System configuration, etc.]
Further, each component of each of the illustrated devices is a functional concept, and does not necessarily have to be physically configured as shown in the figure. That is, the specific form of distribution / integration of each device is not limited to the one shown in the figure, and all or part of them may be functionally or physically distributed / physically in any unit according to various loads and usage conditions. Can be integrated and configured. Further, each processing function performed by each device is realized by a CPU or GPU and a program that is analyzed and executed by the CPU or GPU, or as hardware by wired logic. Can be realized.

Further, among the processes described in the above-described embodiment, all or part of the processes described as being automatically performed can be manually performed, or the processes described as being manually performed can be performed. All or part of it can be done automatically by a known method. In addition, the processing procedure, control procedure, specific name, and information including various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified.

[program]
Further, it is also possible to create a program in which the processing executed by the simulation apparatus described in the above embodiment is described in a language that can be executed by a computer. For example, it is possible to create a simulation program in which the processing executed by the simulation apparatus 100 according to the embodiment is described in a language that can be executed by a computer. In this case, the same effect as that of the above embodiment can be obtained by executing the simulation program by the computer. Further, the same processing as that of the above embodiment may be realized by recording the simulation program on a computer-readable recording medium, reading the simulation program recorded on the recording medium into the computer, and executing the simulation program.

FIG. 6 is a diagram showing an example of a computer that executes a simulation program. As illustrated in FIG. 6, the computer 1000 has, for example, a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. However, each of these parts is connected by a bus 1080.

The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012, as illustrated in FIG. The ROM 1011 stores, for example, a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090, as illustrated in FIG. The disk drive interface 1040 is connected to the disk drive 1100 as illustrated in FIG. For example, a removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120, as illustrated in FIG. The video adapter 1060 is connected, for example, to a display 1130, as illustrated in FIG.

Here, as illustrated in FIG. 6, the hard disk drive 1090 stores, for example, the OS 1091, the application program 1092, the program module 1093, and the program data 1094. That is, the above-mentioned simulation program is stored in, for example, the hard disk drive 1090 as a program module in which a command executed by the computer 1000 is described.

Further, the various data described in the above embodiment are stored as program data in, for example, a memory 1010 or a hard disk drive 1090. Then, the CPU 1020 reads the program module 1093 and the program data 1094 stored in the memory 1010 and the hard disk drive 1090 into the RAM 1012 as needed, and executes various processing procedures.

The program module 1093 and program data 1094 related to the simulation program are not limited to those stored in the hard disk drive 1090, and may be stored in, for example, a detachable storage medium and read by the CPU 1020 via a disk drive or the like. good. Alternatively, the program module 1093 and the program data 1094 related to the simulation program are stored in another computer connected via a network (LAN, WAN (Wide Area Network), etc.) and read by the CPU 1020 via the network interface 1070. May be done.

The above-described embodiments and modifications thereof are included in the invention described in the claims and the equivalent scope thereof, as included in the technique disclosed in the present application.

100 Simulation device 110 Communication unit 111 Display unit 112 Operation unit 120 Storage unit 121 Learning data storage unit 122 Prediction model storage unit 123 Evaluation data storage unit 124 Arrangement information storage unit 125 Learned model storage unit 130 Control unit 131 First reception Unit 132 Setting unit 133 Generation unit 134 Second reception unit 135 Execution unit 136 Judgment unit 137 Output control unit

Claims (8)

The first reception section that accepts input of learning data,
A generation unit that learns using the received training data and generates a prediction model,
A second reception unit that accepts the arrangement of the evaluation data used for the simulation, the generated prediction model, and the reinforcement learning device that performs reinforcement learning in the simulation.
An execution unit that executes the simulation using the evaluation data, the prediction model, and the reinforcement learner based on the received state of the arrangement.
Based on the learning result of the reinforcement learning device in the simulation, it is determined whether or not to regenerate the prediction model, and when it is determined to regenerate the prediction model, the generation unit is referred to. A judgment unit that instructs the regeneration of the prediction model, and
A simulation device characterized by having. The determination unit determines the evaluation value of the error between the predicted value of the prediction model based on the learning result of the reinforcement learning device in the actual environment and the prediction value of the prediction model based on the learning result of the reinforcement learning device in the simulation. Based on this, it is determined whether or not to regenerate the prediction model.
The simulation apparatus according to claim 1. The determination unit determines whether or not to regenerate the prediction model based on the degree of correlation between the prediction value of the prediction model based on the learning result of the reinforcement learning device in the simulation and the measured value.
The simulation apparatus according to claim 1. The learning data includes image data and includes image data.
The generation unit further generates a prediction image model that generates prediction image data based on the image data as the prediction model.
The second reception unit receives the arrangement of the predicted image model and receives it.
The execution unit executes the simulation including the predicted image model, and the execution unit executes the simulation.
Further, it has an output control unit that outputs the predicted image data corresponding to the predicted value of the predicted model based on the learning result of the enhanced learning device in the simulation.
The simulation apparatus according to any one of claims 1 to 3, wherein the simulation apparatus is characterized in that. The learning data includes voice data.
As the prediction model, the generation unit further generates a prediction voice model that generates prediction voice data based on the voice data.
The second reception unit receives the arrangement of the predicted voice model and receives it.
The execution unit executes the simulation including the predictive voice model, and the execution unit executes the simulation.
Further, it has an output control unit that outputs the predicted voice data corresponding to the predicted value of the predicted model based on the learning result of the enhanced learning device in the simulation.
The simulation apparatus according to any one of claims 1 to 4, wherein the simulation apparatus is characterized in that. The determination unit further receives an evaluation of the predicted image data or the predicted voice data output by the output control unit, and determines whether or not to regenerate the prediction model based on the received evaluation.
The simulation apparatus according to claim 4 or 5. The first reception process that accepts the input of learning data,
A generation process that learns using the received training data and generates a prediction model,
A second reception process that accepts the arrangement of the evaluation data used for the simulation, the generated prediction model, and the reinforcement learning device that performs reinforcement learning in the simulation.
An execution step of executing the simulation using the evaluation data, the prediction model, and the reinforcement learner based on the received state of the arrangement.
Based on the learning result of the reinforcement learning device in the simulation, it is determined whether or not to regenerate the prediction model, and when it is determined to regenerate the prediction model, the above-mentioned generation step is performed. A judgment process that instructs the regeneration of the prediction model, and
A simulation method characterized by the simulation device performing. The first reception step that accepts the input of learning data,
A generation step to train using the received training data and generate a predictive model,
A second reception step that accepts the arrangement of the evaluation data used for the simulation, the generated prediction model, and the reinforcement learning device that performs reinforcement learning in the simulation.
An execution step for executing the simulation using the evaluation data, the prediction model, and the reinforcement learner based on the received state of the arrangement.
Based on the learning result of the reinforcement learning device in the simulation, it is determined whether or not to regenerate the prediction model, and when it is determined to regenerate the prediction model, the above-mentioned generation step is performed. A decision step that directs the regeneration of the predictive model, and
A simulation program characterized by having a computer execute.

Images