JP7521336B2 - NETWORK SUBSTITUTION DEVICE, NETWORK SUBSTITUTION METHOD, TRAINED MODEL, ESTIMATION DEVICE, AND ESTIMATION METHOD - Google Patents

Description

The present invention relates to an apparatus and method for formulating the computational process of a trained model, and also to an apparatus and method for estimating the output of a trained model using such a formula.

The use of third-generation AI that uses deep neural networks (hereinafter referred to as "DNN") is being promoted in society. DNN is used, for example, for image recognition of still and video images, voice recognition, natural language processing such as machine translation, and anomaly detection processing that detects signs of abnormalities from time-series data from sensors attached to industrial equipment.

A DNN is a neural network that is compatible with deep learning and has a deeper network hierarchy. A DNN contains multiple layers, including an input layer, multiple hidden layers, and an output layer, and each layer is structured with multiple nodes connected by edges. Each layer has a function called an activation function, and edges can have weights. The output value of each node is calculated from the output value of the node in the previous layer that is connected to that node. In other words, the output value of each node is calculated from the output value of the node in the previous layer, the weight value of the connecting edge, and the activation function of the layer.

Patent document 1 describes a device for achieving a high compression rate of network coefficients (weight coefficients, etc.) of a neural network. The device described in patent document 1 compresses the trained network coefficients while learning, using as training data a set of input data input to the input layer of the trained neural network and output values of the hidden layer or output layer of the neural network.

Patent document 2 describes a device for shortening the processing time of calculations using a genetic algorithm. The device described in patent document 2 uses a neural network to find a fitness function for each solution candidate in a group of solution candidates.

JP 2019-28746 A (published on July 31, 2017) Japanese Patent Application Laid-Open No. 6-161984 (published on June 10, 1994)

However, it would be desirable if the computational processing of DNN could be further accelerated. With the techniques described in Patent Documents 1 and 2, there were cases where the computational processing of neural networks could not be accelerated.

One aspect of the present invention was made in consideration of the above problems, and aims to realize a technology that speeds up the calculations of deep neural networks.

In order to solve the above problem, a network replacement device according to one aspect of the present invention includes one or more processors that execute a determination step and a replacement step. Also, a network replacement method according to one aspect of the present invention includes a determination step and a replacement step.

The identification step is a step of identifying a nonlinear relational equation that expresses the relationship between the output value of a node included in a first layer of a deep neural network composed of multiple layers and the output value of a node included in a second layer, using a genetic algorithm. The replacement step is a step of replacing an operation between the first layer and the second layer in the deep neural network with an operation using the relational equation.

The trained model according to one aspect of the present invention is a trained model obtained from a deep neural network composed of multiple layers, in which the calculations from a first layer to a second layer included in the multiple layers of the deep neural network are replaced by calculations using a nonlinear relational equation identified by a genetic algorithm that represents the relationship between the output value of a node included in the first layer and the output value of a node included in the second layer.

The estimation device according to one aspect of the present invention includes one or more processors that execute the estimation step. The estimation method according to one aspect of the present invention includes an estimation step. The estimation step is a step of estimating output data from input data using the trained model described above.

The network replacement device, network replacement method, and deep neural network according to one aspect of the present invention can speed up the calculations of the deep neural network. Furthermore, the estimation device and estimation method according to one aspect of the present invention can perform estimation processing using a deep neural network replaced by such a network replacement device and network replacement method.

1 is a diagram showing a configuration of a network replacement system according to an embodiment of the present invention; 2 is a block diagram showing a configuration of a network replacement device included in the network replacement system of FIG. 1 . 3 is a flowchart showing the flow of a network replacement method implemented by the network replacement device of FIG. 2 . FIG. 1 illustrates an example of a trained model and a genetic algorithm. 4 is a flowchart showing the flow of a specific step in FIG. 3 . 13 is a flowchart showing the flow of a process performed by the network replacement device. FIG. 13 is a diagram illustrating the contents of a crossover. FIG. 1 illustrates the contents of a subtree mutation. FIG. 1 illustrates the content of a hoist mutation. FIG. 1 illustrates the contents of point mutations. FIG. 4 is a diagram for explaining a specific example of a trained model obtained by the replacement step in FIG. 3 . FIG. 4 is a diagram for explaining a specific example of a trained model obtained by the replacement step in FIG. 3 . FIG. 4 is a diagram for explaining a specific example of a trained model obtained by the replacement step in FIG. 3 . FIG. 4 is a diagram for explaining a specific example of a trained model obtained by the replacement step in FIG. 3 . 2 is a block diagram showing a configuration of an estimation device included in the network replacement system of FIG. 1 . 14 is a flowchart showing the flow of an estimation method implemented by the estimation device of FIG. 13 . 2 is a block diagram showing a configuration of a machine learning device included in the network replacement system of FIG. 1 . 16 is a flowchart showing the flow of a machine learning method implemented by the machine learning device of FIG. 15 . FIG. 2 is a diagram showing a configuration of a casting phase.

[Embodiment 1]
[Network Replacement System]
A network substitution system S according to an embodiment of the present invention will be described with reference to Fig. 1. Fig. 1 is a diagram showing the configuration of the network substitution system S.

The network replacement system S is a system for formulating the relationships between nodes of a DNN and aggregating parts of the DNN. As shown in FIG. 1, the network replacement system S includes a network replacement device 1, a machine learning device 2, and an estimation device 3.

The network replacement device 1 is a device that implements the network replacement method M1. The network replacement method M1 includes a specification step (M12) in which the network replacement device 1 specifies a nonlinear relational expression that expresses the relationship between the output value of a node included in a first layer of a deep neural network composed of multiple layers and the output value of a node included in a second layer using a genetic algorithm, and a substitution step (M13) in which an operation between the first layer and the second layer in the deep neural network is replaced with an operation using the relational expression. As the learned model LM1, for example, an algorithm such as a neural network model such as a convolutional neural network or a recurrent neural network, a regression model such as a linear regression, or a tree model such as a regression tree can be used. The configuration of the network replacement device 1 and the flow of the network replacement method M1 will be described in detail below with reference to the drawings.

Various data may be used as input to the trained model LM1. Examples of input to the trained model LM1 include image data representing still or moving images, audio data representing audio, document data representing documents, and time-series data from sensors attached to industrial equipment. In addition, examples of input to the trained model LM1 include various parameters collected on a casting line.

Various data can be used as the output of the trained model LM1. The output of the trained model LM1 can be, for example, image recognition results, voice recognition results, machine translation results, and the presence or absence of abnormalities. In addition, the output of the trained model LM1 can be, for example, the target value for the CB value (compactability) of the molding sand in the casting line.

The network permutation device 1 includes a data collection device 111. The data collection device 111 temporarily holds input data to the learned model LM1 and output data from the learned model LM1, and inputs the held data to the next stage genetic algorithm GA. The data collection device 111 is, for example, a buffer equipped with information storage means such as a semiconductor memory or a hard disk, and input/output means. Note that, although the example of FIG. 1 illustrates a configuration in which the data collection device 111 is included in the network permutation device 1, the data collection device 111 may be configured as a device separate from the formula identification device 1. The data collection device 111 may be configured as a hardware unit, or may be configured as a combination of hardware and software.

The network replacement device 1 executes the network replacement method M1 to generate a trained model LM2. The trained model LM2 is a trained model for causing a computer to function so as to output output data based on input data, and is a trained model for causing a computer to function so that the output value of a node included in the second layer is calculated by a calculation using the output value of each node included in the first layer and a relational expression. In other words, the trained model LM2 is a trained model obtained from a deep neural network composed of multiple layers, and is a trained model in which the calculation from the first layer to the second layer included in the multiple layers of the trained model LM1 is replaced by a calculation using a nonlinear relational expression identified by a genetic algorithm that represents the relationship between the output value of the node included in the first layer and the output value of the node included in the second layer.

The machine learning device 2 is a device for implementing the machine learning method M2. The machine learning method M2 is a method for constructing a learning dataset DS using input data and constructing a trained model LM1 by machine learning (supervised learning) using the learning dataset DS. Details of the configuration of the machine learning device 2 and the flow of the machine learning method M2 will be described later with reference to the accompanying drawings.

The estimation device 3 is a device for implementing the estimation method M3. The estimation method M3 includes a second estimation step M31 in which the estimation device 3 estimates output data from input data using a trained model LM2 generated by the network replacement method M1. Details of the configuration of the estimation device 3 and the flow of the estimation method M3 will be described later with reference to the accompanying drawings.

The network replacement system S executes a learning phase, a network replacement phase, and an estimation phase. The learning phase, network replacement phase, and estimation phase can be briefly explained as follows.

(1) Learning Phase In the learning phase, a trained model LM1 is constructed. The machine learning device 2 creates training data from input data and output data, and adds the created training data to the training dataset DS. The machine learning device 2 also constructs the trained model LM1 by machine learning using the training data DS. The constructed trained model LM1 is transferred from the machine learning device 2 to the network replacement device 1.

(2) Network Replacement Phase In the network replacement phase, the network replacement device 1 estimates output data using the learned model LM1 based on the input data. The network replacement device 1 also identifies a nonlinear relational expression that expresses the relationship between the output value of a node included in the first layer of the learned model LM1 and the output value of a node included in the second layer by nonlinear regression using a genetic algorithm. The network replacement device 1 also replaces the calculation between the first layer and the second layer in the learned model LM1 with a calculation using the identified relational expression.

(3) Estimation Phase In the estimation phase, the estimation device 3 estimates output data from input data using the learned model LM2 generated by the network substitution device 1.

[Configuration of the Network Replacement Device]
The configuration of the network replacement device 1 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the configuration of the network replacement device 1.

The network replacement device 1 is realized using a general-purpose computer, and includes a processor 11, a primary memory 12, a secondary memory 13, an input/output interface 14, a communication interface 15, and a bus 16. The processor 11, the primary memory 12, the secondary memory 13, the input/output interface 14, and the communication interface 15 are interconnected via the bus 16.

The secondary memory 13 stores a network replacement program P1 and a learned model LM1. The processor 11 expands the network replacement program P1 and the learned model LM1 stored in the secondary memory 13 onto the primary memory 12. The processor 11 then executes each step included in the network replacement method M1 according to the instructions included in the network replacement program P1 expanded onto the primary memory 12. The learned model LM1 expanded onto the primary memory 12 is used when the processor 11 executes an estimation step M11 (described later) of the network replacement method M1. Note that the network replacement program P1 being stored in the secondary memory 13 refers to the source code or an executable file obtained by compiling the source code being stored in the secondary memory 13. Also, the learned model LM1 being stored in the secondary memory 13 refers to the parameters defining the learned model LM1 being stored in the secondary memory 13.

Devices that can be used as processor 11 include, for example, a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), a DSP (Digital Signal Processor), an MPU (Micro Processing Unit), an FPU (Floating point number Processing Unit), a PPU (Physics Processing Unit), a microcontroller, or a combination of these. Processor 11 is sometimes called a "computing device."

In addition, a device that can be used as the primary memory 12 can be, for example, a semiconductor RAM (Random Access Memory). The primary memory 12 is sometimes called a "main memory". In addition, a device that can be used as the secondary memory 13 can be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), an optical disk drive (ODD), a floppy (registered trademark) disk drive (FDD), or a combination thereof. The secondary memory 13 is sometimes called an "auxiliary memory". The secondary memory 13 may be built into the network replacement device 1, or may be built into another computer (for example, a computer constituting a cloud server) connected to the network replacement device 1 via the input/output interface 14 or the communication interface 15. In this embodiment, the storage in the network replacement device 1 is realized by two memories (the primary memory 12 and the secondary memory 13), but is not limited to this. In other words, the storage in the network replacement device 1 may be realized by one memory. In this case, for example, one storage area of the memory can be used as the primary memory 12, and another storage area of the memory can be used as the secondary memory 13.

An input device and/or an output device are connected to the input/output interface 14. Examples of the input/output interface 14 include interfaces such as USB (Universal Serial Bus), ATA (Advanced Technology Attachment), SCSI (Small Computer System Interface), and PCI (Peripheral Component Interconnect). Examples of input devices connected to the input/output interface 14 include a group of sensors provided in a casting line. Data acquired from the group of sensors in the network replacement method M1 is input to the network replacement device 1 and stored in the primary memory 12. Examples of input devices connected to the input/output interface 14 include a keyboard, a mouse, a touchpad, a microphone, or a combination of these. Data acquired from a user in the network replacement method M1 is input to the network replacement device 1 through these input devices and stored in the primary memory 12. Examples of output devices connected to the input/output interface 14 include a display, a projector, a printer, a speaker, a headphone, or a combination of these. Information provided to a user in the network replacement method M1 is output from the network replacement device 1 through these output devices. The network replacement device 1 may incorporate a keyboard that functions as an input device and a display that functions as an output device, like a laptop computer. Alternatively, the network replacement device 1 may incorporate a touch panel that functions as both an input device and an output device, like a tablet computer.

The communication interface 15 is connected to other computers via a network in a wired or wireless manner. Examples of the communication interface 15 include interfaces such as Ethernet (registered trademark) and Wi-Fi (registered trademark). Available networks include a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), a global area network (GAN), or an internetwork including these networks. The internetwork may be an intranet, an extranet, or the Internet. Data (e.g., the trained model LM1) acquired by the network replacement device 1 from other computers (e.g., the machine learning device 2) in the network replacement method M1, and data provided by the network replacement device 1 to other computers in the network replacement method M1 are transmitted and received via these networks.

In this embodiment, a configuration is adopted in which the network replacement method M1 is executed using a single processor (processor 11), but the present invention is not limited to this. In other words, a configuration may be adopted in which the network replacement method M1 is executed using multiple processors. In this case, the multiple processors that cooperate to execute the network replacement method M1 may be provided in a single computer and configured to be able to communicate with each other via a bus, or may be provided in a distributed manner in multiple computers and configured to be able to communicate with each other via a network. As an example, a processor built into a computer that constitutes a cloud server and a processor built into a computer owned by a user of the cloud server may cooperate to execute the network replacement method M1.

In addition, in this embodiment, a configuration is adopted in which the learned model LM1 is stored in a memory (secondary memory 13) built into the same computer as the processor (processor 11) that executes the network replacement method M1, but the present invention is not limited to this. In other words, a configuration may be adopted in which the learned model LM1 is stored in a memory built into a computer different from the processor that executes the network replacement method M1. In this case, the computer with the built-in memory that stores the learned model LM1 is configured to be able to communicate with the computer with the built-in processor that executes the network replacement method M1 via a network. As an example, a mode can be considered in which the learned model LM1 is stored in a memory built into a computer that constitutes a cloud server, and a processor built into a computer owned by a user of the cloud server executes the network replacement method M1.

In addition, in this embodiment, a configuration is adopted in which the learned model LM1 is stored in a single memory (secondary memory 13), but the present invention is not limited to this. That is, a configuration may be adopted in which the learned model LM1 is distributed and stored in multiple memories. In this case, the multiple memories that store the learned model LM1 may be provided in a single computer (which may or may not be a computer with a built-in processor that executes the network replacement method M1), or may be distributed and provided in multiple computers (which may or may not include a computer with a built-in processor that executes the network replacement method M1). As an example, a configuration may be considered in which the learned model LM1 is distributed and stored in memories built into each of the multiple computers that make up the cloud server.

When the data collection device 111 is configured as a device separate from the network replacement device 1, the data collection device 111 includes, for example, a memory (not shown) and a processor (not shown).

[Network replacement method flow]
The flow of the network replacement method M1 will be described with reference to Fig. 3. Fig. 3 is a flowchart showing the flow of the network replacement method M1. The network replacement method M1 includes an estimation step M11, a specification step M12, and a replacement step M13.

The estimation step M11 is a step in which the processor 11 estimates output data from input data using the learned model LM1. In the estimation step M11, the processor 11 inputs the input data to the learned model LM1 to obtain output data.

In the estimation process using the learned model LM1, the output values of the nodes included in the first layer of the learned model LM1 (hereinafter referred to as "first data") and the output values of the nodes included in the second layer (hereinafter referred to as "second data") are temporarily stored in the data collection device 111. The first layer and the second layer are the input layer, the output layer, or the hidden layer. The data stored in the data collection device 111 is used in the subsequent identification step M12.

The identification step M12 is a step in which the processor 11 identifies a nonlinear relational equation that expresses the relationship between the output values of the nodes included in the first layer of the learned model LM1 and the output values of the nodes included in the second layer, using a genetic algorithm GA. The relational equation identified by the processor 11 in the identification step M12 is a function different from the activation function used in the neural network. A specific example of the identification step M12 will be described later with reference to different drawings.

The substitution step M13 is a step in which the processor 11 substitutes the calculation between the first layer and the second layer in the learned model LM1 with a calculation using the relational expression identified in step M12. A specific example of the learned model LM2 obtained by the substitution step M13 will be described later with reference to a different drawing.

(Specific examples of specific steps)
A specific example of the identifying step M12 included in the network replacement method M1 will be described with reference to Fig. 4 and Fig. 5. Fig. 4 is a diagram illustrating a learned model LM11 and a genetic algorithm GA. Fig. 5 is a flowchart showing the flow of the identifying step M12 executed by the processor 11.

The trained model LM11 is a deep neural network composed of multiple layers. The trained model LM11 is an example of the trained model LM1 according to this embodiment. The trained model LM11 includes multiple layers, including an input layer LX to which input data is input, a hidden layer LY1, a hidden layer LY2, and an output layer LZ that outputs output data. Each layer has a structure in which multiple nodes are connected by edges.

Each layer has a function called an activation function, and edges can have weights. The output value of each node is calculated from the output value of the node in the previous layer that is connected to that node. In other words, the output value of each node is calculated from the output value of the node in the previous layer, the weight value of the connecting edge, and the activation function of the layer.

In the example of FIG. 4, the input layer LX includes nodes X1, X2, X3, and X4. That is, in this example, the input data input to the trained model LM11 includes output values x1, x2, x3, and x4 of nodes X1, X2, X3, and X4.

Hidden layer LY1 includes nodes Y11, Y12, and Y13. The output values y11, y12, and y13 of nodes Y11, Y12, and Y13 are calculated using the output values x1, x2, x3, and x4 of the nodes in input layer LX, which is the layer preceding hidden layer LY1.

Hidden layer LY2 includes nodes Y21, Y22, and Y23. The output values y21, y22, and y23 of nodes Y21, Y22, and Y23 are calculated using the output values y11, y12, and y13 of the nodes in hidden layer LY1, which is the layer preceding hidden layer LY2.

The output layer LZ includes node Z1. The output value z1 of node Z1 is calculated using the output values y21, y22, and y23 of the nodes in hidden layer LY2, which is the layer before the output layer LZ. That is, in this example, the output data of the trained model LM11 is the output value z1 of node Z1.

The data collection device 111 temporarily holds input data to the learned model LM1 and output data from the learned model LM1, and inputs the input data and output data to the next-stage genetic algorithm GA in time synchronization. In other words, the data collection device 111 operates as a kind of buffer that compiles data from the learned model LM1 (in certain information units) and inputs it to the next-stage genetic algorithm GA.
The genetic algorithm GA includes a first generation G1 to a fourth generation G4. In the identification step M12 according to this specific example, a relational equation expressing the relationship between the first data and the second data is identified using a genetic algorithm. The genetic algorithm GA is an algorithm that prepares a plurality of individuals i each expressing a solution candidate by a gene, preferentially selects an individual i having a high fitness Di, and searches for a solution by repeating operations such as crossover and mutation. In this embodiment, the individual i is a nonlinear relational equation expressed in a tree structure, and the operators and arguments included in the relational equation are represented by the nodes of the tree. The fitness Di is given by a fitness function.

In this specific example, the nonlinear relational equations for the individual i include, for example, the following three equations (1) to (3).
y11=f ₁ (x1, x2, x3, x4)...(1)
y12=f ₂ (x1, x2, x3, x4)...(2)
y13=f ₃ (x1, x2, x3, x4)...(3 )

In equations (1) to (3), x1 to x4 are output values of each node in the input layer LX (an example of the first layer) . y11 to y13 are output values of nodes Y21 to Y23 in the hidden layer LY2 (an example of the second layer). In other words, equations (1) to (3) are nonlinear relational expressions that express the relationship between the output values of the nodes included in the input layer LX (the first layer) of the trained model LM11 and the output values of the nodes included in the hidden layer LY2 (the second layer).

In this specific example, the processor 31 identifies the relational equation by genetic programming. Genetic programming is an extension of the genetic algorithm, and uses a tree structure to represent gene types. Note that the flow of the identification step M12 shown in FIG. 5 is an example, and the method of identifying the relational equation using the genetic algorithm GA is not limited to the method shown in FIG. 5. Various other methods can be adopted as a method of identifying the relational equation using the genetic algorithm GA.

Figure 5 is a flow chart illustrating the flow of the identification step M12 executed by the processor 11. The processor 11 executes the identification step M12 using a specified module (hereinafter referred to as "module A"). Module A is a module that executes a genetic algorithm. In module A, the processor 11 starts by creating a population of simple random equations that represent the relationships between known independent variables and their dependent variable targets in order to predict new data. The processor 11 then evolves the population to generate the next generation of the population by selecting the most suitable individuals from the population to be subjected to genetic manipulation. Through the above operations, a relational equation that best represents the above relationships is identified.

In this specific example, a module that executes genetic programming is used as module A. Genetic programming is an extension of the genetic algorithm, and uses a tree structure to represent gene types. Note that the flow of the identification step M12 shown in FIG. 5 is an example, and the method of identifying a relational equation using the genetic algorithm GA is not limited to the method shown in FIG. 5. Various other methods can be adopted as a method of identifying a relational equation using the genetic algorithm GA.

In step M121, the processor 11 acquires the first data and the second data of the learned model LM1 from the data collection device 111.

In step M122, the processor 11 acquires parameters to be used in the genetic algorithm GA. The parameters include, for example, the number of individuals to be generated N, the tournament size Nt, the crossover probability Pc, the mutation probability Pms, the number of evolutionary generations Ng, the operator Oj to be used in the syntax tree, the maximum depth d of the syntax tree, and the event occurrence probabilities Pk1 to Pk5. The values of each parameter are input to the network replacement device 1, for example, by the user.

The number of individuals to be generated N represents the number of individuals i to be included in the set. The tournament size Nt is the number of individuals i to be randomly selected from the set of the current generation. The mutation probability Pms is the probability that a gene will mutate. Operators Oi used in the syntax tree are, for example, Max, Min, sqrt (root), log (natural logarithm), +, -, ×, ÷, sin (radians), cos (radians), tan (radians), abs, neg, and inv. Max is an operator that selects the maximum value. Min is an operator that selects the minimum value. neg is an operator that makes the sign negative. inv is an operator that sets arguments close to zero to 0.

The event occurrence probabilities Pk1 to Pk5 are the probabilities that operations m1 to m5 are selected as operations for evolving the next generation set. The processor 11 evolves the next generation set using one of the operations m1 to m5. The sum of the event occurrence probabilities Pk1 to Pk5 is 1. As an example, the values of the event occurrence probabilities Pk1, Pk2, Pk3, Pk4, and Pk5 are "0.1", "0.2", "0.3", "0.4", and "0.1", respectively. Operations m1 to m5 will be described later with reference to different drawings.

In step M123, processor 11 randomly generates N individuals i based on the specified parameters (operator Oi to be used in the syntax tree, maximum depth d of the syntax tree, etc.), and generates a set of N individuals i that will become the initial current generation.

In step M124, the processor 11 calculates the fitness Di of each individual i included in the set of the current generation. The fitness Di is given by a fitness function.

In step M125, processor 11 randomly selects individuals i of tournament size Nt from the current generation set, selects the individual i with the highest fitness Di among them, and adds it to the next generation set. The individual i selected in step M125, i.e., the individual i added to the next generation set, is also called the "winner tree."

Processor 11 repeats the process of step M125 until the number of individuals in the next generation becomes N, the same as the current generation, that is, while the number of individuals in the next generation has not yet reached N (step M126; NO). When the number of individuals in the next generation reaches N (step M126; YES), processor 21 executes the process of step M127.

In step M127, processor 11 executes a process to evolve the next generation set. Details of step M127 will be described later with reference to a different drawing.

In step M130, processor 11 overwrites the next-generation set onto the current-generation set. In step M131, processor 11 determines whether the number of evolutionary generations Ng has been reached. If the number of evolutionary generations Ng has not been reached (step M131; NO), processor 11 returns to the processing of step M124. On the other hand, if the number of evolutionary generations Ng has been reached (step M131; YES), processor 11 proceeds to the processing of step M132.

In step M132, processor 11 identifies an individual i with the highest fitness Di from among the individuals i included in the set of the current generation. In step M133, processor 11 outputs the identified individual i as the optimal solution. For example, processor 11 displays on the display the relational equation that is the selected individual i, or a figure (such as a graph) that represents the relational equation.

FIG. 6 is a flow chart illustrating the flow of step M127 executed by the processor 11. In step M201, the processor 11 selects one of operations m1 to m5 based on the event occurrence probabilities Pk1 to Pk5 set by the user. If operation m1 is selected (step M201; "operation m1"), the processor 11 proceeds to processing of step M202. If operation m2 is selected (step M201; "operation m2"), the processor 11 proceeds to processing of step M211. If operation m3 is selected (step M201; "operation m3"), the processor 11 proceeds to processing of step M221. If operation m4 is selected (step M201; "operation m4"), the processor 11 proceeds to processing of step M231. If operation m5 is selected (step M201; "operation m5"), the processor 11 ends the processing.

Operation m1 is a crossover. Crossover is a method of mixing genetic material between individuals. In the case of crossover, in step M202, processor 11 randomly selects subtrees in each winning tree for the winning trees in the next generation set.

In step M203, processor 11 generates a next-generation set for donors. The processing content of step M203 is the same as the processing content of steps M123 to M125 in FIG. 6. That is, processor 11 first randomly generates N individuals i based on individual parameters specified by the user, and generates a set of N individuals i (hereinafter referred to as the "donor set"). Next, processor 11 calculates the fitness Di of each individual i included in the donor set. Next, processor 11 randomly extracts individuals i in a number equal to the tournament size Nt from the donor set, selects the individual i with the highest fitness Di among them, and adds it to the donor set of the next generation. The individual i selected by this processing is also referred to as the "donor tree". Processor 11 repeats the donor tree selection processing until the number of donor trees of the next generation reaches N.

In step M204, processor 11 randomly selects a subtree included in the donor tree (hereinafter referred to as the "donor subtree").

In step M205, processor 11 replaces the subtrees in the winning tree. In this embodiment, processor 11 removes the subtree selected in step M202 from the winning tree, and transplants the donor subtree selected in step M203 into the location where the subtree was. In other words, processor 11 replaces the subtree included in the winning tree with the donor subtree. The winning tree with the replaced subtree becomes the descendant (individual) of the next generation.

Operation m2 is an operation for mutating a subtree. By mutating a subtree, extinct functions and operators can be reintroduced into the population and diversity can be maintained. In this case, in step M211, processor 11 randomly selects a subtree to be included in the winning tree.

In step M212, processor 11 randomly generates a subtree. In step M213, processor 11 replaces the subtrees in the winning tree. In this embodiment, processor 11 removes the subtree selected in step M202 from the winning tree, and transplants the subtree generated in step M212 into the location of the removed subtree. In other words, processor 11 replaces the subtree included in the winning tree with the subtree generated in step M212. The winning tree with the replaced subtree becomes the descendant (individual) of the next generation.

Operation m3 is a hoist mutation. Hoist mutation is a mutation operation that combats tree bloat. In step M221, the processor randomly selects a subtree to be included in the winning tree. In step M222, processor 11 randomly selects a subtree to be included in the subtree selected in step M221.

In step M223, processor 11 rolls up the subtree selected in step M222 to the position of the original subtree (the subtree selected in step M221). The winning tree into which this subtree has been rolled up becomes the descendant (individual) of the next generation.

Operation m4 is a point mutation. Point mutation is an operation that reintroduces extinct relations and operators into a population in order to maintain diversity. In step M231, processor 11 randomly selects a node of the winning tree. In step M232, processor 11 replaces the node selected in step M231 with another node. As a result, the relational expression represented by the winning tree is replaced with another relational expression that requires the same number of arguments as the original node. The winning tree obtained by the replacement becomes the descendant (individual) of the next generation.

Operation m5 is regeneration, in which the winning tree is replicated and included in the next generation without modification.

Figures 7 to 10 are diagrams illustrating the contents of operations performed on the winning tree. Figure 7 is a diagram illustrating the contents of operation m1 (crossover). In the example of Figure 7, subtree tr111 of winner tree tr11 is replaced with subtree tr121 of donor tree tr12, resulting in winner tree tr13. Winner tree tr13 becomes the descendant (individual) of the next generation.

Figure 8 is a diagram illustrating the contents of operation m2 (subtree mutation). In the example of Figure 8, subtree tr111 of winner tree tr11 is replaced with subtree tr22 to become winner tree tr23. Winner tree tr23 becomes the descendant (individual) of the next generation.

Figure 9 is a diagram illustrating the contents of operation m3 (hoist mutation). In the example of Figure 9, subtree tr1121 of winner tree tr11 is rolled up to the position of subtree tr112, becoming winner tree tr31. Winner tree tr31 becomes the descendant (individual) of the next generation.

Figure 10 is a diagram illustrating the contents of operation m4 (point mutation). In the example of Figure 10, nodes n21 and n34 included in the winner tree tr11 are replaced with nodes n421 and n434, resulting in a winner tree tr41. The winner tree tr41 becomes the descendant (individual) of the next generation.

(Specific example 1 of a trained model obtained by a substitution step)
FIG. 11 is a diagram for explaining specific examples of learned models LM21 and LM22 obtained by the substitution step M13. In the figure, the learned model LM12 is an example of the learned model LM1 according to this embodiment. The learned model LM12 includes an input layer LX, an output layer LY, and hidden layers LA to LD. The input layer LX includes nodes X1 to X12. The output layer LY includes nodes Y1 to Y12. The hidden layer LA includes nodes A1 to A6. The hidden layer LB includes nodes B1 to B6. The hidden layer LC includes nodes C1 to C6. The hidden layer LD includes nodes D1 to D6. The learned models LM21 and LM22 are specific examples of the learned model LM2 obtained by the substitution step M13.

In this specific example, in the identification step M12, the processor 11 identifies a nonlinear relational expression fxb expressing the relationship between the output value (first data) of each node included in the input layer LX (first layer) and the output value (second data) of each node included in the hidden layer LB (second layer) using a genetic algorithm. The relational expression fxb is, for example, the following formula (4). In formula (4), i is a natural number satisfying 1≦i≦6, and b _i is the output value of node Bi included in the hidden layer LB. x1 to x12 are the output values of nodes X1 to X12 included in the input layer LX.
b _i =fxb _i (x1, x2, x3,..., x12)...(4)

In addition, in the above-mentioned identification step M12, the processor 11 identifies a nonlinear relational equation fbd expressing the relationship between the output value (first data) of the node included in the hidden layer LB (first layer) and the output value (second data) of the node included in the hidden layer LD (second layer) using a genetic algorithm. In this case, the relational equation identified by the processor 11 is, for example, the following equation (5). In equation (5), k is a natural number satisfying 1≦k≦6, and d _k is the output value of the node Di included in the hidden layer LD. b1 to b6 are the output values of the nodes B1 to B6 included in the hidden layer LB.
d _k = fbd _k (b1, b2, b3,..., b6)...(5)

In a substitution step M13, the processor 11 substitutes the operations between the first layer and the second layer in the learned model LM12 with operations using the relational expressions fxb and fbd to generate learned models LM21 and LM22.

The trained model LM21 is a trained model in which the calculations between the input layer LX (first layer) and the hidden layer LB (second layer) in the trained model LM12 have been replaced with calculations that use the relational equation fbd. The trained model LM21 includes an input layer LX, a hidden layer LB', LC, LD, and an output layer LY. The hidden layer LB of the trained model LM21 includes nodes B1' to B6'.

In the trained model LM12, the output values of each node in the hidden layer LB are calculated from the output values of each node in the previous layer, the hidden layer LA, the weight values of the connecting edges, and the activation function of the hidden layer LB. In contrast, in the trained model LM21, the output values of each node in the hidden layer LB' (second layer) are calculated by substituting the output values of each node in the input layer LX (first layer) into equation (4) identified by the processor 11 in identification step M12.

The trained model LM22 is a trained model in which the calculations between the first layer (hidden layer LB) and the second layer (hidden layer LD) of the trained model LM21 have been replaced with calculations using the relational equations identified in the identification step M12. The trained model LM22 includes an input layer LX, hidden layers LB' and LD', and an output layer LY. The hidden layer LB' includes nodes B1' to B6'. The hidden layer LD' includes nodes D1' to D6'.

In the trained model LM12 and trained model L21, the output value of each node in the hidden layer LD is calculated from the output value of each node in the previous layer, the hidden layer LC, the weight values of the connecting edges, and the activation function of the hidden layer LD. In contrast, in the trained model LM22, the output value of each node in the hidden layer LD' (second layer) is calculated by substituting the output value of each node in the hidden layer LB' (first layer) into equation (5) identified by the processor 11 in identification step M12.

(Specific example 2 of trained model obtained by substitution step)
12 is a diagram for explaining another specific example of a learned model obtained by the substitution step M13. In the figure, learned models LM23 and LM24 are specific examples of the learned model LM2 obtained by the substitution step M13. In this specific example, in the identification step M12, the processor 11 identifies three types of relational expressions, a relational expression fxa, a relational expression fac, and a relational expression fcy, as nonlinear relational expressions that represent the relationship between the output value of each node included in the first layer and the output value of each node included in the second layer.

The relational expression fxa is a relational expression that expresses the relationship between the output value (first data) of each node included in the input layer LX (first layer) and the output value (second data) of each node included in the hidden layer LA (second layer). The relational expression fxa is, for example, the following formula (6). In formula (6), i is a natural number that satisfies 1≦i≦6, and a _i is the output value of node Ai included in the hidden layer LA. x1 to x12 are the output values of nodes X1 to X12 included in the input layer LX.
a _i = fxa _i (x1, x2, x3,..., x12)...(6)

The relational expression fac is a relational expression that expresses the relationship between the output value (first data) of each node included in the hidden layer LA (first layer) and the output value (second data) of each node included in the hidden layer LD (second layer). The relational expression fac is, for example, the following formula (7). In formula (7), i is a natural number that satisfies 1≦i≦6, and c _i is the output value of node Ci included in the hidden layer LD. a1 to a6 are the output values of nodes A1 to A6 included in the hidden layer LA.
c _i = fac _i (a1, a2, a3,..., a6)...(7)

The relational expression fcy is a relational expression that expresses the relationship between the output value (first data) of each node included in the output layer LY (first layer) and the output value (second data) of each node included in the hidden layer LC (second layer). The relational expression fcy is, for example, the following formula (8). In formula (8), i is a natural number that satisfies 1≦i≦6, and _yi is the output value of node Yi included in the output layer LY. c1 to c6 are the output values of nodes C1 to C6 included in the hidden layer LC.
y _i = fcy _i (c1, c2, c3,..., c6)...(8)

In the above-mentioned substitution step M13, the processor 11 replaces the operations between the first layer and the second layer in the learned model LM12 with operations using the relational equation identified in the identification step M12, thereby generating the learned model LM23.

The trained model LM23 includes an input layer LX, hidden layers LA' and LC', and an output layer LY'. The hidden layer LA' includes nodes A1' to A6'. The hidden layer LC' includes nodes C1' to C6'. The output layer LY' includes nodes Y1' to Y12'. In the trained model LM23, the calculations between the input layer LX (first layer) and the hidden layer LA' (second layer) are replaced with calculations using the relational expression fxa. In the trained model LM23, the calculations between the hidden layer LA' (first layer) and the hidden layer LC' (second layer) are replaced with calculations using the relational expression fac. In the trained model LM23, the calculations between the hidden layer LC' (first layer) and the output layer LY' (second layer) are replaced with calculations using the relational expression fcy.

In the trained model LM12, the output values of each node in the hidden layer LA are calculated from the output values of each node in the previous layer, the input layer LX, the weight values of the connecting edges, and the activation function of the hidden layer LA. In contrast, in the trained model LM23, the output values of each node in the hidden layer LA' (second layer) are calculated by substituting the output values of each node in the input layer LX (first layer) into the relational expression fxa.

In the trained model LM12, the output values of each node in the hidden layer LC are calculated from the output values of each node in the previous layer, the hidden layer LB, the weight values of the connecting edges, and the activation function of the hidden layer LC. In contrast, in the trained model LM23, the output values of each node in the hidden layer LC' (second layer) are calculated by substituting the output values of each node in the hidden layer LA' (first layer) into the relational expression fac.

In the trained model LM12, the output values of each node in the output layer LY are calculated from the output values of each node in the previous layer, the hidden layer LD, the weight values of the connecting edges, and the activation function of the output layer LY. In contrast, in the trained model LM23, the output values of each node in the output layer LY' (second layer) are calculated by substituting the output values of each node in the hidden layer LC' (first layer) into the relational expression fcy.

In this specific example, the processor 11 further uses the relational expression fxa (an example of a first relational expression), the relational expression fac (an example of a second relational expression), and the relational expression fcy to specify a nonlinear relational expression that expresses the relationship between the output value of the node included in the input layer LX (an example of a first layer) and the output value of the node included in the output layer LY (an example of a second layer). In this specific example, the intermediate layers LA and LC are examples of the third layer according to this embodiment. The processor 11 specifies the relational expression fxy that expresses the relationship between x1 to x12 and y1 to y12 by solving the simultaneous equations of the relational expressions fxa, fac, and fcy. The relational expression fxy is expressed, for example, by the following formula (9). In formula (9), i is a natural number that satisfies 1≦i≦12, and _yi is the output value of the node Yi included in the output layer LY. x1 to x12 are the output values of the nodes X1 to X12 included in the input layer LX.
y _i =fxy _i (x1, x2, x3,..., x12)...(9)

The processor 11 replaces the calculations between the input layer LX and the output layer LY in the learned model LM23 with calculations using the relational equation fxy, and generates the learned model LM24. The learned model LM24 includes an input layer LX and an output layer LY. The output layer LY includes nodes Y1'' to Y12''.

In the trained model LM24, the calculations between the input layer LX and the output layer LY' in the trained model LM23 are replaced with calculations using the relational equation fxy. That is, in the trained model LM24, the output value of each node included in the output layer LY'' is calculated by substituting the output value of each node included in the input layer LX into the relational equation fxy.

(Specific example 3 of trained model obtained by substitution step)
13 is a diagram for explaining another specific example of a learned model obtained by the substitution step M13. In the figure, learned models LM26 and LM27 are specific examples of the learned model LM2 obtained by the substitution step M13. In this specific example, in the identification step M12, the processor 11 identifies five types of relational expressions, i.e., relational expression fxa, relational expression fab, relational expression fbc, relational expression fcd, and relational expression fdy, as nonlinear relational expressions expressing the relationship between the output value of each node included in the first layer and the output value of each node included in the second layer.

The relational equation fxa represents the relationship between the output values of each node included in the input layer LX and the output values of each node included in the hidden layer LA.

The relational expression fab is a relational expression that represents the relationship between the output value of each node included in hidden layer LA and the output value of each node included in hidden layer LB. The relational expression fbc is a relational expression that represents the relationship between the output value of each node included in hidden layer LB and the output value of each node included in hidden layer LC. The relational expression fcd is a relational expression that represents the relationship between the output value of each node included in hidden layer LC and the output value of each node included in hidden layer LD. The relational expression fdy is a relational expression that represents the relationship between the output value of each node included in hidden layer LD and the output value of each node included in hidden layer LY.

In the above-mentioned substitution step M13, the processor 11 substitutes the calculations between the first layer and the second layer in the learned model LM12 with calculations using the relational equations identified in the identification step M12 to generate the learned model LM26. The learned model LM26 includes an input layer LX, hidden layers LA' to LD', and an output layer LY'. The hidden layer LA' includes nodes A1' to A6'. The hidden layer LB' includes nodes B1' to B6'. The hidden layer LC' includes nodes C1' to C6'. The hidden layer LD' includes nodes D1' to D6'. The output layer LY' includes nodes Y1' to Y12'.

In the trained model LM26, the calculations between each layer are replaced with calculations using relational expressions. For example, in the trained model LM26, the calculations between the input layer LX and the hidden layer LA' are replaced with calculations using the relational expression fxa. Similarly, in the trained model LM26, the calculations between the hidden layer LA' and the hidden layer LB' are replaced with calculations using the relational expression fab.

In the trained model LM26, the output values of the nodes in each layer are calculated by substituting the output values of each node in the previous layer into the relational expressions fxa to fdy identified by the genetic algorithm.

In this specific example, the processor 11 further identifies a relational equation fxy that expresses the relationship between x1 to x12 and y1 to y12 by solving the simultaneous equations of the relational equations fxa to fdy. The relational equation fxy is expressed, for example, by the above-mentioned equation (9).

The processor 11 replaces the calculations between the input layer LX and the output layer LY' in the learned model LM26 with calculations using the relational equation fxy to generate the learned model LM27. The learned model LM27 includes an input layer LX and an output layer LY". The output layer LY" includes nodes Y1" to Y12".

In the trained model LM27, the calculations between the input layer LX and the output layer LY'' in the trained model LM26 are replaced with calculations using the relational equation fxy. That is, in the trained model LM27, the output value of each node included in the output layer LY'' is calculated by substituting the output value of each node included in the input layer LX into the relational equation fxy.

(Specific example 4 of trained model obtained by substitution step)
14 is a diagram for explaining another specific example of the learned model LM2 obtained by the substitution step M13. In the figure, a learned model LM25 is an example of the learned model LM2 obtained by the substitution step M13. In this specific example, in the identification step M12, the processor 11 identifies a nonlinear relational expression fxy that expresses the relationship between the output value (first data) of each node included in the input layer LX (first layer) and the output value of each node included in the output layer LY (second layer) using a genetic algorithm.

In a substitution step M13, the processor 11 substitutes the operations between the input layer LX and the output layer LY in the learned model LM12 with operations using the relational expression fxy to generate the learned model L25. The learned model LM25 includes an input layer LX and an output layer LY. The output layer LY of the learned model LM25 includes nodes Y1' to Y12'.

The trained model LM25 is a trained model in which the calculations between the input layer LX and the output layer LY in the trained model LM12 have been replaced with calculations using the relational equation fxy. In the trained model LM25, the output value of each node included in the output layer LY' (second layer) is calculated by substituting the output value of each node included in the input layer LX (first layer) into equation (4) identified by the processor 11 in the identification step M12.

[Configuration of the Estimation Device]
The configuration of the estimation device 3 will be described with reference to Fig. 15. Fig. 15 is a block diagram showing the configuration of the estimation device 3.

The estimation device 3 is realized using a general-purpose computer, and includes a processor 31, a primary memory 32, a secondary memory 33, an input/output interface 34, a communication interface 35, and a bus 36. The processor 31, the primary memory 32, the secondary memory 33, the input/output interface 34, and the communication interface 35 are interconnected via the bus 36.

The estimation program P3 is stored in the secondary memory 33. The processor 21 expands the estimation program P3 stored in the secondary memory 23 onto the primary memory 32. The processor 31 then executes each step included in the estimation method M3 (an example of an estimation step) according to instructions included in the estimation program P3 expanded onto the primary memory 32. The learned model LM2 stored in the secondary memory 33 is used in a second estimation step M31 (described below) of the estimation method M3. Note that the estimation program P3 being stored in the secondary memory 33 refers to the source code, or an executable file obtained by compiling the source code, being stored in the secondary memory 33.

Devices that can be used as processor 31 include, for example, a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), a DSP (Digital Signal Processor), an MPU (Micro Processing Unit), an FPU (Floating point number Processing Unit), a PPU (Physics Processing Unit), a microcontroller, or a combination of these. Processor 31 is sometimes called a "computing device."

In addition, a device that can be used as the primary memory 32 can be, for example, a semiconductor RAM (Random Access Memory). The primary memory 32 is sometimes called a "main memory". In addition, a device that can be used as the secondary memory 33 can be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), an optical disk drive (ODD), a floppy disk drive (FDD), or a combination thereof. The secondary memory 33 is sometimes called an "auxiliary memory". The secondary memory 33 may be built into the estimation device 3, or may be built into another computer (for example, a computer constituting a cloud server) connected to the estimation device 3 via the input/output interface 34 or the communication interface 35. In this embodiment, the storage in the estimation device 3 is realized by two memories (the primary memory 32 and the secondary memory 33), but is not limited to this. In other words, the storage in the estimation device 3 may be realized by one memory. In this case, for example, one storage area of the memory can be used as the primary memory 32, and another storage area of the memory can be used as the secondary memory 33.

An input device and/or an output device are connected to the input/output interface 34. Examples of the input/output interface 34 include interfaces such as USB (Universal Serial Bus), ATA (Advanced Technology Attachment), SCSI (Small Computer System Interface), and PCI (Peripheral Component Interconnect). Examples of input devices connected to the input/output interface 34 include a group of sensors provided in the casting line. Data acquired from the group of sensors in the estimation method M3 is input to the estimation device 3 and stored in the primary memory 32. Examples of input devices connected to the input/output interface 34 include a keyboard, a mouse, a touchpad, a microphone, or a combination of these. Data acquired from the user in the estimation method M3 is input to the estimation method M3 via these input devices and stored in the primary memory 32. Examples of output devices connected to the input/output interface 34 include a display, a projector, a printer, a speaker, a headphone, or a combination of these. Information provided to the user in the estimation method M3 is output from the estimation device 3 via these output devices. The estimation device 3 may incorporate a keyboard that functions as an input device and a display that functions as an output device, like a laptop computer. Alternatively, the estimation device 3 may incorporate a touch panel that functions as both an input device and an output device, like a tablet computer.

The communication interface 35 is connected to other computers via a network in a wired or wireless manner. Examples of the communication interface 35 include interfaces such as Ethernet (registered trademark) and Wi-Fi (registered trademark). Available networks include a PAN (Personal Area Network), a LAN (Local Area Network), a CAN (Campus Area Network), a MAN (Metropolitan Area Network), a WAN (Wide Area Network), a GAN (Global Area Network), or an internetwork including these networks. The internetwork may be an intranet, an extranet, or the Internet. Data (e.g., the trained model LM2) provided to the estimation device 3 from other computers (e.g., the network replacement device 1) is transmitted and received via these networks.

In this embodiment, a configuration is adopted in which the estimation method M3 is executed using a single processor (processor 31), but the present invention is not limited to this. That is, a configuration in which the estimation method M3 is executed using multiple processors may be adopted. In this case, the multiple processors that cooperate to execute the estimation method M3 may be provided in a single computer and configured to be able to communicate with each other via a bus, or may be provided in a distributed manner in multiple computers and configured to be able to communicate with each other via a network. As an example, a processor built in a computer that constitutes a cloud server and a processor built in a computer owned by a user of the cloud server may cooperate to execute the estimation method M3.

In addition, in this embodiment, a configuration is adopted in which the learned model LM2 is stored in a memory (secondary memory 33) built into the same computer as the processor (processor 31) that executes the estimation method M3, but the present invention is not limited to this. That is, a configuration may be adopted in which the learned model LM2 is stored in a memory built into a computer different from the processor that executes the estimation method M3. In this case, the computer with the built-in memory that stores the learned model LM2 is configured to be able to communicate with the computer with the built-in processor that executes the estimation method M3 via a network. As an example, a mode in which the learned model LM2 is stored in a memory built into a computer that constitutes a cloud server, and a processor built into a computer owned by a user of the cloud server executes the estimation method M3, etc., can be considered.

In addition, in this embodiment, a configuration is adopted in which the learned model LM2 is stored in a single memory (secondary memory 33), but the present invention is not limited to this. That is, a configuration may be adopted in which the learned model LM2 is distributed and stored in multiple memories. In this case, the multiple memories that store the learned model LM2 may be provided in a single computer (which may or may not be a computer with a built-in processor that executes the estimation method M3), or may be distributed and provided in multiple computers (which may or may not include a computer with a built-in processor that executes the estimation method M3). As an example, a configuration may be considered in which the learned model LM2 is distributed and stored in memories built into each of the multiple computers that make up the cloud server.

[Flow of estimation method]
The flow of the estimation method M3 will be described with reference to Fig. 16. Fig. 16 is a flowchart showing the flow of the estimation method M3. The estimation method M3 includes a second estimation step M31 and an output step M32.

The second estimation step M31 is a step in which the processor 11 estimates output data from input data using the learned model LM2. The output data estimated in the second estimation step M31 is data that is estimated to be output when input data is input to the learned model LM1.

The output step M32 is a step in which the processor 11 outputs the estimated output data. The processor 11 outputs the output data, for example, by displaying an image representing the output data on a display.

According to this embodiment, the number of nodes in a deep neural network can be reduced and its structure can be consolidated, thereby speeding up the calculations of the deep neural network. In addition, by making the calculations related to each node into a relational equation, it is easy to grasp the characteristics of each node.

Furthermore, according to this embodiment, the estimation device 3 estimates the output of the learned model LM1 using the learned model LM2. In other words, the estimation device 3 can estimate the output of the learned model LM1 from the input data by a calculation that is simpler than that of the learned model LM1, without using the learned model LM1.

In addition, according to this embodiment, the learned model is compressed into a nonlinear relational equation, making it possible to visualize the processing of the intermediate layer (hidden layer) in a mathematical formula.

In addition, for example, if the number of nodes, number of layers, and activation function of the DNN are know-how, it is preferable to keep them secret. According to this embodiment, by making the intermediate layers into relational expressions, it is possible to keep the specific configuration of the DNN, such as the number of nodes, number of layers, and what activation function is used, secret. In other words, it is possible to encrypt the configuration of the DNN, making it difficult to analyze.

In addition, in this embodiment, the data collection device 111 buffers the input data and output data of the learned model LM1 and inputs them to the genetic algorithm GA with temporal synchronization. Since it is possible to achieve temporal synchronization between the output from the learned model LM1 and the input to the genetic algorithm GA, it is possible to predict the time until the next stage of the genetic algorithm GA outputs the final solution.

In addition, in this embodiment, the data collection device 111 acts as a kind of buffer that collects data of the learned model LM1 (in certain information units) and inputs it to the next stage of the genetic algorithm GA. This makes it possible to reduce wasted memory space and wasted calculation processing in the generation calculation processing of the next stage of the genetic algorithm GA, and to make the generation calculation processing of the genetic algorithm more efficient.

[Configuration of machine learning device]
The configuration of the machine learning device 2 will be described with reference to Fig. 17. Fig. 17 is a block diagram showing the configuration of the machine learning device 2.

The machine learning device 2 is realized using a general-purpose computer, and includes a processor 21, a primary memory 22, a secondary memory 23, an input/output interface 24, a communication interface 25, and a bus 26. The processor 21, the primary memory 22, the secondary memory 23, the input/output interface 24, and the communication interface 25 are interconnected via the bus 26.

The secondary memory 23 stores the machine learning program P2 and the learning dataset DS. The learning dataset DS is a collection of teacher data DS1, DS2, etc. The processor 21 expands the machine learning program P2 stored in the secondary memory 23 onto the primary memory 22. The processor 21 then executes each step included in the machine learning method M2 according to the instructions included in the machine learning program P2 expanded onto the primary memory 22. The learning dataset DS stored in the secondary memory 23 is constructed in a learning dataset construction step M21 (described later) of the machine learning method M2, and is used in a trained model construction step M22 (described later) of the machine learning method M2. In addition, the trained model LM1 constructed in the trained model construction step M22 of the machine learning method M2 is also stored in the secondary memory 23. Note that the machine learning program P2 being stored in the secondary memory 23 refers to the source code, or an executable file obtained by compiling the source code, being stored in the secondary memory 23. In addition, the learned model LM1 being stored in the secondary memory 23 means that the parameters that define the learned model LM1 are stored in the secondary memory 23.

Devices that can be used as the processor 21 include, for example, a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), a DSP (Digital Signal Processor), an MPU (Micro Processing Unit), an FPU (Floating point number Processing Unit), a PPU (Physics Processing Unit), a microcontroller, or a combination of these. The processor 21 is sometimes called a "computing device."

In addition, a device that can be used as the primary memory 22 can be, for example, a semiconductor RAM (Random Access Memory). The primary memory 22 is sometimes called a "main memory". In addition, a device that can be used as the secondary memory 23 can be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), an optical disk drive (ODD), a floppy disk drive (FDD), or a combination thereof. The secondary memory 23 is sometimes called an "auxiliary memory". The secondary memory 23 may be built into the machine learning device 2, or may be built into another computer (for example, a computer constituting a cloud server) connected to the machine learning device 2 via the input/output interface 24 or the communication interface 25. In this embodiment, the memory in the machine learning device 2 is realized by two memories (the primary memory 22 and the secondary memory 23), but is not limited to this. In other words, the memory in the machine learning device 2 may be realized by one memory. In this case, for example, one storage area of the memory can be used as the primary memory 22, and another storage area of the memory can be used as the secondary memory 23.

An input device and/or an output device are connected to the input/output interface 24. Examples of the input/output interface 24 include interfaces such as USB (Universal Serial Bus), ATA (Advanced Technology Attachment), SCSI (Small Computer System Interface), and PCI (Peripheral Component Interconnect). Examples of input devices connected to the input/output interface 24 include a group of sensors provided in the casting line. Data acquired from the group of sensors in the machine learning method M2 is input to the machine learning device 2 and stored in the primary memory 22. Examples of input devices connected to the input/output interface 24 include a keyboard, a mouse, a touchpad, a microphone, or a combination thereof. Data acquired from a user in the machine learning method M2 is input to the machine learning device 2 via these input devices and stored in the primary memory 22. Examples of output devices connected to the input/output interface 24 include a display, a projector, a printer, a speaker, a headphone, or a combination thereof. Information provided to the user in the machine learning method M2 is output from the machine learning device 2 via these output devices. The machine learning device 2 may incorporate a keyboard that functions as an input device and a display that functions as an output device, like a laptop computer. Alternatively, the machine learning device 2 may incorporate a touch panel that functions as both an input device and an output device, like a tablet computer.

Other computers are connected to the communication interface 25 via a network in a wired or wireless manner. Examples of the communication interface 25 include interfaces such as Ethernet (registered trademark) and Wi-Fi (registered trademark). Available networks include a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), a global area network (GAN), or an internetwork including these networks. The internetwork may be an intranet, an extranet, or the Internet. Data (e.g., the trained model LM1) provided by the machine learning device 2 to other computers (e.g., the network replacement device 1) is transmitted and received via these networks.

In this embodiment, a configuration is adopted in which the machine learning method M2 is executed using a single processor (processor 21), but the present invention is not limited to this. That is, a configuration may be adopted in which the machine learning method M2 is executed using multiple processors. In this case, the multiple processors that cooperate to execute the machine learning method M2 may be provided in a single computer and configured to be able to communicate with each other via a bus, or may be provided in a distributed manner in multiple computers and configured to be able to communicate with each other via a network. As an example, a processor built into a computer that constitutes a cloud server and a processor built into a computer owned by a user of the cloud server may cooperate to execute the machine learning method M2.

In addition, in this embodiment, a configuration is adopted in which the training dataset DS is stored in a memory (secondary memory 23) built into the same computer as the processor (processor 21) that executes the machine learning method M2, but the present invention is not limited to this. That is, a configuration may be adopted in which the training dataset DS is stored in a memory built into a computer other than the processor that executes the machine learning method M2. In this case, the computer with the built-in memory that stores the training dataset DS is configured to be able to communicate with the computer with the built-in processor that executes the machine learning method M2 via a network. As an example, a configuration is conceivable in which the training dataset DS is stored in a memory built into a computer that constitutes a cloud server, and a processor built into a computer owned by a user of the cloud server executes the machine learning method M2.

In addition, in this embodiment, a configuration is adopted in which the training dataset DS is stored in a single memory (secondary memory 23), but the present invention is not limited to this. That is, a configuration may be adopted in which the training dataset DS is distributed and stored in multiple memories. In this case, the multiple memories that store the training dataset DS may be provided in a single computer (which may or may not be a computer with a built-in processor that executes the machine learning method M2), or may be distributed and provided in multiple computers (which may or may not include a computer with a built-in processor that executes the machine learning method M2). As an example, a configuration can be considered in which the training dataset DS is distributed and stored in memories built into each of the multiple computers that make up the cloud server.

In addition, in this embodiment, a configuration is adopted in which the network replacement method M1, the machine learning method M2, and the estimation method M3 are executed using different processors (processor 11, processor 21, and processor 31), but the present invention is not limited to this. That is, the network replacement method M1, the machine learning method M2, and the estimation method M3 may be executed using the same processor. In this case, by executing the machine learning method M2, the learned model LM1 is stored in a memory built into the same computer as this processor. Then, when executing the network replacement method M1, this processor uses the learned model LM1 stored in this memory to generate the learned model LM2. Furthermore, when executing the estimation method M3, this processor estimates output data using the generated learned model LM2.

[Machine learning method flow]
The flow of the machine learning method M2 will be described with reference to Fig. 18. Fig. 18 is a flowchart showing the flow of the machine learning method M2.

The machine learning method M2 includes a learning dataset construction step M21 and a trained model construction step M22.

The learning dataset construction step M21 is a step in which the processor 21 constructs a learning dataset DS, which is a collection of teacher data DS1, DS2, ....

Each training data DSi (i = 1, 2, ...) includes input data and output data. The input data included in the training data DSi is the same as the input data input to the trained model LM1. In the training dataset construction step M21, the processor 21 acquires this data in a manner similar to that of the network replacement device 1. The training data DSi also includes output data as labels. In the training dataset construction step M21, the processor 21 associates the input data with the output data and stores them in the secondary memory 23. The training dataset DS is constructed by the processor 21 repeating the above process.

The trained model construction step M22 is a step in which the processor 21 constructs the trained model LM1. In the trained model construction step M22, the processor 21 constructs the trained model LM1 by supervised learning using the training dataset DS. Then, the processor 21 stores the constructed trained model LM1 in the secondary memory 23.

Example 1
The network replacement system S according to this embodiment is applied to, for example, a casting system. The casting phase C10 performed by the casting system will be described with reference to Fig. 19. Fig. 19 is a diagram showing the configuration of the casting phase C10 in the casting system.

The casting phase C10 is a phase in which casting is performed using casting sand. For example, as shown in FIG. 18, the casting phase C10 can be composed of a kneading process C11, a molding process C12, a demolding process C13, a mold coating process C14, a frame alignment process C15, a pouring process C16, a cooling process C17, and a frame removal process C18.

The kneading process C11 is a process in which additives including resin and a hardener are added to the molding sand and kneaded. The molding process C12 is a process in which a mold is formed by filling a flask with the molding sand kneaded in the kneading process C11. Here, an upper mold corresponding to the upper part of the mold and a lower mold corresponding to the lower part of the mold are each formed. The mold removal process C13 is a process in which the upper mold and the lower mold formed in the mold removal process C12 are separated from the flask. The mold coating process C14 is a process in which a mold wash is applied to the product surfaces of the upper mold and the lower mold removed in the mold removal process C13. The frame alignment process C15 is a process in which the upper mold and the lower mold to which the mold wash has been applied in the mold coating process C14 are combined to obtain a mold. The pouring process C16 is a process in which molten metal is poured into the mold obtained in the frame alignment process C15. The cooling process C17 is a process for cooling the molten metal poured into the mold in the pouring process C16. The cooled molten metal solidifies inside the mold to become a casting. The flask removal process C18 is a process for vibrating the mold to dismantle it into a sand block and remove the casting obtained in the cooling process C17.

The sand blocks obtained in casting phase C10 are recycled into casting sand through, for example, a crushing process, a separation process, a pre-recycling cooling process, a sand recycling process, etc.

In the casting system according to this embodiment, the input data in the trained model LM2 generated using a genetic algorithm includes, for example, sensor output values detected by various sensors installed on the casting line. The output data includes, for example, the CB value, mold strength, a label indicating whether the mold is chipped, and a label indicating whether the product is good or bad.

Example 2
The network replacement system S according to this embodiment is applied to, for example, a sales forecasting system. In the sales forecasting system, the input data in the trained model LM2 generated using a genetic algorithm includes, for example, condition data indicating temperature, day of the week, traffic volume, population, number of households, store area, unit price, number of products available, distance from a station, advertising expenses, SNS hashtag statistics, keyword search statistics, number of people infected with infectious diseases, number of foreign visitors to Japan, exchange rates, stock prices, interest rates, prices, employment statistics, the Bank of Japan Tankan, and household survey. The output data includes, for example, data indicating sales amounts.

Example 3
The network substitution system S according to this embodiment is applied to, for example, a drug discovery system. In the drug discovery system, input data in the trained model LM2 generated using a genetic algorithm includes, for example, molecular model parameters and data indicating the binding affinity of each component. Output data includes, for example, data indicating a drug.

Example 4
The network substitution system S according to the present embodiment is applied to, for example, a material development system. In the material development system, the input data in the trained model LM2 generated using a genetic algorithm includes, for example, the mixing ratio and physical property values of the alloy components. The output data includes, for example, characteristic data representing the characteristics (strength, etc.) of the generated material.

Example 5
The network substitution system S according to the present embodiment is applied to, for example, an image recognition system. In the image recognition system, input data in a trained model LM2 generated using a genetic algorithm includes, for example, data indicating image features. Output data includes, for example, data indicating an image recognition result.

Example 6
The network replacement system S according to the present embodiment is applied to, for example, an equipment malfunction prediction system. In the equipment malfunction prediction system, the input data in the trained model LM2 generated using a genetic algorithm includes, for example, detection values detected by each sensor installed in the equipment. The output data includes, for example, data indicating an estimation result of whether or not a malfunction has occurred in the equipment.

[Modifications]
In the above embodiment, the processor 11 may replace the calculation of a layer in which learning is slow among the layers included in the learned model LM1 with a calculation using a relational expression specified by using a genetic algorithm. The hidden layers of the learned model LM1 may include layers in which learning is slow and layers in which learning is fast. A layer in which learning is slow is a layer that has many perceptrons or multiple weightings. On the other hand, a layer in which learning is fast is a layer that has few perceptrons, no weighting, or only one weighting. The processor 11 may identify a layer in which learning is slow based on parameters that define the learned model LM1, such as the number of perceptrons and the number of weightings, and replace the calculation of the identified layer with a calculation using a relational expression specified by using a genetic algorithm.

〔summary〕
The network substitution device according to aspect 1 includes one or more processors that execute an identification step of using a genetic algorithm to identify a nonlinear relational equation that expresses a relationship between an output value of a node included in a first layer of a deep neural network composed of multiple layers and an output value of a node included in a second layer, and a substitution step of replacing an operation between the first layer and the second layer in the deep neural network with an operation using the relational equation.

According to the above configuration, the network replacement device can speed up the calculations between the first layer and the second layer of the deep neural network. This makes it possible to speed up the calculations of the deep neural network.

The network replacement device according to aspect 2 has the following features in addition to the features of the network replacement device according to aspect 1. That is, in the network replacement device according to aspect 2, in the identifying step, the processor identifies, using a genetic algorithm, a first nonlinear relational expression expressing the relationship between the output value of a node included in the first layer and the output value of a node included in a third layer between the first layer and the second layer, identifies, using a genetic algorithm, a second nonlinear relational expression expressing the relationship between the output value of a node included in the third layer and the output value of a node included in the second layer, and identifies, using the first relational expression and the second relational expression, a nonlinear relational expression expressing the relationship between the output value of a node included in the first layer and the output value of a node included in the second layer.

According to the above configuration, the network replacement device can speed up the calculations between the first layer and the second layer of the deep neural network. This makes it possible to speed up the calculations of the deep neural network.

The network replacement method according to aspect 3 includes an identification step in which one or more processors use a genetic algorithm to identify a nonlinear relational equation that expresses the relationship between the output value of a node included in a first layer of a deep neural network composed of multiple layers and the output value of a node included in a second layer, and a replacement step in which an operation between the first layer and the second layer in the deep neural network is replaced with an operation using the relational equation.

The trained model according to aspect 4 is a trained model for causing a computer to function so as to output output data based on input data obtained from a deep neural network composed of multiple layers, in which an operation from a first layer to a second layer included in the multiple layers of the deep neural network is replaced by an operation using a nonlinear relational expression identified by a genetic algorithm that represents the relationship between the output value of a node included in the first layer and the output value of a node included in the second layer, and the output value of a node included in the second layer is calculated by an operation using the output value of each node included in the first layer and the relational expression.

The estimation device according to aspect 5 includes one or more processors that execute an estimation step of estimating output data from input data using the trained model according to aspect 4.

The above configuration makes it possible to speed up the calculations between the first layer and the second layer of the deep neural network. This makes it possible to speed up the calculations of the deep neural network.

The network replacement method according to aspect 6 includes an estimation step in which one or more processors estimate output data from input data using the trained model according to aspect 4.

The above configuration makes it possible to speed up the calculations between the first layer and the second layer of the deep neural network. This makes it possible to speed up the calculations of the deep neural network.

[Additional Note 1]
In each of the above-described embodiments and modified examples, the machine learning device 2 constructs the learned model LM1, but the learned model LM1 may be constructed in advance by another device other than the machine learning device 2. In this case, the network substitution device 1 performs the above-described network substitution method M1 using the learned model constructed in advance by the other device.

[Additional Note 2]
In the above-described embodiments and modifications, the processor 11 identifies the nonlinear relational expression representing the relationship between the first data and the second data by nonlinear regression using a genetic algorithm. The algorithm for identifying the nonlinear relational expression representing the relationship between the first data and the second data is not limited to the genetic algorithm, and may be another nonlinear algorithm such as logistic regression.

[Additional Note 3]
The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Other embodiments obtained by appropriately combining the technical means disclosed in the above-described embodiments are also included in the technical scope of the present invention.

1 Network replacement device 2 Machine learning device 3 Estimation device 11, 21, 31 Processor 12, 22, 32 Primary memory 13, 23, 33 Secondary memory 14, 24, 34 Input/output interface 15, 25, 35 Communication interface 16, 26, 36 Bus LM1, LM2 Trained model M1 Network replacement method M11 Estimation step M12 Identification step M2 Machine learning method M21 Training dataset construction step M22 Trained model construction step M3 Estimation method M31 Second estimation step M32 Output step S Network replacement system

Claims (6)

a step of identifying, using a genetic algorithm, a nonlinear relational expression that represents a relationship between an output value of a node included in a first layer of a deep neural network composed of multiple layers and an output value of a node included in a second layer, the relational expression having a one-to-one correspondence with each node included in the second layer;
A replacement step of replacing an operation between the first layer and the second layer in the deep neural network with an operation using the relational expression;
and one or more processors executing
A network replacement device comprising: In the identifying step, the processor identifies a first nonlinear relational expression expressing a relationship between an output value of a node included in the first layer and an output value of a node included in a third layer between the first layer and the second layer, using a genetic algorithm;
identifying a second nonlinear relational expression expressing a relationship between an output value of a node included in the third layer and an output value of a node included in the second layer using a genetic algorithm;
2. The network replacement device according to claim 1, wherein the first relational expression and the second relational expression are used to identify a nonlinear relational expression that represents a relationship between an output value of a node included in the first layer and an output value of a node included in the second layer. a step of one or more processors identifying, using a genetic algorithm, a nonlinear relational expression that represents a relationship between an output value of a node included in a first layer of a deep neural network composed of multiple layers and an output value of a node included in a second layer, the relational expression having a one-to-one correspondence with each node included in the second layer;
A replacement step of replacing an operation between the first layer and the second layer in the deep neural network with an operation using the relational expression;
13. A network replacement method comprising: A trained model for causing a computer to output output data based on input data, the trained model being obtained from a deep neural network composed of multiple layers,
A trained model for causing a computer to function such that operations from a first layer to a second layer included in the multiple layers of the deep neural network are replaced by operations using a nonlinear relational equation identified by a genetic algorithm, which represents the relationship between the output values of nodes included in the first layer and the output values of nodes included in the second layer, and which has a one-to-one correspondence with each node included in the second layer, and the output values of nodes included in the second layer are calculated by operations using the output values of each node included in the first layer and the relational equation. The learning method includes: providing one or more processors that execute an estimation step of estimating output data from input data using the learned model according to claim 4;
An estimation device comprising: An estimation step in which one or more processors estimate output data from input data using the trained model according to claim 4;
The estimation method according to claim 1,

Images