JP6754343B2 - Neural network regulators, devices and programs - Google Patents

Description

The present invention relates to an adjusting device that adjusts a neural network so that it can be operated in a low processing power, for example, a portable device.

A neural network is used to make some determination based on an input or to determine the output for an input. Patent Document 1 discloses that a neural network is used to determine the behavior of a person from a signal output by a sensor such as an acceleration sensor attached to the person.

U.S. Pat. No. 9,278,255

In order to obtain a highly accurate output from a neural network, it is necessary to construct a neural network using a large number of neurons. However, such a neural network has a high processing load, which makes it difficult to operate it on a device having a low processing capacity, for example, a portable device.

The present invention provides an adjusting device that adjusts a neural network so that it can be operated even by a device having a low processing capacity.

According to one aspect of the present invention, the adjusting device that adjusts the first neural network to generate the second neural network is a first means for obtaining a first value indicating the importance of each neuron of the first neural network. , The second means for obtaining the second value indicating the calculation cost of each neuron of the first neural network, and the set in which a predetermined number of neurons are selected from all the neurons of the first neural network are included in the set. An evaluation value is calculated based on the first value and the second value of each of the included neurons, and a selection means for selecting one set based on the evaluation value and the predetermined number included in the selected set. The second neural network by deleting the other neurons from the first neural network, leaving the neurons of the above and the neurons necessary for the predetermined number of neurons included in the selected set to perform the calculation. It is characterized by having a generation means for generating a network.

According to the present invention, the adjusting device can adjust the neural network so that it can be operated even in a device having a low processing power.

The block diagram of the neural network used for the explanation of one Embodiment. Explanatory drawing of term used in embodiment. Explanatory drawing of term used in embodiment. The figure which shows the intrinsic importance of each neuron. Explanatory diagram of the calculation method of the total importance of each neuron. The figure which shows the total importance of each neuron. The figure which shows the calculation cost of each neuron. Explanatory drawing of the calculation method of the correlation value of a pair of neurons. Explanatory drawing of the selection method of a neuron. Explanatory drawing of the selection method of a neuron. Explanatory drawing of the selection method of a neuron. Explanatory drawing of the selection method of a neuron. The block diagram of the adjustment apparatus by one Embodiment.

Hereinafter, exemplary embodiments of the present invention will be described with reference to the drawings. The following embodiments are examples, and the present invention is not limited to the contents of the embodiments. Further, in each of the following figures, components that are not necessary for the description of the embodiment will be omitted from the drawings.

FIG. 1 shows a neural network for explaining the present embodiment. As mentioned above, in order to obtain more accurate output, a neural network is constructed using a large number of neurons, but here, in order to simplify the explanation of the principle of the present invention, neuron # 1 This will be described using a neural network having a total of 12 neurons of ~ # 12. The content to be determined based on the output of this neural network is arbitrary, but here, as an example, it is assumed that the content of human behavior is determined based on the output of the neural network. In this case, the neural network of FIG. 1 is generated by learning on a device having high processing power based on the output of, for example, sensors worn by a large number of people. Further, in the present embodiment, human behavior is determined based on the output values of all neurons of the neural network. Therefore, it is assumed that the judgment information indicating the relationship between the output values of each of neurons # 1 to # 12 and the human behavior is also determined in advance by learning performed on a device having high processing power. Then, for example, when the output of the sensor attached to a certain person is given as an input, the behavior of the person is determined based on the output value and the determination information of each of the neurons # 1 to # 12.

The adjusting device according to the present embodiment constructs a new neural network by reducing the number of neurons of the neural network shown in FIG. 1 in order to enable the neural network to operate even with a device having a low processing capacity such as a mobile terminal. Therefore, the number N is determined first. The number N is smaller than the number M of neurons in the original neural network, and this number N is determined based on the processing power of a device having a low ability to operate the neural network.

First, before explaining the processing in the adjusting device according to the present embodiment, the terms used in the present embodiment will be defined. A neuron that directly gives an input to a neuron is called a "parent neuron" of the neuron. That is, the neuron to which a certain path is connected is called the parent neuron of the neuron to which the path is connected. Furthermore, all the neurons included when following the path from a certain neuron to the first layer neuron (neurons # 1 to # 4 in FIG. 1) in the direction opposite to that direction are referred to as the "neuron". It shall be referred to as "all parent neurons" or "all parent neurons". Specifically, the parent neurons of neuron # 9 in FIG. 2 are neurons # 5, # 6 and 7, and all parent neurons of neuron # 9 are neurons # 1 to # 7. The whole parent neuron of a neuron is a neuron that affects the input value of the neuron.

Similarly, a neuron that takes the output of a neuron as a direct input is called a "child neuron" of that neuron. That is, the neuron to which a certain path is connected is called a child neuron of the neuron to which the path is connected. Furthermore, all the neurons included when the path is forwarded from a certain neuron to the final layer neuron (neurons # 10 to # 12 in FIG. 1) are referred to as "all child neurons" or "all child neurons" of the neuron. It shall be called "all child neurons". Specifically, the child neurons of neuron # 2 in FIG. 3 are neurons # 5 and 7, and the whole child neurons of neuron # 2 are neurons # 5, 7 and # 8 to # 12. A whole child neuron of a neuron is a neuron whose input is affected by the output of that neuron.

The coordinator of the present embodiment determines the "importance" for each of the neurons # 1 to # 12. The importance can be determined by any known algorithm, such as Linear Regression or Random Forest. In the following description, the importance of a neuron obtained by a known algorithm such as linear regression is referred to as "inherent importance" in order to distinguish it from the total importance described later. FIG. 4 shows the intrinsic importance of each of the neurons # 1 to # 12 used in the description of this embodiment.

The coordinator determines the "total importance" T for each of the neurons # 1 to # 12 based on their intrinsic importance. The total importance of a neuron is obtained as the sum of the sum of the intrinsic importance of all child neurons of the neuron multiplied by a predetermined coefficient and the intrinsic importance of the neuron. Here, the predetermined coefficient is an integer of 1 or less. For example, if the intrinsic importance of a neuron is A, the sum of the intrinsic importance of all child neurons of the neuron is B, and a predetermined coefficient is γ, the total importance T of the neuron is T = A + γB. .. The table in FIG. 5 shows for each neuron its intrinsic importance, all child neurons, and total importance. The predetermined coefficient γ is set to 0.01 here. For example, since the total child neurons of neuron # 6 are neurons # 9, # 11 and # 12, the sum of their intrinsic importance is 11.2 + 5.4 + 12.3 = 28.9. Therefore, when multiplied by the coefficient α = 0.01, the value is 0.289, and by adding the intrinsic importance of neuron # 6, the total importance of neuron # 6 is calculated to be 4.589. Will be done. FIG. 6 shows the total importance of each neuron # 1 to # 12 in each neuron # 1 to # 12. The calculation method of the total importance of neurons is not limited to the calculation method of this example. For example, the total importance of a certain neuron may be a value obtained by multiplying the sum of the total importance of the child neurons of the neuron by a predetermined coefficient and adding the intrinsic importance of the neuron. In this case, the intrinsic importance is calculated in order from the neurons in the final layer. In this way, if the total importance of a certain neuron is a value that increases as the intrinsic importance of the neuron and the intrinsic importance of each of all child neurons of the neuron increase, the calculation method is another method. You may.

Further, the adjusting device of the present embodiment obtains the "calculation cost" C for each of the neurons # 1 to # 12. The calculation cost C is calculated as the total calculation cost of the parent neuron + 1. FIG. 7 shows the calculation cost of each of the neuros # 1 to # 12. For example, neurons # 1 to # 4 do not have a parent neuron, so their calculation costs are all 1. The parent neurons of neuron # 5 are neurons # 1 to # 4, and the total calculation cost is 4. Therefore, the computational cost of neuron # 5 is 5. Similarly, the computational costs of neurons # 6 and # 7 are 3 and 4, respectively. Further, the parent neurons of neuron # 9 are neurons # 5 to # 7, and the total calculation cost is 12. Therefore, the computational cost of neuron # 9 is 13. The calculation method of the calculation cost of the neuron is not limited to the calculation method of this example. For example, the computational cost of a neuron can be the number of all parent neurons of that neuron plus one. As described above, if the calculation cost of a certain neuron becomes a value that increases as the number of all parent neurons increases, the calculation method may be another method.

Further, the coordinator of the present embodiment obtains a "correlation value" Corr for each combination of two selected neurons # 1 to # 12. For example, the output of each neuron # 1 to # 12 when an arbitrary value is input to the neural network of FIG. 1 is calculated. FIG. 8 shows the output of each neuron # 1 to # 12 when a total of n values are input to the neural network. Here, the average value of the output of neuron # 1 for each input is E (X), and the average value of the output of neuron # 2 is E (Y). Further, let E (X ² ) be the average value of the squares of the outputs of neuron # 1 for each input, and let E (Y ² ) be the average value of the squares of the outputs of neuron # 2. Further, let E (XY) be the average value obtained by multiplying the output of neuron # 1 and the output of neuron # 2 for each input. In this case, the correlation value Corr of the pair of neurons # 1 and # 2 is
Corr = M / N
M = (E (XY) -E (X) E (Y))
N = (E (X ² ) -E (X) ² ) ^0.5 (E (Y ² ) -E (Y) ² ) ^0.5
Can be obtained as. The calculation of the correlation value is a so-called Pearson correlation coefficient, but it is an example, and the calculation method of the correlation value of the pair of neurons is not limited to the calculation method of this example. Specifically, an arbitrary calculation method such that the correlation value of two neurons that output similar values becomes high regardless of the input can be used for the calculation of the correlation value.

The coordinator obtains the evaluation value Ev for each combination in which a predetermined number N is selected from all neurons # 1 to # 12 by the following equation (1).
Ev = ΣT-αΣC-βΣCorr (1)
In equation (1), ΣT indicates that the total importance of each of the N neurons included in the selected set is integrated, and ΣC is the calculation cost of each of the N neurons included in the selected set. Is shown to be integrated. In addition, ΣCorr indicates that the correlation value of each pair selected from N neurons included in the selected pair is integrated. Note that α and β are adjustment coefficients for adjusting the value of the calculation cost and the value of the correlation value to the value of the total importance, respectively, and both are positive values.

For example, when 3 N, adjustment device _calculates an evaluation value Ev for the set each 12 _{C 3.} The calculation of the evaluation value Ev for the set of neurons # 1, 2 and 4 will be described below. It should be noted that α is 0.2 and β is 3. First, the total importance of neurons # 1, 2 and 4, respectively, is 3.929, 11.789 and 1.432, respectively, as shown in FIG. Therefore, the ΣT of the equation (1) is 17.15. The calculation costs of neurons # 1, 2 and 4, respectively, are 1, 1, 1 as shown in FIG. Therefore, since the ΣC of the equation (1) is 3 and α = 0.2, the αΣC of the equation (1) is 0.6.

Further, the correlation value of the pair of neurons # 1 and # 2 is 0.3, the correlation value of the pair of neurons # 1 and # 4 is 0.2, and the correlation value of the pair of neurons # 2 and # 4 is 0.2. Assuming that it is 0.1, the ΣCorr of the equation (1) is 0.6 and β = 3, so the βΣCorr of the equation (1) is 1.8. Therefore, in this case, the evaluation value Ev is 17.15-0.6-1.8 = 14.75.

Adjusting device, in this way, in the case of a neural network with M neurons, and calculates an evaluation value Ev for the set each _M C _N, it is determined to leave a large set of neurons in most evaluation value Ev. After that, the coordinator determines which neurons should be left based on the neurons decided to be left, and deletes the other neurons. In the following, when N = 4 is set and it is decided to keep neurons # 1, # 3, # 6 and # 11 shown by shading in FIG. 9 based on the evaluation value Ev, the neurons to be left and the neurons to be deleted are further described. How to judge will be described.

Since neuron # 11 needs the output of neurons # 8 and # 9 for its calculation, it is determined that neurons # 8 and # 9 must remain. FIG. 10 also shades neurons # 8 and # 9 that are determined to remain. Neurons # 8 and # 9 then need neurons # 5 and # 7 for their calculations, and neurons # 5 and # 7 need neurons # 2 and # 4 for their calculations. Therefore, it is determined that neurons # 2, # 4, # 5 and # 7 must also be left. FIG. 11 shades all neurons that are determined to remain. Therefore, the regulator determines to delete neurons # 10 and # 12, which are not necessary for the calculations in neurons # 1, # 3, # 6 and # 11, which are determined to remain. Therefore, the neural network after deletion is as shown in FIG.

In FIG. 12, a total of nine neurons exist in the neural network, but the neurons actually used as outputs are neurons # 1, # 3, # 6, and # determined to be left based on the evaluation value Ev. There are only 11. That is, neurons # 2, # 4, # 6, # 7 to # 9 are left so that neurons # 1, # 3, # 6 and # 11 decided to keep can perform calculations, and their outputs are The value is not used for judgment. The coordinator then re-learns to create determination information indicating the relationship between the output values of neurons # 1, # 3, # 6 and # 11, which are determined to remain, and human behavior.

For example, the neural network and determination information shown in FIG. 12 generated in this way are set in a portable terminal device such as a smartphone. Here, the number of neurons in the original neural network is set to 12 for the sake of brevity, but in reality, a very large number of neurons are included, depending on the processing capacity of the target terminal device. By setting N, a neural network with a light processing load can be constructed. Since the number of neurons in the neural network actually used by the terminal device is larger than N, N is set in consideration of the increase. Further, in the present embodiment, since the number of neurons used for the determination is N, the determination process based on the output of the neural network is also lightened.

In the present embodiment, the correlation value is also used in the calculation of the evaluation value Ev in the equation (1), but the configuration may be such that the correlation value is not used. That is, the evaluation value Ev can be obtained by the following equation (2).
Ev = ΣT-αΣC (2)
According to the calculation formula of the formula (2), the N neurons determined to be left first have a high total importance and a low calculation cost. In other words, neurons with low total importance and high computational cost are difficult to select. In this way, in a neural network, by eliminating neurons that have high computational cost and are not very important, and by making judgments by important neurons that have low computational cost, high-precision but processing generated by a device with high processing power From a neural network with a heavy load, it is possible to generate a neural network with a low processing load while suppressing deterioration of accuracy.

In the equation (1) using the correlation value, it becomes difficult to select two neurons having a large correlation value. The outputs of two neurons with high correlation values are similar, and even if a pair of two neurons with high correlation values is left, the two outputs are redundant. Therefore, by setting the correlation value of each of the N neurons selected by the equation (1) to be low, it is possible to eliminate neurons that perform redundant output.

FIG. 13 is a configuration diagram of an adjusting device according to the present embodiment. The holding unit 10 holds the trained neural network and the number N. Here, holding a neural network means holding information about each neuron constituting the neural network, the connection state by those paths, and the weight of each path. The importance calculation unit 11 calculates the intrinsic importance of each neuron based on the neural network held by the holding unit, and further calculates the total importance of each neuron based on the intrinsic importance of each neuron. The cost calculation unit 12 calculates the calculation cost of each neuron based on the neural network held by the holding unit. The processing unit 14 determines N neurons having the maximum evaluation value Ev of the equation (2) based on the intrinsic importance of each neuron and the calculation cost. Then, the determined N neurons and the neurons necessary for the calculation of the N neurons are left, and the other neurons are deleted to generate a neural network that can be used in a device having low processing power. In addition, judgment information for making a judgment based on the output of the determined N neurons of the generated neural network is determined by learning. This determination information is also used together with the neural network generated by the device with low processing power. As described above, the evaluation value Ev of the formula (1) can be used instead of the evaluation value Ev of the formula (2). In this case, as shown in FIG. 13, the adjusting device is provided with the correlation calculation unit 13. The correlation calculation unit 13 calculates the correlation value of each of the two neuron pairs.

The adjusting device according to the present invention can be realized by a program that operates a computer as the adjusting device. These computer programs are stored in a computer-readable storage medium or can be distributed over a network.

11: Importance calculation unit, 12: Cost calculation unit, 13: Correlation calculation unit, 14: Processing unit

Claims (10)

An adjustment device that adjusts the first neural network to generate the second neural network.
A first means for obtaining a first value indicating the importance of each neuron in the first neural network, and
A second means for obtaining a second value indicating the calculation cost of each neuron of the first neural network, and
For each group in which a predetermined number of neurons are selected from all the neurons of the first neural network, an evaluation value is calculated based on the first value and the second value of each neuron included in the group. A selection means for selecting one set based on the evaluation value, and
The predetermined number of neurons included in the selected set and the neurons required for the predetermined number of neurons included in the selected set to perform calculations are left, and the other neurons are referred to as the first neural network. By deleting from, the generation means for generating the second neural network and
An adjusting device characterized by being equipped with. The evaluation value of the set shall be a value that increases as the first value of each neuron included in the set increases, and decreases as the second value of each neuron included in the set increases. The adjusting device according to claim 1. The adjusting device according to claim 2, wherein the selection means selects the set having the maximum evaluation value. The adjusting device according to any one of claims 1 to 3, wherein the first value of the neuron is a value that increases as the importance of the neuron increases. The adjusting device according to any one of claims 1 to 4, wherein the first value of the neuron is a value that increases as the importance of the child neuron of the neuron increases. The adjusting device according to any one of claims 1 to 5, wherein the second value of the neuron is a value that increases as the number of all parent neurons of the neuron increases. For each of the two neuron pairs of the first neural network, a third means for obtaining a third value indicating the correlation between the two neurons included in the pair is further provided.
Claim 1 is characterized in that the selection means uses the third value of each of two pairs of neurons selected from the predetermined number of neurons included in the set to calculate the evaluation value of the set. 6. The adjusting device according to any one of 6. The adjusting device according to claim 7, wherein the evaluation value of the set becomes smaller as the third value of the pair of neurons included in the set becomes larger. A device characterized in that the second neural network generated by the adjusting device according to any one of claims 1 to 8 is used. A program comprising operating a computer as the adjusting device according to any one of claims 1 to 8.

Images