JP6855360B2 - Information estimation device and information estimation method - Google Patents

Description

The present invention relates to an information estimation device and an information estimation method that perform estimation processing using a neural network. In particular, the present invention is an information estimation device and an information estimation method that calculate a variance value indicating uncertainty of an estimation result at high speed by providing a dropout layer in a neural network and performing approximate calculation in the activation layer. is there. If the variance value of the estimated value output as the estimation result is large, the estimation result is uncertain (that is, the reliability is low), and if the variance value of the estimated value output as the estimation result is small, the estimation result is It can be said that it is certain (that is, it is highly reliable).

Compared to other estimators, estimators using neural networks (NN: Neural Network) can process a large amount of information such as images and sensor signal data as input data and perform estimation, so they can be used in various fields. It is expected to be applied to.

The neural network has a structure in which layers for processing data are arranged, and each layer is configured so that data is input, the calculation set for that layer is performed, and then the processed data is output. ing. Specifically, the input data from the observed object is first input to the input layer of the neural network, processed, and then output, and then input to each layer (intermediate layer) in order as input data. While being output after the processing is performed, the processing in each layer is repeated, and the data propagates in the neural network. Then, data is output from the output layer, which is the last layer, and that is the estimation result. The input data from the observed object is n-dimensional vector data of the observation target to be estimated. For example, in the case of a black-and-white camera image of 10 pixels in height and width, 10 × 10 = 100 dimensions having elements corresponding to each pixel. (That is, n = 100) vector data.

In each layer of the neural network, the number of dimensions of the input vector data and the number of dimensions of the output vector data can be set to be different. That is, the number of dimensions of vector data increases or decreases each time it passes through each layer. Further, the number of dimensions of the vector data output from the output layer differs depending on the content to be estimated by the designer. For example, when it is desired to estimate the values of "velocity" and "point", the output from the output layer is n = one-dimensional scalar data. Alternatively, when it is desired to classify whether the input image is "pedestrian", "automobile", or "bicycle" from the input image (that is, when it is desired to classify three classes), from the output layer. The output of is n = three-dimensional vector data that stores the "points" indicating which of the above three classes is applicable.

In addition, there are a learning phase and an estimation phase as processing executed by an estimator that performs estimation processing using a neural network.

In the learning phase, the designer uses the training data to calculate and train the weights of neurons in the neural network to produce the desired specific output for input data with a specific pattern.

In the estimation phase, new data that has never been seen, that is, test data, is input to a neural network with the rules learned in the learning phase and estimated, and if the learning is successful, the estimation result is estimated according to the learned concept. put out.

The major difference between a conventional estimator using a neural network and an estimator using other probabilistic methods such as Bayesian estimation is that the neural network only outputs the estimation result as a "value", and the estimation result is estimated. The variance value that represents the uncertainty of is not calculated.

Since the variance value representing uncertainty cannot be calculated in the neural network, it is difficult to make settings such as adopting only reliable estimation results above a certain level with a threshold set, and there is a high possibility of erroneous judgment. It may be included. For example, when using a neural network in an environment that requires high safety, such as estimating the surrounding conditions of a car, if the estimation result contains an erroneous judgment, it will lead to a serious accident. There is a risk of accidental use.

On the other hand, Non-Patent Document 1 below proposes a method of calculating an output value and its variance value in a neural network. Hereinafter, the calculation method disclosed in Non-Patent Document 1 will be described.

The method of calculating the variance value in Non-Patent Document 1 is to calculate the variance of the estimation result by using the dropout, which is originally used to prevent overfitting during learning, also at the time of estimation. Dropout is defined as, for example, as disclosed in Patent Document 1 below, a dropout layer is provided in a layer of a neural network, and each element of vector data input to the dropout layer is independently provided. , It is a method to make it zero with _{a certain probability p drop} decided in advance by the designer.

For example, when the input vector data is 100 dimensions, that is, it consists of 100 elements, _{whether or not the value contained in each element is independently set to zero with a probability p drop} (if it is not set to zero). The value contained in the original element does not change). As a result, statistically, _{the number of elements of 100 × p drop} out of 100 elements has a value of zero. That is, in the dropout, the calculation process is performed in a state where the number of elements corresponding to the _{probability p drop is missing (set to zero).}

At the time of learning, the weight is calculated so that the output result calculated in the state where the element is missing with the _{probability p drop has the minimum difference with respect to the desired correct answer data.} Also, during learning, this calculation is repeated many times. That is, for each element of another vector data input to the dropout layer, _{it is newly determined whether or not the probability p drop} is set to zero, and the probability p _drop is supported for the other vector data. The calculation process is performed with the number of elements missing, and the weight is calculated so that the difference is minimized with respect to the desired correct answer data. By repeating the learning using the dropout for the vector data input in this way, learning is performed so that the same correct answer data can be output as an estimation result even if any element of the vector data is missing. ..

The calculation method using this dropout has conventionally been applied only during learning. That is, in the past, dropouts were sometimes used during learning, but dropouts were not used during estimation.

However, in Non-Patent Document 1, even in the case of estimation calculation, the estimation calculation is repeated many times with the dropout of the input vector data from the same object, so that the estimation calculation is performed together with the output value. A method for calculating the variance value has been introduced. In Non-Patent Document 1, the estimation using this dropout is called a Monte Carlo (MC) dropout. Since the pattern of the element group of the input vector data in which the probability p _drop is zero in the dropout layer is different for each estimation calculation due to the defect, the final estimation result through the subsequent layers is also different each time. In this specification, the fact that the output estimation result is different each time the vector data is input may be expressed as "fluctuation" of the estimation result.

FIG. 1 (a) shows the state of the output value obtained by one calculation, and FIG. 1 (b) shows the state of distribution with respect to the output value obtained by repeating the calculation many times. FIG. 1A shows how the output values obtained by one calculation are plotted on a graph (the horizontal axis represents the values). Further, in FIG. 1 (b), the output values obtained by the calculation of a plurality of times (10 times in this case) are plotted on a graph (the horizontal axis represents the value and the vertical axis schematically represents the frequency). The state of being done is shown.

Further, the distribution of the output values shown in FIG. 1 (b) is counted as a frequency for each value, and the state represented by the histogram is shown in FIG. 1 (c). In FIG. 1 (c), the state of the distribution shown in FIG. 1 (b) is represented by a histogram showing the relationship between the magnitude (horizontal axis) of the output value and its frequency (vertical axis). By repeating the trials many times in this way, the distribution of the output values can be obtained, and FIG. 1 (c) shows a graph of the probability density distribution of the estimated values obtained by statistical processing (FIG. 1 (c)). The graph drawn by the dotted line inside), the mean value, and the estimated variance value are also shown. In this specification, obtaining a probability density distribution by repeating trials many times in this way may be expressed as "calculating in a Monte Carlo manner".

In Non-Patent Document 1, MC calculations are repeated, MC values (about 200) of final output vector data that change each time are collected, and the variance value of those values is calculated by the following formula. The variance value obtained by this equation is defined as the uncertainty with respect to the input data.

In the above equation, x ^* is the input, y ^* is the output, T is the number of calculations (that is, T = MC), and the left side is the variance value of the ^{output y *.} As in the above equation, the left side (variance value) is the average of the constant term τ ^-1 _ID (first term on the right side ^{) related to the initial variance and the variance value of the output y *} (second term on the right side) to the output y ^* . It is represented by the sum of the values obtained by subtracting the square of (the third term on the right side).

Intuitively speaking, this calculation is an element of the vector data by calculating the estimated value of the neural network for the same object many times and randomly setting the value of the input vector data of the dropout layer to zero each time. By creating random defects in the group, the output data from the dropout layer is intentionally shaken. Even if the output data from the dropout layer is intentionally shaken in this way, if the final estimation result output from the output layer does not shake, that is, if the variance is small, the neural network will be able to do so. It can be considered that the reliability is high with respect to the estimated value. On the other hand, on the contrary, when the final estimation result output from the output layer fluctuates greatly, that is, when the variance is large, the neural network can be considered to be unreliable with respect to the estimated value.

International Publication WO201410508666A1

“Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning”, Yarin Gal, Zoubin Ghahramani: June 6, 2015 (available from https://arxiv.org/pdf/1506.02142v1.pdf)

However, as described above, in order to obtain the variance value for one observation target, the calculation after the dropout layer of the neural network must be repeated many times. For example, the number of calculations is required to be MC times (about 200 times). When the number of calculations is reduced, a smooth distribution shape cannot be obtained with respect to the probability density distribution of the output value, and it is difficult to accurately estimate the variance value. On the other hand, if the number of calculations is increased, it becomes possible to estimate a more accurate variance value, but there is a problem that a huge number of calculations imposes a heavy burden on the calculation process in practice because the calculation process takes time and labor. ..

In order to solve the above problems, the present invention uses a neural network to perform estimation processing that can perform high-speed calculation of a variance value indicating uncertainty of estimation results without performing calculation processing a huge number of times. It is an object of the present invention to provide an information estimation device and an information estimation method for performing the above.

In order to achieve the above object, according to the present invention, the present invention has an integrated layer composed of a combination of a dropout layer for deleting a part of input data and an FC layer or a convolution layer for calculating weights. An information estimation device that performs estimation processing using a neural network having an activation layer that performs calculations using a nonlinear function at least one before and after the integrated layer.
Is it possible to set the variance value of the multivariate distribution output from the activated layer to zero through the calculation in the activated layer by referring to the data related to the multivariate distribution input to the activated layer? The data analysis unit that determines whether or not
An estimation processing unit that skips calculations related to a multivariate distribution determined by the data analysis unit to be able to set the variance value to zero when performing calculations in the integrated layer in the estimation processing. ,
An information estimation device having the information is provided.

Further, in order to achieve the above object, according to the present invention, there is an integrated layer composed of a combination of a dropout layer for deleting a part of input data and an FC layer or a convolution layer for calculating weights. In addition, it is an information estimation method that performs estimation processing using a neural network having an activation layer that performs calculations using a nonlinear function at least one before and after the integrated layer.
Is it possible to set the variance value of the multivariate distribution output from the activated layer to zero through the calculation in the activated layer by referring to the data related to the multivariate distribution input to the activated layer? Data analysis steps to determine if
In the estimation process, when the calculation is performed by the integrated layer, the estimation processing step of skipping the calculation related to the multivariate distribution determined that the variance value can be set to zero in the data analysis step is included. ,
An information estimation method to have is provided.

The present invention has an effect that the calculation of the variance value indicating the uncertainty of the estimation result can be performed at high speed without performing a huge number of calculation processes in the estimation technique using the neural network. It is possible to quickly and easily determine the reliability of the estimation result by the network. Further, according to the present invention, it is possible to determine whether or not to adopt the estimation result according to the reliability, and whether or not to fuse with the estimation result by other Bayesian estimation or the like, and the neural network. The applicable range of can be greatly expanded.

It is a graph which shows the output value obtained from the conventional estimator using a neural network, (a) is a graph which shows the state of the output value obtained by one calculation, and (b) is a graph which shows the state of the output value obtained by one calculation a plurality of times. It is a graph which shows the state of the output value obtained by the calculation of, and (c) is a graph which shows the histogram of the output value obtained by a plurality of calculations. It is a figure which shows an example of the partial structure of the neural network handled by the estimation calculation which is the premise of the embodiment of this invention, (a) is the figure of the neural network composed of a DF1 layer, an activation layer and a DF2 layer. A diagram showing an example, (b) is a diagram showing an example of a neural network composed of a DF1 layer and an activation layer, and (c) is a diagram showing an example of a neural network composed of an activation layer and a DF2 layer. Is. It is a figure which shows typically the calculation process of the DF1 layer in the estimation calculation which is the premise of the embodiment of this invention. It is a figure which shows typically the calculation process of the DF2 layer in the estimation calculation which is the premise of the embodiment of this invention. It is a flowchart which shows an example of the calculation process of the mean value and the variance value in the DF1 layer in the estimation calculation which is the premise of the Embodiment of this invention. It is a flowchart which shows an example of the calculation process of the covariance value in the DF1 layer in the estimation calculation which is the premise of the embodiment of this invention. It is a flowchart which shows an example of the calculation process of the mean value and the variance value in the DF2 layer in the estimation calculation which is the premise of the Embodiment of this invention. It is a block diagram which shows an example of the structure of the information estimation apparatus in 1st Embodiment of this invention. In the first embodiment of the present invention, it is a graph related to the ReLU function used as the activation function, (a) is a graph showing an example of the ReLU function used as the activation function, and (b) is a graph showing an example of the ReLU function used as the activation function. The graph showing an example of the distribution input to the activation function (distribution biased to the negative region), (c) shows activation when the distribution input to the activation function is biased to the negative region. A graph showing an example of the output from the function, (d) is a graph showing an example of the distribution (distribution biased to the positive region) input to the activation function, and (e) is input to the activation function. It is a graph which shows an example of the output from the activation function in the distribution where the distribution is biased to the positive region. In the first embodiment of the present invention, it is a graph related to the sigmoid function used as the activation function, (a) is a graph showing an example of the sigmoid function used as the activation function, and (b) is a graph showing an example of the sigmoid function used as the activation function. The graph showing an example of the distribution input to the activation function (distribution biased to the negative region), (c) shows activation when the distribution input to the activation function is biased to the negative region. A graph showing an example of the output from the function, (d) is a graph showing an example of the distribution (distribution biased to the positive region) input to the activation function, and (e) is input to the activation function. It is a graph which shows an example of the output from an activation function in the case where the distribution is biased to a positive region. In the first embodiment of the present invention, it is a diagram schematically showing how the output is approximated by a delta function when a distribution biased to a negative region is input to the activation layer. It is a figure for demonstrating an example of the criteria for determining that the Gaussian distribution is biased to a negative region in the 1st Embodiment of this invention. In the first embodiment of the present invention, it is a figure for demonstrating the relationship between a variance value and a covariance value, and FIG. A diagram showing how the covariance values of are distributed with a certain spread, and (b) is a diagram showing that when one of the variance values for the two indexes is zero, these covariance values become zero. , (C) is a diagram showing that when both the variance values for the two indexes are zero, these covariance values become zero. It is a flowchart which shows an example of the calculation process of the mean value and the dispersion value in the DF1 layer in the 1st Embodiment of this invention. It is a flowchart which shows an example of the calculation process of the covariance value in the DF1 layer in the 1st Embodiment of this invention. It is a flowchart which shows an example of the calculation process of the mean value and the dispersion value in the DF2 layer in the 1st Embodiment of this invention. Experimental results obtained by using the proposed method in the first embodiment of the present invention, and experimental results when other methods are used (experiments using the estimation calculation which is the premise of the embodiment of the present invention). As a result, it is a graph which shows (the result of an experiment using the technique proposed in Non-Patent Document 1).

Hereinafter, the first and second embodiments of the present invention will be described.

First, the outline of the neural network estimation calculation (hereinafter, referred to as the estimation calculation which is the premise of the embodiment of the present invention) which is the premise of the first and second embodiments of the present invention will be described. The presumed calculation that is the premise of the embodiment of the present invention was invented by the same inventor as the present application, and is described in the specification and drawings of Japanese Patent Application No. 2016-252813 previously filed by the applicant. It is the same as the one. Japanese Patent Application No. 2016-252813 has not been published at the time of filing the application of the present application, and does not constitute a prior art for the present invention.

The estimation calculation that is the premise of the embodiment of the present invention takes the burden of calculation processing (time and labor associated with a huge number of calculations) in the method for calculating the variance value in a Monte Carlo manner proposed in Non-Patent Document 1 described above. It was conceived to mitigate, and the variance value is calculated analytically.

In the estimation calculation proposed in Non-Patent Document 1, random calculation results are repeatedly generated using Monte Carlo dropouts at the time of estimation calculation of the neural network, and the calculation results are statistically aggregated to obtain the output value. The method of constructing a distribution and finding the variance value from it is used. In this case, the number of times the calculation is repeated MC needs to be at least MC = 200 times or more empirically, and if it is less than this, an accurate distribution shape or dispersion value cannot be obtained. On the other hand, in the estimation calculation which is the premise of the embodiment of the present invention, it is possible to directly analytically obtain the distribution that the result of the iterative calculation will make without performing the iterative calculation like Monte Carlo. It is possible to calculate the variance value from the distribution.

FIG. 2 shows an example of the partial structure of the neural network handled by the estimation calculation which is the premise of the embodiment of the present invention. A neural network having a dropout layer that loses a part of input data and an integrated layer consisting of a combination of a fully connected (FC) layer that calculates weights is, for example, FIG. 2 (a). It partially has such a structure. That is, from the input side (input vector Xin) to the output side (input vector Xout), a dropout 1 layer, an FC (Fully Connected) 1 layer, an activation layer, a dropout 2 layer, and the last FC (Fully Connected) 2 The layers have a structure arranged in this order, and the value output from the final layer is the estimated value of the neural network.

Since the dropout layer and the FC layer perform continuous calculation processing in an integrated manner, the layers in which the dropout layer and the FC layer are integrated (FC layer with a dropout) are integrated together. It may be called a layer (DF layer). In FIG. 2A, the partial structure of the neural network consists of a DF1 layer (a layer in which a dropout 1 layer and an FC1 layer are combined), an activation layer, and a DF2 layer (a dropout 2 layer and an FC2 layer). However, it is not limited to the structure shown in FIG. 2 (a), and is composed of only the DF1 layer and the activated layer, for example, as shown in FIG. 2 (b). As shown in FIG. 2C, it may be composed of only the activation layer and the DF2 layer.

For example, suppose you want to calculate the variance of an estimated value in a neural network with the partial structure of FIG. 2 (a). It should be noted that the layers before the DF1 layer do not have a layer that performs calculation processing that creates randomness like the dropout layer. In this case, the input value to the DF1 layer is not a random variable having a distribution, but a fixed value like a normal neural network.

Hereinafter, with reference to FIGS. 3 and 4, how the input value is converted into a random variable, propagated in a distribution, and output as an estimated value in the final layer will be specifically described.

As described above, the input value to the DF1 layer is a fixed value. Therefore, the input value Xin ^DF1 _{to the DF1 layer is the n Xin} ^{DF 1-} dimensional vector Xin ^DF1 as shown below. The j-th element (index j) of the vector Xin ^DF1 is represented as ^{Xin DF1} _{j. The notation n Xin} ^DF1 in the specification indicates that the subscript of n is Xin ^DF1 .

As schematically shown in FIG. 3, the input vector Xin ^DF1 is input to the DF1 layer, and the dropout probability set in the dropout ^{1 layer of the DF1} _{layer p drop} DF1 and the weight preset in the FC1 layer. The ^{following calculation is performed using W DF1} and bias b ^DF1 , and the calculation result is output as the output ^{vector Xout DF1.}

According to the estimation calculation which is the premise of the embodiment of the present invention, the output value from the DF1 layer is the n _Xout ^{DF 1-} dimensional vector Xout ^DF1 , and the i-th element of this vector Xout ^{DF1 is as follows.} _{The notation n Xout} ^DF1 in the specification indicates that the subscript of n is Xout ^DF1 .

Furthermore, due to the dropout _{, the} ^Wi , j DF1 Xin ^DF1 _j terms (1 ≤ j ≤ n _Xin ^DF1 ) on _{the right side disappear randomly with the probability p drop} ^{DF1 , so the xout DF1 on the} left side, which is the sum of each term. _i can be calculated as a "sampling sum". The sum of samplings is no longer a fixed value, but a random variable with a multivariate Gaussian distribution consisting of ^{the mean vector μ DF1} _i and the variance-covariance matrix Σ ^DF1 _{i, and is expressed as follows.}

The above is the _{calculation in which the fixed value n Xin} ^{DF 1-} dimensional input vector Xin ^DF1 _{is output as a random variable n Xout} ^{DF 1-} dimensional multivariate Gaussian distribution through the DF1 layer. ..

In the estimation calculation which is the premise of the embodiment of the present invention, the calculation process using the above-mentioned formulas for calculating the mean value, the variance value, and the covariance value is performed in the DF1 layer. When this calculation process is represented by the program process, it is as follows.

5 and 6 show a flowchart of the calculation process represented by the program process as described above, paying particular attention to the loop process. FIG. 5 is a flowchart showing an example of calculation processing of the mean value and the variance value in the DF1 layer in the estimation calculation which is the premise of the embodiment of the present invention.

In the flowchart of FIG. 5, the average value and the variance value of one n _Xout ^DF are calculated for one element of the n _Xout ^{DF of the vector Xout DF1 (loop L1).} Calculation of the mean value mu ^DF1 _i for the i-th element of the vector Xout ^DF1 (index i) comprises n _Xin ^DF1 times of the average value calculation processing (step S101) (loop L2), the i-th element of the vector Xout ^DF1 The calculation of the variance value diag (Σ ^DF1 ) _i with respect to (index i) includes the variance value calculation process (step S102) of _{n Xin} ^{DF once (loop L3).} Therefore, the n _Xout ^{DF 1-} time loop contains the n _Xin ^{DF 1-} time loop, and in order to calculate the mean and variance values for all the elements of the ^{vector Xout DF1, n for each of the mean and variance values.} _Xout ^DF1 × n _Xin ^{DF One} calculation needs to be performed.

Further, FIG. 6 is a flowchart showing an example of the calculation process of the covariance value in the DF1 layer in the estimation calculation which is the premise of the embodiment of the present invention. In the calculation of the covariance value of the vector Xout ^DF1, separate elements as with a vector Xout ^DF1 (e.g. i1 th element) (e.g. i2 th element: however, i1 ≠ i2) for the combination of, n _Xin ^DF1 times It is necessary to execute the covariance value calculation process (step S201) of. That is, in the flowchart of FIG. 6, the n _Xout ^DF1 × n _Xout ^DF1 or combinations, n _Xout ^DF1 × n _Xout ^DF1 times calculation is performed (loop L4 and loop L5), further these loops L4 and loop L5 , N _Xin ^{DF One-} time covariance value calculation process (step S201) is included (loop L6). Therefore, for the calculation of the covariance value, n _Xout ^DF1 × n _Xout ^{DF 1} loop contains n _Xin ^{DF 1} loop, and in order to calculate the covariance value for all the elements of the ^{vector Xout DF1} Approximately n _Xout ^DF1 x n _Xout ^DF1 x n _Xin ^{DF One} calculation needs to be performed.

Since the calculation related to the diagonal term (i1 = i2) is not executed in the calculation of the covariance value, the number of calculations is _{less than n Xout} ^DF1 × n _Xout ^DF1 × n _Xin ^DF1 times, but n _Xout ^DF1 × n _Xout. It is clear that the number of calculations of the order (number of digits) similar to that of ^DF1 × n _Xin ^{DF1 is required.} Also, of course, by using the symmetry of the variance-covariance matrix, it is possible to calculate only one off-diagonal term and the other off-diagonal term by its inversion, but the amount of calculation is It is only reduced by half.

Further, in the estimation calculation which is the premise of the embodiment of the present invention, it is proposed that the input / output data is regarded as a mixed Gaussian distribution. In this case, it is necessary to perform the above-described processing independently for each Gaussian distribution constituting the data, but the description thereof will be omitted here.

For example, in the neural network having the partial structure shown in FIG. 2A, the output value Xin ^DF1 _{(multivariate Gaussian distribution of n Xout} ^DF1 dimension, which is a random variable) output from the DF1 layer is input to the activation layer. .. In the activation layer, for example, after calculation processing by an activation function (non-linear function) such as a ReLu (Rectified Linear Unit) function or a sigmoid function is executed, a vector having the same number of dimensions as the input vector (n ^{Xout DF 1} dimension) is _generated. It is output and input to the DF2 layer. Therefore, the number of dimensions n _Xin ^DF2 of the input vector of the DF2 layer is the same as the _{number of dimensions n Xout} ^DF1 _{of the input vector of the DF2 layer (n Xout} ^DF1 = n _Xin ^DF2 ). _{The notation n Xin} ^DF2 in the specification indicates that the subscript of n is Xin ^DF2 .

The multivariate Gaussian distribution input to the DF2 layer is expressed as follows.

Note that μ _in ^DF2 is an n _Xin ^{DF two-} dimensional vector indicating the average, and the element at the i-th index (1 ≦ i <n _Xin ^DF2 ) is expressed _{as μ in} ^DF2. Further, Σ _in ^DF2 is an n _Xin ^DF2 × n _Xin ^{DF two-} dimensional variance-covariance matrix, and the diagonal term of the index (i, i) th (1 ≦ i ≦ n _Xin ^DF2 ) indicates the variance value. The element is written _{as Σ in} ^DF2 _i. On the other hand, the off-diagonal term at the index (i, j) th (1 ≦ i, j, i ≠ j ≦ n _Xin ^DF2 ) indicates the covariance value, and this element is expressed as Σ _in ^DF2 _{i, j} .

As schematically shown in FIG. 4, the input vector Xin ^DF2 is input to the DF2 layer, and the dropout probability set in advance in the dropout 2 layer of the DF2 layer p _drop ^DF2 , the weight preset in the FC2 layer. The ^{following calculation is performed using W DF2} and bias b ^DF2 , and the calculation result is output as an output ^{vector Xout DF2.}

According to the estimation calculation which is the premise of the embodiment of the present invention, the output vector Xout ^DF2 in the DF2 layer is also considered as the sampling sum by the dropout, and is defined as follows, as in the calculation process in the DF1 layer. n _Xout ^{DF Two-} dimensional multivariate Gaussian distribution. _{The notation n Xout} ^DF2 in the specification indicates that the subscript of n is Xout ^DF2 .

Note that μ _out ^DF2 is an n _Xout ^{DF two-} dimensional vector indicating the average, and the element at the i-th index (1 ≦ i <n _Xout ^DF2 ) is expressed _{as μ out} ^DF2. Further, Σ _out ^DF2 is an n _Xout ^DF2 × n _Xout ^{DF two-} dimensional variance-covariance matrix, and the diagonal term of the index (i, i) th (1 ≦ i ≦ n _Xout ^DF2 ) indicates the variance value. The element is written _{as Σ out} ^DF2 _i. On the other hand, the off-diagonal term at the index (i, j) th (1 ≦ i, j, i ≠ j ≦ n _Xout ^DF2 ) indicates the covariance value, and this element is expressed as Σ _out ^DF2 _{i, j} .

In ^{the covariance part of Xout DF2} , the DF2 layer is the final layer of the neural network, and only the dispersion value indicates the uncertainty of the output value (that is, the estimated value by the neural network) output from the final layer. There is no need to calculate it as out of interest. However, after this, if the DF2 layer is not the final layer and the output value from the DF2 layer is input to a further dropout layer (third dropout layer), the output value of the DF2 layer is also covariant. You need to calculate the value.

In the estimation calculation which is the premise of the embodiment of the present invention, the calculation process using the above-mentioned formula for calculating the mean value and the variance value is performed in the DF2 layer. When this calculation process is represented by the program process, it is as follows.

FIG. 7 shows a flowchart expressing the calculation process represented by the program process as described above, paying particular attention to the loop process. FIG. 7 is a flowchart showing an example of calculation processing of the mean value and the variance value in the DF2 layer in the estimation calculation which is the premise of the embodiment of the present invention. In the flowchart of FIG. 7, the n _Xout ^DF2 five element vector Xout ^DF2, the calculation of n _Xout ^DF2 times of the mean value and the variance value is performed (loop L7). The calculation of the variance values of the _{mean values μ out} ^DF2 _i and ListW ^DF2 μ _in ^DF2 _i for a certain element (for example, the i-th element) of the vector Xout ^DF2 includes the mean value calculation process (step S301) of _{n Xin} ^{DF2 times.} Yes (loop L8). On the other hand, the dispersion value, n _Xin ^DF2 times VarW _Xin ^DF2 _i in the loop _L9, calculation of the sum of _jj (step S302), relates to the combination of n _Xin ^DF2 × n _Xin ^DF2 amino index in loops L10 and loop L11 After executing the calculation of the sum of CovWXin ^DF2 _{i, jj1, jj2, jj1 <jj2} (step S303), the variance value diag (Σ _out ^DF2 ) _i of the output value of the DF2 layer is calculated from these results (step). S304). Therefore, for the calculation of the variance value, the n _Xout ^DF2 loop contains the n _Xin ^DF2 × n _Xin ^DF2 loop, and in order to calculate the variance value for all the elements of the ^{vector Xout DF2, it is approximately n.} _Xout ^DF2 x n _Xin ^DF2 x n _Xin ^{DF Two} calculations need to be performed.

As described above, the estimation calculation which is the premise of the embodiment of the present invention includes the loop processing in which the calculation processing in the DF1 layer and the DF2 layer requires a huge number of calculations. For example, in a typical neural network such as AlexNet, the number of dimensions n _Xin ^FC2 of the input vector of the FC layer is 4096, and the number of dimensions n _Xout ^FC2 of the output vector of the FC layer is 1000. The calculation of the dispersion value alone requires repeated calculation of 1000 × 4096 × 4096 times. Therefore, even in the estimation calculation which is the premise of the embodiment of the present invention, there is a problem that when the number of neurons of the neural network is large and the number of layers is further increased, it is necessary to execute a huge number of calculations. There is.

Hereinafter, the first and second embodiments of the present invention will be described. The estimation calculation in the first embodiment of the present invention has made it possible to calculate the variance value representing the uncertainty at a higher speed by further improving the estimation calculation which is the premise of the above-described embodiment of the present invention. It is a thing. Further, the estimation calculation in the second embodiment of the present invention is an improvement of the estimation calculation in the first embodiment of the present invention, and the calculation is interrupted when the estimation result is determined to be uncertain. This prevents unnecessary calculation processing from being performed.

<First Embodiment>
In the estimation calculation in the first embodiment of the present invention, in the activation layer of the neural network, the input distribution becomes a compressed and crushed distribution such that the mean value and the variance value are almost zero at the output. Focusing on the fact that distribution calculation may not be necessary in the subsequent layers, the mean and variance values can be set (approximate) to zero by the compressed collapsed distribution, and when it is not. By simply dividing the calculation into two, the number of calculations in the loop processing can be reduced and high-speed calculation processing can be realized. When the multivariate distribution (for example, multivariate Gaussian distribution) input to the activated layer is output after the calculation in the activated layer, the mean value and the variance value are set (approximate) to zero. It will be described later that it may have a shape of a possible distribution (crushed distribution).

FIG. 8 is a block diagram showing an example of the configuration of the information estimation device according to the first embodiment of the present invention. The information estimation device 10 shown in FIG. 8 has an estimation calculation unit 20 and a data analysis unit 30. The information estimation device 10 shown in FIG. 8 is an estimator that performs estimation processing using a neural network, and the neural network is a dropout layer that omits a part of input data and an FC layer (Fully Connected) that calculates weights. Layer), it has an activation layer that performs calculations using non-linear functions. The FC layer can be regarded as mathematically equivalent to the convolution layer (convolution layer).

The block diagram shown in FIG. 8 merely represents a function related to the present invention, and in an actual implementation, it may be realized by hardware, software, firmware, or any combination thereof. Functions implemented in software are stored in any computer-readable medium as one or more instructions or codes, and these instructions or codes are stored in a CPU (Central Processing Unit) or GPU (Graphics Processing Unit). It can be executed by a hardware-based processing unit such as a graphics processing unit). Further, the functions related to the present invention may be realized by various devices including an IC (Integrated Circuit), an IC chipset, and the like.

Similar to the conventional neural network, the estimation calculation unit 20 has a function related to estimation calculation which is a premise of the embodiment of the present invention, in addition to a function of performing calculation processing of input data in each layer and outputting an estimation result (a function related to estimation calculation which is a premise of the embodiment of the present invention. For example, it has a function to execute the calculation process described using the flowcharts shown in FIGS. 5 to 7), and the input data is propagated to each layer with what kind of distribution by dropout and output. It is configured to calculate the shape of the distribution and the distribution value output from the final output layer.

On the other hand, the data analysis unit 30 has a function of executing a process characteristic of the first embodiment of the present invention. Specifically, does the data analysis unit 30 output the distribution input to the activation layer of the neural network as a compressed and crushed distribution from the activation layer by the calculation processing in the activation layer? It has a function to judge whether or not. Then, the data analysis unit 30 significantly reduces the number of calculations by controlling the distribution that can be determined to be output as a compressed and crushed distribution from the activation layer so as not to execute the calculation (skip the calculation). Therefore, the calculation time can be shortened and the calculation process can be speeded up.

Hereinafter, an approximation process that takes into account the calculation process in the activated layer, which is executed by the data analysis unit 30, will be described. Since the above description was intended to show the enormous number of calculations of the estimation calculation which is the premise of the embodiment of the present invention, it is assumed that nothing is processed in the activation layer. However, in reality, the calculation process using the activation function is executed in the activation layer. Hereinafter, the ReLU function and the sigmoid function, which are usually used as activation functions, will be described with reference to FIGS. 9 (a) to 9 (e) and FIGS. 10 (a) to 10 (e).

FIG. 9A is a diagram showing an example of the ReLU function used as the activation function. The ReLU function is expressed as follows in the mathematical formula.

max (x, y) is a function that is x when x> y, and y otherwise. That is, the function f (x) = max (x, 0) has an output value of zero when the input value x is negative, and the input value x is the output value x as it is when the input value x is zero or more. It becomes. When a distribution consisting of the above Gaussian functions (Gaussian distribution) is calculated as an input value for this function f (x), what kind of distribution the output value will be, the average value of the input Gaussian distribution. 9 (b) to 9 (e) show the cases depending on the situation.

For example, when the average value of the Gaussian distribution of the input values is negative as shown in FIG. 9 (b), the function f (x) which is the output value is as shown in FIG. 9 (c). In FIG. 9B, since most of the values constituting the Gaussian distribution are in the negative region, the output value f (x) by the ReLU function is almost zero. That is, as shown in FIG. 9 (c), in the input Gaussian distribution, the positive region of the Gaussian distribution of the input value barely retains its shape, but the value of the negative region occupying most of it is zero. It becomes a crushed state near the, and becomes a state of a sharp peak like a delta function.

On the other hand, when the average value of the Gaussian distribution of the input values is positive as shown in FIG. 9 (d), the function f (x) is as shown in FIG. 9 (e). In this case, since most of the values of the Gaussian distribution are in the positive region, most of the output value f (x) by the ReLU function is the same as the input value, and the distribution is almost unchanged. ..

Further, FIG. 10A is a diagram showing an example of a sigmoid function used as an activation function. The sigmoid function is expressed as follows in the mathematical formula.

Similarly, when the distribution (Gaussian distribution) consisting of the Gaussian functions as described above is calculated as the input value for this function f (x), what kind of distribution the output value will be, of the input Gaussian distribution. 10 (b) to 10 (e) show what kind of case may occur depending on the average value.

For example, when the average value of the Gaussian distribution of the input values is negative as shown in FIG. 10 (b), the function f (x) which is the output value is as shown in FIG. 10 (c). Similar to the case where the activation function is the ReLU function, when the activation function is a sigmoid function, the output value f (x) by the sigmoid function is almost the same when most of the Gaussian distribution of the input value is in the negative region. It becomes zero, and as shown in FIG. 10 (c), the distribution of output values is crushed near zero.

On the other hand, when the average value of the Gaussian distribution of the input values is positive as shown in FIG. 10 (d), the function f (x) becomes as shown in FIG. 10 (e). In this case, since most of the values of the Gaussian distribution are in the positive region, the output value f (x) by the sigmoid function is close to the value whose distribution is almost 1 as shown in FIG. 10 (d). It will be in a crushed state.

As described above, in both cases where the activation function f (x) is the ReLU function or the sigmoid function, if most of the input distribution is biased to the negative region, the output value f (x) The distribution of is shaped like a delta function having a concentrated distribution near x = 0. In the first embodiment of the present invention, paying attention to the property of the activation function as described above, the data analysis unit 30 determines whether or not the distribution input to the activation layer is biased to the negative region. However, when it is determined that the distribution input to the activation layer is biased to the negative region, as shown in FIG. 11, the distribution output from the activation layer in response to the input distribution is It is considered that the state of the delta function (mean value = 0, variance value = 0) is reached. If it is determined that the distribution input to the activation layer is not biased to the negative region, the same calculation processing as usual may be performed for the distribution, or approximation using an arbitrary method. Calculations may be performed.

Whether or not the distribution input to the activation layer is biased to the negative region can be determined based on arbitrary conditions, but as an example, the average value μ of the distribution and the width of the distribution, that is, It is possible to make a judgment based on the standard deviation σ (or the square root of the variance value). For example, suppose that the Gaussian function input to the activation layer and the Gaussian function output from the activation layer are defined by the mean value and the standard deviation as follows.

At this time, for example, as shown in FIG. 12, the input mean value μ _in of the Gaussian distribution exists in the negative region, and the standard deviation σ _in representing the distribution width of the Gaussian distribution is multiplied by a negative sign. If it is smaller, it may be determined that the Gaussian distribution is on the negative side. This condition is expressed by the following equation.

_{In some cases, the Gaussian distribution is negative when the input mean value μ in} of the Gaussian distribution is smaller than the value obtained by multiplying the standard deviation σ _in by the negative constant −c, as shown in the following equation. You may judge that it is on the side of.

For example, in the example of the ReLU function shown in FIGS. 9 (a) to 9 (e), whether or not the Gaussian distribution input to the activation layer is biased to the negative region by using the function ActivationLayer as shown below. It is possible to determine the average and dispersion values of the Gaussian distribution output from the activated layer by determining.

The above function ActivationLayer simply determines that the Gaussian distribution input to the activation layer is biased to the negative region (μ _in <−σ _in ), the distribution output from the activation layer. Is considered to be a delta function with zero mean and variance, while otherwise the output distribution remains in the shape of the input Gaussian distribution.

Alternatively, if the Gaussian distribution input to the activation layer can be considered to be biased towards the negative region (eg μ _in <−σ _in ), it returns true, otherwise it returns true. May set a function ActionLayerMakeZero that returns false.

When the above functions are used, the calculation processing in the DF1 layer and the DF2 layer is significantly reduced. Hereinafter, as an example, a neural network having a partial structure shown in FIG. 2A (a neural network composed of a DF1 layer, an activation layer, and a DF2 layer) will be described.

First, in DF1 layer, after calculating the variance of Xout ^DF1, it is necessary to further calculate the covariance values Xout ^DF1. In the estimation calculation that is the premise of the embodiment of the present invention, it is necessary to perform the calculation _{about n Xout} ^DF1 × n _Xout ^DF1 × n _Xin ^{DF 1 time as described above.}

However, the covariance value offdiag (Σ ^DF1 ) _{i1, i2, i1 ≠ i2 is the distribution Xout i1} ^DF1 for each combination of indexes i1 and i2 (however, 1 ≦ i1, i2 ≦ n _Xout ^DF1 , i1 ≠ i2). , Xout _i2 ^DF1 and if one of the variances is zero (the distribution is represented as a delta function), i.e. var (Xout ^DF1 ) _i1 and var (Xout ^DF1 ) _i2 If at least one of the values is zero, the covariance values Cov (Xout ^DF1 ) _{i1 and i2} can be regarded as zero. The situation is shown in FIG. In the combination of the two indexes, when both variance values are not zero as shown in FIG. 13 (a), the covariance values have a distribution with a certain spread, while those in FIGS. 13 (b) and 13 As shown in (c), when at least one of the variance values is zero, the covariance value can be considered to be zero.

Therefore, when calculating the covariance value, it is not necessary to calculate all combinations of the two indexes, and the distribution value that is not biased to the negative region (the variance value of the distribution output from the activation layer is regarded as zero). It is only necessary to calculate for the combination of each other. Focusing on this point, as represented by the processing of the following program (processing in the DF1 layer), a table called ZeroFlag (i) for recording Boolean data is prepared, and the distribution is biased to the negative region. (True) is returned to the index of (distribution value of the distribution output from the activation layer can be regarded as zero), while distribution that is not biased to the negative region (output from the activation layer). False (false) is returned to the index of the distribution (distribution whose variance value of the distribution cannot be regarded as zero).

In addition, as represented by the processing of the following program (processing in the DF1 layer), in the calculation of the covariance value, the table of ZeroFlag (i) is referred to, and the calculation is performed for the index in which true is recorded. While skipping, the calculation is performed only for the index in which false (false) is recorded. As a result, in the calculation process of the covariance value in the DF1 layer, only the one in which false (false) is set in ZeroFlag is the calculation target, so that the number of calculations is significantly reduced.

14 and 15 show flowcharts related to the calculation process in the DF1 layer represented by the program process as described above. Here, the same processing as the flowcharts of FIGS. 5 and 6 described above will not be described.

In the flowchart shown in FIG. 14, first, the process of initializing all the covariance values to zero is performed (step S401). Further, when the mean value μ ^DF1 _i and the variance value diag (Σ ^DF1 ) _{i of} a certain index i are calculated, it is further determined whether the distribution is biased to the negative region, and the distribution is biased to the negative region. If yes, true is recorded in ZeroFlag (i), otherwise false is recorded in ZeroFlag (i) (step S402). Here, as an example, the function ActivationLayerMakeZero is used, for example, when “μin <−σin”, true is recorded in ZeroFlag (i), and when not, false (false) is recorded in ZeroFlag (i). ) Is recorded, but it is not limited to this condition.

Further, in the flowchart shown in FIG. 15, it is determined whether or not ZeroFlag (i1) is false (false) (step S501) and whether or not ZeroFlag (i2) is false (false) (step S502). Only when both ZeroFlag (i1) and ZeroFlag (i2) are false (false), the nonzero covariance value is calculated. At this time, when ZeroFlag (i1) is true, or when ZeroFlag (i2) is true, the calculation process of the covariance value is skipped, so that the number of calculations can be significantly reduced. Can be done. The covariance value for which the calculation process is skipped becomes zero set by the initialization process in step S401.

Further, in the DF2 layer, a calculation process represented by the following program processing (processing in the DF2 layer) is performed. The DF2 layer is the final layer, and as described above, only the variance value indicates the uncertainty of the output value output from the final layer. Therefore, the covariance value is not of interest here. Not calculated.

FIG. 16 shows a flowchart regarding the calculation process in the DF2 layer represented by the program process as described above. Here, the description of the same processing as that of the flowchart of FIG. 7 described above will be omitted.

In the flowchart shown in FIG. 16, first, the initialization process for making all the variance values zero is performed (step S601). Further, when calculating the variance value, it is determined whether or not ZeroFlag (jj) is false (step S602), and only for ^{VarWXin DF2} _{i, jj in which ZeroFlag (jj) is false (false).} On the other hand, if ZeroFlag (jj) is true, the calculation is skipped. Further, it is determined whether or not ZeroFlag (jj1) is false (step S603) and whether or not ZeroFlag (jj2) is false (false) (step S604). ^{Only the combination of CovWXin DF2} _{i, jj1, jj2, jj1 <jj2} in which both jj2) are false (false) is calculated, while ZeroFlag (jj1) is true, or ZeroFlag (jj2) is true. If (true), the calculation is skipped. As a result, in the calculation process of the dispersion value in the DF2 layer, only the one in which false (false) is set in ZeroFlag is the calculation target, so that the number of calculations is significantly reduced.

In the process according to the first embodiment of the present invention described above, it is considered that most of the dispersion values become zero in the calculation process in the activation function when the process of iterative calculation by the loop is regarded as a "matrix" calculation. It can be said that by making the matrix sparse, the calculation amount is reduced and the calculation time is shortened to speed up the calculation.

The above-mentioned process according to the first embodiment of the present invention is described based on the structure of a neural network in which the activation layer is sandwiched between two DF layers as shown in FIG. 2 (a). However, it is not limited to such a structure. In the first embodiment of the present invention, when the distribution input to the activation layer is biased to the negative region, the distribution output from the activation layer (for example, a multivariate mixed Gaussian distribution) is averaged. Focusing on the fact that it can be approximated to a state like a delta function in which the values and variance values are almost zero, the calculation is skipped for elements with such a distribution to reduce the amount of calculation and the calculation time. It is characterized by. Therefore, for example, the structure including the DF layer and the activating layer as shown in FIG. 2 (b) and the structure including the activating layer and the DF layer as shown in FIG. 2 (c) are also of the present invention. It is possible to apply the process of the first embodiment.

In addition, an experiment carried out using the proposed method in the first embodiment of the present invention described above will be described. In the actual experiment, first, the image provided by ImageNet was used and trained as a 1000-class classifier by AlexNet of the neural network. Then, using this neural network, the variance value of the estimated value output by 1000 neurons corresponding to each of 1000 classes was calculated for a certain input image by forward calculation. The experimental results are shown in FIG. In the graph, the vertical axis is the variance value directly linked to the uncertainty, and the horizontal axis corresponds to the first 10 classes out of 1000 classes.

In FIG. 17, three types of bar graphs are arranged for each class, and from the left, the experimental results (reference numeral X) carried out using the proposed method according to the first embodiment of the present invention, and the implementation of the present invention. Experimental results (reference code Y) using the estimation calculation (proposal method in Japanese Patent Application No. 2016-252813), which is the premise of the above form, and experimental results based on the technique proposed in Non-Patent Document 1 (200 Monte Carlo-like). Iterative calculation) (reference code Z) is represented. As shown in FIG. 17, since similar variance values can be calculated in each method, even though the proposed method in the first embodiment of the present invention uses approximate calculation, It can be seen that the accuracy of the estimation result can be maintained. Further, in the estimation calculation which is the premise of the embodiment of the present invention, it took 20 minutes to obtain the estimation result, but on the other hand, according to the estimation calculation by the proposed method in the first embodiment of the present invention. The estimation result can be obtained in 15 seconds, and the calculation speed is greatly improved.

<Second embodiment>
Next, a second embodiment of the present invention will be described. The estimation calculation in the second embodiment of the present invention is calculated when it is found that the variance value representing the uncertainty of the estimation result output from the DF2 layer, which is the final layer, becomes larger than a preset threshold value. By interrupting the processing, the calculation processing is speeded up.

The configuration of the information estimation device 10 according to the second embodiment of the present invention is the same as that of the information estimation device 10 shown in FIG. 8, but the data analysis unit 30 further outputs a dispersion finally output from the neural network. It has a function to judge whether the value (estimated value output from the DF2 layer in FIGS. 2A to 2C) is larger than a certain threshold value and end the calculation according to the judgment result. There is.

The variance value finally output from the neural network represents the uncertainty of the estimation result. For example, by suspending and terminating the calculation process when it is found that the variance value becomes larger than the threshold value for determining the reliability even though the user wants a reliable estimation result to some extent. , Unnecessary calculation processing that will result in unreliable estimation results can be stopped.

The second embodiment of the present invention can be realized by improving the first embodiment of the present invention described above. Focusing again on the dispersion value in the DF2 layer, the dispersion value in the DF2 layer is expressed as follows.

The first term on the right-hand side of the above (Equation 1) _{is always positive because the dropout probability P drop} is less than 1 and the variance value Var is positive. The second term on the right side of (Equation 1) above is the product of the positive coefficient and the sum Σ of ^{VarWXin DF2} _{i, jj.} VarWXin ^DF2 _{i, jj} is the variance value of the index jj of the output Xout ^DF1 _{of the DF1 layer multiplied by (Wi, jj} ^DF2 ) ² , and since the variance value is zero or more, VarWXin ^DF2 _{i, jj} Is always greater than or equal to zero.

Therefore, the second term on the right side of the above (Equation 1) is also always positive.

On the other hand, the third term on the right side of the above (Equation 1) includes the sum Σ of ^{CovWXin DF2} _{i, jj1, jj2, jj1 <jj2. ^{CovWXin DF2 i, jj1, jj2,}} jj1 <jj2 is the covariance value of the index JJ1 and index JJ2 output Xout ^DF1 DF1-layer, which multiplied by _{^{W i, jj1 DF2 · W i}} , jj2 DF2, positive , Zero, or negative.

Here, what the user wants to know is the reliability of the estimation result finally output from the neural network, not the variance value itself finally output from the neural network. It is only necessary to present to the user whether the variance value diag (Σ _out ^DF2 ) _{i of} the DF2 layer directly connected to is larger or smaller than a certain threshold value.

For example, in the addition of the first to third terms on the right side of the above (Equation 1), first, the process of calculating the third term on the right side, which is uncertain whether positive or negative, is performed (step A), and then. By performing the process of adding the remaining right-hand side first and second terms that are always zero or more (step B), it can be efficiently determined that the dispersion value of the DF2 layer becomes larger than a certain threshold value. .. As a result, when it is determined that the dispersion value of the DF2 layer becomes larger than a certain threshold value, the reliability of the estimation result can be regarded as low, and the calculation process related to the estimation result can be interrupted and terminated.

Specifically, first, in the DF1 layer, the mean value and the variance value are calculated in the same manner as in the first embodiment described above, while the covariance value is as follows when expressed by the processing of the program. Do the calculation like.

In the above calculation of the covariance value, as in the first embodiment, the calculation is performed only for those whose ZeroFlag (i) table is false, but further, CovWXin ^DF2 _{i, jj1, jj2} By simultaneously performing the calculation (step A) relating to the sum Σ (TotalCovWXin ^DF2 _{s ) of, jj1 <jj2} , the value of the third term on the right side of the above (Equation 1) is calculated.

Further, in the DF2 layer, the average value is calculated in the same manner as in the first embodiment described above, while the variance value is calculated as follows when expressed by the processing of the program.

In the DF2 layer, when the dispersion value is obtained, the first initialized dispersion value (initial value is zero) is added to the third term (step A) on the right side of the above equation (1) already calculated in the DF1 layer. The calculation result) is added, and then the remaining right-hand side first and second terms, which are zero or more values, are added to this value. In the calculation represented by the processing of the above program, the first term on the right side is first added, and then the second term on the right side is calculated and added, and the calculation result is equal to or higher than a preset predetermined threshold (Threshold). If this happens, it is determined that the final variance value will be greater than the predetermined threshold (that is, an uncertain estimation result that is less reliable than the specified level), and the calculation will be interrupted and terminated. .. At this time, the reason why the calculation is interrupted may be presented to the user, for example, by presenting that the estimation result reliability of the current calculation is low.

INDUSTRIAL APPLICABILITY The present invention can calculate the variance value, which is the uncertainty of the estimation result, at high speed in the estimation device using the neural network, and can be applied to all the techniques related to the neural network. Furthermore, the present invention can greatly expand the scope of application of neural networks, and is greatly applicable in environments where high-speed and reliable processing is required, such as estimation of moving objects such as automobiles and pedestrians. It is possible for the function to be exhibited.

10 Information estimation device 20 Estimation calculation unit 30 Data analysis unit

Claims (12)

It has an integrated layer consisting of a combination of a dropout layer that deletes a part of the input data and an FC layer or a convolution layer that calculates weights, and a nonlinear function at least one before and after the integrated layer. It is an information estimation device that performs estimation processing using a neural network with an activation layer that performs calculations using.
Is it possible to set the variance value of the multivariate distribution output from the activated layer to zero through the calculation in the activated layer by referring to the data related to the multivariate distribution input to the activated layer? The data analysis unit that determines whether or not
An estimation processing unit that skips calculations related to a multivariate distribution determined by the data analysis unit to be able to set the variance value to zero when performing calculations in the integrated layer in the estimation processing. ,
Information estimation device to have. The data analysis unit can set the variance value of the multivariate distribution output from the activation layer to zero when the multivariate distribution input to the activation layer is biased to a negative region. The information estimation device according to claim 1, which is configured to determine that. When the estimation processing unit calculates the covariance value using the two dispersion values in the integrated layer arranged in front of the activation layer, at least one of the two dispersion values is determined. The covariance value calculated using the two variance values when it is related to the data related to the multivariance distribution determined by the data analysis unit to be able to set the variance value to zero. The information estimation device according to claim 1 or 2, which is configured to skip the calculation of. The estimation processing unit is any one of claims 1 to 3 configured to record whether or not the variance value and covariance value of the multivariate distribution input to the activation layer are set to zero. The information estimation device described in. In the integrated layer arranged after the activated layer, the estimation processing unit is output from the integrated layer from the dispersion value and covariance value of the multivariate distribution input to the integrated layer. A claim configured to skip the calculation for data in which the variance and covariance values of the multivariate distribution input to the activation layer are set to zero when calculating the variance value of the variate distribution. The information estimation device according to any one of 1 to 4. When the estimation processing unit calculates the dispersion value of the multivariate distribution output from the integrated layer arranged after the activation layer, the dispersion value of the multivariate distribution output from the integrated layer is calculated. The information estimation device according to any one of claims 1 to 5, which is configured to end the calculation process when it is determined that the value becomes larger than a predetermined threshold value. It has an integrated layer consisting of a combination of a dropout layer that deletes a part of the input data and an FC layer or a convolution layer that calculates weights, and a nonlinear function at least one before and after the integrated layer. This is an information estimation method that performs estimation processing using a neural network with an activation layer that performs calculations using.
Is it possible to set the variance value of the multivariate distribution output from the activated layer to zero through the calculation in the activated layer by referring to the data related to the multivariate distribution input to the activated layer? Data analysis steps to determine if
In the estimation process, when the calculation is performed by the integrated layer, the estimation processing step of skipping the calculation related to the multivariate distribution determined that the variance value can be set to zero in the data analysis step is included. ,
Information estimation method to have. In the data analysis step, when the multivariate distribution input to the activation layer is biased to a negative region, the variance value of the multivariate distribution output from the activation layer can be set to zero. The information estimation method according to claim 7, further comprising a step of determining that. In the estimation processing step, when the covariance value is calculated using the two dispersion values in the integrated layer arranged in front of the activation layer, at least one of the two dispersion values is determined. The covariance value calculated using the two variance values when it is related to the data related to the multivariance distribution determined by the data analysis unit to be able to set the variance value to zero. The information estimation method according to claim 7 or 8, further comprising a step of skipping the calculation of. The estimation processing step further comprises a step of recording whether or not the variance value and covariance value of the multivariate distribution input to the activation layer are set to zero, according to any one of claims 7 to 9. The described information estimation method. In the integration layer arranged after the activation layer, the estimation processing step is output from the integration layer from the dispersion value and covariance value of the multivariate distribution input to the integration layer. Claim 7 further includes a step of skipping the calculation relating to the data in which the variance value and the covariance value of the multivariate distribution input to the activation layer are set to zero when calculating the variance value of the variable distribution. The information estimation method according to any one of 10 to 10. In the estimation processing step, when calculating the dispersion value of the multivariate distribution output from the integrated layer arranged after the activation layer, the dispersion value of the multivariate distribution output from the integrated layer is calculated. The information estimation method according to any one of claims 7 to 11, further comprising a step of ending the calculation process when it is determined that the value becomes larger than a predetermined threshold value.

Images