JP4352340B2 - Neural network output value evaluation method and storage medium - Google Patents

Description

  The present invention relates to a reliability evaluation of an output value of a neural network and a neural network evaluation method that indicates the basis of the evaluation (hereinafter also referred to as this).

  Neural networks are used in many fields such as control, prediction and diagnosis because of their characteristics such as learning ability, excellent non-linearity and pattern matching.

  Various types of neural networks have been proposed, but most of them are in practical use with a three-layer network structure. In a three-layer neural network, learning is usually performed in advance by an algorithm called a back-propagation method, and an internal coupling state is adjusted. This learning is to adjust the connection weight inside the neural network so as to obtain the desired function based on the given data. When the correct output is determined for the input, the neural network correctly outputs it. This is done by adjusting whether or not the coupling load is adjusted so that there is no mistake.

  In the neural network thus learned, when the same input data as the learning data given at the time of learning is given, the same data as at the time of learning data is outputted. Further, when input data close to learning data is given, there is a feature that outputs close to learning data. Therefore, if the input case is the same as that given at the time of learning, the output value is sufficiently reliable, but if it is different from the case used for learning, it is not 100% reliable. Therefore, it is important to evaluate the reliability of the output value for evaluating whether the output value obtained for a given input case is appropriate.

  As a conventional method for evaluating the reliability of an output value of a neural network, for example, Japanese Patent Application No. 4-26562 (Japanese Patent Laid-Open No. 5-225163) “Neural Network System and Neural Network Learning Method” has been proposed.

  The evaluation method disclosed in this publication is characterized in that an evaluation neural network for evaluating the reliability is provided separately from the normally used neural network. This neural network for evaluation outputs 0 when learned data is input, and outputs 1 when unlearned data is input. Therefore, when the evaluation neural network outputs a value close to 0, since the pattern of the input data is a learned pattern, the output value of the neural network is usually highly reliable, and the evaluation neural network is close to 1. When outputting a value, since the pattern of the input data is an unlearned pattern, it is evaluated that the reliability is low.

  As another evaluation method of a neural network, there is a method disclosed in Japanese Patent Laid-Open No. 5-282276 “Neural network output state reliability level determination method and determination system”.

  This evaluation method is an evaluation method for a neural network specialized in pattern recognition, and the reliability is evaluated when the output is digital and multi-output, only one output is 1 and all other outputs are 0. It is an evaluation method that can be done. In this evaluation method, when only one output value is close to 1 and all other output values are close to 0, it is determined that the output result is highly reliable.

  Further, as an output evaluation method of a method for explaining the basis for the output value, there is a method disclosed in Japanese Patent Application Laid-Open No. 10-74188 “Data learning apparatus and plant control apparatus”.

  In this evaluation method, similar data is searched from learning cases in which the output values of the neural network are stored in advance, and if similar data is found, this is used as the basis for prediction. The output value is described as data.

  Neural networks are used in many fields because of their learning ability, nonlinear ability, pattern matching ability, etc., but because the internal structure is nonlinear and complex, is the output value of the neural network highly reliable? It was difficult to judge and evaluate. It was also difficult to easily explain why the output value was output.

  For example, in each of the conventional evaluation methods described above, in the first method disclosed in “Neural Network System and Neural Network Learning Method”, an evaluation neural network for determining whether an input pattern has been learned or not learned is provided. Since it is necessary to prepare separately, there is a disadvantage that the system becomes complicated. Further, there is a problem regarding the reliability of the evaluation neural network itself. For example, if the evaluation neural network is not well trained, the reliability evaluation value itself is low in reliability.

  Further, in the method disclosed in the second “method and system for determining the reliability level of the neural network output state” described above, the structure of the neural network has multiple outputs, and only one of the outputs is 1 or 0.9. This is an evaluation method that can be used only when it is specialized in a digital case where all other outputs are 0 or 0.1, and should be applied to neural networks having other structures. I can't.

  Further, the evaluation method disclosed in the third “data learning device and plant control device” described above is a method of calculating the similarity between the input data to be explained and the input data of the past case from the Euclidean distance, and is simple. In addition, this method is easy to explain to the operator. However, since the weighting of a plurality of input data is not taken into consideration, there is a possibility that the output value of the case extracted as a similar case with respect to the output value of the neural network is greatly different from the actual value.

  As a simple example to explain this point, a problem example using a regression equation is given. In the regression equation y = 100 * x1 + x2 (both x1 and x2 are in the range of 0 to 1), the explanation target case is (x1, x2) = (1, 1), and the past case is case 1 (x1, x2) = (1 , 0), Case 2 (x1, x2) = (0.5, 1), Case 2 is more similar to Case 1 than Case 1 when judged by Euclidean distance. There is a problem that is considered. In order to avoid this, it is understood that in the regression equation, the importance of the input factor x2 is 1/100 compared to x1, and it is necessary to perform a search that places importance on the input factor x1.

  An object of the present invention is to provide a method for explaining and evaluating a neural network output value in view of the above problems. It is another object of the present invention to provide a neural network evaluation method that can easily show the basis of the neural network output value to the operator.

  The present invention is an evaluation method that uses a computer to evaluate an output value output from a learned neural network using a learning case as an input as the similarity between the evaluation case and the learning case. A plurality of learning examples used for learning of the neural network are stored in a memory, and the importance of each input factor constituting the input of the neural network is determined from the connection state of the neural network, and the evaluation target case and The degree of similarity with each learning case stored in the memory is obtained using the degree of importance.

  The present invention also extracts a connected state of a learned neural network using a plurality of cases as learning cases when used by a computer, obtains the importance of each input factor of the neural network from the connected state, and In order to cause the computer to calculate the similarity between the input case to the neural network to be evaluated based on the degree and the learning case, and to extract the similar case with respect to the evaluation case based on the similarity This computer-readable recording medium stores the program.

  According to the present invention, since the internal state is complicated, it is difficult to easily explain why the output value is calculated. In the neural network, it is possible to provide one method for explaining the output value. .

  In the present invention, similar cases of the input cases are extracted from the learning cases of the neural network. From the extracted similar cases, for example, “There are similar cases of XX in the learning data. The basis of the output result can be explained. Furthermore, the similarity with the similar case can be shown as an index of reliability for the output result.

  Further, according to the present invention, the similarity can be extracted by determining the importance of the neural network input factor from the connection state of the neural network, and similar cases can be extracted according to the importance of the input data. Therefore, highly accurate explanation and evaluation are possible.

  FIG. 1 is a diagram showing a basic configuration of a neural network evaluation apparatus according to this embodiment.

  The neural network 20 is used after learning in advance a large number of cases in which input and output are paired. When the learned neural network 20 receives the same input data as the learning case, it outputs the same output value as the learning case. Further, even if the input data is a pattern that does not exist in the learning case, there is a feature that a value close to the output value of the learning case having the input value close to the pattern is output. The evaluation device 10 of the present embodiment uses this feature to search and extract similar cases of input data from the cases used by the neural network 20 for learning.

  The neural network 20 changes the internal coupling state by learning. There is a feature that the coupling of the internal state becomes strong for certain important input data, and the coupling of the internal state becomes weak or no coupling for unnecessary input data. In other words, not all input data is calculated in the same way, and there are some types of input data that greatly affect the output result and those that do not have much influence. In consideration of this point, in this embodiment, similar cases are extracted by determining the importance of each input factor of the neural network from the connection state inside the neural network, and obtaining the similarity based on the importance. This makes it possible to search for similar cases that place importance on input factors with high importance and neglect input factors with low importance.

  If a similar case that is very close to the input pattern is found in the learning data by this search, the reliability of the output value of the neural network is considered high, and the value is output because the similar case has already been learned. Can be explained. Conversely, if a similar case with a high degree of similarity is not found, the reliability of the output value of the neural network is assumed to be low, and it can be explained that the value is output because there is no input pattern similar to the learning case.

  The evaluation device 10 of FIG. 1 includes an importance level determination unit 11, a similarity calculation unit 12, a learning case storage unit 13, and an output evaluation unit 14.

The learning case storage unit 13 inputs a plurality of learning cases used during learning of the neural network 20 that evaluates the output value as input values (x _1, x ₂ ,... X _i ) and output values output as a result thereof. (Y ₁ , y ₂ ,... Y _o ) are stored in association with each other. The importance determination unit 11 reads the connection state in the neural network 20 and obtains the importance for each input factor x _{1 to} x _i of the neural network 20 from the state of the internal connection. Since the neural network 20 has already been learned and its connection state does not change, the importance level determination unit 11 does not read out the connection state of the neural network 20 from the neural network 20 but stores it in advance. As good as The similarity calculation unit 12 calculates the importance of each of the input factors x _{1 to} x _i obtained by the importance determination unit 11 by using the similarity between the case to be evaluated and each learning case stored in the learning case storage unit 13. Find using.

  The output value evaluation unit 14 sets a learning case with the highest similarity obtained by the similarity calculation unit 12 or a learning case with a similarity equal to or higher than a certain value as a similar case, and outputs the similar case as an output value evaluation.

  In the present embodiment, the evaluation neural network 20 is an output value for a three-layer hierarchical neural network having a configuration including i input layer elements, h intermediate layer elements, and o output layer elements as shown in FIG. It is assumed that the reliability of the system is evaluated.

In the following description, x represents input data to the neural network, and y represents output data. Further, n pieces of learning data used for learning of the neural network 20 having the number of inputs i and the number of outputs o shown in FIG. 2 are represented by x ₁₁ to x _1i and y _{11 to} y _1o.
:
:
x _{n1 to} x _ni , y _{n1 to} y _no
These are stored in the learning case storage unit 13 as learning examples. Then a case description target output output data y ₁ ~y _o when the input data of the input data x ₁ ~x _i Toko neural network 20, which is learned by the training examples is given, as the evaluation Assume that the output values are explained by outputting similar cases and their similarities.

In the evaluation apparatus 10, first, the importance level determination unit 11 determines the importance level for each of the input factors x _{1 to} x _i of the neural network 20 from the connection state of the neural network 20. Then, using this importance, the similarity calculation unit 12 calculates the similarity of each learning case in the learning case storage unit 13. As a result, it is possible to search for similar cases based on the degree of connectivity within the neural network 20, and to extract similar cases that place importance on input factors with high importance and neglect input factors with low importance.

There are various methods for determining the importance. One example is shown below. In the case of a neural network consisting of i input layer elements i, intermediate layer elements h, and output layer elements o in FIG. 2, and the coupling between the layers is w and v, the input factor x _k (k = 1 to i) The importance can be expressed by the following formula, for example.

  Here, the expression (1) is an expression expressing the importance by paying attention only to the coupling state of the input layer and the intermediate layer, and the expression (2) is the input layer and the intermediate layer, the intermediate layer and the output layer 2 This is an expression expressing the importance by paying attention to two combined states. Good determination of importance can be performed by the determination method according to any one of the equations (1) and (2). The importance level determination unit 11 reads the internal coupling state from the neural network 20 and obtains the importance level of each input factor by the above formulas and the like.

  Using the importance obtained in this way, the similarity calculation unit 12 calculates the similarity between each learning case of the neural network 20 stored in the learning case storage unit 13 and the input case serving as an explanation target. The similarity can be calculated from the difference between the input factors x to be compared and the importance of the input factors, and various specific calculation formulas can be considered. An example is shown below.

The above equations (3) to (6) are obtained by calculating the similarity _j between the input cases x _{1 to} x _i and the learning cases x _{j1 to} x _ji (j = 1 to n) stored in the learning case storage unit 13. It is a formula showing. In the equations (3) to (6), the similarity j means that the similarity is higher as the value is larger. Regardless of which of these formulas (3) to (6) is used, the similarity can be calculated in consideration of the importance of each input factor.

  Also, from the similarity j expressed by the equations (3) to (6), it can be seen that the larger the value is, the higher the similarity is. However, since it is indicated by a negative value, how much similarity is shown. It is difficult for operators to understand. In view of this, it is conceivable that the output value is more easily understood by the operator by expressing the degree of similarity as a percentage. Various formulas can be considered for this purpose. For example, the following formula is obtained from the formula (4).

  By such an expression, the similarity between the input case and each learning case is represented by a degree of 0 to 100%. By displaying this similarity degree together with each learning case, the operator becomes a target. It is possible to easily grasp how many similar cases exist as learning examples. In this equation (7), it is assumed that the input data x is normalized to a value between 0 and 1.

  Each learning case is output from the similarity calculation unit 12 to the evaluation unit 14 together with the similarity to the case to be explained, and the evaluation unit 14 learns the case having the highest similarity among these, or learning with a similarity greater than a certain value. The case is output as an evaluation for the output value together with the similarity as a similar case.

  In the above configuration, the case where the degree of similarity is higher than a certain value is simply extracted, but in this method, when there are a plurality of similar cases with the same or almost the same degree, a plurality of cases are output. The operator does not know which case output value should be believed. In the case of a configuration in which only one case is extracted as a similar case, the output value of the learning case may not match the actual output value of the neural network.

  As a configuration for coping with this, the evaluation unit 14 may extract a plurality of similar cases having a high degree of similarity, average them, and output this as a value as an explanation result. As a method of this extraction, there can be considered a method in which the degree of similarity is equal to or higher than a certain value, a ratio of the degree of similarity to be the maximum similarity is higher than a specific value, and a specific number from the top in descending order of similarity.

  In the case of this configuration, since an average value of a plurality of similar cases having a high degree of similarity is taken, there is a feature that the value as an explanation result is one. This also increases the degree of coincidence between the output of the neural network and the explanation result.

  FIG. 3 is a flowchart showing an input case evaluation process performed by the evaluation apparatus 10.

In the evaluation apparatus 10, first, when input data x (x _{1 to} x _i ) is input to the neural network 20 to be explained, the internal connection state is read from the network 20 as step S1. Then, the importance degree for each input factor of the neural network 20 is calculated from the coupled state by the equation (1) and the like (step S2).

  Next, in step S3, the cases used for learning of the neural network 20 are sequentially read out from the learning case storage unit 13, and using the importance for each input factor obtained in step S2, equations (3) to (7), etc. The similarity with the input value x of the neural network 20 is calculated by Then, in step S5, the one with the highest similarity or one with a similarity greater than a certain value is output as a similar case.

  FIG. 4 is a flowchart showing the processing of the evaluation apparatus in the case of a configuration in which an average of a plurality of similar cases is taken and one value as an explanation result is output.

  In the processing of FIG. 4, after the input data x is input to the neural network 20, the processing operations from step S11 to step S14 for obtaining similar cases between this input case and each case in the learning case storage unit 13 are as shown in FIG. Since this is the same as the processing from step S1 to step S4 in the case of 3, description is omitted.

  After the degree of similarity between the input case and each learning case is obtained in the process up to step S14, next, a range of the degree of similarity for extracting the similar case is set as step S15. This similarity degree is determined to be, for example, 90% or more by the operator's input, and a learning example having a similarity degree within this range is searched and extracted as step S16. Then, in step S17, the load average is performed on the plurality of similar cases according to the degree of similarity, and the result is output.

  Next, more specific system configuration examples and examples using the evaluation apparatus 10 according to the present embodiment described so far will be described.

  In the following description, the case where the evaluation apparatus of FIG. 1 is used in a system that performs prediction of water inflow into a dam using a neural network is taken as an example.

  FIG. 5 is a diagram illustrating a configuration example of the prediction system.

  The prediction system 100 is a system that predicts the inflow amount to the dam from the measurement data of the flow rate and the rainfall collected from each measurement site via the communication network 200 by the neural network 20. The evaluation result by 10 is displayed to the operator.

  The prediction system 100 includes an evaluation device 110, a neural network 120, a data accumulation unit 130, a measurement data storage unit 140, an output processing unit 150, a display unit 160, and a prediction result storage unit 170. Among them, the evaluation unit 110 and the neural network 120 are the same as the evaluation device 10 and the neural network 20 of FIG. In addition, the neural network 120 is learned in advance so as to output a difference in the inflow amount of the dam from each measurement data as a prediction result according to past cases.

  The data accumulation unit 130 manages measurement data collected from each measurement location via the communication network 200 and stores the measurement data in the measurement data storage unit 140. Further, the data accumulation unit 130 outputs measurement data of the current value from the communication network 200 and past measurement data in the data accumulation unit 130 as input data to the neural network 120. The output processing unit 150 stores the output data from the neural network 120 and the evaluation device 110 in the prediction result storage unit 170, processes the data into image data such as a graph format, and outputs and displays it on the display unit 160. The prediction result storage unit 170 stores the prediction result by the neural network 120 in association with the evaluation by the evaluation device 110, and the data stored in the prediction result storage unit 170 is used for re-education of the neural network 120 or the like. It is done.

  FIG. 6 shows input factors and output factors of the neural network 120 in the prediction system 100. In the figure, (0) is the measured value of the current time, (-1), (-2),. . Is 1 hour ago, 2 hours ago,. . The measured value is shown.

  In the prediction system 100 of this embodiment, the neural network 120 calculates the rainfall from the current time to 20 hours ago, the inflow to the dam up to 3 hours ago, and the flow from the river 1, river 2 and river 3 upstream of the dam. These data are input from the data accumulation unit 130 to the neural network 120. Then, the difference in the amount of inflow into the dam is output as an output factor of the neural network 120, and a change in the amount of inflow of the dam one hour ahead is predicted. Along with the predicted value, similar cases are extracted and shown from past cases used for learning of the neural network 120 to explain the reliability and the basis of the predicted result.

FIG. 7 is a diagram showing the importance of each of the input factors x _k (k = 1 to 35) of the neural network 120 shown in FIG. Based on this importance value, the prediction system 100 obtains the similarity between the input case to be explained and each learning case. The degree of importance in the figure is obtained using equation (1).

  The similarity is obtained from the following equation using the importance shown in FIG.

  Further, the following expression is obtained by expressing the degree of similarity as a percentage of the expression (8).

  However, in Expression (9), the input value x is normalized to a value of 0 to 1.

  FIG. 8 and FIG. 9 show display examples of prediction results displayed on the display unit 160 and similar cases that are the basis thereof. FIG. 8 is a display example of the prediction result performed at 21:00. The graph shows the predicted value of the flow rate to the dam after 1 hour together with data indicating the state of the basin for the past 20 hours used as input data for prediction. It is displayed. In the figure, the bar graph shows the rainfall at each hour, and the line graph shows the flow of the dam and the flow of each upstream river. The x part in the figure indicates the current time point (21:00 time point), and the measurement data for the past 20 hours (1 to 21:00) and the change in the inflow amount 1 hour ahead (22:00 time point) from here. The predicted value for minutes is shown.

  FIG. 9 shows a similar case that is the basis of the prediction result of FIG. 8, and a past case having the highest degree of similarity is extracted from those used as learning cases, and is displayed in the same format as the prediction result of FIG. It is displayed.

If the extracted similar case is similar to the input case of the explanation target, the output value of the neural network indicates that it is close to the similar case, and if the extracted similar case is not similar to the input case of the explanation target Shows a tendency that the output value of the neural network is different from the past cases. In the case of FIG. 8 and FIG. 9, the similarity obtained from the equation (8) is 96.71%, and the similarity between the input data used for the prediction and the extracted case is high. . Also, the amount of change in the amount of inflow into the dam after 1 hour (22:00) is both over 100 [m ³ / s], which are very similar. From this result, the operator can determine that the prediction result of FIG. 8 is highly reliable. In the above example, only one similar case is displayed and output with respect to the predicted value in FIG. 8, but a plurality of similar cases having a degree of similarity greater than or equal to a specific value may be displayed and output.

As described above, in the prediction system of the present embodiment, the output (predicted value) from the neural network 120 can be explained by the learning case (past case) and the reliability can be evaluated. For example, with respect to the prediction result of FIG. 8, assuming that FIG. 9 is a similar case, “the change amount of the inflow amount of the similar case in the learning case is more than 100 [m ³ / s]. It can be explained that the value is also about 100 [m ³ / s]. The degree of similarity with similar cases extracted from learning cases can also be expressed as a percentage, so this value can be used as an index of reliability, and the operator can easily grasp the reliability of the predicted value. I can do it.

  Next, a case will be described in which, instead of simply outputting an example having the highest similarity as described above, a plurality of cases are extracted as similar cases, and an average of these cases is used as an evaluation for a prediction result by a neural network. .

  As a method of extracting this case, after obtaining the similarity between the input case and each learning case, for example, in order from the highest similarity (the top three cases, etc.), the similarity is a specific value or more (90% or more) Etc.), and a method of a certain ratio or more with respect to the maximum similarity of similar examples (for example, 90% when the similarity of a case having a high similarity is 90%, that is, 81% or more) Can be done.

  As a simple example, let us consider the explanation of the output value of a neural network having the addition of two inputs and one output as learning data. However, the similarity judgment formula is

It was. For the sake of simplicity, the importance of input 1 and input 2 is both 1.

  Based on the above assumptions, an input example to be explained is shown in FIG. 10, and a learning example of the neural network is shown in FIG. FIG. 11 shows the similarity between each learning case and the case to be explained in FIG. 10 obtained from the equation (10). In the figure, case 2 and case 3 are cases having the highest similarity with a similarity of −1, and the two learning cases have the same similarity. The output value of case 2 is 2 and the output value of case 3 is 4, which is different from 3.0 which is the output value of the neural network in the explanation object shown in FIG.

For these two cases having the highest degree of similarity, when the weighted average with the degree of similarity as a weight is taken, output value (explanation value from the past case) = [| similarity of case 2 | × output value of case 2+ | Similarity of Case 3 | × Output Value of Case 3] / | Similarity of Case 2 + Similarity of Case 3 | = 3.0
And coincides with the output value of the neural network.

  In this way, by explaining with the weighted average with the similarity degree as a weight, the explanation result by the past case becomes a good value (a value closer to the output value of the neural network).

  FIG. 12 is a diagram illustrating a system environment of an information processing system in which a prediction system using an evaluation apparatus and a neural network according to the present embodiment is constructed.

  As shown in FIG. 12, the information processing system includes a CPU 301, a main storage device 302, a hard disk device 303, an input / output device 304 including a display, a keyboard, a mouse, and the like, a network connection device 305 such as a modem and a LAN adapter, and a CD and DVD. Further, a portable storage medium 307 such as an optical disk or a floppy disk is mounted, and a medium reading device 306 that reads out the stored contents from the portable storage medium 307 is connected to each other via a bus 308.

  In the information processing system of FIG. 12, the program and data stored in the portable storage medium 307 are read out by the medium reading device 306 and loaded or stored in the main storage device 302 or the hard disk 303. Each processing according to the present embodiment can be realized by software by the CPU 301 loading and executing the program and data on the main storage device 302.

  In this information processing system, application software may be exchanged using the portable storage medium 307. Therefore, the present invention is not limited to the evaluation device for the output value of the neural network or the prediction system using the neural network, but a computer for causing the computer to perform the functions of the above-described embodiment of the present invention when used by the computer. Can be configured as a readable storage medium.

  In this case, as the “storage medium”, for example, as shown in FIG. 13, it is detachable from a medium driving device 407 such as a CD-ROM, a floppy disk (or an MO, DVD, removable magnetic disk, etc.). A portable storage medium 406, a storage device (database or the like) 402 in an information provision processing apparatus (server or the like) held by an external information provider transmitted via the network line 403, or a main body 404 of the information processing apparatus 401 Internal memory (RAM or external storage device) 405 and the like. A program stored in the portable storage medium 406 or the storage device (database or the like) 402 is loaded into a memory (RAM or external storage device or the like) 405 in the main body 404 and executed, and each of the above-described embodiments. Realize the function.

It is a figure which shows the structure of an evaluation apparatus. It is a figure which shows the structure of the neural network used as a premise. It is a flowchart which shows the evaluation process of the input example by an evaluation apparatus. It is a flowchart which shows the process of the evaluation apparatus in the case of the structure which averages several similar cases and outputs one value which is an explanatory result. It is a figure which shows the structural example of the prediction system which estimates the inflow amount of a dam. It is a figure which shows the input factor and output factor of the neural network in a prediction system. It is a figure which shows the importance with respect to each input factor. It is a figure which shows the example of the prediction result displayed on a display part. It is a figure which shows the similar example used as the basis of the prediction result displayed on a display part. It is a figure which shows the input example used as description object. It is a figure which shows the learning example of a neural network. 1 is an environment diagram of an information processing system in which the present system is used. It is a figure which shows the example of a storage medium.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Evaluation apparatus 11 Importance determination part 12 Similarity calculation part 13 Learning example memory | storage part 14 Evaluation part 20, 120 Neural network 100 Prediction system 110 Evaluation part 120 Neural network 130 Data accumulation part 140 Measurement data storage part 150 Output processing part 160 Display Unit 170 prediction result storage unit 200 communication network

Claims (2)

An evaluation method using a computer to evaluate an output value output from a learned neural network with a learning case as an input as a similarity between the evaluation case and the learning case,
The computer is
A plurality of learning examples used for learning of the neural network are stored in a memory, and the importance of each input factor constituting the input of the neural network is determined from the connection state of the neural network,
A method for evaluating an output value of a neural network, wherein a similarity between the evaluation target case and each learning case stored in the memory is obtained using the importance. When used by computer
Extracting the connected state of learned neural networks using multiple cases as learning cases,
Obtain the importance of each input factor of the neural network from the combined state,
Based on the importance, the similarity between the input case to the neural network to be evaluated and the learning case is obtained,
A computer-readable recording medium storing a program for causing the computer to extract a similar case for an evaluation target case based on the similarity.