JP7493813B2 - Explanation generation device, explanation generation method and program - Google Patents

Description

The present invention relates to an explanation generation device, an explanation generation method, and a program.
This application claims priority based on Japanese Patent Application No. 2019-137811, filed on July 26, 2019, the contents of which are incorporated herein by reference.

One of the machine learning techniques is neural networks such as deep learning (for example, Patent Document 1).

Japanese Patent Publication No. 2019-105883

In general, even if the structure of a neural network is analyzed, it is not possible to explain the processing that the neural network performs in natural language. In particular, even if the structure of a neural network with a complex structure is analyzed, it is not possible to explain the processing that the neural network performs in natural language.
For example, when judging the validity of a process performed by a neural network, it is preferable to be able to explain the process performed by the neural network in natural language.

One example of the objectives of the present invention is to provide an explanation generation device, an explanation generation method, and a program that can obtain a natural language explanation of the processing performed by a neural network.

According to a first aspect of the present invention, an explanation generation device includes a linear sum acquisition unit that performs a structural transformation to simplify the structure of a neural network, and generates a model in the form of a linear sum corresponding to the output nodes of the neural network before the structural transformation based on the neural network after the structural transformation, and an explanation generation unit that generates, based on the model in the form of a linear sum, an explanation in natural language of the processing performed by the neural network before the structural transformation .

The linear sum acquisition unit may generate the model including input variables indicating features in the input data to the neural network, and the explanation generation unit may use a natural language description of the features to generate a natural language explanation of the processing performed by the neural network.

According to a second aspect of the present invention, an explanation generating method includes a computer performing a structural transformation to simplify the structure of a neural network, generating a model in the form of a linear sum corresponding to output nodes of the neural network before the structural transformation based on the neural network after the structural transformation, and generating a natural language explanation of the processing performed by the neural network before the structural transformation based on the model in the form of a linear sum.

According to a third aspect of the present invention, a program is provided for causing a computer to perform a structural transformation to simplify the structure of a neural network, and based on the structurally transformed neural network, generate a model in the form of a linear sum corresponding to the output nodes of the neural network before the structural transformation, and generate, based on the model in the form of a linear sum, a natural language explanation of the processing performed by the neural network before the structural transformation .

According to the present invention, it is possible to obtain a natural language description of the processing performed by a neural network.

1 is a schematic block diagram illustrating an example of a functional configuration of an explanation generating device according to an embodiment. FIG. 13 illustrates an example of weighted sums at output nodes of an illustrative neural network according to an embodiment. FIG. 2 is a diagram illustrating an example of a model acquired by the explanation generation device according to the embodiment. 11 is a diagram illustrating an example of a procedure of a process performed by a linear sum acquisition unit according to the embodiment. FIG. 11 is a diagram illustrating an example of a procedure of a process performed by a linear sum acquisition unit according to the embodiment. FIG.

The following describes embodiments of the present invention, but the following embodiments do not limit the scope of the invention. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.
Fig. 1 is a schematic block diagram showing an example of a functional configuration of an explanation generating device according to an embodiment. In the configuration shown in Fig. 1, the explanation generating device 100 includes a communication unit 110, a display unit 120, an operation input unit 130, a storage unit 180, and a control unit 190. The control unit 190 includes a linear sum acquisition unit 191, an explanation generating unit 194, and a neural network execution unit 195. The linear sum acquisition unit 191 includes a structure conversion unit 192 and a variable processing unit 193.

The explanation generation device 100 generates an explanation in natural language of a neural network (NN) to be explained.
Generally, even if the structure of a neural network is analyzed, the processing performed by the neural network cannot be explained in natural language. In particular, even if the structure of a neural network with a complex structure such as a deep neural network (DNN) is analyzed, the processing performed by the neural network cannot be explained in natural language.
On the other hand, there is a need to make machine learning such as neural networks explainable in natural language. For example, in order to evaluate the validity of a calculation result by a neural network, it is preferable to be able to explain in natural language the process by which the neural network calculates the calculation result.

Therefore, the explanation generation device 100 generates an explanation in natural language of the processing performed by the neural network to be explained.
To generate an explanation, the explanation generating device 100 obtains a model in the form of a linear sum according to the output nodes (Nodes) of the neural network to be explained. The model in the form of a linear sum according to the output nodes of the neural network is a model that calculates a weighted sum that is the same as or an approximation of the calculated value of the weighted sum at the output nodes of the neural network. The model in the form of a linear sum is also simply called a model.

2 shows an example of weighted sums at the output nodes of the neural network under discussion, which shows an output node N21 of the neural network under discussion and M nodes N11 to N1M (M is a positive integer) that output values to the output node N21.
The value of output x _i from node N1i (i is an integer of 1≦i≦M) is multiplied by weight v _i to obtain v _i x _i , which is input to output node N21. Output node N21 calculates the sum s of v _i x _i as shown in formula (1).

The calculation of the sum s shown in equation (1) corresponds to the calculation of the weighted sum at output node N21.
The output node N21 further compares the calculated sum s with a threshold Th and outputs a value according to the comparison result. For example, if s≧Th, the output node N21 may output a value of 1, and if s<Th, the output node N21 may output a value of 0.

3 is a diagram showing an example of a model acquired by the explanation generation device 100. FIG 3 shows an example of a model acquired by the explanation generation device 100 for the output node N21 in FIG.
3, the values of N input variables _F1 , _F2 , ..., _FN (N is a positive integer) are multiplied by weights _w1 , _w2 , ..., _wN , respectively. Then, in this model, the N values _w1F1 , _w2F2 , _... , _{wN FN} obtained by the multiplication are _summed and output.
A function f representing an operation according to the model shown in FIG.

When comparing the weighted sum at the output node exemplified in formula (1) with the weighted sum by the model in the form of a linear sum exemplified in formula (2), the number and values of the inputs may be different. Specifically, the value of M in formula (1) may be different from the value of N in formula (2). Also, the value of x _i in formula (1) may be different from the value of F _i in formula (2). Also, the number and values of the weights may be different. Therefore, the value of v _i in formula (1) may be different from the value of w _i in formula (2).

The values of the model's input variables _F1 , _F2 , ..., _FN are determined according to the values of the outputs _x1 , _x2 , ..., _xM from each node from node N11 to N1M, and in that case, the value of the model's output shown in equation (2) should be the same as or approximately equal to the weighted sum value at the output node shown in equation (1).
For example, if the input to the neural network to be explained is each pixel value in image data, features obtained from the image data may be used as input to the model in the form of a linear sum.

Furthermore, for example, consider a case where the values of each node N11 to N1M in the neural network of Figure 2 are obtained by converting input data using a multi-layer circuit. An example of such a case is when image data is received as input data, and the values of each node N11 to N1M are obtained through conversion using a multi-layer circuit.

In this case, it is not possible to explain what the values of the nodes N11 to N1M represent, and it is conceivable that a useful explanation cannot be given about the processing performed by the neural network. A useful explanation here is, for example, an explanation that can be understood by humans.
On the other hand, if the input to the model in Figure 3 is features such as the intensity of red, explaining the conditions on the features can hopefully provide a useful explanation for the processing performed by the neural network.

Based on the obtained model, the explanation generation device 100 generates an explanation in natural language of the processing performed by the neural network to be explained. When the input variables _F1 , _F2 , ..., _FN of the model are variables indicating features that can be expressed in natural language, the explanation is generated by expressing in natural language a combination of the features indicated by these variables and the magnitude indicated by the weights. Being able to express a feature in natural language means, for example, that the feature can be named.

For example, when the value of the input variable F _i indicates the strength of red of a color component and the value of the weight w _i is a positive value, the explanation generation device 100 may express the condition indicated by w _i F _i as "strong red". The explanation generation device 100 may combine such a condition with a target for judgment based on the output value of the neural network, and output an explanation in the form of "If red is strong, XX is large, and..., it is judged to be △△".
The explanation generation device 100 may be configured using a computer such as a personal computer (PC) or a workstation (WS).

The communication unit 110 communicates with other devices. For example, the communication unit 110 receives the neural network to be explained from the other device. For further example, the communication unit 110 receives the neural network to be explained in a data format that can be executed by the neural network execution unit 195.

The display unit 120 has a display screen, such as a liquid crystal panel or an LED (Light Emitting Diode) panel, and displays various images. For example, the display unit 120 displays a natural language explanation of the processing performed by the neural network of the explanation target generated by the explanation generation device 100. However, the method by which the explanation generation device 100 outputs the explanation is not limited to a specific method. For example, the communication unit 110 may transmit the explanation to another device.

The operation input unit 130 includes input devices such as a keyboard and a mouse, and accepts user operations. For example, the operation input unit 130 accepts user operations that instruct the explanation generation device 100 to perform processing, such as a user operation that instructs the explanation generation device 100 to generate an explanation.

The storage unit 180 stores various data. For example, the storage unit 180 stores a neural network of an explanation target received by the communication unit 110. The function of the storage unit 180 is executed using a storage device included in the explanation generation device 100.
The control unit 190 executes various processes by controlling each unit of the explanation generation device 100. The functions of the control unit 190 are executed by the CPU included in the explanation generation device 100 reading out a program from the storage unit 180 and executing it.

The linear sum acquisition unit 191 acquires a model in the form of a linear sum that generates an output according to the output of the neural network to be explained, as described above for the explanation generation device 100.
For example, the linear sum acquisition unit 191 may approximate the neural network to be explained with a neural network having a simpler structure, and extract the weighted sum calculation by the output nodes of the obtained neural network with the simpler structure as a model in the form of a linear sum.

Alternatively, the linear sum acquisition unit 191 may convert the neural network to be explained into an equivalent neural network with a simpler structure. The linear sum acquisition unit 191 may then extract the weighted sum calculation by the output nodes of the resulting neural network with a simpler structure as a model in the form of a linear sum. Equivalence here means outputting the same value for the same input value. For example, if the same value is input to two equivalent neural networks, the two equivalent neural networks will output the same value.

Alternatively, the linear sum acquisition unit 191 may extract the output nodes of the neural network to be explained as they are as a model in the form of a linear sum. For example, if the structure of the neural network to be explained is originally a simple structure, the linear sum acquisition unit 191 may extract the calculation of the weighted sum by the output nodes of the neural network to be explained as they are as a model in the form of a linear sum.

When the neural network to be described has multiple output nodes, the linear sum acquisition unit 191 generates a model in the form of a linear sum for each output node. The linear sum acquisition unit 191 may generate a model in the form of a linear sum for all output nodes, or may generate a model in the form of a linear sum for only some of the output nodes.

The structure conversion unit 192 converts the structure of the neural network to be explained. In particular, the structure conversion unit 192 simplifies the structure of the neural network to be explained. Specifically, the structure conversion unit 192 reduces at least one of the number of nodes and the number of arcs in the neural network to be explained. An arc here is an information transmission path that connects two nodes.

As described above with respect to the linear sum acquisition unit 191, the structure conversion unit 192 may convert the neural network to be explained into a neural network with a simpler structure that approximates the neural network, or may convert the neural network to be explained into an equivalent neural network with a simpler structure.
Alternatively, when the linear sum acquisition unit 191 extracts the output node of the neural network to be explained as a model in the form of a linear sum as is, the structure conversion unit 192 does not convert the neural network to be explained, but leaves the structure as it is.

When the neural network to be described has multiple output nodes, the linear sum acquisition unit 191 generates a model in the form of a linear sum for each output node, and the structural transformation unit 192 generates a neural network after structural transformation for each output node. When the linear sum acquisition unit 191 generates a model in the form of a linear sum for only some of the multiple output nodes, the structural transformation unit 192 may generate a neural network after structural transformation for only those output nodes.

The variable processing unit 193 performs processing so that the input variables of the model in the form of a linear sum acquired by the linear sum acquisition unit 191 indicate features that can be expressed in language. For example, the variable processing unit 193 may configure a model in the form of a linear sum that approximates the neural network acquired by the processing of the structure conversion unit 192 using input variables that indicate features that can be explained in natural language, such as "intensity of red" or "size of XX."
When the linear sum acquisition unit 191 generates a plurality of models in the form of a linear sum, the variable processing unit 193 performs processing for each model so that the input variables of the model indicate features that can be expressed in language.

The explanation generation unit 194 generates an explanation in natural language of the processing performed by the neural network to be explained, based on the model in the form of a linear sum acquired by the linear sum acquisition unit 191. As described above for the explanation generation device 100, when the input variables _F1 , _F2 , ..., _FN of the model are variables indicating features that can be explained in natural language, the explanation generation unit 194 generates an explanation by expressing in natural language a combination of the features indicated by these variables and the magnitude indicated by the weights.

The neural network execution unit 195 executes a neural network. For example, when input data for the neural network to be explained is given, the neural network execution unit 195 executes the neural network to be explained and calculates an output for the given input data.

Fig. 4 is a diagram showing an example of the procedure of the process performed by the linear sum acquisition unit 191. Fig. 4 shows an example of classifying neural networks based on the simplicity of their structure and the ease of explaining the input variables of the model in the form of the corresponding linear sum.
The linear sum acquisition unit 191 performs processing to make the processing performed by the neural network to be explained easier to explain. The linear sum acquisition unit 191 performs structural transformation and processing to make the input variables of the model relatively easy to explain on the neural network to be explained, so as to obtain a neural network with a relatively simple structure and with input variables of the model in the form of a corresponding linear sum that are relatively easy to explain. The structural transformation here refers to processing to transform the structure of the neural network. The processing to make the input variables of the model relatively easy to explain is referred to as input variable processing.

The linear sum acquisition unit 191 may perform input variable processing after performing structural transformation on the neural network to be explained. Alternatively, the linear sum acquisition unit 191 may perform input variable processing on the neural network to be explained and then perform structural transformation. Alternatively, the linear sum acquisition unit 191 may perform structural transformation and input variable processing simultaneously on the neural network to be explained.

There may be cases where the process performed by the linear sum acquisition unit 191 on the neural network to be explained corresponds to both the structural transformation and the input variable process. For example, when the linear sum acquisition unit 191 transforms the structure of the neural network to be explained, so that the input to the output node becomes easier to explain, the structural transformation in this case can be said to correspond to the input variable process.
In the configuration example of FIG. 1, a portion of the linear sum acquisition unit 191 that performs structural transformation is shown as a structural transformation unit 192 , and a portion of the linear sum acquisition unit 191 that processes input variables is shown as a variable processing unit 193 .

Fig. 5 is a diagram showing an example of the procedure of the process performed by the linear sum acquisition unit 191. Fig. 5 shows an example of the case where the linear sum acquisition unit 191 performs input variable processing after performing a structural transformation on the neural network to be described.
5, the structure conversion unit 192 converts the structure of the neural network to be explained (step S11). Through this process, the structure conversion unit 192 converts the neural network to be explained into a neural network with a simpler structure.

Next, the linear sum acquisition unit 191 generates a model in the form of a linear sum based on the neural network obtained by the structural transformation performed by the structural transformation unit 192 (step S12).
For example, the linear sum acquisition unit 191 may model the calculation of the weighted sum of the output nodes of the neural network. In this case, the model generated by the linear sum acquisition unit 191 represents the output from the node preceding the output node as the input variables of the model. Examples of the output from the node preceding the output node include _x1 , _x2 , ..., _xM in Fig. 2. Examples of the input variables of the model include _F1 , _F2 , ..., _FN in Fig. 3.

In addition, the model generated by the linear sum acquisition unit 191 expresses the weights of the arcs between the output node and the previous node as weights in the linear sum. Examples of the weights of the arcs between the output node and the previous node include _v1 , _v2 , ..., _vM in Fig. 2. Examples of the weights in the linear sum include _w1 , _w2 , ..., _wN in Fig. 3.

Next, the variable processing unit 193 performs input variable processing so that the input variables of the model generated by the linear sum acquisition unit 191 are easier to explain (step S13). For example, the variable processing unit 193 replaces a combination w _i F _i of the input variables of the model and the weights applied to the input variables with a feature obtained from an input to the neural network to be explained and a variable applied to the feature. The model after the replacement may be a model equivalent to the model before the replacement. Alternatively, the model after the replacement may be a model that approximates the model before the replacement.

In the process of step S13, one or both of the number of combinations w ₁ F ₁ , w ₂ F ₂ , . . . , w _{NF N} of model input variables and weights and the values of the weights w ₁ , w ₂ , . . . , w _N are changed.
The linear sum acquisition unit 191 may perform the process of step S12 and the process of step S13 simultaneously. In addition, if the input variables of the model generated by the linear sum acquisition unit 191 in step S12 are relatively easy to explain, the process of step S13 may be suppressed. In this case, suppressing the process means not performing the process.

An example of the process performed by the linear sum acquisition unit 191 will be further described.
<Structural transformation using genetic programming>
The structure conversion unit 192 may convert the structure of the neural network to be explained using a genetic programming (GP) technique, which is a type of evolutionary computation method of stochastic optimization. By this structure conversion, the structure conversion unit 192 may convert the neural network to be explained into a network-type function that approximates the neural network. The genetic programming technique can be used as a technique for approximating a function by combining modules (components) that are easy to explain and that are prepared in advance. In this case, it is expected that an explanation can be given to a network-type function that is configured by a combination of explainable modules.

For example, when the input is an image, the user may prepare several explainable components for constructing a function after structural transformation, such as several types of image processing filters, and provide these to the structural transformation unit 192. The structural transformation unit 192 expresses the structure of the combination of these components as a network structure or a tree structure, and generates a new structure using a genetic programming technique. The structural transformation unit 192 may use all of the provided components, or may allow some components not to be used. Furthermore, the structural transformation unit 192 may use one component only once in one tree structure, or may allow one component to be used multiple times.

The structural transformation unit 192 repeatedly generates a tree structure using a genetic programming technique until a neural network that approximates the neural network to be explained with greater accuracy than specified conditions is obtained as a structurally transformed neural network represented as a network structure or tree structure.

It is expected that the structure of the network will be simplified by the structure conversion unit 192 generating a network type function that uses components to approximate the neural network to be explained.
In addition, by simplifying the structure of the neural network, the processing from the input node to the node preceding the output node is also relatively simple, and it is expected that the processing will be relatively easy to explain in natural language. Furthermore, by executing the processing from the input node to the node preceding the output node by combining components, it is expected that the processing will be relatively easy to explain in natural language. In this way, the above structural transformation is also expected to have an effect on input variable processing.

<Structural transformation by pruning neural networks>
The structure conversion unit 192 may convert the structure of the neural network by deleting nodes and arcs that do not affect the output of the neural network from the neural network to be explained. Alternatively, the structure conversion unit 192 may delete nodes and arcs that have a small effect on the output of the neural network in addition to or instead of the nodes and arcs that do not affect the output of the neural network.

For example, a node that does not have a path to an output node is a node that does not affect the output node. Also, when the weight value of an arc is 0, it can be considered as the absence of the arc. The structure conversion unit 192 may delete nodes and arcs that do not affect the output node.
In addition, when the magnitude (absolute value) of the weight of an arc is small, the information transmitted between nodes on that arc is considered to have little effect on subsequent nodes, which are nodes that receive information.
Therefore, the structure conversion unit 192 may delete arcs whose weights are equal to or smaller than a predetermined threshold value.

The structure conversion unit 192 deletes either or both of the nodes and arcs from the neural network, thereby making the structure of the neural network relatively simple.
In addition, by simplifying the structure of the neural network, the processing from the input node to the node preceding the output node is also relatively simple, and it is expected that the processing will be relatively easy to explain in natural language. In this way, the above structural transformation is also expected to have an effect on input variable processing.

<Input variable processing by feature quantification of input variables>
The variable processing unit 193 may use variables indicating feature quantities as input variables of a model in the form of a linear sum.
For example, a user provides a means for calculating a certain feature amount from an input to the neural network to be explained to the variable processing unit 193. The means here may be provided as a calculation algorithm.

The variable processing unit 193 temporarily replaces the input variables of the model in the form of a linear sum generated by the linear sum acquisition unit 191 with variables indicating a given feature amount. The variable processing unit 193 judges whether the model after replacement approximates the model before replacement or whether the replacement is an equivalent conversion. The variable processing unit 193 judges various replacement patterns, detects a replacement in which the model after replacement approximates the model before replacement or the replacement is an equivalent replacement, and adopts that replacement.
Alternatively, as described above, when the linear sum acquisition unit 191 generates a model, the variables indicating the feature amounts may be used as input variables of the model.

Since the input variables of the model in the form of a linear sum are variables that indicate features, it is expected that a natural language explanation of the features can be used to explain the processing performed by the neural network being explained.
Regarding the provision of a natural language explanation to the explanation generation device 100, when a user provides a means for calculating a certain feature to the variable processing unit 193, the means may be linked to the explanation and a natural language explanation of the feature may be provided.

Alternatively, the user may prepare filter components for extracting features, and the variable processing unit 193 may combine the components using genetic programming techniques to generate a filter that outputs features, as in the case of the structural transformation using genetic programming described above.
In this case, the user may determine whether the obtained filter output represents a feature, and if so, what kind of feature it represents. For example, the variable processing unit 193 may input some sample data to the generated filter, and present the sample data and the filter output for the sample data to the user. The user may grasp a feature common to samples with large output values of the filter as a feature extracted by the filter, and provide the explanation generating device 100 with a natural language explanation of the feature, such as "intensity of red."

<Input variable processing by identifying characteristics exhibited by the input variables>
The variable processing unit 193 may identify the characteristics indicated by the input variables without replacing the input variables of the model.
For example, the variable processing unit 193 may extract samples in which the magnitude of a value of an input variable is greater than or equal to a predetermined threshold from among multiple samples of input data to the neural network to be explained, and present the samples to the user. Then, the user may provide the explanation generation device 100 with a natural language explanation of a feature common to the presented samples. The explanation generation device 100 uses the provided explanation as an explanation of the feature indicated by the input variable.
As a sample of input data to the neural network to be explained, input data in the training data applied to the neural network to be explained can be used.

Alternatively, the variable processing unit 193 may present to the user a set of samples in which only one of the model's input variables differs in value and the other input variables have the same values. The user may then provide to the explanation generation device 100 a natural language explanation of the different features of the two presented samples. The explanation generation device 100 uses the provided explanation as an explanation of the feature indicated by that input variable (the input variable whose value differs in the two samples).

Here, the variable processing unit 193 may generate a set of samples in which only one input variable among the input variables of the model is different in value and the other input variables have the same values.
When the value of the input variable of the model indicates the output value of a node preceding the output node in the neural network, the variable processing unit 193 may generate input data to the neural network such that the value of the input variable of the model becomes the desired value, using a method similar to backpropagation.

In this case, the variable processing unit 193 calculates the difference between the values of the input variables of the model when the sample data is applied to the neural network and the desired values. Then, instead of adjusting the weight values based on the difference using the backpropagation method, the variable processing unit 193 adjusts the output value of the previous node based on the difference. The variable processing unit 193 repeats this process up to the input data, and by repeating it for multiple sample data, calculates input data to the neural network such that the values of the input variables of the model become the desired values.

An example of the process in which the explanation generating unit 194 generates an explanation in natural language will be further described.
<Conversion of linear sums into natural language explanations>
The explanation generator 194 treats an input variable with a positive (+) weight value as an input variable indicating a positive condition. Positive conditions are used, for example, in the form of "XX is strong,""XX is large,""XX is abundant," and "XX is very much."
Furthermore, the explanation generator 194 treats an input variable with a negative (-) weight value to be multiplied as an input variable indicating a negative condition. Negative conditions are used, for example, in the form of "weak XX", "small XX", "little XX", "not much XX", etc.
The "〇〇" is replaced with a natural language that indicates the input variable (or its characteristics), such as the intensity of red.

In addition, the explanation generation unit 194 may use all of the input variables for the explanation, or may select some input variables based on the weights and the values of the input variables and use the selected input variables for the explanation.
Here, it is considered that the larger the weight, the greater the influence that the value of the corresponding input variable has on the model output. Therefore, the explanation generating unit 194 may exclude variables whose multiplied weight is equal to or smaller than a predetermined threshold from the scope of explanation.

Therefore, the explanation generator 194 may compare the value of the multiplied weight w with a threshold a (a>0) and process the explanations for the model's input variables as follows:
(1) If w>a: Treat the input variable as an input variable indicating a positive condition.
(2) If −a≦w≦a: The input variable is excluded from the explanation.
(3) If w<-a: The input variable is treated as an input variable indicating a negative condition.

Here, the user may be able to change the value of threshold a. If the user desires a concise explanation, the user sets the value of threshold a to a positive value with a relatively large magnitude (absolute value). An example of a relatively large value here is 0.9. On the other hand, if the user desires a more detailed explanation, the user sets the value of threshold a to a positive value with a relatively small magnitude. An example of a relatively small value here is 0.5.

For example, if the characteristic indicated by the input variable is the intensity of the red color component, and the value of the weight to be multiplied is positive and has a large absolute value, the explanation generator 194 generates a positive condition such as "very red."
In addition, when the feature indicated by the input variable is the complexity of the texture, and the value of the weight to be multiplied is negative and its absolute value is large, the explanation generating unit 194 generates a negative condition such as "the texture is not very complex."

The explanation generation unit 194 combines the features indicated using the input variables adopted as the subject of the explanation, for example, with an and condition, to generate an explanation of the neural network of the subject of the explanation, such as, for example, "If the color is strong red, the texture is not very complex, ..., and XX is large, then it is determined to be △△."

It should be noted that a program for implementing all or part of the functions of the calculations and controls performed by the explanation generation device 100 may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed to process each part. Note that the term "computer system" here includes hardware such as the OS and peripheral devices.
Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs (Read Only Memory), and CD-ROMs (Compact Disc Read Only Memory), as well as storage devices such as hard disks built into computer systems. The above-mentioned program may be for realizing part of the above-mentioned functions, or may be capable of realizing the above-mentioned functions in combination with a program already recorded in the computer system.

Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment and also includes design modifications within the scope that does not deviate from the gist of the present invention.

Embodiments of the present invention may be applied to an explanation generation device, an explanation generation method and a program.

REFERENCE SIGNS LIST 100 Explanation generation device 110 Communication unit 120 Display unit 130 Operation input unit 180 Storage unit 190 Control unit 191 Linear sum acquisition unit 192 Structure conversion unit 193 Variable processing unit 194 Explanation generation unit 195 Neural network execution unit

Claims (4)

a linear sum acquisition unit that performs structural transformation to simplify the structure of the neural network, and generates a model in the form of a linear sum according to the output nodes of the neural network before the structural transformation based on the structurally transformed neural network;
an explanation generating unit that generates an explanation in natural language of the processing performed by the neural network before structural transformation based on the model in the form of a linear sum;
An explanation generation device comprising: The linear sum acquisition unit generates the model including input variables indicating features in input data to the neural network;
the explanation generation unit uses the natural language explanation of the feature quantity to generate a natural language explanation of the process performed by the neural network.
The explanation generation apparatus according to claim 1 . The computer
performing structural transformation to simplify the structure of the neural network, and generating a model in the form of a linear sum according to the output nodes of the neural network before the structural transformation based on the neural network after the structural transformation;
generating a natural language description of the processing performed by the neural network before structural transformation based on the linear sum model;
A method for generating explanations comprising: On the computer ,
performing structural transformation to simplify the structure of the neural network, and generating a model in the form of a linear sum according to the output nodes of the neural network before the structural transformation based on the neural network after the structural transformation;
generating a natural language explanation of the processing performed by the neural network before structural transformation based on the linear sum model;
A program for executing.

Images