JP7732299B2 - Learning device, learning method, and program - Google Patents

Description

The present invention relates to a learning device, a learning method, and a program.

Neural networks (hereafter referred to as "NNs") have high performance in areas such as image recognition. However, NNs are generally composed of a huge number of parameters and complex models, making it difficult to interpret the relationship between the NN parameters and the output results from the NN. To solve this problem, several methods have been proposed for obtaining NNs with high interpretability. Note that "high interpretability" can also be said to mean "high consistency with human senses."

For example, one known technique involves manually labeling a heat map that indicates the areas that a neural network model should focus on for making decisions, and then training the model to match that heat map, thereby obtaining a model that is easy for humans to interpret (see, for example, Non-Patent Document 1). Furthermore, if the heat map obtained from a model has low interpretability, a model with higher interpretability can be obtained by retraining the model so that it does not match the heat map.

There is also a known method for improving the accuracy of a neural network by introducing into the network a mechanism for extracting regions of interest from input data that the neural network will use to make decisions (see, for example, Non-Patent Document 2). The regions of interest obtained using this method can be manually corrected, and the neural network can be retrained so that the corrected regions of interest match the neural network's regions of interest, thereby improving the interpretability and accuracy of the neural network.

Andrew Ross and two others, "Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations," [online], [Retrieved September 8, 2021], Internet <https://arxiv.org/abs/1703.03717> Masahiro Mitsuhara and 6 others, "Embedding Human Knowledge into Deep Neural Network via Attention Map," [online], [Retrieved September 8, 2021], Internet <https://arxiv.org/abs/1905.03540> "Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization", [online], [Retrieved September 8, 2021], Internet <https://arxiv.org/abs/1610.02391v3>

However, the methods described in Non-Patent Document 1 and Non-Patent Document 2, in which heat map labels are prepared manually, require a high human resource cost for labeling.

On the other hand, a method that does not require labeling is the method described in Non-Patent Document 1, in which a model is retrained so that it does not match the heat map of a trained model. However, this method has the problem that retraining is likely to result in a decrease in accuracy. Furthermore, this method has the problem that the degree of match of the heat map decreases uniformly for all data, which may actually reduce the interpretability of the heat map for individual data.

Therefore, it is desirable to provide technology that can obtain highly interpretable and accurate models while reducing human costs.

In order to solve the above problem, according to one aspect of the present invention, there are provided an input unit that acquires first input data and a correct value of the first input data; an inference unit that outputs a first inference value corresponding to each of a plurality of inference models based on the first input data and a plurality of inference models; an explanation unit that outputs first explanation information corresponding to each of the plurality of inference models that indicates the magnitude of contribution of the first input data to the first inference value; an inference evaluation unit that obtains an inference evaluation result based on the correct value and the first inference value; an explanation evaluation unit that obtains an explanation evaluation result based on the degree of agreement between the first explanation information corresponding to each of the plurality of inference models; and a weighting unit that adjusts first weighting parameters of the plurality of inference models based on the inference evaluation result and the explanation evaluation result so that the degree of agreement between the first explanation information becomes smaller. and an update unit that updates the first weight parameter, wherein the input unit acquires second input data, the inference unit outputs a second inference value corresponding to each of the plurality of trained models based on the second input data and a plurality of trained models that are a plurality of estimation models after the first weight parameter has been updated, and the explanation unit outputs second explanation information corresponding to each of the plurality of trained models that indicates the magnitude of contribution of the second input data to the second inference value, and the learning device comprises: a presentation control unit that controls presentation of the second inference value and the second explanation information to a user, and an operation unit that receives selected model information from the user that indicates one or more trained models selected by the user from the plurality of trained models .

The learning device may also include a recording control unit that controls the recording of information indicating one or more trained models selected by the user from the plurality of trained models.

The explanation evaluation result may take a smaller value the greater the degree of match between the first explanation information corresponding to each of the multiple inference models.

The explanation evaluation unit may obtain the explanation evaluation result based on the inner product of vectors normalized from the first explanation information corresponding to each of the plurality of inference models.

The explanation evaluation unit may binarize the first explanation information to generate a mask for each of the multiple inference models, calculate the product of the mask generated from the first explanation information corresponding to an inference model other than itself and the first explanation information corresponding to its own inference model, and obtain the explanation evaluation result based on the sum of the products for each of the multiple inference models.

The explanation section may include a function capable of backpropagation.

The explanation unit may have a second weight parameter, and the update unit may update the second weight parameter using a backpropagation algorithm.

At least one of the plurality of inference models may include a neural network. Note that a neural network is merely one example of a machine learning algorithm. Therefore, other machine learning algorithms may be used instead of a neural network.

The update unit may update the first weight parameter based on the sum of the inference evaluation result and the explanation evaluation result.

The first explanatory information may be a heat map showing the magnitude of the contribution of the first input data to the first inferred value.

According to another aspect of the present invention, a method for generating a first inference value corresponding to each of a plurality of inference models based on the first input data and the plurality of inference models is provided. The method further comprises: obtaining first input data and a correct answer value for the first input data; outputting first explanation information corresponding to each of the plurality of inference models, the first explanation information indicating the magnitude of contribution of the first input data to the first inference value; obtaining an inference evaluation result based on the correct answer value and the first inference value; obtaining an explanation evaluation result based on the degree of agreement between the first explanation information corresponding to each of the plurality of inference models; and adjusting the first explanation information of the plurality of inference models so that the degree of agreement between the first explanation information is reduced based on the inference evaluation result and the explanation evaluation result. a learning method including: updating a first weight parameter; acquiring second input data; outputting a second inference value corresponding to each of a plurality of trained models based on the second input data and a plurality of trained models that are a plurality of estimation models after the first weight parameter is updated; outputting second explanatory information corresponding to each of the plurality of trained models indicating a magnitude of contribution of the second input data to the second inference value; controlling presentation of the second inference value and the second explanatory information to a user; and receiving selected model information from the user indicating one or more trained models selected by the user from the plurality of trained models .

According to another aspect of the present invention, a computer includes an input unit that acquires first input data and a correct value of the first input data, an inference unit that outputs a first inference value corresponding to each of a plurality of inference models based on the first input data and a plurality of inference models, an explanation unit that outputs first explanation information corresponding to each of the plurality of inference models that indicates the magnitude of contribution of the first input data to the first inference value, an inference evaluation unit that obtains an inference evaluation result based on the correct value and the first inference value, an explanation evaluation unit that obtains an explanation evaluation result based on the degree of agreement between the first explanation information corresponding to each of the plurality of inference models, and an update unit that updates first weight parameters of the plurality of inference models based on the inference evaluation result and the explanation evaluation result so as to reduce the degree of agreement between the first explanation information. a presentation control unit that controls presentation of the second inference value and the second explanation information to a user; and an operation unit that receives, from the user, selected model information indicating one or more trained models selected by the user from the plurality of trained models.

As described above, the present invention provides technology that can obtain highly interpretable and accurate models while reducing human costs.

FIG. 2 is a diagram illustrating an example of a functional configuration of a learning device according to an embodiment of the present invention. 10A and 10B are diagrams for explaining a method for obtaining explanation evaluation results by multiplying a mask obtained by binarizing a heat map with another heat map. 10 is a flowchart showing an example of operation in a learning stage of the learning device according to the embodiment. 10 is a flowchart illustrating an example of operation of the learning device according to the embodiment in a test stage. FIG. 1 is a diagram illustrating a hardware configuration of an information processing device as an example of a learning device.

Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Note that in this specification and drawings, components that have substantially the same functional configuration will be assigned the same reference numerals, and redundant explanations will be omitted.

In addition, in this specification and drawings, multiple components having substantially the same functional configuration may be distinguished by adding different numbers after the same reference symbol. However, if there is no need to particularly distinguish between multiple components having substantially the same functional configuration, only the same reference symbol will be used. Furthermore, similar components in different embodiments may be distinguished by adding different letters after the same reference symbol. However, if there is no need to particularly distinguish between similar components in different embodiments, only the same reference symbol will be used.

(0. Overview of the embodiment)
An overview of an embodiment of the present invention will be described. In the embodiment of the present invention, a learning device that performs neural network learning based on a combination of input data (learning data) and a correct answer value will be described. However, a neural network is merely one example of a machine learning algorithm. Therefore, other machine learning algorithms may be used instead of a neural network. For example, an SVM (Support Vector Machine) may be used as another example of a machine learning algorithm.

1. DETAILED DESCRIPTION OF THE EMBODIMENTS
An embodiment of the present invention will now be described in detail.

(1.1. Example of the configuration of the learning device)
1 is a diagram showing an example of the functional configuration of a learning device 10 according to an embodiment of the present invention. As shown in Fig. 1, the learning device 10 according to an embodiment of the present invention includes an input unit 101, an inference unit 102, an explanation unit 103, an inference evaluation unit 104, an explanation evaluation unit 105, an update unit 106, a presentation control unit 107, a recording control unit 108, a display unit 121, and an operation unit 122.

In an embodiment of the present invention, it is primarily assumed that the inference unit 102 includes n inference models (n is an integer greater than 1), i.e., a "first inference model" through an "nth inference model." Furthermore, in an embodiment of the present invention, it is primarily assumed that each of the first through nth inference models includes a neural network. Hereinafter, neural network will also be referred to as "NN."

The NNs included in each of the first to nth inference models use a weight parameter 110 (first weight parameter). In this case, the NNs included in each of the first to nth inference models may have a common structure and use different weight parameters 110 (first weight parameters). Alternatively, the NNs included in each of the first to nth inference models may have different structures.

In addition, at least one of the first to nth inference models may include a neural network. For example, some of the first to nth inference models may include a neural network, and other parts of the first to nth inference models may include other machine learning algorithms instead of a neural network.

Furthermore, in the embodiment of the present invention, it is mainly assumed that the explanation unit 103 is configured to include a neural network. The neural network included in the explanation unit 103 uses a weight parameter (second weight parameter).

The dataset 100, the weight parameters 110 (first weight parameters) for the first to nth inference models, and the weight parameters (second weight parameters) held by the explanation unit 103 are stored in a storage unit (not shown). This storage unit may be configured with memory such as RAM (Random Access Memory), a hard disk drive, or flash memory.

The input unit 101, inference unit 102, explanation unit 103, inference evaluation unit 104, explanation evaluation unit 105, update unit 106, presentation control unit 107, and recording control unit 108 each include a computing device such as a CPU (Central Processing Unit) or GPU (Graphics Processing Unit), and their functions can be realized when a program stored in ROM (Read Only Memory) is loaded into RAM and executed by the computing device. In this case, a computer-readable recording medium on which the program is recorded can also be provided. Alternatively, these blocks may be composed of dedicated hardware or a combination of multiple hardware components. Data required for calculations by the computing device is stored as appropriate in a storage unit (not shown).

In the initial state, initial values are set for the weight parameters 110 of the first to nth inference models and the weight parameters held by the explanation unit 103. For example, the initial values set for these may be random values, but any values are also acceptable. For example, the initial values set for these may be learned values obtained in advance through learning.

(Dataset 100)
The dataset 100 includes a plurality of pieces of input data (first input data) used in the learning stage and correct values for each of the plurality of pieces of input data. The plurality of pieces of input data used in the learning stage may correspond to training data. Furthermore, the dataset 100 includes a plurality of pieces of input data (second input data) used in the testing stage. The plurality of pieces of input data used in the testing stage may correspond to testing data.

It is generally assumed that test data is prepared as separate data from training data. However, test data may also include part of the training data.

Furthermore, in embodiments of the present invention, it is primarily assumed that the input data is image data (especially still image data). However, the type of input data is not particularly limited, and data other than image data can also be used as input data. For example, the input data may be video data containing multiple frames, or audio data.

(Input unit 101)
In the learning stage, the input unit 101 sequentially acquires combinations of input data and correct values to be used in the learning stage from the dataset 100. The input unit 101 sequentially outputs the combinations of input data and correct values to be used in the learning stage to the inference unit 102. In addition, in the testing stage, the input unit 101 sequentially acquires input data to be used in testing from the dataset 100. The input unit 101 sequentially outputs the input data to be used in the testing stage to the inference unit 102.

For example, when the input unit 101 has acquired and output all combinations of input data and correct answers used in the learning stage from the dataset 100, it may repeat the operation of acquiring and outputting those combinations again from the beginning a predetermined number of times. In such a case, the blocks subsequent to the input unit 101 may also sequentially repeat their respective processes based on the re-input. On the other hand, for example, when the input unit 101 has acquired and output all input data used in the testing stage from the dataset 100, it may end the acquisition of input data.

(Inference unit 102)
In the learning stage, the inference unit 102 obtains inference values (first inference values) corresponding to each of the first to n-th inference models based on the input data input from the input unit 101 and the first to n-th inference models. Similarly, in the testing stage, the inference unit 102 obtains inference values (second inference values) corresponding to each of the first to n-th inference models based on the input data input from the input unit 101 and the first to n-th inference models.

The weight parameters 110 used by the first to nth inference models are stored in a storage unit (not shown). Therefore, the inference unit 102 acquires the weight parameters 110 from the storage unit (not shown), and performs inference using the first to nth inference models based on the acquired weight parameters 110 and the input data input from the input unit 101.

In this specification, obtaining output from a NN based on input to the NN is broadly referred to as "inference."

As an example, if the function representing the i-th inference model is Fi (i is an integer from 1 to n) and the input to the i-th inference model is x, the output from the i-th inference model can be expressed as Fi(x).

As will be explained later, the explanation method (i.e., the method for generating explanation information) used by the explanation unit 103 may include an explanation method that requires information such as feature quantities (intermediate feature quantities) output from each of the first to nth inference models in addition to the inference value. In such cases, the inference unit 102 may output to the explanation unit 103, along with the inference value, the feature quantities output from each of the intermediate layers of the first to nth inference models.

The specific configuration of the first to nth inference models is not particularly limited. However, it is preferable that the format of each output from the first to nth inference models be set to match the format of the correct value corresponding to the input data. For example, if the correct value is the class of a classification problem, it is preferable that each output from the first to nth inference models be a one-hot vector whose length is equal to the number of classes.

In the learning stage, the inference unit 102 outputs inference values corresponding to each of the first through n-th inference models to the explanation unit 103 and the inference evaluation unit 104. On the other hand, in the testing stage, the inference unit 102 outputs inference values corresponding to each of the first through n-th inference models to the explanation unit 103 and the presentation control unit 107.

(Explanation unit 103)
The explanation unit 103 generates explanation information that explains the basis for determining the inference values input from the inference unit 102 for each of the first to n-th inference models.

Here, the explanatory information is information indicating the magnitude of the contribution of the input data to the inferred value input from the inference unit 102. Below, we will mainly explain the case where the explanatory information is a heat map that indicates the magnitude of the contribution of the input data to the inferred value for each region (e.g., pixels that make up an image) or variable. The heat map can indicate important regions or variables of the input data that contributed to the judgment.

If the input data is image data, the heat map can be represented by a two-dimensional vector. Alternatively, if the input data is tabular data, the heat map can be represented by a one-dimensional vector.

The heat map may be generated in any manner. For example, the explanation unit 103 may generate a heat map based on the inferred values input from the inference unit 102. Alternatively, as described above, there may be cases where not only the inferred values but also feature quantities are input from the inference unit 102 to the explanation unit 103. In such cases, the explanation unit 103 may generate a heat map based on the inferred values and feature quantities input from the inference unit 102.

For example, the explanation unit 103 may include a function capable of backpropagation. In this case, as will be described later, the weight parameters held by the explanation unit 103 may be updated by the update unit 106 using the backpropagation method. In other words, the explanation unit 103 may generate a heat map using the weight parameters updated using the backpropagation method.

The so-called Grad-CAM method described in Non-Patent Document 3 can be used as an explanation method for generating a heat map using weight parameters updated using the backpropagation method. Grad-CAM is an explanation method that outputs a heat map that shows the areas of the input to a neural network that have a high contribution to the inferred value. Other explanation methods that can be used include Vanilla Gradient and SmoothGrad.

As mentioned above, the inference value corresponding to the i-th inference model can be expressed as Fi(x). As an example, if the function indicating the heat map generation process is G, the heat map Ti(x) corresponding to the i-th inference model generated by the explanation unit 103 can be expressed as in the following equation (1).

Ti(x)=G(Fi(x))...(1)

In the learning stage, the explanation unit 103 outputs the generated n heat maps (first explanation information) to the explanation evaluation unit 105. Meanwhile, in the testing stage, the explanation unit 103 outputs the generated n heat maps (second explanation information) to the presentation control unit 107.

(Inference evaluation unit 104)
The inference evaluation unit 104 obtains an inference evaluation result based on the inference values corresponding to each of the first to nth inference models input from the inference unit 102 and the correct answer value acquired by the input unit 101. More specifically, the inference evaluation unit 104 obtains an inference evaluation result by comparing the inference values corresponding to each of the first to nth inference models with the correct answer value acquired by the input unit 101.

In an embodiment of the present invention, it is assumed that the inference evaluation unit 104 calculates the sum of loss functions corresponding to the inferred value and the correct value for the first to nth inference models as loss function L1, which is an example of an inference evaluation result. Here, the loss function corresponding to the inferred value and the correct value is not limited to a specific function, and a loss function similar to that used in general neural networks may be used. For example, the loss function corresponding to the inferred value and the correct value may be a cross-entropy error based on the difference between the correct value and the inferred value.

The inference evaluation unit 104 outputs the inference evaluation results to the update unit 106.

(Explanation evaluation unit 105)
The explanation evaluation unit 105 obtains an explanation evaluation result based on the heat maps corresponding to the first to n-th inference models input from the explanation unit 103. More specifically, the explanation evaluation unit 105 compares the heat maps corresponding to the first to n-th inference models with each other. Then, the explanation evaluation unit 105 obtains an explanation evaluation result based on the degree of agreement between the n heat maps as the comparison results.

In an embodiment of the present invention, a loss function is primarily assumed in which the explanation evaluation result takes a smaller value as the degree of agreement between n heat maps increases. Note that the degree of agreement between n heat maps may be rephrased as the degree of discrepancy indicating the degree to which the n heat maps differ from each other. In such a case, the loss function may take a smaller value as the degree of discrepancy between the n heat maps decreases.

There are no limitations on the method for obtaining explanation evaluation results from n heat maps. Here, we will explain two methods for obtaining explanation evaluation results: one method for obtaining explanation evaluation results by multiplying a mask that binarizes a heat map with another heat map, and one method for obtaining explanation evaluation results by taking the dot product of normalized heat maps.

Figure 2 is a diagram illustrating a method for obtaining explanation evaluation results by multiplying a mask obtained by binarizing a heat map with another heat map. For ease of explanation, the example shown in Figure 2 assumes that n = 2, i.e., that the inference unit 102 has a first inference model and a second inference model.

Referring to Figure 2, the first inference model outputs an inferred value and a heat map H1. Meanwhile, the second inference model outputs an inferred value and a heat map H2. In Figure 2, in heat maps H1 and H2, areas of the input data that contribute more to the inferred value are shown in darker colors.

The explanation evaluation unit 105 binarizes the heat map H1 to generate a mask M1, and binarizes the heat map H2 to generate a mask M2. Note that binarization can be performed by setting the value of elements (e.g., pixels constituting the heat map) that are equal to or greater than a threshold c to 1, and the value of elements that are less than the threshold c to 0. In Figure 2, the binary values 1 and 0 are represented by black and white, respectively.

The explanation evaluation unit 105 calculates, for each element, the product of the heat map H1 output from the first inference model and the mask M2 generated from the heat map H2 output from the second inference model. Similarly, the explanation evaluation unit 105 calculates, for each element, the product of the heat map H2 output from the second inference model and the mask M1 generated from the heat map H1 output from the first inference model. This allows a set of products corresponding to each element to be obtained for each inference model.

The explanation evaluation unit 105 calculates the sum of products by adding up the products corresponding to each element for all inference models. Then, the explanation evaluation unit 105 calculates a total value by adding up the sums of products calculated in this way for all elements. The explanation evaluation unit 105 sets this total value as the loss function L2, an example of the explanation evaluation result.

The case where n=2 was explained with reference to Figure 2. If n is any integer greater than 1, the explanation is as follows:

That is, the explanation evaluation unit 105 generates a mask Mi(x) by binarizing the values of each element of the heat map Ti(x) for i = 1 to n. Next, for each inference model, the explanation evaluation unit 105 calculates the product of the heat map Ti(x) output from its own inference model and the sum of the masks M1(x) to Mi-1(x) and Mi+1(x) to Mn(x) generated from the heat maps corresponding to inference models other than its own, for each element.

The explanation evaluation unit 105 calculates the sum of products by adding up the products corresponding to each element for the first to nth inference models. The explanation evaluation unit 105 then obtains the explanation evaluation result based on the sum of products calculated in this way. More specifically, the explanation evaluation unit 105 calculates a total value by adding up the sums of products for all elements. The explanation evaluation unit 105 sets this total value as the loss function L2, an example of the explanation evaluation result.

This loss function L2 is the sum of the areas in each heat map that have values above a threshold in heat maps other than the target. By training to reduce the value of this loss function L2, n inference models with low heat map matching are obtained. Note that the loss function L2 in this case can be expressed as in equation (2) below. In equation (2), e represents the element number. Here, the heat map Ti(x) may be normalized by dividing it by the size of the heat map Ti(x), |Ti(x)|. The heat map Ti(x) may also be multiplied by an activation function such as sigmoid.

With reference to Figure 2, we explained a method for obtaining explanation evaluation results by multiplying a mask created by binarizing a heat map with another heat map. Next, we will explain a method for obtaining explanation evaluation results by taking the dot product of normalized heat maps.

For i = 1 to n, the explanation evaluation unit 105 normalizes the heat map Ti(x) by dividing it by the size of the heat map Ti(x), |Ti(x)|, to generate normalized vectors for i = 1 to n. The explanation evaluation unit 105 then obtains an explanation evaluation result based on the dot product of the normalized vectors for i = 1 to n. More specifically, the explanation evaluation unit 105 calculates a total value by adding up the dot products for all elements. The explanation evaluation unit 105 sets this total value as the loss function L2, an example of the explanation evaluation result.

The larger the dot product of the normalized vectors, the larger the value of this loss function L2. A large dot product of normalized vectors means that the degree of match between heat maps is high. Therefore, by training to reduce the value of this loss function L2, n inference models with low degrees of match between heat maps are obtained. Note that the loss function L2 in this case can be expressed as in the following equation (3). In equation (3), e represents the element number.

The explanation evaluation unit 105 outputs the explanation evaluation results to the update unit 106.

(Update unit 106)
The update unit 106 updates the weight parameters 110 used by each of the first to n-th inference models based on the inference evaluation result input from the inference evaluation unit 104 and the explanation evaluation result input from the explanation evaluation unit 105. This allows the weight parameters 110 to be updated so that the inference values output from each of the first to n-th inference models approach the correct value and so that the degree of agreement between the n heat maps output from the explanation unit 103 decreases. The weight parameters 110 may be updated by backpropagation.

For example, the update unit 106 may add the inference evaluation result input from the inference evaluation unit 104 and the explanation evaluation result input from the explanation evaluation unit 105, and update the weight parameter 110 based on the addition result. At this time, the update unit 106 may update the weight parameter 110 by backpropagation using the calculated addition result as the error. As described above, if the inference evaluation result is expressed as loss function L1 and the explanation evaluation result is expressed as loss function L2, the addition result is L1 + L2.

Furthermore, the update unit 106 may update the weight parameters held by the explanation unit 103. More specifically, if the explanation unit 103 includes a function capable of backpropagation, the update unit 106 may update the weight parameters held by the explanation unit 103 by backpropagation based on the inference evaluation result and the explanation evaluation result.

The learning termination condition (i.e., the weight parameter update termination condition) is not particularly limited and may be any condition indicating that a certain degree of learning has been performed on the first through nth inference models. Specifically, the learning termination condition may include a condition that the value of the loss function L1 + L2 is smaller than a threshold. Alternatively, the learning termination condition may include a condition that the change in the value of the loss function L1 + L2 is smaller than a threshold (a condition that the value of the loss function L1 + L2 has converged). Alternatively, the learning termination condition may include a condition that the weight parameters have been updated a predetermined number of times. Alternatively, if the inference evaluation unit 104 calculates accuracy (e.g., accuracy rate) based on the correct answer value and the inferred value, the learning termination condition may include a condition that the accuracy exceeds a predetermined percentage (e.g., 90%).

(Presentation control unit 107)
In the test phase, the presentation control unit 107 controls the display unit 121 so that inference values corresponding to the first to nth inference models input from the inference unit 102 and n heat maps input from the explanation unit 103 are presented to the user. More specifically, the presentation control unit 107 controls the display unit 121 so that inference values corresponding to the first to nth inference models and n heat maps are displayed. Note that a configuration in which n heat maps are displayed but inference values corresponding to the first to nth inference models are not displayed may also be envisioned.

(Display section 121)
The display unit 121 is configured by a display and has a function of displaying various information under the control of the presentation control unit 107. For example, the display unit 121 can display n inferred values and n heat maps. Here, the form of the display unit 121 is not particularly limited. For example, the display unit 121 may be a liquid crystal display (LCD) device, an OLED (organic light emitting diode) device, or a display device such as a lamp.

(Operation unit 122)
The operation unit 122 accepts operations by the user. For example, suppose the user refers to n inference values and n heat maps and finds one or more inference models (hereinafter also referred to as "selection models") with high interpretability from the n inference models. At this time, the user inputs information indicating the selection model (hereinafter also referred to as "selection model information") into the operation unit 122, and the operation unit 122 accepts selection model information 123. For example, the selection model information 123 may be a number indicating the selection model.

Note that in the embodiment of the present invention, it is primarily assumed that the operation unit 122 is a mouse and keyboard. However, the form of the operation unit 122 is not particularly limited. For example, the operation unit 122 may be a touch panel or another input device.

(Recording control unit 108)
The recording control unit 108 controls the recording of the selection model information 123 received from the user by the operation unit 122. More specifically, the recording control unit 108 stores the selection model information 123 received from the user by the operation unit 122 in a storage unit (not shown). The selection model information 123 is later acquired from the storage unit (not shown), and the selection model indicated by the selection model information 123 can be used as a trained model with high interpretability.

The test termination conditions are not particularly limited, and may be any conditions that indicate that the user has performed the test a sufficient number of times. Specifically, the test termination conditions may include a condition that the user has confirmed the inference results a predetermined number of times or more during the test phase.

The above describes an example configuration of the learning device 10 according to an embodiment of the present invention.

(1.2. Actions in the learning stage)
The flow of operations in the learning stage of the learning device 10 according to the embodiment of the present invention will be described with reference to Fig. 3. Fig. 3 is a flowchart showing an example of operations in the learning stage of the learning device 10 according to the embodiment of the present invention.

First, as shown in FIG. 3, the input unit 101 acquires a combination of input data (i.e., learning data) and correct values from the dataset 100. Furthermore, the inference unit 102 acquires weight parameters 110 corresponding to each of the n inference models (S11). The inference unit 102 performs inference based on the input data acquired by the input unit 101 and the n inference models (S12), and outputs the n inference values obtained by the inference to the inference evaluation unit 104 and the explanation unit 103, respectively.

The explanation unit 103 generates a heat map that explains the basis for determining each of the n inference values based on the n inference values input from the inference unit 102 (S13). The explanation unit 103 outputs the generated n heat maps to the explanation evaluation unit 105.

The inference evaluation unit 104 evaluates the n inference values input from the inference unit 102 based on the correct answer value acquired by the input unit 101 to obtain an inference evaluation result. More specifically, the inference evaluation unit 104 calculates a loss function corresponding to the correct answer value and the n inference values as the inference evaluation result. The inference evaluation unit 104 outputs the calculated inference evaluation result to the update unit 106.

The explanation evaluation unit 105 obtains an explanation evaluation result based on the degree of agreement between the n heat maps input from the explanation unit 103. More specifically, the explanation evaluation unit 105 calculates a loss function according to the degree of agreement between the n heat maps input from the explanation unit 103 as the explanation evaluation result. The explanation evaluation unit 105 outputs the calculated explanation evaluation result to the update unit 106 (S14).

The update unit 106 updates the weight parameters 110 corresponding to each of the first to nth inference models based on the inference evaluation results input from the inference evaluation unit 104 and the explanation evaluation results input from the explanation evaluation unit 105 (S15). More specifically, the update unit 106 updates the weight parameters 110 using backpropagation based on the inference evaluation results and explanation evaluation results. Furthermore, the update unit 106 updates the weight parameters held by the explanation unit 103 using backpropagation based on the inference evaluation results and explanation evaluation results.

Each time the update unit 106 finishes updating the weight parameters based on input data, it determines whether the learning termination condition has been met (S16). If it determines that the learning termination condition has not been met (NO in S16), operation proceeds to S11, the input unit 101 acquires the next input data, and the inference unit 102, explanation unit 103, inference evaluation unit 104, explanation evaluation unit 105, and update unit 106 each re-execute their respective processes based on the next input data. On the other hand, if the update unit 106 determines that the learning termination condition has been met (YES in S16), learning is terminated.

The above describes the operational flow during the learning stage of the learning device 10 according to an embodiment of the present invention.

(1.3. Operation in the test phase)
The flow of operations in the test phase of the learning device 10 according to the embodiment of the present invention will be described with reference to Fig. 4. Fig. 4 is a flowchart showing an example of operations in the test phase of the learning device 10 according to the embodiment of the present invention.

First, as shown in FIG. 4, the input unit 101 acquires a combination of input data (i.e., test data) and correct values from the dataset 100. Furthermore, the inference unit 102 acquires weight parameters 110 corresponding to each of the n inference models (S21). The inference unit 102 performs inference based on the input data acquired by the input unit 101 and the n inference models (S22), and outputs the n inference values obtained by the inference to the explanation unit 103 and the presentation control unit 107, respectively.

The explanation unit 103 generates a heat map that explains the basis for determining each of the n inferred values based on the n inferred values input from the inference unit 102 (S23). The explanation unit 103 outputs the generated n heat maps to the presentation control unit 107.

The presentation control unit 107 controls the display unit 121 so that the n inferred values input from the inference unit 102 and the n heat maps input from the explanation unit 103 are presented to the user. The display unit 121 displays the n inferred values and the n heat maps in accordance with the control of the presentation control unit 107 (S24).

The operation unit 122 receives from the user information (selection model information 123) indicating one or more inference models determined to have high interpretability from the n inference models. The recording control unit 108 controls the recording of the selection model information 123 received from the user by the operation unit 122 (S25). A storage unit (not shown) stores the selection model information 123 in accordance with the control of the recording control unit 108.

The recording control unit 108 determines whether the test termination condition has been met (S26) each time it finishes recording control of the selection model information 123 based on the input data. If it determines that the test termination condition has not been met ("NO" in S26), operation proceeds to S21, the input unit 101 acquires the next input data, and the inference unit 102, explanation unit 103, presentation control unit 107, and recording control unit 108 each re-execute their respective processes based on the next input data. On the other hand, if the recording control unit 108 determines that the test termination condition has been met ("YES" in S26), the test is terminated.

The above describes the operational flow during the test phase of the learning device 10 according to an embodiment of the present invention.

(1.4. Effects of the embodiment)
As described above, according to an embodiment of the present invention, learning can be performed so that the inference values output from each of the first to nth inference models approach the correct value and so that the degree of similarity between the n heat maps output as explanatory information decreases. This makes it possible to obtain inference models that output multiple heat maps that are different from each other. This allows a user to select and use a model that outputs a heat map with higher interpretability from among the n models.

The above describes the effects achieved by embodiments of the present invention.

(2. Hardware configuration example)
Next, an example of the hardware configuration of the learning device 10 according to an embodiment of the present invention will be described. Below, an example of the hardware configuration of an information processing device 900 will be described as an example of the hardware configuration of the learning device 10 according to an embodiment of the present invention. Note that the example of the hardware configuration of the information processing device 900 described below is merely one example of the hardware configuration of the learning device 10. Therefore, the hardware configuration of the learning device 10 may be such that unnecessary components are deleted from the hardware configuration of the information processing device 900 described below, or new components are added.

Figure 5 is a diagram showing the hardware configuration of an information processing device 900 as an example of a learning device 10 according to an embodiment of the present invention. The information processing device 900 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, a host bus 904, a bridge 905, an external bus 906, an interface 907, an input device 908, an output device 909, a storage device 910, and a communication device 911.

The CPU 901 functions as an arithmetic processing device and control device, and controls the overall operation of the information processing device 900 in accordance with various programs. The CPU 901 may also be a microprocessor. The ROM 902 stores programs used by the CPU 901, calculation parameters, etc. The RAM 903 temporarily stores programs used in the execution of the CPU 901, and parameters that change as appropriate during the execution. These are interconnected by a host bus 904, which is composed of a CPU bus, etc.

The host bus 904 is connected to an external bus 906, such as a PCI (Peripheral Component Interconnect/Interface) bus, via a bridge 905. Note that the host bus 904, bridge 905, and external bus 906 do not necessarily need to be configured separately; these functions may be implemented on a single bus.

The input device 908 is composed of input means for the user to input information, such as a mouse, keyboard, touch panel, buttons, microphone, switches, and levers, and an input control circuit that generates input signals based on user input and outputs them to the CPU 901. By operating this input device 908, the user operating the information processing device 900 can input various data to the information processing device 900 and instruct processing operations.

The output device 909 includes, for example, display devices such as a CRT (Cathode Ray Tube) display device, a liquid crystal display (LCD) device, an OLED (Organic Light Emitting Diode) device, a lamp, and an audio output device such as a speaker.

The storage device 910 is a device for storing data. The storage device 910 may include a storage medium, a recording device that records data on the storage medium, a reading device that reads data from the storage medium, and a deletion device that deletes data recorded on the storage medium. The storage device 910 is configured, for example, with an HDD (Hard Disk Drive). This storage device 910 drives the hard disk and stores programs executed by the CPU 901 and various data.

The communication device 911 is, for example, a communication interface configured with a communication device for connecting to a network. Furthermore, the communication device 911 may support either wireless communication or wired communication.

The above describes an example of the hardware configuration of the learning device 10 according to an embodiment of the present invention.

(3. Summary)
Although the preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, the present invention is not limited to these examples. It is clear that a person skilled in the art to which the present invention pertains can conceive of various modifications and alterations within the scope of the technical ideas set forth in the claims, and it is understood that these also naturally fall within the technical scope of the present invention.

For example, the above example mainly assumes that the learning device 10 learns n inference models simultaneously. However, the learning device 10 does not have to learn all n inference models simultaneously. For example, a trained inference model may be used as part of the n inference models. In this case, the weight parameters of the trained inference model may be fixed to a constant value without being updated.

Furthermore, in the above example, it is mainly assumed that the explanation unit 103 uses one type of heat map generation method. However, the explanation unit 103 may use multiple types of heat map generation methods. In this case, the explanation unit 103 may output to the update unit 106 the total value of the loss based on the degree of agreement between the heat maps for the multiple types of heat map generation methods as an example of the explanation evaluation result.

REFERENCE SIGNS LIST 10 Learning device 100 Data set 101 Input unit 102 Inference unit 103 Explanation unit 104 Inference evaluation unit 105 Explanation evaluation unit 106 Update unit 107 Presentation control unit 108 Recording control unit 110 Weight parameter 121 Display unit 122 Operation unit 123 Selection model information

Claims (12)

an input unit that acquires first input data and a correct answer value of the first input data;
an inference unit that outputs a first inference value corresponding to each of a plurality of inference models based on the first input data and a plurality of inference models;
an explanation unit that outputs first explanation information corresponding to each of the plurality of inference models, the first explanation information indicating the magnitude of contribution of the first input data to the first inferred value;
an inference evaluation unit that obtains an inference evaluation result based on the correct answer value and the first inference value;
an explanation evaluation unit that obtains an explanation evaluation result based on the degree of agreement between first explanation information corresponding to each of the plurality of inference models;
an update unit that updates first weight parameters of the plurality of inference models based on the inference evaluation result and the explanation evaluation result so as to reduce the degree of agreement between the first explanation information;
A learning device comprising:
the input unit acquires second input data;
the inference unit outputs second inference values corresponding to the plurality of trained models, which are the plurality of estimation models after updating the first weight parameters, based on the second input data and the plurality of trained models;
the explanation unit outputs second explanation information corresponding to each of the plurality of trained models, the second explanation information indicating a magnitude of contribution of the second input data to the second inferred value;
the learning device includes: a presentation control unit that controls presentation of the second inferred value and the second explanation information to a user;
an operation unit that receives, from the user, selected model information indicating one or more trained models selected by the user from the plurality of trained models;
A learning device comprising: The learning device
A recording control unit that controls recording of information indicating the one or more trained models,
The learning device according to claim 1 . the explanation evaluation result takes a smaller value as the degree of agreement between the first explanation information corresponding to each of the plurality of inference models increases;
The learning device according to claim 1 or 2 . the explanation evaluation unit obtains the explanation evaluation result based on an inner product of vectors normalized from first explanation information corresponding to each of the plurality of inference models;
The learning device according to claim 3 . The explanation evaluation unit binarizes the first explanation information to generate a mask for each of the multiple inference models, calculates the product of the mask generated from the first explanation information corresponding to an inference model other than itself and the first explanation information corresponding to its own inference model, and obtains the explanation evaluation result based on the sum of the products for each of the multiple inference models.
The learning device according to claim 3 . The explanation section includes a function capable of backpropagation.
The learning device according to any one of claims 1 to 5 . the explanation section has a second weighting parameter;
the update unit updates the second weight parameters by backpropagation.
The learning device according to claim 6 . At least one of the plurality of inference models includes a neural network.
The learning device according to any one of claims 1 to 7 . the update unit updates the first weight parameter based on an addition result of the inference evaluation result and the explanation evaluation result.
The learning device according to any one of claims 1 to 8 . the first explanatory information is a heat map showing the magnitude of contribution of the first input data to the first inferred value;
The learning device according to any one of claims 1 to 9 . Obtaining first input data and a correct answer value of the first input data;
outputting a first inference value corresponding to each of the plurality of inference models based on the first input data and the plurality of inference models;
outputting first explanatory information corresponding to each of the plurality of inference models indicating the magnitude of contribution of the first input data to the first inferred value;
obtaining an inference evaluation result based on the correct answer value and the first inference value;
obtaining an explanation evaluation result based on the degree of agreement between first explanation information corresponding to each of the plurality of inference models;
updating first weight parameters of the plurality of inference models based on the inference evaluation result and the explanation evaluation result so that the degree of agreement between the first explanation information becomes smaller;
obtaining second input data;
outputting second inference values corresponding to each of the plurality of trained models based on the second input data and the plurality of trained models, which are the plurality of estimation models after updating the first weight parameters; and
outputting second explanation information corresponding to each of the plurality of trained models, the second explanation information indicating a magnitude of contribution of the second input data to the second inferred value;
controlling presentation of the second inference value and the second explanatory information to a user;
receiving selected model information from the user indicating one or more trained models selected by the user from the plurality of trained models;
including, learning methods. Computer,
an input unit that acquires first input data and a correct answer value of the first input data;
an inference unit that outputs a first inference value corresponding to each of a plurality of inference models based on the first input data and a plurality of inference models;
an explanation unit that outputs first explanation information corresponding to each of the plurality of inference models, the first explanation information indicating the magnitude of contribution of the first input data to the first inferred value;
an inference evaluation unit that obtains an inference evaluation result based on the correct answer value and the first inference value;
an explanation evaluation unit that obtains an explanation evaluation result based on the degree of agreement between first explanation information corresponding to each of the plurality of inference models;
an update unit that updates first weight parameters of the plurality of inference models based on the inference evaluation result and the explanation evaluation result so as to reduce the degree of agreement between the first explanation information;
A learning device comprising:
the input unit acquires second input data;
the inference unit outputs second inference values corresponding to the plurality of trained models, which are the plurality of estimation models after updating the first weight parameters, based on the second input data and the plurality of trained models;
the explanation unit outputs second explanation information corresponding to each of the plurality of trained models, the second explanation information indicating a magnitude of contribution of the second input data to the second inferred value;
the learning device includes: a presentation control unit that controls presentation of the second inferred value and the second explanation information to a user;
an operation unit that receives, from the user, selected model information indicating one or more trained models selected by the user from the plurality of trained models;
A program that functions as a learning device .