JP7179714B2 - Explanation device, explanation method and explanation program - Google Patents

Description

The present invention relates to an apparatus, method, and program for explaining the basis of classification results by machine learning models.

Conventionally, a classification method using a machine learning model has been provided for classifying, for example, whether the content of a sentence is positive or negative. However, because the mechanism of classification by a machine learning model is invisible, it is difficult to infer the classification factors from the classification results.
Therefore, methods have been proposed for indicating important factors that have contributed to the classification in the classification results (see, for example, Non-Patent Documents 1 to 4).

M. T. Ribeiro, S.; Singh, and C.I. Ｇｕｅｓｔｒｉｎ， “‘Ｗｈｙ Ｓｈｏｕｌｄ Ｉ Ｔｒｕｓｔ Ｙｏｕ？’： Ｅｘｐｌａｉｎｉｎｇ ｔｈｅ Ｐｒｅｄｉｃｔｉｏｎｓ ｏｆ Ａｎｙ Ｃｌａｓｓｉｆｉｅｒ，” ｉｎ Ｐｒｏｃｅｅｄｉｎｇｓ ｏｆ ｔｈｅ ２２ｎｄ ＡＣＭ ＳＩＧＫＤＤ Ｉｎｔｅｒｎａｔｉｏｎａｌ Ｃｏｎｆｅｒｅｎｃｅ ｏｎ Ｋｎｏｗｌｅｄｇｅ Ｄｉｓｃｏｖｅｒｙ ａｎｄ Ｄａｔａ Ｍｉｎｉｎｇ， Ｎｅｗ Ｙｏｒｋ， ＮＹ， ＵＳＡ， ２０１６， ｐｐ． 1135-1144. S. M. Lundberg and S. -I. Lee, "A Unified Approach to Interpreting Model Predictions," in Advances in Neural Information Processing Systems 30, I.M. Guyon, U.S.A. V. Luxburg, S.; Bengio, H.; Wallach, R.; Fergus, S.; Vishwanathan, and R.I. Garnett, Eds. Curran Associates, Inc. , 2017, pp. 4765-4774. J. Chen, L. Song, M. J. Wainwright, andM. I. Jordan, "L-Shapley and C-Shapley: Efficient Model Interpretation for Structured Data," in International Conference on Learning Presentations, 2019. M. T. Ribeiro, S.; Singh, and C.I. Guestrin, "Anchors: High-Precision Model-Agnostic Explanations," in Thirty-Second AAAI Conference on Artificial Intelligence, 2018.

However, although the conventional method provides the degree of importance of each word, it does not consider the position and order of the words, so it is not possible to provide an explanation based on the combination of words. Although grammatical relationships are also taken into account in Non-Patent Document 4, it relies on grammatical models and lacks flexibility.

SUMMARY OF THE INVENTION An object of the present invention is to provide an explanation device, an explanation method, and an explanation program capable of appropriately outputting the grounds of classification results by a machine learning model.

The explanation device according to the present invention inputs a text in which two consecutive words are randomly rearranged into a machine learning model after fixing the arrangement of two consecutive words, and calculates the classification prediction value as the result of inputting the text into the machine learning model. A link-by-link prediction value generator that obtains multiple prediction values by link for each link that indicates a sequence of two words, and a text in which the position of one word is fixed and other words are randomly rearranged. A word-by-word prediction value generating unit that obtains a plurality of classification prediction values, which are the results of inputting to the learning model, as word-by-word prediction values for each word, and a relative magnitude of the link-by-link prediction value for each link. a link importance calculation unit for calculating the link importance based on the degree of importance of the word; a word importance calculation unit for calculating the word importance based on the relative magnitude of the word-by-word prediction value for each word; a component selection unit that selects a component consisting of two or more consecutive words based on the link importance and the word importance; and a component importance calculation unit that calculates the importance.

The link importance calculation unit calculates, as the link importance, a value obtained by normalizing the difference between the average value for each link and the overall average value of the predicted values for each link, and the word importance calculation unit may calculate, as the word importance, a value obtained by normalizing the difference between the average value for each word and the overall average value of the word-by-word prediction values.

When the average value for each link is equal to or greater than the overall average value, the link importance calculation unit calculates the standard deviation of the predicted value for each link that is equal to or greater than the overall average value for the difference. After dividing, a value obtained by multiplying the overall standard deviation is calculated, and if the average value for each link is less than the overall average value, the average of the link-specific predicted values for the difference After dividing by the standard deviation of the value less than the value, the value obtained by multiplying the overall standard deviation is calculated, and the word importance calculation unit calculates, when the average value for each word is equal to or greater than the overall average value, After dividing the difference by the standard deviation of the values above the overall average among the word-specific prediction values, the value obtained by multiplying the overall standard deviation is calculated, and the average value for each word is the overall value. If it is less than the average value, the difference may be divided by the standard deviation of the word-by-word prediction values that are less than the overall average value, and then multiplied by the overall standard deviation to calculate a value. .

The component selector may select the link as part of the component if the link importance exceeds an average of the word importance of two words related by the link.

The component selection unit selects the link as part of the component when the sum of the link importance and the average of the word importance of two words related by the link exceeds a threshold. You may choose.

The explanation device may include an output unit that outputs the importance calculated for each of the components and words not included in the components.

In the explanation method according to the present invention, after fixing the sequence of two consecutive words, a text in which other words are randomly rearranged is input to a machine learning model, and the prediction value of the classification is calculated as the above two words. A link-by-link prediction value generation step of obtaining multiple prediction values by link for each link that indicates a sequence of two words, and a text in which the position of one word is fixed and other words are randomly rearranged are machine-generated. A word-by-word predicted value generation step of obtaining a plurality of predicted values of the classification that are the results of inputting to the learning model as word-by-word predicted values for each of the words; a link importance calculation step of calculating a link importance based on the degree of importance of each word; a word importance calculation step of calculating a word importance based on the relative magnitude of the word-by-word prediction value for each word; a component selection step of selecting a component consisting of two or more consecutive words based on the link importance and the word importance; and a component importance calculating step of calculating the importance.

An explanation program according to the present invention is for causing a computer to function as the explanation device.

ADVANTAGE OF THE INVENTION According to this invention, the basis of the classification result by a machine-learning model can be output appropriately.

It is a figure which shows the example of presentation by the explanation method in embodiment in comparison with the conventional method. It is a figure which shows the functional structure of the explanatory device in embodiment. 4 is a first flowchart illustrating the flow of processing by the explanation device in the embodiment; 9 is a second flowchart illustrating the flow of processing by the explanation device in the embodiment; 9 is a third flowchart illustrating the flow of processing by the explanation device in the embodiment; 10 is a fourth flowchart illustrating the flow of processing by the explanation device in the embodiment; 10 is a fifth flowchart illustrating the flow of processing by the explanation device in the embodiment;

An example of an embodiment of the present invention will be described below.
In the explanation method of the present embodiment, by assigning an importance value to each text element for the classification result of the text input to the trained machine learning model, the basis for deriving this result is explained. be.
In this explanation method, the type of model is not limited, and the grounds for the output result can be explained after the fact for a trained model.

FIG. 1 is a diagram showing a presentation example according to the explanation method of this embodiment in comparison with a conventional method.
In a machine learning model that performs positive or negative semantic analysis, for example, for an input text "This movie is not bad", each word "This", "movie", "is", "not", Importance was calculated for "bad". In this example, ``not'' is explained as ``Medium-importance-negative'', and ``bad'' is explained as ``High-importance-negative''. It is shown.

However, the sequence of words "not bad" is semantically positive, and the classification result by the machine learning model is also positive.
In this embodiment, a component consisting of a plurality of words is introduced, and the importance of each component is presented. In this example, "high-importance-positive" is given for "not bad".

In this embodiment, when a text is divided into a plurality of words, the relationship between consecutive words is called a link, and a plurality of words related by one or more links determined to be important by a method described later are configure components;

FIG. 2 is a diagram showing the functional configuration of the explanation device 1 according to this embodiment.
The explanation device 1 is an information processing device (computer) such as a server device or a personal computer, and includes a control unit 10, a storage unit 20, input/output devices for various data, communication devices, and the like.

The control unit 10 is a part that controls the entire explanation device 1, and implements each function in the present embodiment by appropriately reading and executing various programs stored in the storage unit 20. FIG. The control unit 10 may be a CPU.

The storage unit 20 is a storage area for various programs and various data for causing the hardware group to function as the explanation device 1, and may be a ROM, RAM, flash memory, hard disk drive (HDD), or the like. Specifically, the storage unit 20 stores a program (description program) for causing the control unit 10 to execute each function of the present embodiment, a target machine learning model, and the like.

The control unit 10 includes a link-by-link prediction value generation unit 11, a word-by-word prediction value generation unit 12, a link importance calculation unit 13, a word importance calculation unit 14, a component selection unit 15, and a component importance calculation unit. 16 and an output unit 17 . The explanation device 1 uses these functional units to generate information explaining the classification result by the machine learning model, and provides the information to the user.

The link-by-link prediction value generation unit 11 fixes the sequence of two consecutive words in the text input to the learned machine learning model, and then generates the text in which other words are randomly rearranged by the same machine learning model. A plurality of predicted values of the classification, which are the results of inputting to the learning model, are obtained as predicted values for each link for each link indicating a sequence of two words.

The word-by-word prediction value generating unit 12 inputs a text in which the position of one word is fixed and other words are randomly rearranged into the same machine learning model, and the prediction value of the classification is generated by word For each word, a plurality of prediction values are acquired for each word.

Here, the predicted value such as positive or negative is output as a numerical value. For example, if it is positive, the degree is indicated by a positive value, and if it is negative, the degree is indicated by a negative value. you can

The link importance calculator 13 calculates the link importance based on the relative magnitude of the link-by-link predicted value for each link. In other words, when the absolute value of the predicted value by the text rearranged while maintaining the link is relatively large, it is considered that the link contributes greatly to the classification result of the original text. Largely calculated.

For example, the link importance calculator 13 calculates a value obtained by normalizing the difference between the average value of each link and the average value of the entire predictive value for each link as the link importance.
Specifically, when the average value for each link is equal to or greater than the overall average value, the link importance calculation unit 13 calculates the standard deviation After dividing by , calculate the value multiplied by the overall standard deviation. Similarly, when the average value for each link is less than the overall average value, the link importance calculation unit 13 divides the difference by the standard deviation of the predicted values for each link that are less than the overall average value. After that, calculate the value multiplied by the overall standard deviation.

The word importance calculation unit 14 calculates word importance based on the relative magnitude of the word-by-word prediction value for each word. In other words, if the absolute value of the predicted value of the relocated text while maintaining the word position is relatively large, it is considered that the word contributes significantly to the classification result of the original text. calculated to be large.

For example, the word importance calculation unit 14 calculates, as the word importance, a value obtained by normalizing the difference between the average value for each word and the overall average value of the prediction values for each word.
Specifically, when the average value for each word is equal to or greater than the overall average value, the word importance calculation unit 14 calculates the standard deviation After dividing by , calculate the value multiplied by the overall standard deviation. Similarly, when the average value for each word is less than the overall average value, the word importance calculation unit 14 divides the difference by the standard deviation of the word-based prediction values that are less than the overall average value. After that, calculate the value multiplied by the overall standard deviation.

The component selection unit 15 selects a component consisting of two or more consecutive words based on the link importance and word importance.
Specifically, the component selector selects a link as part of a component if, for example, the link importance of a link exceeds the average of the word importance of two words related by this link. select.
For example, when the sum of the link importance of a certain link and the average of the word importance of two words related by this link exceeds a threshold, the component selection unit 15 selects this link as a component. Select as part of

Here, when a link is selected, a component consisting of two words related by this link is constructed, and when two consecutive links are selected, three words related by these links are constructed. Components consisting of words are constructed. Thus, a component consists of two or more consecutive words.

The component importance calculator 16 calculates the importance of this component based on the link importance and word importance associated with the component.

The output unit 17 presents the importance to the user by outputting the calculated importance on a display or the like for each of the selected component and words not included in any component.

Next, the contents of processing by the explanation device 1 will be described in detail.
3 to 7 are flowcharts illustrating the flow of processing by the explanation device 1 in this embodiment.

Arrays LinkPred and LinkHashmap[Key] for storing values predicted by the machine learning model are prepared in advance. However, Key is an index indicating the order of links in the original text, and Key=0, 1, .
Similarly, arrays FeaturePred and FeatureHashmap[Key] are provided. However, Key is an index indicating the order of words in the original text, and Key=0, 1, .

As parameters, the user designates the number of loops for acquiring the samples of the prediction values for each link and the number of loops for acquiring the samples of the prediction values for each word.

In step S1, the link-by-link prediction value generator 11 sequentially selects one link included in the original text. Let i be the index of the selected link.
In step S2, the link-by-link predicted value generator 11 repeats steps S3 to S4 for the number of loops specified in advance by a parameter.

In step S3, the link-by-link prediction value generator 11 fixes the sequence of the two words related by the link selected in step S1, and generates a text in which other words are randomly rearranged.
For example, for "this movie is not bad", if the link with index i=3, that is, the string of words "not bad" is fixed, then "not bad this is movie", "movie not bad is this" text is generated.

In step S4, the link-by-link predicted value generation unit 11 inputs the text generated in step S3 to the machine learning model, and acquires the output as the link-by-link predicted value.
This predicted value for each link is stored in LinkPred and LinkHashmap[i].

In step S5, the link-by-link predicted value generator 11 determines whether or not all links have been selected. If the determination is YES, the process moves to step S6, and if the determination is NO, the process moves to step S1.

In step S6, the word-by-word prediction value generator 12 sequentially selects one word included in the original text. Let i be the index of the selected word.
In step S7, the word-by-word prediction value generator 12 repeats the following steps S8 to S9 by the number of loops specified in advance by a parameter.

In step S8, the word-by-word prediction value generator 12 fixes the position of the word selected in step S6 and generates a text in which other words are randomly rearranged.
For example, for "this movie is not bad", if the position of the word "not" with index i=3 is fixed, the text "movie is bad not this", "bad this is not movie", etc. is generated.

In step S9, the word-by-word predicted value generation unit 12 inputs the text generated in step S8 to the machine learning model, and acquires the output as word-by-word predicted values.
This word-by-word prediction value is stored in FeaturePred and FeatureHashmap[i].

In step S10, the word-by-word prediction value generator 12 determines whether or not all words have been selected. If the determination is YES, the process moves to step S11, and if the determination is NO, the process moves to step S6.

In step S11, the link importance calculation unit 13 calculates the variance Lvar of values less than the average value and the variance Uvar of values equal to or greater than the average value among the link-by-link prediction values stored in LinkPred.

In step S12, the link importance calculator 13 sequentially selects one link included in the original text. Let i be the index of the selected link.
In step S13, the link importance calculation unit 13 determines whether or not the average value of the predicted values included in LinkHashmap[i] is smaller than the average value of the predicted values included in LinkPred. If the determination is YES, the process moves to step S14, and if the determination is NO, the process moves to step S15.

と計算する。 In step S14, the link importance calculation unit 13 uses Lvar to calculate the weight of the link with the index i as follows:

to calculate.

と計算する。 In step S15, the link importance calculation unit 13 uses Uvar to calculate the weight of the link with the index i as follows:

to calculate.

In step S16, the link importance calculator 13 determines whether or not all links have been selected. If the determination is YES, the process moves to step S17, and if the determination is NO, the process moves to step S12.

In step S17, the word importance calculation unit 14 calculates the variance Lvar of values less than the average value and the variance Uvar of values equal to or greater than the average value among the word-by-word prediction values stored in FeaturePred.

In step S18, the word importance calculator 14 sequentially selects one word included in the original text. Let i be the index of the selected word.
In step S19, the word importance calculator 14 determines whether or not the average value of the predicted values included in FeatureHashmap[i] is smaller than the average value of the predicted values included in FeaturePred. If the determination is YES, the process moves to step S20, and if the determination is NO, the process moves to step S21.

と計算する。 In step S20, the word importance calculation unit 14 uses Lvar to calculate the weight of the word with the index i as follows:

to calculate.

と計算する。 In step S21, the word importance calculation unit 14 uses Uvar to calculate the weight of the word with the index i as follows:

to calculate.

In step S22, the word importance calculator 14 determines whether or not all words have been selected. If the determination is YES, the process moves to step S23, and if the determination is NO, the process moves to step S18.

In step S23, the component selection unit 15 sequentially selects one link included in the original text. Let i be the index of the selected link.

In step S24, the component selection unit 15 calculates the average of the absolute values of the importance of the two words on both sides linked by the link with the index i, that is, the words with the indexes i and i+1.

Here, the importance of a word with an index of i is a value obtained by multiplying the word weight FeatureWeight "i" described above by the overall standard deviation FeatureDev of the word-by-word prediction values stored in FeaturePred.
As with the importance of words, the importance of a link with an index of i is obtained by multiplying the above-mentioned link weight LinkWeight "i" by the overall standard deviation LinkDev of the predicted values for each link stored in LinkPred. defined as a value.

In step S25, the component selection unit 15 determines whether the absolute value of the importance of the link with index i is greater than the average value calculated in step S24. If this determination is YES, that is,
abs(LinkWeight[i])*LinkDev>mean(abs(FeatureWeight[i]), abs(FeatureWeight[i+1]))*FeatureDev
If , the process moves to step S27, and if the determination is NO, the process moves to step S26.

In step S26, the component selection unit 15 compares the importance of the link with the index i and the average value of the importance of the two words associated with this link, that is, the words with the indexes i and i+1. Determine whether the absolute value of the sum is greater than a threshold.
For example, the component selection unit 15 uses values obtained by adding standard deviations LinkDev and FeatureDev to Linkweight and FeatureWeight, respectively,
abs(mean(FeatureWeight[i], FeatureWeight[i+1])*FeatureDev+LinkWeight[i]*LinkDev)>LinkDev*(LinkDev+abs(mean(LinkWeight)))+FeatureDev*(FeatureDev+abs(FeatureWeight)(FeatureWeight)(FeatureDev)
is established. If the determination is YES, the process moves to step S27, and if the determination is NO, the process moves to step S28.

In step S27, the component selection unit 15 determines that the link with the index i is important, and stores it as the link forming the component.

In step S28, the component selection unit 15 determines whether or not all links have been selected. If the determination is YES, the process moves to step S29, and if the determination is NO, the process moves to step S23.

In step S29, the component selection unit 15 selects, as components, a plurality of continuous words related by the important links stored in step S27. At this time, if the important links are continuous, a component consisting of three or more words is selected.

ここで、Ｆ及びＬは、それぞれコンポーネントを構成する単語及びリンクのインデックス集合である。
また、Ｅは、コンポーネントの両端の単語とその外側とを関係付けるリンクのインデックス集合である。｜Ｅ｜は、Ｅの要素数を示し、コンポーネントがテキストの始め又は終わりに位置する場合は１となり、その他の場合は２となる。 In step S30, the component importance calculator 16 calculates importance I for each of the components selected in step S29, for example, using the following equation.

where F and L are the index sets of words and links that make up the components, respectively.
Also, E is an index set of links that relate the words at both ends of the component to the outside thereof. |E| indicates the number of elements in E and is 1 if the component is located at the beginning or end of the text, 2 otherwise.

In step S31, the output unit presents the selected component and words not included in the component to the user by separately displaying them together with the calculated importance. This indicates the important words or components that make up the input text as the basis for the classification results of the machine learning model.

According to the present embodiment, the explanation device 1 inputs the text in which the sequence of two consecutive words is fixed and other words are randomly rearranged into the machine learning model. and a prediction value for each word, which is the result of inputting text in which the position of one word is fixed and other words are randomly rearranged into a machine learning model. Then, the explanation device 1 calculates consecutive two or more is selected, and the importance of the component is calculated based on the link importance and the word importance.

Therefore, the explanation device 1 collects samples of predicted values using a plurality of texts having a fixed word or link in common, and calculates the degree of importance indicating the degree of contribution of this word or link to the classification result. can be calculated. By calculating the importance of the link, a component composed of a plurality of words is found, and the importance of this component is calculated.
As a result, the explanation device 1 can combine the meaning of a single word with the importance of a single word as the basis for the text classification result for a machine learning model that is a black box whose behavior is unknown. It is possible to indicate the importance of each component, and to appropriately output the grounds for the classification results of the machine learning model.

In addition, the explanation device 1 normalizes and calculates the difference between the average value of the collected predicted values for each link or word and the overall average value as the degree of importance of a link or word.
As a result, the explanation device 1 can suppress the influence of the distribution of predicted values and appropriately evaluate links and words.

At this time, the explanation device 1 determines whether the average value of the predicted values for each link or word is greater than or equal to the overall average value or less than the average value as the degree of importance of the link or word. After dividing by the standard deviation of , normalize by multiplying by the overall standard deviation. Thereby, the explanation device 1 can appropriately evaluate the importance in both classifications such as positive or negative.

The explanation device 1 determines that this link is important and selects it as part of the component if the link importance exceeds the average of the word importance of the two words related by this link.
Since the degree of importance when the link is fixed is calculated to be higher than when the word is fixed, it is clear that the order of the words indicated by the link is important. can appropriately select components that contain

The explanation device 1 selects this link as part of the component when the sum of the link importance and the average of the word importance of the two words related by this link exceeds a threshold.
If the overall importance of the link and word exceeds a predetermined level, it is clear that the component as this combination is important, and the explanation device 1 can appropriately select such a component.

The explanatory device 1 outputs the calculated importance by displaying it on a display or the like, with each of the selected component and words not included in any component as a unit. As a result, the explanation device 1 can appropriately present to the user, in an easy-to-understand manner, the important components and the independently important words as the basis for the classification result by the machine learning model.

Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments. Moreover, the effects described in the above-described embodiments are merely enumerations of the most suitable effects produced by the present invention, and the effects of the present invention are not limited to those described in the embodiments.

The risk explanation method by the explanation device 1 is implemented by software. When it is implemented by software, a program constituting this software is installed in an information processing device (computer). Further, these programs may be recorded on removable media such as CD-ROMs and distributed to users, or may be distributed by being downloaded to users' computers via a network. Furthermore, these programs may be provided to the user's computer as a web service through the network without being downloaded.

1 explanation device 10 control unit 11 predicted value generation unit for each link 12 predicted value generation unit for each word 13 link importance calculation unit 14 word importance calculation unit 15 component selection unit 16 component importance calculation unit 17 output unit 20 storage unit

Claims (8)

After fixing the sequence of two consecutive words, the prediction value of the classification, which is the result of inputting the text in which other words are randomly rearranged into the machine learning model, is calculated for each link that indicates the sequence of the two words. a prediction value generation unit for each link that acquires multiple prediction values for each link;
After fixing the position of one word, a text in which other words are randomly rearranged is input to a machine learning model, and a plurality of classification prediction values are obtained for each word as word-specific prediction values. a word-by-word prediction value generator that
a link importance calculation unit that calculates a link importance based on the relative magnitude of the link-by-link predicted value for each of the links;
a word importance calculation unit for calculating a word importance based on the relative magnitude of the word-by-word prediction value for each word;
a component selection unit that selects a component consisting of two or more consecutive words based on the link importance and the word importance;
and a component importance calculator that calculates the importance of the component based on the link importance and the word importance related to the component. The link importance calculation unit calculates, as the link importance, a value obtained by normalizing the difference between the average value for each link and the overall average value of the predicted values for each link,
2. The explanation according to claim 1, wherein the word importance calculation unit calculates, as the word importance, a value obtained by normalizing a difference between the average value for each word and the overall average value of the word-by-word prediction values. Device. The link importance calculation unit
When the average value for each link is greater than or equal to the overall average value, the difference is divided by the standard deviation of the values greater than or equal to the overall average value among the link-specific prediction values, and then the overall standard deviation Calculate the value multiplied by
When the average value for each link is less than the overall average value, the difference is divided by the standard deviation of the values less than the overall average value among the link-specific predicted values, and then the overall standard deviation Calculate the value multiplied by
The word importance calculation unit
When the average value for each word is greater than or equal to the overall average value, the difference is divided by the standard deviation of the word-specific prediction value greater than or equal to the overall average value, and then the overall standard deviation Calculate the value multiplied by
When the average value for each word is less than the overall average value, the difference is divided by the standard deviation of the word-specific prediction value that is less than the overall average value, and then the overall standard deviation 3. The explanation device according to claim 2, wherein the value obtained by multiplying the . 2. From claim 1, wherein the component selection unit selects the link as part of the component when the link importance exceeds an average of the word importance of two words related by the link. An instructional device according to any one of claims 3 to 4. The component selection unit selects the link as part of the component when the sum of the link importance and the average of the word importance of two words related by the link exceeds a threshold. 5. An instructional device according to any one of claims 1 to 4 to select. 6. The explanation device according to any one of claims 1 to 5, further comprising an output unit for outputting a calculated degree of importance for each of the components and words not included in the components. After fixing the sequence of two consecutive words, the prediction value of the classification, which is the result of inputting the text in which other words are randomly rearranged into the machine learning model, is calculated for each link that indicates the sequence of the two words. a predicted value generation step for each link that acquires a plurality of predicted values for each link;
After fixing the position of one word, a text in which other words are randomly rearranged is input to a machine learning model, and a plurality of classification prediction values are obtained for each word as word-specific prediction values. a word-by-word prediction value generation step for
a link importance calculation step of calculating a link importance based on the relative magnitude of the link-by-link predicted value for each of the links;
a word importance calculation step of calculating word importance based on the relative magnitude of the word-by-word prediction value for each word;
a component selection step of selecting a component consisting of two or more consecutive words based on the link importance and the word importance;
and a component importance calculating step of calculating the importance of the component based on the link importance and the word importance related to the component. An explanation program for causing a computer to function as the explanation device according to any one of claims 1 to 5.

Images