JP7525449B2 - EXPLAINABILITY DEVICE, EXPLAINABILITY METHOD, AND EXPLAINABILITY PROGRAM - Google Patents

Description

The present invention relates to a method for explaining the results of estimation using a machine learning model.

Conventionally, C-Shapley and the like have been known as methods for making inference results by machine learning models explainable (for example, see Non-Patent Document 1). The explainability method calculates an explanation degree that indicates the extent to which a feature of a sample to be inferred affects an inference result.
When applying an explainability technique to text, a corpus is used with labeled sentences or texts, and a learning model is generated with the words that appear as features.The explainability technique is then applied to the learning model and the sentences or texts that are samples to be explained, to obtain the explainability of each feature of the sample.

Jianbo Chen, Le Song, Martin J. Wainwright, and Michael I. Jordan, L-SHAPLEY AND C-SHAPLEY: EFFICIENT MODEL INTERPRETATION FOR STRUCTURED DATA, ICLR 2019. Shomir Wilson, Florian Schaub, Aswarth Abhilash Dara, Frederick Liu, Sushain Cherivirala, Pedro Giovanni Leon, Mads Schaarup Andersen, Sebastian Zimmeck, Kanthashree Mysore Sathyendra, N. Cameron Russell, Thomas B. Norton, Eduard Hovy, Joel Reidenberg, and Norman Sadeh . The Creation and Analysis of a Website Privacy Policy Corpus. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, 2016.

Incidentally, there are corpora in the training samples where a single document is composed of multiple semantically separated segments. For example, OPP-115, a corpus of privacy policies, has such a structure, and labels are assigned to the segments (see Non-Patent Document 2).
When dealing with such corpora, there are cases where we want to know which segments in a document contribute to inferring the label that indicates the content of the entire document, rather than which words in the segments contributed to inferring the label of the segment. In such cases, even if we applied conventional explainability methods using a learning model generated with the words that appear as features, it was not possible to extend the method to calculate the explainability for the segments rather than the words contained in the sample.

The present invention aims to provide an explainability method that can calculate the degree of explainability for a segment.

The explainability device according to the present invention is an explainability device for a model that has learned labels assigned to segments using d-dimensional learning data based on segments that make up a document, and includes an overall confidence level calculation unit that repeatedly connects some features extracted from each of c randomly selected segments out of the multiple segments that make up a target document to generate a d-dimensional feature vector and inputs the d-dimensional feature vector into the model, and calculates the overall confidence level of the target document for each label by obtaining multiple outputs of the model and performing statistical processing; and an overall confidence level calculation unit that randomly selects c segments that include the target segment out of the multiple segments that make up the target document, and generates a d-dimensional feature sequence whose elements are some features extracted from each of the selected segments, and calculates the confidence level of the target segment. The model includes a neighborhood calculation unit that generates multiple neighborhoods of the target segment, a neighborhood first certainty calculation unit that inputs the feature sequence of the neighborhood to the model and calculates a first certainty of the neighborhood for each label, a neighborhood first importance calculation unit that calculates a first importance of the neighborhood based on the product of the first certainty for each label and the overall certainty, a neighborhood second certainty calculation unit that converts a portion of the feature sequence of the neighborhood extracted from the target segment to a predetermined value and then inputs the converted value to the model and calculates a second certainty of the neighborhood for each label, a neighborhood second importance calculation unit that calculates a second importance of the neighborhood based on the product of the second certainty for each label and the overall certainty, and an explanation calculation unit that calculates an explanation of the target segment based on the difference between the first importance and the second importance.

The overall confidence calculation unit and the neighborhood calculation unit may extract d/c feature quantities for each segment according to a predetermined rule.

The predetermined rule may be to extract features corresponding to words that rank highly in terms of an index based on frequency of occurrence.

The neighborhood calculation unit may generate c feature sequences by varying the number of segments selected before the target segment and the number of segments selected after the target segment.

The explainability method according to the present invention is an explainability method for a model that has learned labels assigned to segments using d-dimensional learning data based on segments that make up a document, and includes an overall confidence level calculation step of: randomly selecting c segments from among the multiple segments that make up the target document, concatenating some of the features extracted from each of the segments to generate a d-dimensional feature vector, and inputting the d-dimensional feature vector into the model, and obtaining multiple outputs of the model and performing statistical processing to calculate the overall confidence level of the target document for each label; and randomly selecting c segments from among the multiple segments that make up the target document, including the target segment, and generating a d-dimensional feature sequence having some of the features extracted from each of the selected segments as elements, as a neighborhood of the target segment, The computer executes a neighborhood calculation step of generating a neighborhood feature sequence, a neighborhood first confidence calculation step of inputting the neighborhood feature sequence to the model and calculating a first confidence level of the neighborhood for each label, a neighborhood first importance calculation step of calculating a first importance level of the neighborhood based on a product of the first confidence level for each label and the overall confidence level, a neighborhood second confidence calculation step of converting a portion of the neighborhood feature sequence extracted from the target segment to a predetermined value and inputting the converted value to the model and calculating a second confidence level of the neighborhood for each label, a neighborhood second importance calculation step of calculating a second importance level of the neighborhood based on a product of the second confidence level for each label and the overall confidence level, and an explanation calculation step of calculating an explanation level of the target segment based on a difference between the first importance level and the second importance level.

The explainability program of the present invention is for causing a computer to function as the explainability device.

According to the present invention, it is possible to calculate the degree of explanation for a segment.

FIG. 2 is a diagram illustrating a functional configuration of an explainable device according to an embodiment. 11 is a flowchart illustrating a neighborhood calculation process according to an embodiment. 1 is a flowchart showing the flow of an explainability method according to a conventional device. 1 is a flowchart showing the flow of an explainability method in an embodiment.

An example of an embodiment of the present invention will now be described.
In the explainability method of this embodiment, the method of calculating the explainability for normal words is expanded to calculate the explainability for segments that make up a document.
First, a general explainability method will be described, and then the extended content in this embodiment will be described.

In this embodiment, machine learning on text is assumed. The corpus consists of sentences (segments) and their labels, and the sentences consist of multiple words. Here, the labels are assumed to be binary labels of 0 or 1, but are not limited to this and may be multi-valued.

The basic implementation of machine learning on text is as follows:
(1) A dictionary is created from words that appear in sentences contained in the corpus, i.e., an ID is assigned to each word that appears.
(2) The sentences contained in the corpus are converted into an ID array using a dictionary.
(3) A learning algorithm is executed using a set of pairs of ID sequences and labels converted from the sentences as input to obtain a learning model.

The prediction algorithm takes an ID sequence representing a sentence and a learning model as input, and outputs a predicted label or the confidence (probability) of each label (0 and 1).
At this time, in C-Shapley, the explanation degree is calculated as follows.

Let m be the learning model that takes d-dimensional input data x = [ _x1 , ..., _xd ] based on an ID array as input and returns the confidence of the predicted value y ∈ {0, 1}, where m is a probabilistic function, _Pm (y|x) = P(m(x) = y).
Here, s where s ⊂ {1, 2, ..., d} and k ∈ s is called the feature set around k. Also, for input data x, a sequence x _s = {x _s1 , ..., x _sn } obtained by extracting only the features specified in the feature set s = {s ₁ , ..., _sn } is called a feature sequence. Note that the feature sequence x _s will be referred to as S below.
Moreover, x _k in S is replaced with 0 to obtain S ^* _k .

For a learning model m and input data x, the importance v _x (S) of a feature sequence S is defined as follows.
v _x (S) = Σ _i log (P _m (y _i | S))・P _m (y _i | x) ... (1)

The explanation level φ ^N _x (k) for input data x, feature k, and a neighborhood N of k (a set of feature sequences S) is expressed as follows.
φ ^N _x (k):=Σ _S∈N c(S)・(v _x (S)−v _x (S ^* _k )),
c(S)=2/((|S|+2)(|S|+1)|S|)...(2)
Here, the object of the explanatory degree is each word contained in the document.

In this embodiment, a document that can be divided into multiple segments is handled, and the target of the explainability is the segment. However, the labels of the training data are assigned to the segments, and the generation of the training model is performed in the usual manner. That is, the training data is a d-dimensional vector and a label based on the ID array of the segment.
Here, we will show a method for making the segments that make up a document explainable by using this learning model and the document to be explained as input. Note that the input document is assumed to have been semantically segmented in advance. Alternatively, an existing segmentation tool may be used.

FIG. 1 is a diagram showing the functional configuration of an explainable device 1 according to the present embodiment.
The explanation enabling device 1 is an information processing device (computer) including a control unit 10, a storage unit 20, and various input/output devices and communication devices.

The control unit 10 is a part that controls the entire explainability device 1, and realizes each function in this embodiment by appropriately reading and executing various programs stored in the storage unit 20. The control unit 10 may be a CPU.

The storage unit 20 is a storage area for various programs for causing the hardware group to function as the explainability device 1, various data, etc., and may be a ROM, RAM, flash memory, hard disk drive (HDD), etc.
Specifically, the storage unit 20 stores a program (explainability program) for causing the control unit 10 to execute each function of this embodiment, as well as learning data, learning models, and the like.

The control unit 10 includes an overall certainty calculation unit 11, a neighborhood calculation unit 12, a neighborhood first certainty calculation unit 13, a neighborhood first importance calculation unit 14, a neighborhood second certainty calculation unit 15, a neighborhood second importance calculation unit 16, and an explanation calculation unit 17, and outputs the explanation of the target segment included in the document using these functional units.

The learning model used by the explainability device 1 here learns the labels assigned to the segments using d-dimensional feature vectors based on the segments that make up the documents that serve as the learning data. In other words, the number of dimensions of the feature vectors that are input during learning is d.

The overall certainty factor calculation unit 11 calculates the certainty factor P _m (y|x) of the entire target document as follows.
First, the overall certainty calculation unit 11 randomly selects c segments from among the multiple segments that make up the target document, compresses the features of each of the selected c segments to a dimension of d/c, that is, extracts d/c features (IDs). Next, the overall certainty calculation unit 11 concatenates the compressed features to generate a d-dimensional feature vector, which is input to the model to obtain the certainty for each label.
The overall confidence level calculation unit 11 repeats this process j times, statistically processes (for example, averages) the obtained confidence levels, and calculates the overall confidence level of the target document for each label.
For example, for a binary label of 0 or 1, P _m (1|x) and P _m (0|x)=1-P _m (1|x) are calculated.

Here, the overall confidence calculation unit 11 extracts d/c feature quantities for each segment according to a predetermined rule, for example, extracting feature quantities corresponding to the first d/c words or words ranked highly by an index based on frequency of occurrence (e.g., TF-IDF).

The neighborhood calculation unit 12 calculates the neighborhood N of the k-th target segment in the target document as follows.
That is, the neighborhood calculation unit 12 randomly selects c segments including the target segment from among the multiple segments that make up the target document, and generates multiple d-dimensional feature sequences, each of which has some of the feature quantities extracted from each of the selected segments as elements, as the neighborhood of the target segment.

The neighborhood calculation unit 12 may generate c feature sequences, for example, by sequentially varying the number of segments selected before the target segment and the number of segments selected after the target segment.

FIG. 2 is a flowchart illustrating a neighborhood calculation process according to the present embodiment.
In step S1, the neighborhood calculation unit 12 selects a target segment seg from among a plurality of segments included in a target document, where seg is an array of feature amounts (IDs).
In step S2, the neighborhood calculation unit 12 initializes a variable i to one.

In step S3, the neighborhood calculation unit 12 randomly selects i-1 segments (seg ₁ , seg ₂ , ..., seg _i-1 ) before seg and c-i segments (seg _i+1 , seg _i+2 , ..., seg _c ) after seg in the target document.

In step S4, the neighborhood calculation unit 12 compresses the target segment seg and the segment seg _j selected in step S3 to a length d/c, as in the above-mentioned overall confidence calculation unit, to obtain seg' and seg' _j .
Here, the neighborhood calculation unit 12 calculates a feature sequence in which seg′ and seg′ _j are connected in order,
S _i =seg' ₁ , seg' ₂ , ..., seg' _i-1 , seg', seg' _i+1 , seg' _i+2 , ..., seg' _c
In addition, let S ^* _i be the result of replacing all elements of S _i with length d/c corresponding to seg with a string of 0s.

In step S5, the neighborhood calculation unit 12 adds 1 to i.
In step S6, the neighborhood calculation unit 12 judges whether i>c. If the judgment is YES, the process proceeds to step S7, and if the judgment is NO, the process returns to step S3.

In step S7, the neighborhood calculation unit 12 outputs the neighborhood N={S ₁ , S ₂ , . . . , S _c } and ends the process.

Note that this process is just one example of a procedure for calculating the neighborhood N, and the method for selecting c segments is not limited to this.

The neighborhood first confidence level calculation unit 13 inputs the feature sequence S _i included in the neighborhood N to the model, and calculates the first confidence level P _m (y|S _i ) (y={0, 1}) for each label S _i .
For example, for a binary label of 0 or 1, P _m (1|S _i )=m(S _i ) and P _m (0|S _i )=1-m(S _i ) are calculated.

The neighborhood first importance calculation unit 14 calculates the first importance v _x (S _i ) for each S _i according to the above-mentioned formula (1) based on the product of the first certainty for each label and the overall certainty.

The neighborhood second confidence level calculation unit 15 converts all parts of the feature sequence S _i in the neighborhood N that are extracted from the target segment seg to a predetermined value (e.g., 0), then inputs them into the model and calculates the second confidence level P _m (y|S ^* _i ) for each label S _i .

The neighborhood second importance calculation unit 16 calculates the second importance v _x (S ^* _i ) for each S _i according to the above-mentioned formula (1) based on the product of the second certainty for each label and the overall certainty.

The explanation calculation unit 17 calculates the explanation of the target segment based on the difference between the first importance and the second importance according to the above-mentioned formula (2).

The explainability method realized by these functional units is an extension of the conventional method. For comparison, the conventional procedure is shown below.
FIG. 3 is a flowchart showing the flow of an explainable method according to a conventional device.

In step S11, the conventional device generates a learning model from a corpus of labeled segments (learning data).

In step S12, the conventional device selects a sample segment (test data).

In step S13, the conventional device uses the generated learning model to calculate the confidence of the selected segment.

In step S14, the conventional device uses the generated learning model to calculate the confidence level of the neighborhood of the word to be explained.

In step S15, the conventional device calculates the importance of the same neighborhood as in step S14.

In step S16, the conventional device calculates the confidence level when the target word is removed from the same neighborhood as in step S14.

In step S17, the conventional device calculates the importance when the target word is removed from the neighborhood, as in step S16.

In step S18, the conventional device calculates the explanatory level of the target word based on the difference between the importance calculated in step S15 and the importance calculated in step S17.

FIG. 4 is a flowchart showing the flow of the explainability method in this embodiment.
In step S21, the explainable device 1 generates a learning model from a corpus consisting of labeled segments (learning data).

In step S22, the explainability device 1 selects a sample document (test data). Note that this document has been divided into multiple segments in advance.

In step S23, the explainability device 1 compresses the feature vectors extracted from the selected documents into the dimensions of the training data.

In step S24, the explainability device 1 uses the generated learning model to calculate the confidence level of the document compressed in step S23.

In step S25, the explainability device 1 calculates a neighborhood that includes the segment to be explained.

In step S26, the explainability device 1 uses the generated learning model to calculate the confidence level of the vicinity of the segment to be explained.

In step S27, the explainability device 1 calculates the importance of the same neighborhood as in step S26.

In step S28, the explainability device 1 calculates the confidence level when the target segment is removed from the same neighborhood as in step S26.

In step S29, the explainability device 1 calculates the importance when the target segment is removed from the neighborhood, similar to step S28.

In step S30, the explainability device 1 calculates the explainability of the target segment based on the difference between the importance calculated in step S27 and the importance calculated in step S29.

According to this embodiment, the explainability device 1 uses a model trained on a segment-by-segment basis to calculate the confidence level by compressing the dimensionality of a document consisting of multiple segments, and then similarly compresses the dimensionality of the vicinity of the target segment in the document to calculate the confidence level and importance, and calculates the explainability of the target segment based on the difference with the importance when the target segment is excluded from this vicinity.
In this way, the explainability device 1 can extend the conventional method to calculate the explainability of a segment. That is, the degree of influence of each segment on the estimation result of the entire document is explained, and, for example, labeling of the segments is also possible.

During dimensionality reduction, d/c words are extracted for each segment according to a predetermined rule, and a feature vector with the number of dimensions d of the input to the model is generated from the c segments.
This allows the explainable device 1 to use a segment-based model to evaluate the entire document and the neighborhood of the target segment.
At this time, for example, words with high evaluations based on an index based on the frequency of occurrence such as TF-IDF are extracted, thereby generating an appropriate feature vector.

In addition, when calculating the neighborhood, the explainability device 1 can efficiently generate c feature sequences by varying the number of segments selected before the target segment and the number of segments selected after the target segment.

The above-mentioned embodiment, for example, makes it possible to expand the scope of application of machine learning models, thereby contributing to Goal 9 of the United Nations-led Sustainable Development Goals (SDGs), which is to "build resilient infrastructure, promote sustainable industrialization and foster innovation."

Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments. Furthermore, the effects described in the above-described embodiments are merely a list of the most favorable effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments.

The explainability method using the explainability device 1 is realized by software. When realized by software, the programs that make up this software are installed in an information processing device (computer). These programs may be recorded on removable media such as CD-ROMs and distributed to users, or may be distributed by being downloaded to the user's computer via a network. Furthermore, these programs may be provided to the user's computer as a web service via a network without being downloaded.

Reference Signs List 1 Explainability device 10 Control unit 11 Overall certainty calculation unit 12 Neighborhood calculation unit 13 Neighborhood first certainty calculation unit 14 Neighborhood first importance calculation unit 15 Neighborhood second certainty calculation unit 16 Neighborhood second importance calculation unit 17 Explainability calculation unit 20 Storage unit

Claims (6)

An explainability device for a model that has learned labels assigned to segments using d-dimensional learning data based on segments constituting a document, comprising:
an overall confidence level calculation unit that calculates an overall confidence level of the target document for each label by repeatedly generating a d-dimensional feature vector by concatenating some feature values extracted from each of c randomly selected segments out of a plurality of segments constituting the target document, and inputting the d-dimensional feature vector into the model, and acquiring and statistically processing a plurality of outputs of the model;
a neighborhood calculation unit that randomly selects c segments including a target segment from among a plurality of segments constituting the target document, and generates a plurality of d-dimensional feature sequences having some feature quantities extracted from each of the selected segments as elements as neighborhoods of the target segment;
a neighborhood first confidence level calculation unit that inputs the neighborhood feature sequence into the model and calculates a first confidence level of the neighborhood for each label;
a neighborhood first importance calculation unit that calculates a first importance of the neighborhood based on a product of the first certainty for each label and the overall certainty;
a neighborhood second certainty calculation unit that converts a portion of the neighborhood feature sequence extracted from the target segment into a predetermined value, inputs the converted value into the model, and calculates a neighborhood second certainty for each label;
a neighborhood second importance calculation unit that calculates a second importance of the neighborhood based on a product of the second certainty for each label and the overall certainty;
An explainability calculation unit that calculates an explainability of the target segment based on a difference between the first importance and the second importance. The explainability device according to claim 1, wherein the overall confidence calculation unit and the neighborhood calculation unit extract d/c feature quantities for each segment according to a predetermined rule. The explainability device according to claim 2, wherein the predetermined rule is to extract features corresponding to words ranked highly in terms of an index based on frequency of occurrence. An explainability device according to any one of claims 1 to 3, wherein the neighborhood calculation unit generates c feature sequences by varying the number of segments selected before the target segment and the number of segments selected after the target segment. A method for making an explanation possible for a model that has learned labels assigned to segments using d-dimensional learning data based on segments that constitute a document, the method comprising the steps of:
an overall confidence level calculation step of generating a d-dimensional feature vector by concatenating some feature amounts extracted from each of c randomly selected segments out of a plurality of segments constituting a target document, and inputting the d-dimensional feature vector into the model repeatedly, and calculating an overall confidence level of the target document for each label by acquiring a plurality of outputs of the model and performing statistical processing;
a neighborhood calculation step of randomly selecting c segments including a target segment from among a plurality of segments constituting the target document, and generating a plurality of d-dimensional feature sequences, each of which has a portion of feature quantities extracted from each of the selected segments as elements, as neighborhoods of the target segment;
a neighborhood first confidence calculation step of inputting the neighborhood feature sequence into the model and calculating a neighborhood first confidence for each label;
a neighborhood first importance calculation step of calculating a first importance of the neighborhood based on a product of the first certainty for each label and the overall certainty;
a neighborhood second confidence calculation step of converting a portion of the neighborhood feature sequence extracted from the target segment into a predetermined value, inputting the converted value into the model, and calculating a neighborhood second confidence for each label;
a neighborhood second importance calculation step of calculating a second importance of the neighborhood based on a product of the second certainty for each label and the overall certainty;
an explanation degree calculation step of calculating an explanation degree of the target segment based on a difference between the first importance and the second importance. An explanation enabling program for causing a computer to function as an explanation enabling device according to any one of claims 1 to 4.

Images