JP7621852B2 - Document information extraction system and document information extraction method - Google Patents

Description

The present invention relates to a document information extraction system and a document information extraction method.

Patent Document 1 describes an information processing device that selects a feature to be used for classifying input data from a plurality of feature values extracted from the input data. The information processing device generates a combination of a plurality of feature values extracted from the input data, calculates a first evaluation value for evaluating whether the generated combination is suitable for determining the classification of the input data, generates a plurality of parameters to be used when calculating a second evaluation value for evaluating the combination of feature values, calculates a second evaluation value based on the first evaluation value for each of the plurality of parameters, selects a feature based on the second evaluation value for each of the plurality of parameters, generates a subset of the feature values, and enables selection of the feature when selecting a combination of feature values.

Patent Document 2 describes a feature classification system that classifies features in speech recognition and image recognition. The feature classification system includes a feature converter that converts given features and learns discrimination criteria for feature conversion using correct class labels, which are training data, and a classifier that classifies the converted features converted by the feature converter. In order to take into account the error tendency of classifiers in other feature classification systems, the feature converter uses the posterior probability of classifiers in other feature classification systems during the learning process and performs feature conversion different from that of other feature converters.

JP 2017-10318 A International Publication No. 2016/021060

One method for extracting useful information from a huge amount of documents is to use machine learning mechanisms such as named entity extraction to extract words and related words from documents. However, to extract high-quality information using machine learning mechanisms, it is necessary to set the model's features appropriately. The features used to extract information from documents here are, for example, the properties of a sentence, such as the final word of a word or the words that appear to the left and right of a word. To set these features appropriately, not only is knowledge about the information to be extracted (domain knowledge), but knowledge and experience of machine learning are also required, and the high human burden involved in setting the features is an issue.

Patent Document 1 describes a technology for selecting features to be used in classifying input data. However, the technology described in this document is related to a technology for extracting a group of features, such as the average and variance of pixel values, from an image of an object to be inspected, and is not related to a technology for extracting useful information from a document. In addition, this document selects features from combinations of multiple features extracted from input data using a mechanical brute force method, and it is necessary to prepare a considerable amount of training data in order to achieve extraction accuracy.

Patent Document 2 describes a technique for classifying features in speech recognition and image recognition. However, this document only describes a technique for adjusting a classifier used to classify given features, and does not describe a technique for setting the features.

The present invention has been made in view of the above background, and aims to provide a document information extraction system and a document information extraction method that are equipped with a mechanism for supporting the setting of model features used when extracting useful information from documents using machine learning methods.

One aspect of the present invention for achieving the above object is a document information extraction system, comprising: a document management unit for managing a document group; a document information extraction unit for extracting information from documents of the document group using a machine learning model; a storage unit for storing a plurality of feature amounts constituting parameters of the model and an action type which is information indicating characteristics of an action when extracting the information for each of the feature amounts; a parameter setting reception unit for receiving a setting of the action type or the weight of the evaluation target feature amount while displaying an evaluation target feature amount which is a feature amount to be evaluated among the plurality of feature amounts, one or more sentences from which the evaluation target feature amount is extracted, and a weight currently set for the evaluation target feature amount; the parameter updating unit updating the parameters based on the action type or the weight assigned to each of the features to be evaluated; a feature display priority setting unit determining, for each of the features to be evaluated, a feature display priority which is a priority for displaying the feature to be evaluated based on at least one of the similarity between the feature to be evaluated and other features having the same action type in the document group, the co-occurrence with the other features in the document group, the weight currently set for the feature to be evaluated, and the frequency of occurrence of the feature to be evaluated in the document group; and a parameter setting receiving unit receiving the setting of the action type or the weight for the feature to be evaluated while displaying the feature to be evaluated in accordance with the feature display priority .

Other problems and solutions disclosed in this application will be made clear in the detailed description of the invention and the drawings.

The present invention can assist in setting the features of a model used when extracting useful information from documents using machine learning methods.

FIG. 1 is a system flow diagram illustrating a main configuration of a document information extraction system. FIG. 1 is a block diagram illustrating a main configuration of a document information extraction system. 11 is a diagram illustrating a method for evaluating the likelihood of a feature being positive for use in determining a priority level. FIG. 13 is an example of a feature amount evaluation table. 1 is an example of a feature amount frequency table. 1 is an example of a feature similarity table. 13 is an example of a feature co-occurrence table. 13 is an example of a display priority calculation table. 13 is an example of a feature amount display priority table. 13 is an example of a parameter setting screen. 13 is an example of a before-and-after update comparison screen. 13 is an example of a feature extraction source information table. This is an example of character decoration. 13 is an example of a mapping target feature table. 13 is an example of a feature amount evaluation setting information table. 13 is an example of a feature extraction status display table. 13 is a flowchart illustrating a feature amount evaluation setting information table generating process. 13 is a flowchart illustrating a feature extraction status display table generating process. 1 is an example of an information processing device used to realize a document information extraction system.

The following describes the embodiments with reference to the drawings. Note that the following description and drawings are examples for explaining the present invention, and some parts have been omitted or simplified as appropriate for clarity of explanation. The present invention can also be implemented in various other forms. Unless otherwise specified, each component may be singular or plural.

In the following description, the same or similar configurations may be assigned the same reference numerals, and duplicate descriptions may be omitted. In the following description, the letter "S" before a reference numeral denotes a processing step. In the following description, various types of information may be described using expressions such as "table" and "information", but the various types of information may be expressed using data structures other than these.

Hereinafter, as one embodiment of the present invention, an information processing system (hereinafter referred to as "document information extraction system 1") that uses a machine learning method to extract useful information from a huge number of documents (a group of documents) and provides it to a user will be described. The document information extraction system 1 is used for data-driven material development using so-called MI (Materials Informatics), which extracts information useful for planning guidelines for research and development of new materials, such as correlations between the structure and properties or functions of materials, from a huge group of documents (patent documents, various papers, business reports, corporate reports, news, etc.), and provides the extracted results to a user.

Figure 1 is a system flow diagram explaining the main components of the document information extraction system 1. Figure 2 is a block diagram explaining the main components of the document information extraction system 1. As shown in Figure 1, the document information extraction system 1 has the functions of a document information extraction unit 170 and a parameter setting unit 180. Of these, the document information extraction unit 170 extracts information from accumulated documents 111 using a machine learning model (learning model) and presents it to the user. In addition, the parameter setting unit 180 sets the parameters of the model (feature amount, weight of feature amount) in an interactive format with the user via a user interface.

As shown in FIG. 1 or 2, the document information extraction unit 170 includes a word/related word extraction unit 171 and a document information presentation unit 172. The word/related word extraction unit 171 refers to the initial dictionary 112 and the word/related word dictionary 113, and extracts information on words, related words of words, synonyms of words, etc. (hereinafter referred to as "words/related words 114") from the document 111 using a machine learning mechanism. The machine learning mechanism may be, for example, a named entity extraction technology realized using a recurrent neural network (RNN) or a long short-term memory (LSTM). However, the type of machine learning is not necessarily limited.

The above model is expressed, for example, by a matrix or vector including information on features and the weight of each feature. Features of the above model that extract information from document 111 include, for example, features of surrounding words (left and right words, parts of speech of left and right words, dependencies), features of the word itself (starting word (prefix), ending word (suffix), number of characters, uppercase and lowercase letters, parts of speech, type of character (numbers, kanji, etc.)), etc.

The document 111 is managed by the document management unit 160 shown in FIG. 2. The document 111 includes a large number of documents (for example, documents acquired via the Internet by web scraping or web crawling, and documents registered by a user). In this embodiment, a document is a collection of one or more sentences, one or more paragraphs, one or more clauses or phrases, two or more words, etc., described on a specific topic. In the following, an example will be described in which various information processing performed by the document information extraction system 1 is performed on a document basis, but the processing unit is not necessarily limited. In the following, the document 111 is managed as text data (text data), but the management method of the document 111 is not necessarily limited. The document that is the subject of the various information processing described below is, for example, appropriately decomposed into morphemes by morphological analysis technology, and each morpheme is appropriately converted into a distributed representation (word embedding) and handled.

The initial dictionary 112 and the word/related word dictionary 113 referred to by the document information extraction unit 170 are managed by the dictionary management unit 161 shown in FIG. 2. In the initial dictionary 112, for example, words indicating the directionality of the object to be extracted by the machine learning are set. For example, when information about temperature is extracted from the document 111 related to thermochemistry, words such as "30 degrees" and "10 degrees" are set in the initial dictionary 112. In this case, the document information extraction unit 170 extracts, for example, "30°C" and "10°C" as related words. The word/related word dictionary 113 manages information that associates words with related words and synonyms. The contents of the initial dictionary 112 and the word/related word dictionary 113 may be set by the user, for example, or may be set using a corpus, dictionary, etc. obtained from the Internet, etc. The user prepares the initial dictionary 112 and the word/related word dictionary 113 with contents corresponding to, for example, the technical field from which useful information is to be obtained and the type of information to be extracted.

The document information presenting unit 172 provides the user with the words/related words 114 extracted by the word/related word extraction unit 171 and information generated based on the extracted words/related words 114 (hereinafter referred to as "document extraction information 115"). The user uses the document extraction information 115 provided by the document information presenting unit 172, for example, via a user interface provided in the document information extraction system 1 or another information processing device communicatively connected to the document information extraction system 1.

The parameter setting unit 180 has the functions of a feature information generation unit 181, a feature display priority setting unit 182, a parameter setting screen generation unit 183, a parameter setting reception unit 184, and a parameter update unit 185.

Of these, the feature information generation unit 181 acquires the parameters of the current model (features and weights of each feature) (hereinafter referred to as "current parameters 155") from the word/related word extraction unit 171, and generates feature information 120 (feature evaluation table 121, feature frequency table 122, feature similarity table 123, feature co-occurrence table 124, and feature extraction source information table 125), which is information referenced by the feature display priority setting unit 182 and the like, based on the acquired current parameters 155. Details of the feature information 120 will be described later.

The feature amount display priority setting unit 182 generates a display priority calculation table 131 used to calculate the display priority of features (hereinafter referred to as "feature amount display priority") when setting model parameters through interactive processing with the user, based on the feature amount information 120 generated by the feature amount information generating unit 181. The feature amount display priority setting unit 182 determines the feature amount display priority based on the display priority calculation table 131, and stores the determined feature amount display priority in the feature amount display priority table 132.

The parameter setting screen generating unit 183 displays features in an order according to the feature display priority, and generates a screen (parameter setting screen 1000 shown in FIG. 10) that accepts information related to the setting of parameters of the displayed features from the user. When generating the parameter setting screen 1000, the parameter setting screen generating unit 183 refers to or generates the parameter setting screen information 140 (character decoration definition table 141, mapping target feature table 142, feature evaluation setting information table 143, and feature extraction status display table 144). Details of the parameter setting screen information 140 will be described later.

The parameter setting reception unit 184 receives information regarding parameter settings from the user while displaying the parameter setting screen 1000.

The parameter update unit 185 updates the feature amount evaluation table 121 based on information received from the user by the parameter setting reception unit 184 (the storage unit 110 stores the contents of both the feature amount evaluation table 121 before and after the update). The parameter update unit 185 also stores information on the difference between the feature amount evaluation table 121 before and after the update (hereinafter, “parameter update information 152
"), and updates the parameters of the model used by the word/related word extraction unit 171 based on the generated parameter update information 152.

Next, a method of calculating feature display priorities performed by the feature display priority setting unit 182 will be described. The feature display priority setting unit 182 sets feature display priorities so that feature amounts that the user is to check with priority (feature amounts that are considered to have a large effect on the accuracy of extraction of useful information from the document 111) are preferentially displayed on the parameter setting screen 1000. Specifically, the feature display priority setting unit 182 sets a high feature display priority for feature amounts that fall under any of the following (1) to (3). Note that, hereinafter, a feature amount that acts to extract information that should be extracted from the document 111 is referred to as a "positive feature amount", a feature amount that acts to prevent unuseful information from being extracted from the document 111 is referred to as a "negative feature amount", and a feature amount that does not fall under either the "positive feature amount" or the "negative feature amount" is referred to as a "neutral feature amount".
(1) A positive feature, in which the weight of the feature is set to near zero or a negative value in the current parameters 155 (hereinafter, referred to as an “erroneously evaluated positive feature”).
(2) A negative feature is a feature whose weight is set large (near zero or positive, etc.) in the current parameters 155 (hereinafter, referred to as an “erroneously evaluated negative feature”).
(3) A neutral feature is a feature whose absolute value is set to be large in the current parameters 155 (hereinafter, referred to as a “mis-evaluated neutral feature”).

For example, the feature display priority setting unit 182 evaluates an index (hereinafter referred to as "likelihood of correctness") indicating the possibility that a feature falls under the category of (1) mis-evaluated correct feature, based on the similarity between the feature to be evaluated (hereinafter referred to as "the feature") and other correct features used in the current model (hereinafter referred to as "comparison correct features"), the degree of co-occurrence between the feature and the comparison correct features, the weight of the feature in the current parameters 155, and the frequency of occurrence of the feature in the document 111.

For example, the feature display priority setting unit 182 evaluates an index indicating the possibility of falling under (2) above (hereinafter referred to as "negative likelihood") based on the similarity between the feature to be evaluated (hereinafter referred to as "the feature") and other negative features used in the current model (hereinafter referred to as "comparative negative features"), the co-occurrence between the feature and the comparative negative features, the weight of the feature in the current parameters 155, and the frequency of occurrence of the feature in the document 111.

For example, the feature display priority setting unit 182 evaluates an index (hereinafter referred to as "neutrality") indicating the possibility that the feature falls under (3) above, based on the similarity between the feature to be evaluated (hereinafter referred to as "the feature") and other neutral features used in the current model (hereinafter referred to as "comparison neutral features"), the co-occurrence of the feature and the comparison neutral features, the absolute value of the weight of the feature in the current parameters 155, and the frequency of occurrence of the feature in the document 111.

In addition, in both of the above cases, the frequency of occurrence in document 111 of the feature to be evaluated is used for the evaluation because a feature that occurs less frequently in document 111 is considered to have less impact on the accuracy of information extraction from document 111.

3 is a diagram showing an example of the concept of evaluating applicability to (1) above. The diagram illustrates an example of a case where the "likelihood of positivity" of a feature for extracting a word representing "temperature" from an example document 111 is calculated. As shown in the diagram, "30 degrees", "10 degrees", etc. are set in the initial dictionary 112 to allow a model to learn words representing temperature, and "Final word: degree" (a word ending with "degree") and "Left word: temperature" (a word with the character string "temperature" on the left side) are included as comparative positive features.

The "Final word: kai" (a word ending in kai), "Final word: ℃" (a word ending in ℃), and "Left word: kajishi" (a word with the character string "kajishi" on the left side) shown in the figure are all feature quantities to be evaluated for applicability to (1) above. Note that in this example, it is assumed that each feature quantity to be evaluated has a sufficiently high frequency of occurrence in document 111.

In this example, the feature to be evaluated, "final word: times", is similar only to the comparative positive feature "final word: degree", and the feature to be evaluated, "left word: decreases", is co-occurring only with the comparative positive feature "left word: temperature is". In addition, the feature to be evaluated, "final word: °C", is similar to the comparative positive feature "final word: degree", and also co-occurs with the comparative positive feature "left word: temperature is". For this reason, in this example, the feature "final word: °C" is evaluated as more likely to fall under (1) above, i.e., more "likely to be positive", than the other two features ("final word: times", "left word: decreases").

The feature display priority setting unit 182 determines the feature display priority to be used when presenting to the user based on the "likelihood of positiveness," "likelihood of negativeness," and "likelihood of neutrality" evaluated for each feature as described above. For example, the feature display priority setting unit 182 sets a high feature display priority to a feature having a high "likelihood of positiveness" (exceeding a preset threshold). For another example, the feature display priority setting unit 182 sets a high feature display priority to a feature having a high "likelihood of negativeness" (exceeding a preset threshold). For another example, the feature display priority setting unit 182 sets a high feature display priority to a feature having a high "likelihood of neutrality" (exceeding a preset threshold). Note that the feature display priority setting unit 182 may set the feature display priority of each feature by comprehensively evaluating, for example, two or more of "likelihood of positiveness," "likelihood of negativeness," and "likelihood of neutrality."

Next, a method for calculating the feature display priority will be described with reference to specific tables. When calculating the feature display priority, the feature display priority setting unit 182 refers to or generates (including the meaning of updating) the feature information 120 (feature evaluation table 121, feature frequency table 122, feature similarity table 123, feature co-occurrence table 124).

Figure 4 shows an example of the feature evaluation table 121. The feature evaluation table 121 manages information related to the weight and evaluation of each feature of the model, either currently or after it has been changed by the user. As shown in the figure, the feature evaluation table 121 is made up of multiple entries (records) each having the items of feature 1211, weight 1212, and evaluation 1213. One entry in the feature evaluation table 121 corresponds to one feature.

Of the above items, feature amount 1211 stores a feature amount. Weight 1212 stores the weight of the feature amount. Evaluation 1213 sets information indicating the type (hereinafter referred to as the "action type") according to the action characteristics of the feature amount ("positive" for a positive feature amount, "negative" for a negative feature amount, and "neutral" for a neutral feature amount). For features for which the action type has not been set, "-" is set in evaluation 1213.

5 shows an example of the feature frequency table 122. In the feature frequency table 122, the frequency of occurrence in the document 111 (document group) of each feature used in the current model is managed. As shown in the figure, the feature frequency table 122 is made up of a plurality of entries (records) having items of feature 1221 and frequency 1222. One entry in the feature frequency table 122 corresponds to one feature. Of the above items, the feature 1221 stores the feature. The frequency 1222 stores the frequency of occurrence of the feature in the document 111.

Figure 6 shows an example of feature similarity table 123. In feature similarity table 123, similarities between different features that make up a model are managed. As shown in the figure, feature similarity table 123 is made up of multiple entries (records) each having a first feature 1231, a second feature 1232, and a similarity 1233. One entry in feature similarity table 123 corresponds to one set of features.

Of the above items, the first feature 1231 stores one of the above pairs of features (hereinafter referred to as the "first feature"). The second feature 1232 stores the other of the above pairs of features (hereinafter referred to as the "second feature"). The similarity 1233 stores the similarity between the first feature and the second feature. The feature information generation unit 181 calculates the similarity based on, for example, cosine similarity or Levenshtein distance.

Figure 7 shows an example of the feature co-occurrence table 124. The feature co-occurrence table 124 manages the co-occurrence of different features that make up a model. As shown in the figure, the feature co-occurrence table 124 is made up of multiple entries (records) each having a first feature 1241, a second feature 1242, and a co-occurrence 1243. One entry in the feature co-occurrence table 124 corresponds to one set of features.

Of the above items, the first feature 1241 stores one feature of the above pair (hereinafter referred to as the "first feature"). The second feature 1242 stores the other feature of the above pair (hereinafter referred to as the "second feature"). The co-occurrence 1243 stores the co-occurrence of the first feature and the second feature. Note that the feature information generation unit 181 generates word embeddings (e.g., embeddings generated by word2vec, BERT, etc.) for words, for example, and calculates the co-occurrence of two words using the generated embeddings of each word.

Figure 8 shows an example of the display priority calculation table 131. The display priority calculation table 131 manages information based on the contents of the feature evaluation table 121, feature frequency table 122, feature similarity table 123, and feature co-occurrence table 124. As shown in the figure, the display priority calculation table 131 is composed of multiple entries (records) having each of the items of a first feature 1311, a second feature 1312, a similarity 1313, a co-occurrence 1314, a first feature weight 1315, a second feature weight 1316, and a second feature frequency 1317. One entry in the display priority calculation table 131 corresponds to one set of features.

Of the above items, the first feature 1311 stores one of the above pairs of features (hereinafter referred to as the "first feature"). The second feature 1312 stores the other of the above pairs of features (hereinafter referred to as the "second feature"). The similarity 1313 stores the similarity between the first feature and the second feature. The co-occurrence 1314 stores the co-occurrence between the first feature and the second feature. The first feature weight 1315 stores the weight of the first feature. The second feature weight 1316 stores the weight of the second feature. The second feature frequency 1317 stores the frequency of occurrence of the second feature in the document 111.

9 shows an example of the feature amount display priority table 132. The feature amount display priority table 132 manages information used to set the feature amount display priority of each feature amount, which is generated by the feature amount display priority setting unit 182 based on the display priority calculation table 131. As shown in the figure, the feature amount display priority table 132 is made up of a plurality of entries (records) having the items of feature amount 1321, positive likelihood 1322, negative likelihood 1323, and neutral likelihood 1324. One entry in the feature amount display priority table 132 corresponds to one feature amount.

Of the above items, feature amount 1321 stores the feature amount.

The likelihood of positiveness 1322 stores a value (likelihood of positiveness) calculated from the following formula based on the display priority calculation table 131. In the following formula, the feature amount i corresponds to the second feature amount 1312 in the display priority calculation table 131, and the feature amount j corresponds to the first feature amount 1311 in the same table. In the following formula, a, b, c, and d are constants.

The negative likelihood 1322 stores a value (negative likelihood) calculated from the following formula based on the display priority calculation table 131. In the following formula, the feature amount i corresponds to the second feature amount 1312 in the display priority calculation table 131, and the feature amount j corresponds to the first feature amount 1311 in the same table. In the following formula, a, b, c, and d are constants.

The neutrality likelihood 1323 stores a value (neutrality likelihood) calculated from the following formula based on the display priority calculation table 131. In the following formula, the feature amount i corresponds to the second feature amount 1312 in the display priority calculation table 131, and the feature amount j corresponds to the first feature amount 1311 in the same table. In the following formula, a, b, c, and d are constants.

As described above, when the parameter setting screen generating unit 183 sets parameters interactively with the user, it determines the feature display priority based on the positive likelihood 1322, negative likelihood 1323, and neutral likelihood 1324 determined as described above.

As described above, the document information extraction system 1 determines at least one of the positive, negative, and neutral likelihoods for each feature, and sets the feature display priority based on the determined value, so that it can preferentially present to the user features that can be effectively obtained by adjustment (features that are likely to extract more appropriate words or related words from document 111), allowing the user to adjust (tune) parameters efficiently.

Next, we will explain how the document information extraction system 1 sets parameters through interactive processing with the user. For simplicity, we will use an example of extracting words from document 111.

Figure 10 is an example of a screen (hereinafter referred to as "parameter setting screen 1000") that the parameter setting screen generating unit 183 displays when setting parameters through interactive processing with the user. As shown in the figure, the parameter setting screen 1000 has a feature evaluation setting field 1010, a feature extraction status display field 1020, a before-and-after update comparison button 1030, and a confirm button 1040.

In the feature evaluation setting field 1010, features that the user is prompted to confirm are displayed in order of highest feature display priority in the feature display priority table 132. As shown in the figure, the feature evaluation setting field 1010 is made up of multiple entries having the following items: feature 1011, weight 1012, text 1013, and evaluation 1014. One entry in the feature evaluation setting field 1010 corresponds to one feature.

Of the above items, feature 1011 displays the feature that the user is prompted to confirm. Weight 1012 displays the weight set for that feature. The user can evaluate that feature while checking the weight set for that feature. The user can also manually change the value of weight 1012 using this screen.

Sentence 1013 displays one or more sentences (sentences that are elements of document 111) from which words based on the feature have been extracted. The parts of sentence 1013 that correspond to the feature are highlighted (underlined and italicized in this example), allowing the user to easily grasp the position of the feature in the sentence.

In the evaluation 1014, the action type set for the feature is displayed. For features for which an action type has been set in the past, the feature already set in the evaluation 1014 is automatically displayed as a default value. For features for which an action type has not been set in the past, for example, the action type determined by the parameter setting unit 180 based on the contents of the feature display priority table 132 is displayed as a default value in the evaluation 1014. For example, for features for which the positive likelihood 1322 exceeds a preset threshold, "positive" is displayed as a default value in the evaluation 1014, for features for which the negative likelihood 1323 exceeds a preset threshold, "negative" is displayed as a default value in the evaluation 1014, and for features for which the neutral likelihood 1324 exceeds a preset threshold, "neutral" is displayed as a default value. Note that the method of automatically determining the default value is not necessarily limited. The content of the evaluation 1014 can also be manually changed by the user using the screen. For example, the user changes the content of the evaluation 1014 by operating the action type ("positive", "negative", "neutral") displayed as a pull-down menu.

The feature extraction status display field 1020 displays information showing the status (state) of extraction of each feature displayed in the feature evaluation setting field 1010. As shown in the figure, the feature extraction status display field 1020 is composed of multiple entries having the following items: document ID 1021, extracted words 1022, source text 1023, and unmapped features 1024. One entry in the feature extraction status display field 1020 corresponds to one word extracted from document 111.

Of the above items, document ID 1021 displays the document ID (in this example, the file name of the document from which the extraction was made) that is the identifier of the document from which the extraction was made. Extracted words 1022 displays the words extracted based on the features 1011 displayed in the feature evaluation setting field 1010.

The source sentence 1023 displays the sentence from which the word was extracted. As shown in the figure, the parts of the source sentence 1023 that correspond to the features are decorated with character decoration (solid underline, frame, italics, dashed underline, no decoration) according to the weight of the feature. The features to which character decoration is applied in the source sentence 1023 are the features defined in a mapping target feature table 142, which will be described later. In this example, the weights (effects) set for each feature are differentiated by changing the type of character decoration, but other methods such as color coding may also be used to differentiate.

In this example, among the character decorations, a solid underline is applied to features that have a large effect on extracting words that indicate temperature (positive features), a frame is applied to features that have a large effect on extracting words that indicate temperature (positive features), italics are applied to features that have a large effect on not extracting words that are not temperatures (negative features), dashed underlines are applied to features that have a large effect on not extracting words that are not temperatures (negative features), and no decoration is applied to features that are neither positive nor negative features (neutral features).

The non-mapping feature 1024 displays other features that contributed to the extraction of the word (in this example, the number of characters). Examples of other features displayed in the non-mapping feature 1024 include, for example, "number or letter," "English or Japanese," "Roman alphabet or kanji," and "uppercase or lowercase."

When the user operates the before-and-after comparison button 1030, the parameter setting unit 180 displays a screen (hereinafter referred to as the "before-and-after comparison screen 1100") that shows a comparison between the contents of the feature evaluation setting field 1010 before the user performs the change operation and the contents of the feature evaluation setting field 1010 after the user performs the change operation.

FIG. 11 shows an example of a before-and-after comparison screen 1100. As shown in the figure, the illustrated before-and-after comparison screen 1100 has a feature amount evaluation setting field (before update) 1111 in which the contents of the feature amount evaluation setting field 1010 before the update are displayed, and a feature amount evaluation setting field (after update) 1112 in which the contents of the feature amount evaluation setting field 1010 before the update are displayed. By referring to the before-and-after comparison screen 1100, the user can easily confirm how he or she changed the evaluation and weight of each feature amount. When the user changes the contents of the feature amount evaluation 1014, the feature amount display priority setting unit 182 recalculates the feature amount display priority based on the evaluation (action type) and weight after the update, and the display order of the feature amounts is also updated.

Returning to FIG. 10, when the user operates the confirm button 1040, the parameter update unit 185 generates parameter update information 152 based on the settings on the parameter setting screen 1000 (weight 1012, evaluation 1014), and updates the parameters of the model used by the word/related word extraction unit 171 based on the generated parameter update information 152. The parameter update unit 185 may automatically set the weight of the corresponding feature based on the evaluation 1014 on the parameter setting screen 1000. In this case, the weight automatically set for each feature is set based on, for example, the positive likelihood 1322, negative likelihood 1323, and neutral likelihood 1324 in the feature display priority table 132.

As described above, in the feature amount evaluation setting field 1010 of the parameter setting screen 1000, the feature amount 1011 and the original sentence 1013 from which the feature amount is extracted are displayed, so that the user can set an appropriate evaluation 1014 for the feature amount. In addition, the feature amount extraction status display field 1020 displays information on the extraction status of the feature amount displayed in the feature amount evaluation setting field 1010 (extracted words 1022, original sentence 1023, non-mapping feature amount 1024), so that the user can visually confirm how the feature amount works when extracting words. Even a user with little knowledge or experience of machine learning can efficiently and appropriately set the feature amount by referring to this information. Therefore, for example, it is possible to effectively utilize the abundant knowledge and experience of a domain user who is inexperienced in machine learning but is familiar with the target information field.

Next, examples of various tables used to generate the parameter setting screen 1000 and the pre- and post-update comparison screen 1100 are shown.

Figure 12 is an example of the feature extraction source information table 125. The feature extraction source information table 125 manages information related to the source from which features are extracted. As shown in the figure, the feature extraction source information table 125 is made up of multiple entries (records) having the following items: document ID 1251, word 1252, source sentence 1253, word start point 1254, word end point 1255, feature 1256, feature start point 1257, and feature end point 1258. One entry in the feature extraction source information table 125 corresponds to one feature.

Of the above items, document ID 1251 stores the document ID of the document from which the feature was extracted. Word 1252 stores the word extracted by the feature. Source sentence 1253 stores the sentence from which the word was extracted. Word start point 1254 and word end point 1255 store the position (start point, end point) of the word in the document. Feature 1256 stores the feature. Feature start point 1257 and feature end point 1258 store the position (start point, end point) at which the feature appears in the source sentence.

Figure 13 is an example of a character decoration definition table 141. The character decoration definition table 141 manages information indicating character decoration according to the weight of a feature. The contents of the character decoration definition table 141 are set, for example, by a system administrator or a user. As shown in the figure, the character decoration definition table 141 is made up of multiple records having items of weight 1411 and character decoration 1412. Information indicating the range of weights is stored in weight 1411. Furthermore, information indicating the type of character decoration is stored in character decoration 1412.

Figure 14 is an example of a mapping target feature table 142. The mapping target feature table 142 manages features (target features 1421) to which character enhancement is to be applied, among the features displayed in the source text 1023 and non-mapping features 1024 in the feature extraction status display field 1020 of the parameter setting screen 1000. The contents of the mapping target feature table 142 are set, for example, by a system administrator or a user.

Figure 15 is an example of the feature amount evaluation setting information table 143. The feature amount evaluation setting information table 143 manages information to be displayed in the feature amount evaluation setting field 1010 of the parameter setting screen 1000. As shown in the figure, the feature amount evaluation setting information table 143 is made up of multiple entries (records) each having the following items: feature amount 1431, weight 1432, source sentence 1433, feature amount start point 1434, feature amount end point 1435, and evaluation 1436. One entry in the feature amount evaluation setting information table 143 corresponds to one feature amount.

Of the above items, feature 1431 stores the feature to be evaluated. Weight 1432 stores the weight of the feature. Source sentence 1433 stores the sentence from which the feature was extracted. Feature start point 1434 and feature end point 1435 each store information indicating the position of the feature in source sentence 1433 (start point, end point). Evaluation 1436 is set to the default value mentioned above or an evaluation set by the user (either "positive", "negative", or "neutral").

16 is an example of the feature extraction status display table 144. The feature extraction status display table 144 manages information to be displayed in the feature extraction status display field 1020 of the parameter setting screen 1000. As shown in the figure, the feature extraction status display table 144 manages information to be displayed in the feature extraction status display field 1020 of the parameter setting screen 1000.
The feature extraction status display table 144 is made up of a plurality of records having the following items: D 1441, word 1442, source sentence 1443, word start point 1444, word end point 1445, feature 1446, feature start point 1447, feature end point 1448, weight 1449, character decoration 1450, and unmapped feature 1451. One entry in the feature extraction status display table 144 corresponds to one word extracted from the document 111.

Document ID 1441 stores the document ID of document 111 from which the word (word 1442) was extracted. Word 1442 stores the word extracted using the feature (feature 1446). Source sentence 1443 stores the sentence from which the word was extracted. Word start point 1444 and word end point 1445 store the position (start point, end point) of the word in source sentence 1443. Feature 1446 stores the feature used to extract the word. Feature start point 1447 and feature end point 1448 each store information (start point, end point) indicating the position of the feature in source sentence 1443.

Weight 1449 stores the weight set for the feature (feature 1446). Character decoration 1450 stores information indicating the character decoration to be applied when displaying the feature. Non-mapping feature 1451 stores other features that contributed to the extraction of the word and are not defined in the mapping target feature table 142.

Next, we will explain the processing performed by the parameter setting unit 180 when generating the feature evaluation setting information table 143 and the feature extraction status display table 144.

Figure 17 is a flowchart explaining the process performed by the parameter setting unit 180 when generating the feature evaluation setting information table 143 (hereinafter referred to as "feature evaluation setting information table generation process S1700"). Below, the feature evaluation setting information table generation process S1700 will be explained with reference to this figure.

First, the parameter setting unit 180 reads the feature extraction source information table 125 and the feature evaluation table 121 (S1711 to S1712).

Next, the parameter setting unit 180 selects one entry from the feature evaluation table 121 (S1713).

Next, the parameter setting unit 180 adds an entry to the feature evaluation setting information table, in which the content associated with the source text 1253 in the feature extraction source information table 125 corresponding to the feature of the entry selected in S1713 is set (S1714).

The parameter setting unit 180 then determines whether or not all entries in the feature evaluation table S1713 have been selected in S1713 (S1715). If all entries have not been selected (S1715: NO), the process returns to S1713, where an unselected entry is selected and the same process as above is performed. On the other hand, if all entries have been selected (S1715: YES), the feature evaluation setting information table generation process S1700 ends.

Figure 18 is a flowchart explaining the process performed by the parameter setting unit 180 when generating the feature extraction status display table 144 (hereinafter referred to as "feature extraction status display table generation process S1800"). Below, the feature extraction status display table generation process S1800 will be explained with reference to this figure.

First, the parameter setting unit 180 extracts the feature extraction source information table 125, the feature evaluation table 121, the character decoration definition table 141, and the mapping target feature table 142.
is read (S1811 to S1814).

Next, the parameter setting unit 180 selects one entry from the feature extraction source information table 125 and adds one entry to the feature extraction status display table 144 using the contents of the selected entry (S1815).

Next, the parameter setting unit 180 stores the weight of the corresponding feature in the feature evaluation table 121 in the weight 1449 of the added entry (S1816).

Next, the parameter setting unit 180 stores in the character decoration 1450 of the added entry a character decoration corresponding to the weight stored in S1816 in the character decoration definition table 141 (S1817).

Then, the parameter setting unit 180 determines whether the mapping target feature table 142 contains a feature that is the same as the feature 1446 of the entry added in S1815 (S1818). If the same feature is present (S1818: YES), the parameter setting unit 180 sets the non-mapping feature 1451 of the added entry to blank, and then the process proceeds to S1820. If the same feature is not present (S1818: NO), the parameter setting unit 180 stores the feature in the non-mapping feature 1451 of the added entry (S1819), and then the process proceeds to S1820.

The parameter setting unit 180 then determines whether or not all entries in the feature extraction source information table 125 have been selected in S1815 (S1820). If all entries have not been selected (S1820: NO), the process returns to S1815, where an unselected entry is selected and the same process as above is performed. On the other hand, if all entries have been selected (S1815: YES), the feature extraction status display table generation process S1800 ends.

Next, we will explain the information processing device (computer) used to configure the document information extraction system 1 described above.

Figure 19 shows an example of an information processing device used in the configuration of the document information extraction system 1. As shown in the figure, the illustrated information processing device 10 includes a processor 11, a main memory device 12, an auxiliary memory device 13, an input device 14, an output device 15, and a communication device 16.

The document information extraction system 1 may be realized, for example, by using a plurality of information processing devices 10 connected to each other so as to be able to communicate with each other. In addition, all or a part of the information processing device 10 may be realized by using virtual information processing resources provided by using a virtualization technology, a process space separation technology, or the like, such as a virtual server provided by a cloud system. In addition, all or a part of the functions provided by the information processing device 10 may be realized by a service provided by the cloud system via an API (Application Programming Interface), for example.

Furthermore, all or part of the functions provided by the information processing device 10 may be realized using, for example, SaaS (Software as a Service), PaaS (Platform as a Service), IaaS (Infrastructure as a Service), etc.

The processor 11 shown in FIG. 1 is, for example, a CPU (Central Processing Unit), an MPU (Multi-Processing Unit),
(Micro Processing Unit), GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit),
It is constructed using AI (Artificial Intelligence) chips, etc.

The main memory device 12 is a device for storing programs and data, and is, for example, a ROM (Read Only Memory).
Only Memory), RAM (Random Access Memory), NVRAM (Non Volatile RAM), etc.

The auxiliary storage device 13 is, for example, a solid state drive (SSD), a hard disk drive, an optical storage device (CD (Compact Disc), DVD (Digital Versatile Disc), etc.), a storage system, an IC card, a read/write device for a recording medium such as an SD card or an optical recording medium, a storage area of a cloud server, etc. Programs and data can be read into the auxiliary storage device 13 via a recording medium reader or a communication device 16. The programs and data stored (memorized) in the auxiliary storage device 13 are read into the main storage device 12 as needed.

The input device 14 is an interface that accepts input from the outside, and is, for example, a keyboard, a mouse, a touch panel, a card reader, a pen-input tablet, a voice input device, etc.

The output device 15 is an interface that outputs various information such as the progress of processing and the results of processing. The output device 15 is, for example, a display device (liquid crystal monitor, LCD (Liquid Crystal Display), graphic card, etc.) that visualizes the various information described above, a device that converts the various information described above into audio (audio output device (speaker, etc.)), or a device that converts the various information described above into text (printer, etc.). Note that, for example, the information processing device 10 may be configured to input and output information to and from other devices via the communication device 16.

The input device 14 and the output device 15 constitute a user interface that realizes interactive processing with the user (accepting information, presenting information, etc.).

The communication device 16 is a device that realizes communication with other devices. The communication device 16 is a wired or wireless communication interface that realizes communication with other devices via a communication network 5 (such as a local area network (LAN), a wide area network (WAN), the Internet, a public communication network, or a dedicated line), and is, for example, a network interface card (NIC), a wireless communication module, or a USB module.

For example, an operating system, a file system, a DBMS (DataBase Management System) (relational database, NoSQL, etc.), a KVS (Key-Value Store), etc. may be introduced into the information processing device 10. The document information extraction system 1 stores the various types of information (data) described above, for example, as a table of a database or a file managed by a file system.

The various functions provided by the document information extraction system 1 are realized by the processor 11 reading and executing a program stored in the main memory device 12, or by the hardware (FPGA, ASIC, AI chip, etc.) that constitutes the document information extraction system 1. In addition, the various functions of the document information extraction system 1 are realized using, for example, various publicly known data mining methods such as text data mining, various publicly known natural language processing methods (morphological analysis, syntactic analysis, semantic analysis, context analysis, feature extraction, word distributed representation, named entity extraction, text classification, sequence labeling), and various publicly known machine learning methods (deep learning (DNN (Deep Neural Network), RNN (Recurrent Neural Network), etc.)).

As described above in detail, when setting parameters (feature quantities, weights), the document information extraction system 1 of this embodiment accepts the setting of an action type or weight while displaying the feature quantities to be evaluated, the one or more sentences from which the feature quantities to be evaluated are extracted, and the weights set for the feature quantities to be evaluated. This allows the user to efficiently and appropriately set the feature quantities to be evaluated while checking the original sentences and weights (while checking what settings have been made to what feature quantities).

In addition, the portion of the feature to be evaluated is highlighted in the text 1013 in the feature evaluation setting field 1010 of the parameter setting screen 1000, so the user can easily check how the feature to be evaluated affects the document.

In addition, in the source text 1023 of the feature extraction status display field 1020 of the parameter setting screen 1000, the part of the text to be evaluated that is displayed is highlighted with character decoration, colors, etc. that distinguish between different action types, so that the user can efficiently and appropriately set the feature to be evaluated while recognizing the action characteristics of the feature to be evaluated for each action type.

In addition, the feature extraction status display field 1020 on the parameter setting screen 1000 displays the words extracted from the document, allowing the user to efficiently and appropriately set the features to be evaluated while understanding how the features work and what words are extracted.

In addition, the feature extraction status display field 1020 on the parameter setting screen 1000 also displays other features (non-mapping features 1024) that influenced the extraction of information from the document using the feature to be evaluated, so the user can efficiently and appropriately set the feature to be evaluated while also referring to the other features.

In this way, the document information extraction system 1 of this embodiment can assist in setting the features of a model used when extracting useful information from a document using a machine learning method. Therefore, for example, even a person who has knowledge (domain knowledge) about the field of information to be extracted but has no knowledge or experience of machine learning can efficiently and appropriately set the features, and the knowledge of the domain user can be effectively utilized to appropriately set the features.

Although one embodiment of the present invention has been described above, it goes without saying that the present invention is not limited to the above embodiment and can be modified in various ways without departing from the spirit of the invention. For example, the above embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to having all of the configurations described. In addition, it is possible to add, delete, or replace part of the configuration of the above embodiment with other configurations.

For example, in the above, various types of processing are performed on a document basis, but the processing unit is not necessarily limited, and the document information extraction system 1 of this embodiment can perform similar processing on a sentence (a sentence consisting of one or more sentences) basis instead of a document.

In the above, the document information extraction unit 170 extracts information from the accumulated documents 111 using a machine learning model (learning model) and presents the extracted information to a system administrator or a user. However, for example, information may be extracted using extraction rules defined by the system administrator or user and presented to the system administrator or user. In that case, for example, in the parameter setting screen 1000 shown in FIG. 10, the non-mapping feature 1024 may display information indicating whether or not the rule is matched.

In addition, the above-mentioned configurations, functional units, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as the program, table, file, etc. that realizes each function can be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive),
It can be placed on a recording medium such as an IC card, an SD card, or a DVD.

Furthermore, the layout of the various functional units, various processing units, and various databases of each information processing device described above is merely an example. The layout of the various functional units, various processing units, and various databases can be changed to an optimal layout in terms of the performance, processing efficiency, communication efficiency, etc. of the hardware and software equipped in these devices.

In addition, the configuration (schema, etc.) of the database that stores the various types of data mentioned above can be flexibly changed from the perspective of efficient use of resources, improved processing efficiency, improved access efficiency, improved search efficiency, etc.

1 Document information extraction system, 110 Storage unit, 111 Document, 112 Initial dictionary, 113
Word/related word dictionary, 114 Word/related word, 115 Document extraction information, 120 Feature information, 121 Feature evaluation table, 122 Feature frequency table, 123 Feature similarity table, 124 Feature co-occurrence table, 125 Feature extraction source information table, 131
Display priority calculation table, 132 feature amount display priority table, 140 parameter setting screen information, 141 character decoration definition table, 142 mapping target feature amount table, 143 feature amount evaluation setting information table, 144 feature amount extraction status display table, 152 parameter update information, 155 current parameters, 160 document management unit, 161 dictionary management unit, 170 document information extraction unit, 171 word/related word extraction unit, 172 document information presentation unit, 180 parameter setting unit, 181 feature amount information generation unit, 182 feature amount display priority setting unit, 183 parameter setting screen generation unit, 184 parameter setting reception unit, 185 parameter update unit, 1000 parameter setting screen, 1100 before-and-after update comparison screen, S1700 feature amount evaluation setting information table generation process, S1800 feature amount extraction status display table generation process

Claims (13)

The information processing device is configured to:
a document management unit for managing a group of documents;
a document information extraction unit that extracts information from documents of the document set using a machine learning model;
A storage unit that stores a plurality of feature amounts constituting parameters of the model and an action type that is information indicating the characteristics of an action when extracting the information for each of the feature amounts;
a parameter setting receiving unit that receives a setting of the action type or the weight of the evaluation target feature while displaying an evaluation target feature which is a feature to be evaluated among the plurality of feature amounts, one or more sentences from which the evaluation target feature is extracted, and a weight currently set for the evaluation target feature;
a parameter update unit that updates the parameters based on the received action type or weight;
a feature display priority setting unit that determines, for each of the evaluation target features, a feature display priority that is a priority for displaying the evaluation target feature based on at least one of a similarity between the evaluation target feature and other feature features having the same action type in the document set, a co-occurrence with the other feature features in the document set, a weight currently set for the evaluation target feature, and an appearance frequency of the evaluation target feature in the document set;
a parameter setting receiving unit that receives a setting of the action type or the weight of the evaluation target feature quantity while displaying the evaluation target feature quantity in accordance with the feature quantity display priority;
A document information extraction system comprising: The document information extraction system according to claim 1,
The action type includes at least one of a positive type indicating that the feature quantity acts to extract information that should be extracted from the document group, a negative type indicating that the feature quantity acts to prevent information that should not be extracted from the document group from being extracted, and a neutral type indicating that the action of the feature quantity does not fall into either the positive type or the negative type.
Document information extraction system. The document information extraction system according to claim 1,
the parameter setting receiving unit highlights a portion of the evaluation target feature quantity in the text to be displayed.
Document information extraction system. The document information extraction system according to claim 3,
the parameter setting receiving unit highlights the portion of the evaluation target feature in the sentence to be displayed, distinguishing the portion for each of the action types of the evaluation target feature.
Document information extraction system. The document information extraction system according to claim 4,
the parameter setting receiving unit highlights the portion of the evaluation target feature in the text to be displayed by applying different character decoration or different colors for each of the action types of the evaluation target feature.
Document information extraction system. The document information extraction system according to claim 1,
the parameter setting receiving unit further displays other feature quantities that have influenced the extraction of information extracted from the document by the evaluation target feature quantity.
Document information extraction system. The document information extraction system according to claim 1,
the parameter setting receiving unit further displays information extracted from the document based on the evaluation target feature.
Document information extraction system. The document information extraction system according to claim 1 ,
The action type includes at least one of a positive type indicating that the feature quantity acts to extract information that should be extracted from the document group, a negative type indicating that the feature quantity acts to prevent information that should not be extracted from the document group from being extracted, and a neutral type indicating that the action of the feature quantity does not fall into either the positive type or the negative type.
Document information extraction system. The document information extraction system according to claim 8 ,
the feature display priority setting unit sets, for the evaluation target feature classified into the positive action type, a feature display priority higher as the evaluation target feature has a higher similarity with other feature quantities having a common action type in the document group, the co-occurrence with the other feature quantities is higher, and the currently set weight is smaller;
Document information extraction system. The document information extraction system according to claim 8 ,
the feature display priority setting unit sets, for the evaluation target feature classified into the negative action type, a feature display priority higher as a similarity between the evaluation target feature and other feature quantities having a common action type in the document group increases, a co-occurrence with the other feature quantities increases, and a currently set weight increases.
Document information extraction system. The document information extraction system according to claim 8 ,
the feature display priority setting unit sets, for the evaluation target feature classified into the neutral action type, a feature display priority higher as a similarity between the evaluation target feature and other feature quantities having a common action type in the document group increases, a co-occurrence with the other feature quantities increases, and a currently set weight increases.
Document information extraction system. The document information extraction system according to claim 1,
The information is a word or a related word of a word.
Document information extraction system. An information processing device,
Managing a collection of documents;
extracting information from documents of the document set using a machine learning model;
A step of storing a plurality of feature quantities constituting parameters of the model and an action type which is information indicating characteristics of an action when extracting the information for each of the feature quantities;
a step of accepting a setting of the action type or the weight of the evaluation target feature while displaying an evaluation target feature which is a feature to be evaluated among the plurality of feature amounts, one or more sentences from which the evaluation target feature is extracted, and a weight currently set for the evaluation target feature;
updating the parameters based on the received action type or weight ;
determining, for each of the features to be evaluated, a feature display priority which is a priority for displaying the feature to be evaluated based on at least one of a similarity between the feature to be evaluated and other features having the same action type in the document set, a co-occurrence with the other features in the document set, a weight currently set for the feature to be evaluated, and an appearance frequency of the feature to be evaluated in the document set; and
a step of receiving a setting of the action type or the weight of the evaluation target feature quantity while displaying the evaluation target feature quantity in accordance with the feature quantity display priority;
A document information extraction method.

Images