JP4925293B2 - Certainty assigning device, method and program - Google Patents

Description

  In the present invention, for example, when a search result document such as google (registered trademark) is presented from the top of the search, a certainty factor (accuracy, recall, F value, correct answer rate, etc.) is given to each document. It is given. The certainty factor (correct answer rate) is a value that indicates whether the document is correct as a search result. If it is 50%, the probability is half, and if it is 100%, it is almost 100%. The present invention relates to a certainty degree assigning device, a method, and a program for a search result to be automatically assigned.

Conventionally, there has been a system in which a document is searched by a keyword, and search results are ordered and output based on the appearance probability of the keyword (see Patent Document 1).
Japanese Patent No. 3799447

  In the conventional system for outputting the search results in order, there is no technique for giving a certainty factor (accuracy rate, etc.) to each document in an effective manner.

  SUMMARY OF THE INVENTION An object of the present invention is to solve the above problems, and to automatically assign a certainty factor (accuracy rate, etc.) to each document when a search result document is presented from the top of the search.

  FIG. 7 is an explanatory diagram of the certainty degree assigning device. In FIG. 7, 1 is an input unit (input unit), 6 is an output unit (output unit), 10 is a document classification device (problem solving unit), 11 is a correspondence table creation unit (correspondence creation unit), and 12 is a certainty factor. An assigning unit (confidence giving means) and 13 are storage units (correspondence table).

  The present invention has the following means in order to solve the conventional problems.

  (1): An input means 1 for inputting a question, a problem solving means 10 for solving the inputted problem and extracting a plurality of the answers, and outputting the extracted answer and a predetermined value, and an answer given in advance When a plurality of questions are prepared, and each of the questions is input to the problem solving means 10 and each answer is output, the certain value, the answer, and the certainty of the answer are obtained (that is, this belief) The degree is determined according to the definition of the certainty by comparing the answer with the question to which the answer has been assigned in advance, and checking how much the answer is output), and the correspondence between the predetermined value and the certainty The correspondence creation means 11 to be created, and when the new problem is inputted from the input means 1 and the answers are ordered by the problem solving means 10 and outputted, the predetermined value at which a certain answer is outputted is obtained, and the correspondence relation From That by applying a confidence answer and a certainty factor providing unit 12 to be output. For this reason, the certainty degree of the output answer can be given and it can be judged easily which answer is reliable.

  (2): In the certainty factor assigning apparatus of (1), a precision ratio that is a ratio of correct outputs among all outputs is used as the certainty factor. For this reason, it is possible to easily determine the accuracy (accuracy) to the output classification or document.

  (3): In the certainty factor assigning apparatus according to (1), a recall ratio, which is a ratio of correct output, is used as the certainty factor. For this reason, it is possible to easily determine the amount of correct leakage from the recall.

  (4): In the certainty factor assigning apparatus of (1), an F value that is the inverse of the average of the reciprocal of the recall rate and the reciprocal of the relevance rate is used as the certainty factor. For this reason, the certainty factor which considered the amount of precision (accuracy) leakage amount can be provided using F value.

  (5): In the certainty degree assigning apparatus according to the above (1) to (4), the number that is output with the certainty degree imparted by the certainty degree imparting means 12 is the number that maximizes the F value. For this reason, it is possible to output only classifications or documents with high certainty.

  (6): In the certainty factor assigning apparatus of (1), the correct answer rate of each answer is used as the certainty factor. For this reason, the correct answer rate (certainty factor) of each output answer can be given, and it can be judged easily which answer is reliable.

  (7): In the certainty degree assigning device of (6), when preparing a plurality of questions to which an answer has been assigned in advance, and inputting each of the questions to the problem solving means 10 and outputting each answer, Finding the predetermined value at which the answer is output at the last minute, checking whether the answer output at the last minute is correct, obtaining the correct answer rate at the predetermined value, and how to determine whether the answer is correct or incorrect Machine learning means for machine learning and storing the learning result, and the certainty factor giving means 12 uses the learning result as the correspondence. For this reason, the correct answer rate of the output correct answer can be easily given by machine learning.

  (8): In the certainty factor assigning apparatus of (7), a predetermined value of a plurality of viewpoints is used as the predetermined value, and a case of a correct answer or an incorrect answer when the predetermined value of the plurality of viewpoints is used in the machine learning means. Let machine learning. For this reason, more accurate machine learning can be performed using predetermined values from a plurality of viewpoints.

  (9): In the certainty assigning apparatus according to (1) to (8), the problem solving means 10 is a document classification apparatus, the problem is a document to which a classification is assigned, and the answer is a classification of the document. is there. For this reason, a certainty factor can be given to the classification to be output.

  (10): In the certainty degree assigning device according to (1) to (8), the problem solving means 10 is an information search device, the problem is a question document, and the answer is searched from the question document. Document. For this reason, a certainty factor can be given to the output answer document.

  (11): In the certainty degree assigning devices of (1) to (10), when the problem solving means 10 obtains a score and outputs an answer, the score of a certain answer is set as the predetermined value as the first answer. Use the value divided by the score (kp). For this reason, certainty can give certainty to the answer.

  (12): In the certainty factor assigning apparatus of (1) to (10), the problem solving means 10 uses the output rank (kj) as the predetermined value when outputting the answer. For this reason, certainty can be given to the answer by kj.

  (13): In the certainty degree assigning device according to (1) to (10), when the problem solving means 10 obtains a score and outputs an answer, the score (kl) is used as the predetermined value. For this reason, certainty can be given to the answer by kl.

  The present invention has the following effects.

  (1): When the correspondence creation means creates a correspondence between the predetermined value and the certainty degree from the question to which the answer has been assigned in advance, and inputs a new question and orders and outputs the answers by the problem solving means Since a certain value is output for a certain answer, and the certainty of the certain answer is given and output from the correspondence, the certainty of the answer can be given, and which answer can be trusted Easy to judge.

  (2): As the certainty factor, since the relevance ratio that is the ratio of correct output among all outputs is used, the relevance ratio (accuracy) to the output classification or document can be easily determined.

  (3): As the certainty factor, since the recall rate that is the ratio of the correct answer output among the number of correct answers is used, the correct answer leakage amount can be easily determined by the recall rate.

  (4): Since the F value, which is the average of the reciprocal of the reproducibility and the reciprocal of the relevance rate, is used as the certainty factor, the F value is used to determine the certainty factor considering the precision (accuracy) leakage amount. Can be granted.

  (5): Since the number to which the certainty factor is given and output by the certainty factor assigning unit is the number that maximizes the F value, it is possible to output only a classification or a document with a high certainty factor.

  (6): Since the accuracy rate of each answer is used as the certainty factor, the correct answer rate (confidence factor) of each output answer can be given, and it can be easily determined which answer is reliable. .

  (7) Since the learning result is provided with machine learning means and the learning result is used as the correspondence relationship, the correct answer rate of the output correct answer can be easily given by machine learning.

  (8): As the predetermined value, a predetermined value of a plurality of viewpoints is used, and the machine learning means causes the machine learning unit to machine-learn cases of correct answers or incorrect answers when the predetermined values of the plurality of viewpoints are satisfied. More accurate machine learning can be performed.

  (9): Since the problem solving means is a document classification device, the problem is a document to which a classification is given, and the answer is a classification of the document, a certainty factor can be given to the outputted classification. .

  (10): Since the problem solving means is an information retrieval device, the problem is a question document, and the answer is a document retrieved from the question document, a certainty factor is given to the output answer document. Can be granted.

  (11): In the case where the problem solving means obtains a score and outputs an answer, a value (kp) obtained by dividing the score of a certain answer by the score of the first answer is used as the predetermined value. The certainty can be given to.

  (12): Since the output order (kj) is used as the predetermined value when the answer is outputted by the problem solving means, the certainty can be given to the answer by kj.

  (13): In the case where the problem solving means obtains a score and outputs an answer, the score (kl) is used as the predetermined value, so that the certainty can be given to the answer by kl.

  The present invention automatically assigns certainty (accuracy, recall, F value, correct answer rate, etc.) to each document when presenting the search result document from the top of the search in the information search result. As a method of giving, a set of correct answers is prepared in advance, and a correspondence table showing how accurate the correct answer set is in any case. Then, when a new document arrives, the situation of the document is examined and the certainty factor is obtained from the previously obtained table. In addition to the table, other similar methods are possible. Also, anything other than document search can be handled as long as the output is listed.

§1: Explanation of assigning confidence based on a table The present invention collects documents similar to a document to be classified by a BM25 or SMART method that is known with high accuracy in a search, and classifies the document group with a high appearance frequency. Confidence is given when classifying the document. In particular, handle a multi-class classification problem in which multiple classifications are assigned to a single document, and refer to the certainty of which classification among the classifications with the highest frequency of occurrence as the classification of the document. Can do.

(1): Description of Document Classification Device FIG. 1 is an explanatory diagram of a document classification device. 1, the document classification apparatus includes an input unit (input unit) 1, a document extraction unit (document extraction unit) 2, a document similarity calculation unit (document similarity calculation unit) 3, and a score calculation unit (score calculation unit). 4. A classification set extraction unit (classification set extraction unit) 5 and an output unit (output unit) 6 are provided.

  The input unit 1 is input means for inputting a document such as a patent document. The document extracting unit 2 is a document extracting unit that extracts a document (k pieces) similar to a document to be classified. The document similarity calculation unit 3 is a document similarity calculation unit that calculates the similarity between documents. The score calculation unit 4 is score calculation means for calculating a classification score. The classification set extraction unit 5 is a classification set extraction unit that extracts a classification set of documents to be classified (with a score equal to or higher than a specified value) based on the classification score. The output unit 6 is output means for outputting (screen display, printing) the classification of the document to be classified. The output of the output unit 6 includes not being displayed on the screen but also outputting to another program inside the program or calculating as a variable value inside the program.

(2): Description of Patent Document Classification Device Patent documents (patent documents) are classified by IPC, FI, F-term, and the like. In particular, the F-term distinguishes a certain technical scope (theme) from various technical viewpoints from many viewpoints. For example, there are many purposes, applications, structures, materials, manufacturing methods, processing methods, control means, etc. It is based on a term list that distinguishes technologies from the technical point of view. For this reason, a plurality of F-terms (patent classifications) are usually assigned to one patent document. Hereinafter, a description will be given of a case where a patent document is used as a document.

FIG. 2 is an explanatory diagram of the patent document classification apparatus. 2, the patent document classification device includes an input unit (input unit) 1, a KDOC extraction unit (KDOC extraction unit) 2, a document similarity calculation unit (document similarity calculation unit) 3, and a score (Score _M1 (x) ) A calculation unit (score calculation unit) 4, an F-term x set extraction unit (F-term x set extraction unit) 5, and an output unit (output unit) 6 are provided.

The input unit 1 is input means for inputting a patent document. The KDOC extraction unit 2 is a KDOC extraction unit that extracts patent documents (k) similar to the patent documents to be classified. Here, KDOC is k extracted patent documents. The document similarity calculation unit 3 is a document similarity calculation unit that calculates the similarity between patent documents. The score (Score _M1 (x)) calculating unit 4 is score calculating means for calculating a patent classification score (Score _M1 (x)). The F-term x set extraction unit 5 is classification set extraction means for extracting a set of F-term x of patent documents to be classified based on patent classification scores. The output unit 6 is an output unit that outputs a set of F-term x of patent documents to be classified.

(3): Explanation of Patent Document Classification Processing FIG. 3 is a flowchart of patent document classification processing. Hereinafter, description will be given according to the processing S1 to S5 of FIG.

  S1: A patent document to be classified is input to the input unit 1.

  S2: The KDOC extraction unit 2 extracts k patent documents (KDOC) similar to the inputted patent document to be classified. Here, the document similarity calculation unit 3 obtains the similarity between the inputted patent document to be classified and the patent document set (in a storage means such as a database) given as learning data. The patent document set given as learning data is a document set to which a correct F-term classification is assigned. The ruby-ir toolkit was used to extract k patent documents. k is a value determined by experiments.

S3: The score (Score _M1 (x)) calculating unit 4 calculates a patent classification score (Score _M1 (x)).

  S4: The F-term x set extraction unit 5 extracts a set of F-term x (scores greater than or equal to a specified value) of patent documents to be classified based on patent classification scores.

  S5: The output unit 6 outputs a set of F-term x of patent documents to be classified.

  FIG. 4 is a process flowchart for obtaining the similarity between the input patent document and the selected patent document. Hereinafter, a description will be given according to the processes S11 to S12 of FIG.

  S11: The document similarity calculation unit 3 extracts keywords from the input patent document. As this keyword, nouns were extracted using morphological analysis technology.

S12: The document similarity calculation unit 3 next selects a patent document including at least one of the above keywords from all the patent documents having a given input theme (theme may not be given) in the learning data. Take out and calculate the Sim _SMART of the taken out patent document. This Sim _SMART is used as the similarity between each patent document in the learning data.

(4): Explanation of F-term x set extraction There are four methods for extracting the F-term x set as follows.

a) Description of Method 1 First, the patent classification device (KDOC extraction unit 2) first sets k patent documents most similar to the input as a set of patent documents (correct F-term classification assigned as learning data). From the document set). These k patent documents will be referred to as KDOC. The ruby-ir toolkit was used to retrieve the documents. k is a value determined by experiments.

(Reference for ruby-ir toolkit)
ruby-ir-eng, "Masao Utiyama", "Information Retrieval Module for Ruby", 2005,
("Www2.nict.go.jp/jt/a132/members/mutiyama/software")
Next, the patent classification device (score calculation unit 4) calculates the score (Score _M1 (x)) of F-term x by sorting KDOCs according to the following equation (1).

here,
role (x, i) = 1 (if i-th document has F-term x classification)
= 0 (otherwise)
Here, score _doc (i) is the value of the similarity of the document in which the similarity between the input document and the selected document is the i-th largest, and k _r is a constant determined by experiment. You can also simplify score _doc (i) as follows:

score _doc (i) = 1001-i
The patent classification device (classification set extraction unit 5) finally extracts a set of F-term x that satisfies the following expression (2).

{X | Score _M1 (x) ≧ k _p × max _y Score _M1 (y)} (2)
However, k _p is a constant determined by experiment. This is the classification required by the set of F-term x extracted.

Explanation of usage example of Method 1 (F-term1, F-term2, etc. below are F-terms used in each document)
Document A Similarity with input document 100 F-term1
Document B Similarity with input document 90 F-term1 F-term2
Document C Similarity with input document 80 F-term1
Document D Similarity with input document 70 F-term3
And kr = 0.99,
The score of F-term1 is 100 + 90 * 0.99 + 80 * 0.99 ^ 2 = 267.5
F-term2 score is 90 * 0.99 = 89.1
F-term3 score is 70 * 0.99 ^ 3 = 67.9
It becomes.

  If kp = 0.9, the classification of scores with a score of 240.8 or higher, which is 0.9 times the top score of 267.5, is extracted. In this case, only F-term1 satisfies it, so only F-term1 is taken out as an answer.

b) Description of Method 2 First, the document classification device extracts KDOC as in Method 1. The document classifier then counts how many documents F-term x appears in KDOC. If this number is written as F _KDOC (x), the document classification device will eventually extract a set of F-term x that satisfies the following expression.

{X | F _KDOC (x) ≧ k _u × k},
However, k _u is a constant determined by experiment. However, when k _u = 0.5, this method is identical to the original k-neighbor method.

c) Description of Method 3 First, the document classification device extracts KDOC as in Method 1. The document classifier then calculates F _KDOC (x). The document classification device finally takes out k _f F-terms in descending order of the value of F _KDOC (x), and determines them as the classification to be obtained. Here, k _f is a constant determined by experiment.

(5): Explanation of correspondence table When kp, k _u , k _f are changed in the above methods 1 to 3, the number of F-terms to be taken out changes. If there is correct data (correct F-term is given) in the input document, create a correspondence table for each changed kp, k _u , k _f and certainty (accuracy, recall, F value) can do.

For example, in the case of correspondence between kp when using Method 1 and the accuracy (accuracy rate) of F-term of patent documents,
F-term accuracy selected when kp = 0.9 → 95%
F-term accuracy selected when kp = 0.8 → 85%
F-term accuracy selected when kp = 0.7 → 80%
F-term accuracy selected when kp = 0.6 → 75%
F-term accuracy selected when kp = 0.5 → 65%
F-term accuracy selected when kp = 0.4 → 50%
F-term accuracy selected when kp = 0.3 → 45%
F-term accuracy selected when kp = 0.2 → 20%
F-term accuracy selected when kp = 0.1 → 10%
The above correspondence is output for each input document (patent document with the correct F-term). Accordingly, the accuracy (relevance ratio) is output for each patent document input to the patent document classification apparatus, and may have different accuracy for each patent document, so the average accuracy of each patent document is taken. For example, the accuracy of each patent document when kp = 0.9 is taken. In the case of the recall rate and F value, the correspondence table is created by taking the average of each patent document in the same manner as the accuracy.

  FIG. 5 is an explanatory diagram of the correspondence between kp and F value. In FIG. 5, the correspondence between kp and F value (F-measure) is that the F value increases from kp to 0.1 to 0.3, and the F value decreases from 0.4 to 0.9. The F value is maximum when kp is 0.3. The Dry run data was used to determine the parameters for each method. The experimental results with Formal run data show the performance of the method.

  FIG. 6 is an explanatory diagram of the correspondence between kp, recall, and accuracy. In FIG. 6, the horizontal axis represents the recall (Recall), the vertical axis represents the accuracy (Precision), and the black dot number in the graph represents the value of kp. In this figure, the accuracy decreases as the recall rate increases. In other words, it can be seen that as kp decreases (the number of F-terms selected increases), the accuracy decreases and the recall rate increases.

(6): Description of Calculation of Similarity Between Documents The following four methods can be used to calculate the similarity between each patent document in the learning data and the input patent document.

a) Description of SMART First, the document classification device extracts a keyword from an input patent document. As keywords, nouns are extracted using morphological analysis technology. Next, a document including at least one keyword is extracted from all patent documents having a given input theme in the learning data. The document classification device (document similarity calculation unit 3) uses the following equation (3) to calculate Sim _SMART of each extracted document. Sim _SMART is used as the similarity between the input document and each patent document in the learning data.

  In this expression, T means the set of keywords that appear in both the input patent document and the retrieved patent document, tf means the number of times the keyword t appears in the retrieved document, and avtf is extracted Qtf means the number of occurrences of keyword t in the input document, utf means the number of different keywords in the extracted document, and pivot means learning Means the average of the number of different keywords per document in all documents in the data, N means the total number of patent documents with a given input theme classification in the training data, and n is the number of documents in which the keyword t appeared Means number.

  SMART is one of the keyword weighting methods for information retrieval (cited reference; Singhal et al., 1996; Singhal, 1997).

b) Description of BM25 The document classification device first extracts keywords from the input patent document. As keywords, nouns were extracted using morphological analysis technology. Next, a document including at least one of the keywords is extracted from all patent documents having a given input theme classification in the learning data. The document classification device (document similarity calculation unit 3) uses the following equation (6) to calculate Sim _BM25 of each extracted document. Sim _BM25 is used as the similarity between the input document and each patent document in the learning data.

T, tf, qtf, N, n are the same as those in SMART. dl is the length of the retrieved article, avdl is the average of the length of the article in all documents, k ₁ , k _{3 and} b are constants determined by experiments. As default values of ruby-ir toolkit, values of k ₁ = 1, k ₃ = 1000, b = 1 were used. Instead of log {(N-n + 0.5) / (n + 0.5)} in the original formula of BM25, log (N / n) was used. This is because the original formula outputs a negative score. It was confirmed that the formula corrected in the experiment gave higher accuracy.

BM25 is one of the keyword weighting methods for information retrieval (cited reference; Robertson et al., 1994).
c) Description of Tfidf First, the document classification device extracts keywords from the input patent document. As keywords, nouns were extracted using morphological analysis technology. Next, a document including at least one of the keywords is extracted from all documents having a given input theme classification in the learning data. The document classification device (document similarity calculation unit 3) uses the following equation (9) to calculate Sim _Tfidf of each extracted document. Sim _Tfidf is used as the similarity between the input document and each document in the training data.

  In this equation, T 1, tf, N and n are the same as those of SMART.

d) Description of Overlap First, the document classification device extracts keywords from the input patent document. As keywords, nouns were extracted using morphological analysis technology. Next, a document including at least one of the keywords is extracted from all documents having a given input theme classification in the learning data. The document classification device (document similarity calculation unit 3) uses the following equation (10) to calculate the Sim _Overlap of each extracted document. Sim _Overlap is used as the similarity between the input document and each document in the training data.

  In this equation, T is the same as that of SMART.

(7): Explanation of Evaluation of Document Search Result When a theme classification of a patent document is given, an F-term classification of the input Japanese patent document is obtained. For this evaluation, F-measure (F value) can be used as shown in FIG. F-measure is the reciprocal of the average of the reciprocal of the recall (Recall) and the reciprocal of the precision (Precision). The recall is the ratio of correct output among correct answers (the larger the recall is, the smaller the correct answer leaks), and the relevance ratio is the ratio of correct outputs among all outputs. Expressed as an expression:

(8): Explanation of word recognition a) Explanation of morphological analysis system In order to divide Japanese into words, a morphological analysis system performed by a word extraction unit is required. Here, ChaSen will be explained (published on the morphological analysis system Chasen http://chasen.aist-nara.ac.jp/index.html.jp developed at Nara Institute of Technology).

  This splits the Japanese sentence and also estimates the part of speech of each word. For example, if “go to school” is entered, the following results can be obtained.

School Gakkou School Noun-General To He To particle-Case particle-General Go Iku Go Verb-Independence
In this way, each line is divided so that one word is included, and reading and part-of-speech information are given to each word.

b) Explanation of English part-of-speech tagging The following part-of-speech tagging systems in English are famous.

Eric Brill, Transformation-Based Error-Driven Learning and
Natural Language Processing: A Case Study in Part-of-Speech Tagging,
Computational Linguistics, Vol. 21, No. 4, p.543-565, 1995.
This estimates the part of speech of each word in an English sentence.

(9): Explanation of assigning confidence based on a table a) Explanation when using kp Collect a large number of sets of questions and answers in advance. The problem is a patent that should have an F-term, and the answer is the F-term of that patent. This is called evaluation data. For each kp, the document classification device outputs an F-term with the above evaluation data and evaluates it, and obtains certainty such as accuracy (accuracy), recall, and F value. Furthermore, the F-term accuracy (matching rate), recall, and average F value of all patents corresponding to the same kp are obtained. Then, a correspondence table of kp, accuracy (accuracy rate), recall rate, and F value is completed.

  Next, when a new patent enters the document classification device, an F-term is output. Find the kp at which each F-term is output. This method is as follows.

  The value obtained by dividing the score (Score) of an F-term by the score of the first F-term (the F-term with the highest score) is the kp at which that F-term is output. (Refer to equation (2), depending on the definition of kp). The score is obtained using equation (1) or the like.

  When kp of each F-term is obtained, the accuracy (accuracy), recall, and F value corresponding to each F-term are displayed based on the previous correspondence table. The accuracy (accuracy), recall, and F value for the F-term group up to that F-term (not the accuracy (accuracy), recall, and F value of each F-term). Each F-term will be explained later.

  Hereinafter, description will be given based on the drawings. FIG. 7 is an explanatory diagram of the certainty degree assigning device. In FIG. 7, the certainty degree assigning device includes an input unit 1, an output unit 6, a document classification device (problem solving unit) 10, a correspondence table creating unit (corresponding relationship creating unit) 11, a certainty degree assigning unit 12, and a storage unit ( Correspondence table) 13 is provided.

  The input unit 1 is input means for inputting information. The output unit 6 is output means for outputting information. The document classification device 10 is document classification means (problem solving means) for classifying the documents described above (see FIGS. 1 and 2). The correspondence table creating unit (corresponding relationship creating means) 11 is a correspondence (table) creating means for creating a correspondence (table) between kp, accuracy (accuracy), recall, and F value. The certainty factor assigning unit 12 is a certainty factor assigning unit that assigns certainty factors such as accuracy (accuracy rate), recall rate, F value, and correct answer rate to the classification given by the document classification device 10. The storage unit (correspondence table) 13 is storage means for storing the correspondence table created by the correspondence table creation unit 11.

  FIG. 8 is a flowchart of the correspondence table creation process. Hereinafter, a description will be given according to processing S21 to S25 of FIG.

  S 21: A large number of pairs of questions and answers (here, patent documents and their F-terms) are input in advance from the input unit 1 and stored in the storage means of the document classification device 10.

  S22: The document classification device 10 searches for another patent document similar to the one input patent document and determines the classification (F-term is determined).

  S23: The document classification device 10 calculates the score of the obtained classification (F-term) according to how many patent documents the classification (F-term) of the other similar patent document appears.

  S24: The correspondence table creation unit 11 obtains the certainty of each classification (F-term) output from the document classification device 10 when kp is changed.

  S25: The correspondence table creation unit 11 assigns a classification (obtains an F-term) to the patent document input in S21 by the document classification device 10, obtains a certainty factor by changing kp, and further sets the same kp. An average value of certainty factors of all corresponding patent documents is obtained, and a correspondence table is created.

  FIG. 9 is a flowchart of the certainty factor giving process. Hereinafter, a description will be given according to processing S31 to S35 of FIG.

  S31: A new document (patent document to which no F-term is assigned) is input from the input unit 1.

  S32: The document classification device 10 searches for a patent document similar to the input patent document (patent document input in the process S21) to determine the classification (determines F-term).

  S33: The document classification apparatus 10 calculates the score of the assigned classification (F-term) based on how many patent documents the classification (F-term) of the similar patent document appears in.

  S34: The certainty factor assigning unit 12 obtains kp at which each classification (F-term) is barely output.

  S35: The certainty factor assigning unit 12 assigns the certainty factor corresponding to the obtained kp from the correspondence table of the storage unit 13 to each F-term and outputs it from the output unit.

  Thus, the present invention relates to document classification. Documents similar to the document to be classified are collected by the BM25 and SMART methods that are known with high accuracy in the search, and the documents are classified into categories having a high appearance frequency in the document group. In particular, it handles Multi-class classification problems where multiple classifications are assigned to a single document, and it is easy to determine which classification is the classification of the document from among the classifications with a high appearance frequency. can do.

  The patent document has a code for classifying a plurality of patents. The code is generally given manually, but if the present invention is used, the code can be given even to some extent automatically, and there is an effect of reducing the manual work.

  In addition, the number of classifications (F-terms) to which the certainty factor is given and output by the certainty factor granting unit 12 is a place where the accuracy (matching rate) is equal to or greater than a certain value (specified value) up to the maximum F value. Thus, unnecessary output can be reduced by performing output up to a certain value (specified value) or less.

b) Explanation in the case of using the output order In the case of the method using the output order, the classification (F-term) output by the document classification device is set to the output system up to kj. This system solves the problem of the evaluation data by changing some values of kj, and obtains the accuracy (precision), recall, and F value. Then, a correspondence table of kj (rank), accuracy (precision), recall, and F value is completed.

  When a new patent comes in, F-term is output by the previous method on the document classification device. Find the kj at which each F-term is output. Then, the output order is kj (this is different depending on the definition of kj, and this part is different in other methods).

  Once the kj of each Fterm is obtained, the accuracy (precision), recall, and F value corresponding to each Fterm are displayed together based on the above correspondence table. Note that this is the accuracy (precision), recall, and F value for the documents up to the F-term. (This is not the accuracy (accuracy), recall, or F value of each F-term. For individual F-terms, see the calculation of individual values below.)

  FIG. 10 is a correspondence table creation process flowchart. Hereinafter, description will be made in accordance with steps S41 to S45 of FIG. 10 (see FIG. 7 for the certainty factor imparting apparatus, where kj is used instead of kp described in FIG. 7).

  S41: A large number of pairs of questions and answers (here, patent documents and their F-terms) are input in advance from the input unit 1 and stored in the storage means of the document classification device 10.

  S42: The document classification device 10 searches for another patent document similar to the one input patent document and determines the classification (F-term is determined).

  S43: The document classification apparatus 10 calculates the rank kj of the obtained classification (F-term) based on how many patent documents the classification (F-term) of the other similar patent document appears in.

  S44: The correspondence table creation unit 11 obtains the certainty of each classification (F-term) output from the document classification device 10 when kj is changed.

  S45: The correspondence table creation unit 11 assigns a classification (obtains F-term) to all the patent documents input in S41 by using the document classification device 10, obtains a certainty factor by changing kj, and sets the same kj. An average value of certainty factors of all corresponding patent documents is obtained, and a correspondence table is created.

  FIG. 11 is a flowchart of the certainty factor giving process. Hereinafter, description will be made in accordance with steps S51 to S55 in FIG. 11 (see FIG. 7 for the certainty factor imparting apparatus, but here, kj is used instead of kp described in FIG. 7).

  S51: A new document (patent document to which no F-term is assigned) is input from the input unit 1.

  S52: The document classification device 10 searches for a patent document similar to the input patent document (the patent document input in the process S41) to determine the classification (F-term is determined).

  S53: The document classification apparatus 10 calculates the rank kj of the obtained classification (F-term) based on how many patent documents the classification (F-term) of the similar patent document appears in.

  S54: The certainty factor assigning unit 12 obtains kj at which each F-term is barely output.

  S55: The certainty factor assigning unit 12 assigns the certainty factor corresponding to the obtained kj from the correspondence table of the storage unit 13 to each F-term and outputs it from the output unit.

c) Explanation when using the score (Score) In the case of the method using the score, the classification (F-term) output by the document classification device is up to a score of kl or more. This system solves the problem of evaluation data by changing some values of kl, and obtains accuracy (precision), recall, and F value. Then, a correspondence table of kl (score), accuracy (precision), recall, and F value is completed.

  When a new patent comes in, the document classification device outputs an F-term using the previous method. Find the kl at which each F-term is output. Here, the score of each F-term is kl. (This is due to the definition of kl. In other ways this part is different).

  If the kl of each Fterm is obtained, the accuracy (precision), recall, and F value corresponding to each Fterm are displayed based on the previous correspondence table. (Note that the accuracy (accuracy), recall, and F value for the document group up to that F-term. The accuracy (accuracy), recall, and F value of each F-term are not. (For term, see the calculation of individual values below).

  FIG. 12 is a flowchart of the correspondence table creation process. Hereinafter, description will be made in accordance with steps S61 to S65 of FIG. 12 (see FIG. 7 for the certainty factor imparting device, where kl is used instead of kp described in FIG. 7).

  S61: A large number of pairs of questions and answers (here, patent documents and their F-terms) are input in advance from the input unit 1 and stored in the storage means of the document classification device 10.

  S62: The document classification device 10 searches for another patent document similar to the one input patent document and determines the classification (F-term is determined).

  S63: The document classification device 10 calculates the score (kl) of the obtained classification (F-term) based on how many patent documents the classification (F-term) of the other similar patent document appears in. To do.

  S64: The correspondence table creation unit 11 obtains the certainty of each classification (F-term) output from the document classification device 10 when kl is changed.

  S65: The correspondence table creation unit 11 assigns a classification (obtains an F-term) to the patent document input in S61 by the document classification device 10 to obtain a certainty factor by changing kl, and further sets the same kl. An average value of certainty factors of all corresponding patent documents is obtained, and a correspondence table is created.

  FIG. 13 is a flowchart of the certainty factor giving process. Hereinafter, description will be made in accordance with steps S71 to S75 in FIG. 13 (see FIG. 7 for the certainty factor imparting device, where kl is used instead of kp described in FIG. 7).

  S71: A new document (patent document to which no F-term is assigned) is input from the input unit 1.

  S72: The document classification device 10 searches for a patent document similar to the input patent document (the patent document input in the process S61) to determine the classification (determines F-term).

  S73: The document classification device 10 calculates a score kl of the obtained classification (F-term) according to how many patent documents the classification (F-term) of the similar patent document appears.

  S74: The certainty factor assigning unit 12 obtains kl at which each F-term is output at the last minute.

  S75: The certainty factor assigning unit 12 assigns the certainty factor corresponding to the obtained kl from the correspondence table of the storage unit 13 to each F-term and outputs it from the output unit.

  The method of using kp, ranking, and score has been described above, but other systems can be used as long as they are systems that output in order.

§2: Explanation in case of information retrieval (1): Explanation of information retrieval system (information retrieval apparatus) Examples of techniques for retrieving documents from keywords (document retrieval techniques) include the following.

(Explanation of how to extract articles containing more word groups A)
There is the following formula as basic knowledge of information retrieval. Here, the score (D) is large.

(1) Explanation of basic method (tf / idf method)
score (D) = Σ (tf (w, D) * log (N / df (w)))
w ∈ W and W is the set of keywords entered by the user
tf (w, D) is the number of occurrences of w in document D
df (w) is the number of documents in which W appears in all documents N is the total number of documents
Documents with high score (D) are output as search results.

(2) Explanation of Okson weighting by Robertson et al.
Murata Masaki, Ma Aoi, Uchimoto Kiyotaka, Osaku Hiromi, Uchiyama Masao, Isahara Hitoshi "Information Retrieval Using Location Information and Field Information" Natural Language Processing (Journal of the Language Processing Society) April 2000, Volume 7, Issue 2 , p.141-p.160
(1) is known to have good performance. The product of the tf term and idf term before taking the product with Σ in Equation (1) is Okapi's weighting method, and this value is used as the word weight.

Okapi expression
Score (D) = Σ (tf (w, D) / (tf (w, D) + length / delta) * log (N / df (w)))
Add by w ∈W
length is the length of article D, delta is the average length of articles,
The length of the article uses the number of bytes of the article and the number of words included in the article.

  Further, the following information search can be performed.

(Okapi references) SE Robertson, S. Walker, S. Jones, MM Hancock-Beaulieu, and M. Gatford Okapi at TREC-3, TREC-3, 1994
(SMART reference)
Amit Singhal AT & T at TREC-6, TREC-6, 1997
As a more advanced information retrieval method, these Okapi and SMART formulas may be used instead of just formulas using tf / idf.

  In these methods, more accurate information retrieval can be performed using not only tf / idf but also the length of the article.

  Rocchio's formula can also be used in this method of extracting articles that contain more words.

(Reference)
"JJ Rocchio", "Relevance feedback in information retrieval", "The SMART retrieval System", "Edited by G. Salton", "Prentice Hall, Inc.", "page 313-323", 1971
This method can be used instead of log (N / df (w))
{E (t) + k_af * (RatioC (t)-RatioD (t))} * log (N / df (w))
use.

E (t) = 1 (keyword from the original search)
= 0 (otherwise)
RatioC (t) is the rate of occurrence of t in article group B
RatioD (t) is the rate of occurrence of t in article group C
Score (D) is obtained by replacing log (N / df (w)) with the above equation, and the larger the value, the more the word group A is extracted.

  A set W of words w to be added when Σ of Score (D) is added is both the original keyword and the word group A. However, the original keyword and the word group A should not overlap.

  Another method is to add Σ of Score (D). The set W of words w is only the word group A. However, the original keyword and the word group A should not overlap.

  Here, Roccio's formula is used in a complicated manner, but simply, the larger the sum of the number of occurrences of words in word group A, the more the word group A may be extracted. Alternatively, the larger the difference in the appearance of the word group A, the more the word group A may be taken out as an article.

(2): Explanation of giving confidence Collect a large number of sets of questions and answers in advance. The problem is an information retrieval question (for example, taking out an article about a corporate merger), and the answer is a group of articles corresponding to the question. One information retrieval system (information retrieval apparatus) as described in (1) above is prepared, which is called evaluation data.

  From the question, morphological analysis is performed, a noun is extracted as a keyword, and an article is extracted by the information search system using the keyword. Then, each article has a Score (D) value if it is an Okapi expression, and the one with this value is output.

a) Explanation of using kp value
In the case of a method using the value of kp, Score_max is the maximum value of Score (D) for a certain question. Then, the document up to Score_max * kp is output. This system solves the problem of the evaluation data by changing some values of kp, and obtains the value of (confidence) such as accuracy (relevance ratio), recall ratio, and F value. Then, a correspondence table of kp, accuracy (precision), recall, and F value is completed.

  Next comes a new information retrieval question. The document is output by the previous method (information retrieval system). Find the kp at which each document is output.

  This method is as follows.

  The value obtained by dividing the score of a document by the score of the first document (the document with the largest score) is the kp at which the document is output. (This is due to the definition of kp. This method is different in the ranking method and other methods).

  When kp of each document is obtained, the accuracy (relevance ratio), reproduction ratio, and F value corresponding to each document are displayed on the basis of the above correspondence table. (Note that this is the accuracy (matching rate), recall, and F value for the group of documents up to that document. It is not the accuracy (matching rate), recall, and F value of individual documents. For the calculation of individual values).

  FIG. 14 is a flowchart of the correspondence table creation process. Hereinafter, description will be made in accordance with steps S81 to S85 in FIG. 14 (see FIG. 7 for the certainty factor imparting apparatus, but here, an information retrieval system (information retrieval apparatus) is used instead of the document classification apparatus in FIG. 7). .

  S81: A large number of sets of questions (questions) and articles (answers) are input in advance from the input unit 1 and stored in the storage means of the information search system.

  S82: The information search system performs a morphological analysis from one input question, extracts a noun as a keyword, and performs information search of the input article (answer) using the keyword. Take out the article.

  S83: The information retrieval system outputs up to a document (article) of Score_max * kp.

  S84: The correspondence table creation unit 11 obtains the certainty factor of each article output from the information search system when kp is changed.

  S85: The correspondence table creation unit 11 outputs articles by the information search system for all the questions input in S81, obtains the certainty by changing kp, and further determines the certainty of all articles corresponding to the same kp. An average value is obtained and a correspondence table is created (the correspondence table is stored in the storage unit 13).

  FIG. 15 is a flowchart of certainty factor assignment processing. Hereinafter, a description will be given according to processing S91 to S94 in FIG.

  S91: A new information search question is input from the input unit 1.

  S92: The information search system performs morphological analysis from the input question, extracts a noun as a keyword, performs an information search using the keyword, and extracts an article.

  S93: The certainty factor assigning unit 12 obtains kp at which each article is output by the information search system.

  S94: The certainty factor assigning unit 12 assigns the certainty factor corresponding to the obtained kp from the correspondence table of the storage unit 13 to the article and outputs it from the output unit.

b) Explanation in the case of using the output order In the case of the method of using the output order, the documents (articles) up to kj are used as the output system. This is to change some values of kj and solve the problem of the evaluation data by this information retrieval system to obtain the accuracy (precision), recall, and F value. Then, a correspondence table of kj, accuracy (precision), recall, and F value is completed.

  Next, when a new information retrieval question comes in, the document is output by the previous method (when creating the correspondence table). Then, kj at which each document is output is obtained. The order of output is kj. (This is due to the definition of kj. In other ways this part is different). If kj of each document is obtained, the accuracy (relevance ratio), reproduction ratio, and F value corresponding to each document are displayed on the basis of the above correspondence table. (Note that it is accuracy (relevance rate), recall rate, and F value for a group of documents up to that document, not the accuracy (relevance rate), recall rate, and F value of each document. (See the calculation of individual values below).

  FIG. 16 is a flowchart of the correspondence table creation process. Hereinafter, description will be made in accordance with steps S101 to S105 in FIG. 16 (see FIG. 7 for the certainty factor assigning device, where an information retrieval system is used instead of the document classification device of FIG. 7 and kj is used instead of kp). Is).

  S101: A large number of sets of questions (questions) and articles (answers) are input in advance from the input unit 1 and stored in the storage means of the information search system.

  S102: The information search system performs a morphological analysis from one inputted question, extracts a noun as a keyword, performs an information search of an article (answer) using the keyword, and extracts an article.

  S103: The information retrieval system outputs documents (articles) up to kj.

  S104: The correspondence table creation unit 11 obtains the certainty factor of each document (article) output from the information search system when kj is changed.

  S105: The correspondence table creation unit 11 outputs a document (article) with the information search system for all the questions input in S101, obtains a certainty factor by changing kj, and all documents corresponding to the same kj ( The average value of the certainty of the article is obtained and a correspondence table is created (the correspondence table is stored in the storage unit 13).

  FIG. 17 is a flowchart of certainty factor assignment processing. Hereinafter, description will be made in accordance with steps S111 to S114 in FIG. 17 (see FIG. 7 for the certainty factor assigning device, where an information retrieval system is used in place of the document classification device in FIG. 7 and kj is used in place of kp. Is).

  S111: A new information search question is input from the input unit 1.

  S112: The information search system performs morphological analysis from the input question, extracts a noun as a keyword, performs an information search using the keyword, and extracts an article.

  S113: The certainty factor assigning unit 12 obtains kj at which each article is barely output by the information search system.

  S114: The certainty factor assigning unit 12 assigns the certainty factor corresponding to the obtained kj from the correspondence table of the storage unit 13 to the article and outputs the article from the output unit.

c) Explanation of using score (Score)
In the case of the method using Score, the output system is up to the document whose Score is kl or more. This information retrieval system solves the problem of evaluation data by changing the value of kl, and obtains the accuracy (precision), recall, and F value. Then, the correspondence table of Score kl, accuracy (precision), recall, and F-number is completed.

  Next comes a new information retrieval question. Output the document using the previous method (method for creating the correspondence table). Here, kl at which each document is output is obtained. Then, the score of each document becomes kl. (This is due to the definition of kl. In other ways this part is different).

Once the kl of each document is obtained, the accuracy (relevance rate), recall rate, and F value corresponding to each document are displayed based on the previous correspondence table. (Note that the accuracy (relevance rate), recall rate, and F value for the document (article) group up to that document. Not the accuracy (relevance rate), recall rate, and F value of each document. (See the case of calculating individual values below)
FIG. 18 is a correspondence table creation process flowchart. Hereinafter, description will be made in accordance with steps S121 to S125 of FIG. 18 (see FIG. 7 for the certainty factor assigning device, where an information retrieval system is used instead of the document classification device of FIG. 7 and kl is used instead of kp). Is).

  S121: A large number of sets of questions (questions) and articles (answers) are input in advance from the input unit 1 and stored in the storage means of the information search system.

  S122: The information search system performs a morphological analysis from one input question, extracts a noun as a keyword, performs an information search for an article (answer) using the keyword, and extracts an article.

  S123: The information search system outputs up to documents (articles) whose Score is kl or more.

  S124: The correspondence table creation unit 11 obtains the certainty factor of each document (article) output from the information search system when kl is changed.

  S125: The correspondence table creation unit 11 outputs a document (article) by the information search system for all the questions input in S121, obtains a certainty factor by changing kl, and further calculates all documents corresponding to the same kl ( The average value of the certainty of the article is obtained and a correspondence table is created (the correspondence table is stored in the storage unit 13).

  FIG. 19 is a flowchart of certainty factor assignment processing. Hereinafter, description will be made in accordance with steps S131 to S134 in FIG. 19 (see FIG. 7 for the certainty factor assigning apparatus, where an information retrieval system is used in place of the document classification apparatus in FIG. 7 and kl is used in place of kp. Is).

  S131: A new information search question is input from the input unit 1.

  S132: The information search system performs morphological analysis from the input question, extracts a noun as a keyword, performs an information search using the keyword, and extracts an article.

  S133: The certainty factor assigning unit 12 obtains kl at which each article is output with the information search system.

  S134: The certainty factor assigning unit 12 assigns the certainty factor corresponding to the obtained kl from the correspondence table of the storage unit 13 to the article and outputs it from the output unit.

d) Explanation of kl score normalization
As kl, a score normalized may be used. There are several methods for score normalization. (1): The number of words is added in Okapi's formula Σ described in the description of the information retrieval system, but the original score is divided by the number of words as the simplest normalization.

  Next, assuming an article in which the word W appears once, the original score is divided by the score of the article.

  Alternatively, the method itself may be changed to stop the addition by Σ, make a vector, and then obtain the score to obtain the same effect as normalization.

  For example, vectors of all kinds of words are created as elements, and the values of the elements of each vector are obtained using the expression inside the Σ of the Okapi expression. To create a vector for the document to be searched. The angle of these vectors is taken as a score. Using the angle has the same effect as normalization.

  In the case of F-term, BM25 and Okapi are almost the same formula, but add only the word by Σ of BM25 formula (6), but the method of dividing the original score by the number of words is the simplest regular It can be as

  Next, assuming an article in which the word W appears once, the original score is divided by the article score.

  Alternatively, the method itself may be changed to stop the addition by Σ, make a vector, and then obtain the score to obtain the same effect as normalization. For example, vectors for all types of words are created as elements, and the values of the elements of each vector are expressed using equation (8) for input words and using equation (7) for document words. Then, a vector is created for the input keyword, and a vector is created for the document to be searched. The angle of these vectors is taken as a score. Using the angle has the same effect as normalization.

e) Explanation of using two predetermined values from different viewpoints For example, consider using two of kp and kl. Consider using kp = 0, 0.1, 0.2,, 1.0 and kj = 1, 2, 3,, 1000. For all these combinations, find the average confidence and create a correspondence table.

Confidence for kp = 0, kj = 1 ...
Confidence for kp = 0.1 and kj = 1 ...
...
Confidence for kp = 0, kj = 2 ...
Confidence for kp = 0.1, kj = 2 ...
...
...
Confidence when kp = 0, kj = 1000 ...
Confidence for kp = 0.1 and kj = 1000 ...
...
The correspondence table is obtained as above.
Here, a new problem is entered. Then, the answer is output. Find kp and kj at the time when the answer is output. These kp and kj are obtained by the same method as in the case of one. If kp and kj are known, the above correspondence table is examined, and the certainty factor in that case is obtained and output. If there is no data in the correspondence table that is exactly the same as kp and kj at the time when the answer is output, interpolation is performed.

  For example, suppose that after this, kp is kp1 and kj is kj1 for a new input. The values for kp1 and kj1 are not on the table. Then, some kind of interpolation processing is required. In that case, you can use the value of the combination of kp1 closest to kp1 and kj1 closest to kj, or you can use kp1 Two kp sandwiched between two kj sandwiched between kj1 may be used, and the average of four combinations of all combinations of kp and kj selected from two kp and two kj may be used.

Also, the two data kp2, kp3 (kp2>kp> kp3) sandwiching kp1 in the table and the two data kj2, kj3 (kj2>kj> kj3) sandwiching kj1 P (2,2) for certainty for kp2, kj2, p (3,2) for certainty for kp3, kj2, and p (2,3) for certainty for kp2, kj3 The confidence for kp3, kj3 is p (3,3)
r (2,2) = sqrt ((kp-kp2) ^ 2 + a (kj-kj2) ^ 2)
r (3,2) = sqrt ((kp-kp3) ^ 2 + a (kj-kj2) ^ 2)
r (2,3) = sqrt ((kp-kp2) ^ 2 + a (kj-kj3) ^ 2)
r (3,3) = sqrt ((kp-kp3) ^ 2 + a (kj-kj3) ^ 2)
As
p (2,2) / r (2,2) + p (3,2) / r (3,2) + p (2,3) / r (2,3) + p (3,3) / r (3,3)
The
1 / r (2,2) + 1 / r (3,2) + 1 / r (2,3) + 1 / r (3,3)
Divided by may be used for confidence.

  Here, a is a constant, which is determined in advance by experiment or given in advance by the system user. ^ Means power and sqrt means square root. A similar method may be used. Other interpolation methods may be used. The same applies when three or more such as kp kj kl are used.

§3: Description of calculation of individual values (1): Description of using document classification device a) Description of using kp values In the case of calculation of individual values, a large number of pairs of questions and answers are preliminarily used. To collect. The problem is the patent that should use the F-term, and the answer is the F-term of that patent. This is called evaluation data.

  The document classification device (patent document classification device) outputs an F-term with the evaluation data. Here, kp is calculated for each F-term. This method is as follows.

  The value obtained by dividing the score (Score) of an F-term by the score of the first F-term (the F-term with the highest score) is the kp at which that F-term is output. (Refer to equation (2), which depends on the definition of kp, and note that this method is different in the ranking method and other methods.) The score is obtained using equation (1) and the like.

  It is checked for each F-term of the output evaluation data that is output, and the correct answer rate at each kp is obtained. Furthermore, the average value of the correct answer rate of the F-term of all the evaluation data (patent documents) corresponding to the same kp is obtained. Then, the correspondence table between kp and accuracy rate is completed.

  When a new patent (without classification) is entered, the document classification device outputs an F-term. Find the kp at which each F-term is output. This method is the same as in the case of creating the above correspondence table. The value obtained by dividing the score of a certain F-term by the score of the first F-term (the F-term with the highest score) is the last. Output kp. (Refer to equation (2), which depends on the definition of kp, and note that this method is different in the ranking method and other methods.) The score is obtained using equation (1) and the like.

If kp of each F-term is obtained, the correct answer rate corresponding to each F-term is attached and displayed based on the previous correspondence table. (Note that this accuracy rate is the accuracy rate of each F-term. The accuracy (accuracy rate), recall rate, F value, etc. for the F-term group up to that F-term are different. )
FIG. 20 is a flowchart of the correspondence table creation process. Hereinafter, description will be made according to the processing S141 to S145 of FIG.

  S141: A large number of pairs of questions and answers (here, patent documents and their F-terms) are input in advance from the input unit 1 and stored in the storage means of the document classification device 10.

  S142: The document classification device 10 searches for another patent document similar to the one input patent document and determines the classification (determines F-term).

  S143: The document classification apparatus 10 calculates the score of the obtained classification (F-term) based on how many patent documents the classification (F-term) of the other similar patent document appears.

  S144: Correspondence table creation unit 11 obtains kp at which each classification (F-term) is output, checks whether it is correct for each classification (F-term), and obtains kp at each kp. Find the accuracy rate.

  S145: The correspondence table creation unit 11 assigns a classification (F-term is determined) by the document classification apparatus 10 to all the patent documents input in S141, and each classification (F-term) is output at the limit kp Further, an average value of correct answer rates of all patent documents corresponding to the same kp is obtained, and a correspondence table is created (the correspondence table is stored in the storage means 13).

  FIG. 21 is a flowchart showing the certainty factor giving process. Hereinafter, a description will be given according to processing S151 to S155 of FIG.

  S151: A new document (patent document to which no F-term is assigned) is input from the input unit 1.

  S152: The document classification device 10 searches for a patent document similar to the input patent document (the patent document input in the process S141) to determine the classification (F-term is determined).

  S153: The document classification apparatus 10 calculates the score of the assigned classification (F-term) according to the number of patent documents in which the classification (F-term) of the similar patent document appears.

  S154: The certainty factor assigning unit 12 obtains kp at which each classification (F-term) is barely output.

  S155: The certainty factor assigning unit 12 assigns the certainty factor corresponding to the kp obtained from the correspondence table of the storage unit 13 to each F-term and outputs it from the output unit.

b) Explanation in the case of using the output order In the case of the method using the output order, the F-term is the output system (the patent document classification device). This system solves the problem of the evaluation data, checks whether the F-term of the kj-th output is present or wrong, and obtains the accuracy rate of the kj-th output. Then, the correspondence table between kj and accuracy rate is completed.

  When a new patent comes in, F-term is output by the previous method (patent document classification device). Then, kj that is output at the end of each F-term is obtained. The order in which the F-term is output is kj. (This is due to the definition of kj. In other ways this part is different).

  If kj of each Fterm is obtained, the correct answer rate corresponding to each Fterm is displayed based on the above correspondence table.

  FIG. 22 is a flowchart of the correspondence table creation process. In the following, description will be given in accordance with steps S161 to S165 of FIG. 22 (see FIG. 7 for the certainty factor imparting apparatus).

  S161: A large number of sets of questions and answers (here, patent documents and their F-terms) are input in advance from the input unit 1 and stored in the storage means of the document classification device 10.

  S162: The document classification device 10 searches for another patent document similar to the one input patent document to determine the classification (F-term is determined).

  S163: The document classification device 10 calculates the rank kj of the obtained classification (F-term) based on the number of patent documents in which the classification (F-term) of the other similar patent document appears.

  S164: The correspondence table creation unit 11 checks whether the kj-th classification (F-term) is output or is incorrect, and obtains the accuracy rate of the kj-th output.

  S165: The correspondence table creation unit 11 obtains the accuracy rate of the kj-th output for all the patent documents input in S161, and further obtains the average value of the accuracy rates of all patent documents corresponding to the same kj A table is created (the correspondence table is stored in the storage means 13).

  FIG. 23 is a flowchart of certainty factor assignment processing. In the following, description will be made according to processing S171 to S175 in FIG. 23 (see FIG. 7 for the certainty factor imparting apparatus).

  S171: A new document (patent document to which no F-term is assigned) is input from the input unit 1.

  S172: The document classification device 10 searches for a patent document similar to the input patent document (the patent document input in the process S161) to determine the classification (F-term is determined).

  S173: The document classification device 10 calculates the rank kj of the obtained classification (F-term) based on how many patent documents the classification (F-term) of the similar patent document appears in.

  S174: The certainty factor assigning unit 12 obtains kj at which each F-term is barely output.

  S175: The certainty factor assigning unit 12 assigns the certainty factor corresponding to the obtained kj from the correspondence table of the storage unit 13 to each F-term and outputs it from the output unit.

c) Explanation when using score (score) In the case of a method using a score, an F-term output system (the patent document classification device) is used. This system solves the problem of evaluation data and evaluates each output F-term. If the F-term score is kl, check whether the F-term exists, and find the correct answer rate for kl. Find this for every kl. Then, the correspondence table between kl and accuracy rate is completed.

When a new patent (without F-term) is entered, the patent document classification device outputs the F-term. Here, kl that each F-term is output is obtained. Then the score of each F-term becomes kl. (It depends on the definition of kl. In other methods, this part is different.)
When the kl of each Fterm is obtained, the correct answer rate corresponding to each Fterm is displayed based on the previous correspondence table.

  FIG. 24 is a flowchart of the correspondence table creation process. Hereinafter, description will be made according to processing S181 to S185 in FIG. 24 (see FIG. 7 for the certainty factor imparting apparatus).

  S181: A large number of sets of questions and answers (here, patent documents and their F-terms) are input in advance from the input unit 1 and stored in the storage means of the document classification device 10.

  S182: The document classification device 10 searches for another patent document similar to the inputted one patent document to determine the classification (determines F-term).

  S183: The document classification apparatus 10 calculates the score (kl) of the obtained classification (F-term) based on how many patent documents the classification (F-term) of the other similar patent document appears. To do.

  S184: The correspondence table creation unit 11 examines whether or not the F-term score is kl, and determines whether the F-term is present, and obtains the correct answer rate in the case of kl.

  S185: The correspondence table creation unit 11 obtains the correct answer rate for every kl of the classification (F-term) of the patent document input in S181. Then, the correspondence table between kl and accuracy rate is completed (the correspondence table is stored in the storage means 13).

  FIG. 25 is a flowchart of the certainty degree giving process. In the following, description will be made in accordance with steps S191 to S195 of FIG. 25 (see FIG. 7 for the certainty factor imparting apparatus).

  S191: A new document (patent document to which no F-term is assigned) is input from the input unit 1.

  S192: The document classification device 10 searches for a patent document similar to the input patent document (the patent document input in the process S181) to determine the classification (determines an F-term).

  S193: The document classification device 10 calculates the score (kl) of the obtained classification (F-term) based on how many patent documents the classification (F-term) of the similar patent document appears in.

  S194: The certainty factor assigning unit 12 obtains kl from which each F-term is output.

  S195: The certainty factor assigning unit 12 assigns the correct answer rate corresponding to the obtained kl from the correspondence table of the storage unit 13 to each F-term and outputs it from the output unit.

(2): Explanation in the case of using the information retrieval apparatus Collect a large number of sets of questions and answers in advance. The problem is an information retrieval question (for example, taking out an article about a corporate merger), and the answer is an article (document) group corresponding to the question. One information retrieval system (information retrieval apparatus) as described above is prepared, which is called evaluation data.

  From the question, morphological analysis is performed, nouns are extracted as keywords, and articles are extracted by the information search system using the keywords. Then, each article has a Score (D) value if it is an Okapi expression, and the one with this value is output.

a) Explanation of using kp value
In the case of a method using the value of kp, Score_max is the maximum value of Score (D) for a certain question. Then, it outputs up to an article (document) of Score_max * kp.

  The information retrieval system outputs an article (document) group with the evaluation data. Here, kp at which each article (document) is barely output is obtained. This method is as follows.

  The value obtained by dividing the score of an article (Score) by the score of the first article (the article with the highest score) is the kp at which that article is output. (Refer to equation (2), which depends on the definition of kp, and note that this method is different in the ranking method and other methods.) The score is obtained using equation (1) and the like.

  For each article of the output evaluation data, it is checked whether it is correct, and the correct answer rate at each kp is obtained. Furthermore, the average value of the correct answer rates of the articles of all the evaluation data (questions) corresponding to the same kp is obtained. Then, the correspondence table between kp and accuracy rate is completed.

  When a new information retrieval question comes in, an article is output by the information retrieval system. Find the kp at which each article is output. This method is the same as in the case of creating the correspondence table, and the value obtained by dividing the score of an article by the score of the first article (the article with the highest score) is the kp at which that article is output. (Refer to equation (2), which depends on the definition of kp, and note that this method is different in the ranking method and other methods.) The score is obtained using equation (1) and the like.

When kp of each article is obtained, the correct answer rate corresponding to each article is displayed based on the previous correspondence table. (Note that this accuracy rate is the accuracy rate of each F-term. The accuracy (accuracy rate), recall rate, F value, etc. for the F-term group up to that F-term are different. )
FIG. 26 is a flowchart of the correspondence table creation process. In the following, description will be made in accordance with steps S201 to S205 in FIG. 26 (see FIG. 7 for the certainty factor assigning apparatus, but an information retrieval system (apparatus) is used instead of the document classification apparatus in FIG. 7).

  S201: A large number of sets of questions and answers (here, information search questions and articles corresponding to the questions) are input in advance from the input unit 1 and stored in the storage means of the information search system.

  S202: The information search system performs a morphological analysis from one input question, extracts a noun as a keyword, and performs information search of the input article group using the keyword to search for an article. Take out.

  S203: The information search system outputs up to Score_max * kp document (article).

  S204: The correspondence table creation unit 11 obtains kp at which each article is output at the limit, checks whether it is correct for each article, and obtains the correct rate at each kp.

  S205: The correspondence table creation unit 11 searches the articles with the information search system for all the questions input in S201, obtains kp at which each article is output, and further, all patent documents corresponding to the same kp. Is calculated, and a correspondence table is created (the correspondence table is stored in the storage means 13).

  FIG. 27 is a flowchart of certainty factor assignment processing. Hereinafter, a description will be given according to processing S211 to S214 in FIG.

  S211: A new information search question is input from the input unit 1.

  S212: The information search system performs morphological analysis from the input question, extracts a noun as a keyword, performs an information search using the keyword, and extracts an article.

  S213: The certainty factor assigning unit 12 obtains kp at which each article is output at the last minute.

  S214: The certainty factor assigning unit 12 assigns each article a correct answer rate, which is the certainty factor corresponding to the kp obtained from the correspondence table of the storage unit 13, and outputs it from the output unit.

b) Explanation of using output order In the case of a method using output order, an information retrieval system is used. This system solves the problem of the evaluation data, checks whether the kj-th output article exists or is wrong, and calculates the correct answer rate of the kj-th output. Then, the correspondence table between kj and accuracy rate is completed.

  When a new patent comes in, the article is output by the previous method (patent information retrieval system). Then, kj from which each article is output is obtained. The order in which the F-term is output is kj. (This is due to the definition of kj. In other ways this part is different).

  If kj of each Fterm is obtained, the correct answer rate corresponding to each Fterm is displayed based on the above correspondence table.

  FIG. 28 is a correspondence table creation process flowchart. In the following, description will be made in accordance with steps S221 to S225 in FIG. 28 (see FIG. 7 for the certainty factor assigning apparatus, but an information retrieval system (apparatus) is used instead of the document classification apparatus in FIG. 7).

  S221: A large number of sets of questions and answers (here, information search questions and articles corresponding to the questions) are input in advance from the input unit 1 and stored in the storage means of the information search system.

  S222: The information search system performs a morphological analysis from one input question, extracts a noun as a keyword, performs information search of the input article group using the keyword, and searches for an article. Take out.

  S223: The information retrieval system outputs up to the rank kj of the extracted articles.

  S224: The correspondence table creation unit 11 checks whether the kj-th article is output or is incorrect, and obtains the accuracy rate of the kj-th output.

  S225: The correspondence table creation unit 11 obtains the accuracy rate of the kj-th output for all the questions input in S221, further obtains the average value of the accuracy rates of all articles corresponding to the same kj, and obtains the correspondence table. Create (correspondence table is stored in the storage means 13).

  FIG. 29 is a flowchart of certainty factor assignment processing. In the following, description will be made in accordance with steps S231 to S234 in FIG. 29 (see FIG. 7 for the certainty factor assigning apparatus, but an information retrieval system (apparatus) is used instead of the document classification apparatus in FIG. 7).

  S231: A new information search question is input from the input unit 1.

  S232: The information retrieval system performs morphological analysis from the input question, extracts a noun as a keyword, performs an information search using the keyword, and extracts an article.

  S233: The certainty factor assigning unit 12 obtains kj at which each article is barely output.

  S234: The certainty factor assigning unit 12 assigns to each article a correct answer rate that is a certainty factor corresponding to the kj obtained from the correspondence table of the storage unit 13, and outputs it from the output unit.

c) Explanation in the case of using the score (Score) In the case of the method of using the score, the information search system is used. This system solves the problem of evaluation data and evaluates each output article. For articles with a score of kl, check if the article is present and find the correct answer rate for kl. Find this for every kl. Then, the correspondence table between kl and accuracy rate is completed.

When a new information retrieval question comes in, an article is output by the information retrieval system. Here, kl at which each article is barely output is obtained. Then, the score of each article becomes kl. (It depends on the definition of kl. In other methods, this part is different.)
If the kl of each article is obtained, the correct answer rate corresponding to each article is displayed based on the previous correspondence table.

  FIG. 30 is a flowchart of the correspondence table creation process. Hereinafter, description will be made in accordance with steps S241 to S245 in FIG. 30 (see FIG. 7 for the certainty factor imparting apparatus, except that an information retrieval system (apparatus) is used instead of the document classification apparatus in FIG. 7).

  S241: A large number of sets of questions and answers (here, information search questions and articles corresponding to the questions) are input from the input unit 1 and stored in storage means of the information search system.

  S242: The information search system performs a morphological analysis from one input question, extracts a noun as a keyword, performs information search of the input article group using the keyword, and searches for an article. Take out.

  S243: The information search system outputs an article having a score of kl or higher.

  S244: The correspondence table creation unit 11 checks whether there is an article for an article whose score is kl, and obtains a correct answer rate in the case of kl.

  S245: The correspondence table creation unit 11 outputs articles for all the questions input in S241, calculates the average value of the correct answer rates corresponding to the same kl, and completes the correspondence table between kl and correct answer rates (correspondence The table is stored in the storage means 13).

  FIG. 31 is a flowchart showing the certainty factor giving process. In the following, description will be made in accordance with steps S251 to S254 in FIG. 31 (see FIG. 7 for the certainty factor assigning apparatus, but an information retrieval system (apparatus) is used instead of the document classification apparatus in FIG. 7).

  S251: A new information search question is input from the input unit 1.

  S252: The information search system performs morphological analysis from the input question, extracts a noun as a keyword, performs an information search using the keyword, and extracts an article.

  S253: The certainty factor assigning unit 12 obtains kl at which each article is output at the last minute.

  S254: The certainty factor assigning unit 12 assigns each article a correct answer rate that is the certainty factor corresponding to the obtained kl from the correspondence table of the storage unit 13, and outputs it from the output unit.

  The method of using kp, ranking, and score has been described above, but other methods that output in order can be used.

(3): Explanation of data interpolation and correction Assume that, for example, a correspondence table between kp and accuracy rate can be created by a method based on a table (correspondence table). After this, the correct rate for new input and kp of kp1 is needed from the table, but the value of kp1 is not on the table. Then, some kind of interpolation processing is required. In that case, the part of kp that is closest to kp1 in the table can be used instead of kp1, or the two rows of kp that sandwich kp1 in the table can be used. It is also possible to use the average of the accuracy rates of the two rows of data as the accuracy rate of kp1.

In addition, two rows of data kp2 and kp3 (kp2>kp> kp3) sandwiching kp1 in the table are used, and the accuracy rates pr2 and pr3 of the two rows of data are used [( kp -kp3) pr2 + (kp2-kp) pr3] / [(kp2-kp) + (kp -kp3)]
May be the correct answer rate. The accuracy rate corresponding to kp may be obtained by other complement processing.

  In addition, the correct answer rate corresponding to kp is obtained by an approximate expression obtained by single regression equation approximation, polynomial approximation, logarithmic approximation, exponential approximation or the like for a pair of kp and accuracy rate. (See, for example, “Time Series Analysis and Forecasting with Excel” (Ohm) Chapter 2 “Single Regression Analysis” Chapter 3 “Multiple Regression Analysis”). Further, correction processing other than the regression analysis approximation may be performed. Data interpolation and correction are the same for other data such as kl.

§4: Explanation in the case of using machine learning a) Detailed explanation of the machine learning method FIG. 32 is an explanatory diagram of the machine learning method. 32, the machine learning method includes teacher data storage means 21, solution-feature pair extraction means 22, machine learning means 23, learning result storage means 24, expression pair extraction means 25, feature extraction means 26, and solution estimation means 27. The output means 28 is provided.

  Here, a method of machine learning by the machine learning means 23 will be described. The machine learning method prepares many sets of problem-solution pairs, learns them, learns what kind of solution the problem becomes, and uses the learning result to create a new problem. This is a method that allows the solution to be estimated (for example, see the following references (1) to (3)).

Reference (1): Masaki Murata, Language Processing Based on Machine Learning, Faculty of Science and Engineering, Ryukoku University. Invited lecture. 2004.http: //www2.nict.go.jp/jt/a132/members/murata/ps/rk1-siryou.pdf
Reference (2): Japanese-English translation of tense aspect modality using support vector machine, Maki Murata, Ma Ao, Kiyotaka Uchimoto, Hitoshi Isahara, IEICE Society for Language Understanding and Communication NLC2000-78, 2001 Year.
Reference (3): CRL in the SENSEVAL2J dictionary task, Masaki Murata, Masao Uchiyama, Kiyotaka Uchimoto, Ma Aoi, Hitoshi Isahara, IEICE Language Understanding and Communication Study Group NLC2001-40, 2001.
In order to convey the problem situation to the machine, what kind of problem is required, features (elements constituting the problem with information used for analysis) are required. The problem is expressed by the feature. For example, in the problem of estimating the tense of Japanese sentence ending expressions, the problem: “He speaks.” --- If the solution “present” is given, an example of a feature is “He speaks.” . "" Speaking. "" Su "". "

  In other words, the machine learning method prepares many sets of feature set-solution pairs, performs learning, learns what kind of solution the feature set becomes, and uses the learning result. This is a method of extracting a set of features from a new problem and inferring a solution in the case of the feature.

  The machine learning means 23 uses techniques such as a k-nearest neighbor method, a simple Bayes method, a decision list method, a maximum entropy method, and a support vector machine method as a machine learning method.

  The k-nearest neighbor method is a method for obtaining a classification destination (solution) by using the k most similar cases instead of the most similar case, and by majority decision of the k cases. k is a predetermined integer number, and generally an odd number between 1 and 9 is used.

  The Simple Bayes method is a method of estimating the probability of each classification based on Bayes' theorem and determining the classification having the highest probability value as a classification destination.

  In the simple Bayes method, the probability of outputting the classification a in the context b is given by the following equation (11).

Here, the context b is a set of features f _j (εF, 1 ≦ j ≦ k) set in advance. p (b) is the appearance probability of the context b. Here, since it is independent of the classification a and is a constant, it is not calculated. P (a) (where P is a tilde at the top of p) and P (f _i | a) are the probabilities estimated from the teacher data, respectively, and the appearance probability of class a, and the feature f for class a means the probability of having _i . When the value obtained by performing maximum likelihood estimation as P (f _i | a) is used, the value often becomes zero, and it may be difficult to determine the classification destination because the value of Expression (12) is zero. Therefore, smoothing is performed. Here, what smoothed using the following formula | equation (13) is used.

Here, freq (f _i , a) means the number of cases having the feature f _i and the classification a, and freq (a) means the number of cases having the classification a.

  The decision list method uses features and combinations of classification destinations as rules, stores them in the list in a predetermined priority order, and when input to be detected is given, from the highest priority in the list This is a method in which input data is compared with the feature of the rule, and the classification destination of the rule having the same feature is set as the classification destination of the input.

In the decision list method, the probability value of each classification is obtained using only one of the features f _j (εF, 1 ≦ j ≦ k) set in advance as a context. The probability of outputting classification a in a context b is given by

p (a | b) = p (a | fmax) (14)
However, fmax is given by the following equation.

P (a _i | f _j ) (where P is a tilde at the top of p) is the rate of appearance of the classification a _i when the feature f _j is in the context.

In the maximum entropy method, when F is a set of features f _j (1 ≦ j ≦ k) set in advance, an expression (entropy) that satisfies the predetermined conditional expression (expression (16)) below ( In this method, the probability distribution p (a, b) when 17) is maximized is obtained, and the classification having the largest probability value is obtained from the probabilities of the respective classifications obtained according to the probability distribution.

However, A and B mean a set of classifications and contexts, and g _j (a, b) is a function that is 1 if the context b has a feature f _j and the classification is a, and is 0 otherwise. means. Further, P (a _i | f _j ) (where P is a tilde at the top of p) means the rate of appearance of (a, b) in the known data.

  Formula (16) is to obtain the expected value of the frequency of the output and feature pair by multiplying the probability p and the function g meaning the appearance of the pair of output and feature, and the expected value in the known data on the right side And the expected value calculated based on the probability distribution calculated on the left side is the constraint, and entropy maximization (smoothing of the probability distribution) is performed to determine the probability distribution of the output and the context. Details of the maximum entropy method are described in the following references (4) and (5).

Reference (4): Eric Sven Ristad, Maximum Entropy Modeling for Natural Language, (ACL / EACL Tutorial Program, Madrid, 1997
Reference (5): Eric Sven Ristad, Maximum Entropy Modeling Toolkit, Release 1.6beta, (http://www.mnemonic.com/software/memt,1998))
The support vector machine method is a method of classifying data composed of two classifications by dividing a space by a hyperplane.

  FIG. 33 is a conceptual diagram of margin maximization in the support vector machine method. In FIG. 33, a white circle means a positive example, a black circle means a negative example, a solid line means a hyperplane that divides the space, and a broken line means a surface that represents the boundary of the margin area. FIG. 33A is a conceptual diagram when the interval between the positive example and the negative example is narrow (small margin), and FIG. 33B is a conceptual diagram when the interval between the positive example and the negative example is wide (large margin). is there.

  At this time, if the two classifications consist of positive and negative examples, the larger the interval (margin) between the positive and negative examples in the learning data, the less likely it is to make an incorrect classification with open data. As shown in FIG. 33B, a hyperplane that maximizes this margin is obtained, and classification is performed using the hyperplane.

  Basically, it is as described above. Usually, an extension of the method that the training data may contain a small number of cases in the inner area of the margin, or an extension that makes the linear part of the hyperplane nonlinear ( The one with the introduction of the kernel function is used.

  This extended method is equivalent to classification using the following discriminant function, and the two classes can be discriminated depending on whether the output value of the discriminant function is positive or negative.

Where x is the context (set of features) to be identified, and x _i and y _j (i = 1,..., L, y _j ∈ {1, -1}) mean the context and classification destination of the learning data. And the function sgn is
sgn (x) = 1 (x ≧ 0)
-1 (otherwise)
In addition, each α _i is for maximizing Expression (19) under the constraints of Expression (20) and Expression (21).

  The function K is called a kernel function, and various functions are used. In this embodiment, the following polynomial is used.

K (x, y) = (x · y + 1) d (22)
C and d are constants set experimentally. For example, C was fixed at 1 throughout all treatments. Moreover, two types of 1 and 2 are tried for d. Here, x _i satisfying α _i > 0 is called a support vector, and the portion taking the sum of Expression (18) is usually calculated using only this case. That is, only actual cases called support vectors are used for actual analysis.

  Details of the extended support vector machine method are described in the following references (6) and (7).

Reference (6): Nello Cristianini and John Shawe-Taylor, An Introduction to Support Vector Machines and other kernel-based learning methods, (Cambridge University Press, 2000)
Reference (7): Taku Kudoh, Tinysvm: Support Vector machines, (http://cl.aist-nara.ac.jp/taku-ku//software/Tiny SVM / index.html, 2000)
The support vector machine method handles data with two classifications. Therefore, when handling cases with three or more classifications, a pair-wise method or a one-VS rest method is usually used in combination with this.

  In the pairwise method, in the case of data having n classifications, every pair (n (n-1) / 2) of two different classification destinations is generated, and a binary classifier indicates which is better for each pair. That is, it is obtained by the support vector machine method processing module and finally obtains the classification destination by majority decision of the classification destination by n (n−1) / 2 binary classification.

  For example, when there are three classification destinations a, b, and c, the one VS rest method generates three sets of classification destination a and other, classification destination b and other, classification destination c and other, The learning process is performed on the set of the support vector machine method. Then, in the estimation process based on the learning result, the learning results of the three sets of support vector machines are used. See how the three support vector machines are estimated as candidates to be estimated. Of the three support vector machines, it is the non-other classification target and the separation plane of the support vector machine. This is a method for obtaining a classification destination of a thing farthest from the object. For example, when a candidate is farthest from the separation plane in the support vector machine created by the learning process of “classification destination a and other”, the candidate classification destination is estimated as a.

  The method of obtaining the degree of the solution (classification destination) that is likely to be about each expression pair estimated by the solution estimation unit 27 differs depending on various methods used as a machine learning method by the machine learning unit 23.

  For example, in the embodiment of the present invention, when the machine learning means 23 uses the k-nearest neighbor method as a machine learning technique, the machine learning means 23 is a set of features extracted from the cases among the cases of the teacher data. The similarity between cases based on the ratio of overlapping features (the number of the same features) is defined, and the defined similarity and the case are stored in the learning result storage unit 24 as learning result information. Keep it.

  Then, the solution estimation means 27 refers to the similarity and the case defined in the learning result storage means 24 when a new expression pair (candidate) is extracted by the expression pair extraction means 25, and the expression pair extraction means 27 For the candidates of the expression pairs extracted by 25, k cases are selected from the cases in the learning result storage means 24 in descending order of the similarity of the candidates, and the classification destination determined by the majority decision in the selected k cases is selected. Estimated as the classification target (solution) of the expression pair candidate. That is, in the solution estimation means 27, the degree of what kind of solution (classification destination) is likely to be obtained for each expression pair is the number of votes of the majority vote in the selected k cases, here “to be extracted”. The number of votes obtained by classification.

  When the simple Bayes method is used as the machine learning method, the machine learning unit 23 stores, in the learning result storage unit 24, a set of the solution of the case and a set of features as learning result information for the case of the teacher data. Remember. Then, the solution estimation means 27, when a new expression pair (candidate) is extracted by the expression pair extraction means 25, based on the combination of the learning result information solution and the feature set in the learning result storage means 24. Based on the Bayes' theorem, the probability of each classification in the case of the feature pair of the expression pair candidates acquired by the feature extraction means 26 is calculated, and the classification having the largest probability value is selected as the candidate for the expression pair. It is estimated as the classification (solution) of the features of. That is, in the solution estimation means 27, the degree of ease of becoming a certain solution in the case of a set of features of expression pair candidates is set as the probability of each classification, in this case, the probability of “to be extracted”.

  Further, when the decision list method is used as the machine learning method, the machine learning unit 23 stores, in the learning result storage unit 24, a list in which rules of features and classification destinations are arranged in a predetermined priority order for the example of the teacher data. Remember. Then, when a new expression pair (candidate) is extracted by the expression pair extraction unit 15, the solution estimation unit 27 selects the extracted expression pair candidates in descending order of priority in the list of the learning result storage unit 24. The feature is compared with the feature of the rule, and the classification destination of the rule having the identical feature is estimated as the candidate classification destination (solution). That is, the solution estimation means 27 divides the degree of the likelihood of becoming a solution in the case of a set of candidate expression pairs into a predetermined priority or a numerical value or scale corresponding thereto, in this case, “to be extracted”. Priority in the list of probabilities.

  When the maximum entropy method is used as the machine learning method, the machine learning means 23 specifies a class that can be a solution from the example of the teacher data, and maximizes an expression that satisfies a predetermined conditional expression and shows entropy. Then, a probability distribution composed of a set of features and a binomial classification that can be a solution is obtained and stored in the learning result storage unit 24. Then, when a new expression pair (candidate) is extracted by the expression pair extraction unit 25, the solution estimation unit 27 uses the probability distribution of the learning result storage unit 24 to identify the feature of the extracted expression pair candidate. The probability of the classification that can be the solution for the set of is determined, the classification that can be the solution having the largest probability value is identified, and the identified classification is estimated as the candidate solution. That is, in the solution estimation means 27, the degree of ease of becoming a certain solution in the case of a set of features of expression pair candidates is set as the probability of each classification, in this case, the probability of “to be extracted”.

  When the support vector machine method is used as the machine learning method, the machine learning means 23 identifies a class that can be a solution from the example of the teacher data, divides the class into a positive example and a negative example, A hyperplane that maximizes the interval between the positive and negative examples of a case and divides the positive and negative examples by a hyperplane in a space whose dimension is a set of case features according to a predetermined execution function using And stored in the learning result storage means 24. When a new expression pair (candidate) is extracted by the expression pair extraction unit 25, the solution estimation unit 27 uses the hyperplane of the learning result storage unit 24 to identify the feature of the extracted expression pair candidate. Whether the set is on the positive example side or the negative example side in the space divided by the hyperplane is specified, and the classification determined based on the specified result is estimated as the candidate solution. That is, in the solution estimation means 27, the degree of easiness to be a solution in the case of a set of candidate expression pairs is set as the distance from the separation plane to the space of the positive example (expression pair to be extracted). . More specifically, when the expression pair to be extracted is a positive example and the expression pair that should not be extracted is a negative example, the case located in the space on the positive example side with respect to the separation plane is referred to as “example to be extracted”. The distance from the separation plane of the case is determined as the degree of the case.

b) Explanation when using machine learning (when using a document classification device)
In the case of the machine learning method using the certainty factor assigning device, a large number of sets of questions and answers are collected in advance. The problem is the patent that should use the F-term, and the answer is the F-term of that patent. This is called evaluation data.

  Here, an F-term is output with the evaluation data using the document classification device. Then, the kp at which each F-term is output is obtained. This method is as follows.

  The value obtained by dividing the score (Score) of an F-term by the score of the first F-term (the F-term with the highest score) is the kp at which that F-term is output. Also, the F-term rank kj is also obtained. Also, find the score of that F-term = kl. A score is calculated | required by said Formula (1) etc.

  Check if it is correct for each F-term. If the answer is correct, the answer is correct when kp, kj, and kl, and the answer is incorrect when kp, kj, and kl are not correct.

  Create the above example for each output F-term. Next, machine learning (machine learning means 23) is used. Machine learning is performed using learning data (solution-feature pair extraction means 22) as examples of correct answers when kp, kj, and kl and incorrect answers when kp, kj, and kl. Here, kp, kj, and kl are features. The correct answer and the incorrect answer are classification destinations to be obtained.

  By machine learning, it learns what kind of kp, kj, kl is correct, what kind of kp, kj, kl is likely to be incorrect, and stores it in the learning result (learning result storage means 24).

  Here comes a new patent (no F-term granted). F-term is output using the document classification device. Then, the kp at which each F-term is output is obtained. This method is as follows.

  The value obtained by dividing the score of an F-term by the score of the first F-term (the F-term with the highest score) is the kp at which that F-term is output. Also, the F-term rank kj is also obtained. Also find the score of that F-term = kl. A score is calculated | required by said Formula (1) etc.

  Based on the previous learning result, a certainty factor that is likely to be correct in the case of kp, kj, and kl at this time is obtained (solution estimation means 27). Here, machine learning (machine learning means) capable of outputting a certainty factor is used.

  This certainty factor is displayed as a correct answer rate corresponding to each F-term. (Note that this accuracy rate is the accuracy rate of each F-term. It is different from the accuracy (accuracy rate), recall rate, F value, etc. for the F-term group up to that F-term) .

  Here, kp, kj, and kl are used as machine learning features, but only some of them may be used as features, and conversely, other features may be added to this feature. A combination of items may be used as a feature.

  For example, using a word or character string included in a patent document group, a feature as to whether the word is included in the corresponding patent or a feature as to whether the character string is included in the corresponding patent may be used. .

c) Explanation when using machine learning (when using information retrieval system)
In the case of the machine learning method using the certainty factor assigning device, a large number of sets of questions and answers are collected in advance. The problem is an information retrieval question, and the answer is an article group corresponding to the question. This is called evaluation data.

  Here, an article is output with the evaluation data using the information retrieval system. Then, the kp at which each article is output is obtained. This method is as follows.

  The value obtained by dividing the score (Score) of an article by the score of the first article (the article with the highest score) is the kp at which the article is output. Also, the ranking kj of the article is obtained. Also find the score of the article = kl. A score is calculated | required by said Formula (1) etc.

  Check each article to see if it is correct. If the answer is correct, the answer is correct when kp, kj, and kl, and the answer is incorrect when kp, kj, and kl are not correct.

  Create the above case for each output article. Next, machine learning (machine learning means 23) is used. Machine learning is performed using learning data (solution-feature pair extraction means 22) as examples of correct answers when kp, kj, and kl and incorrect answers when kp, kj, and kl. Here, kp, kj, and kl are features. The correct answer and the incorrect answer are classification destinations to be obtained.

  By machine learning, it learns what kind of kp, kj, kl is correct, what kind of kp, kj, kl is likely to be incorrect, and stores it in the learning result (learning result storage means 24).

  Here comes the question for new information retrieval. An article is output using the information retrieval system. Then, the kp at which each article is output is obtained. This method is as follows.

  The value obtained by dividing the score of an article by the score of the first article (the article with the highest score) is the kp at which that article is output. Also, the ranking kj of the article is obtained. Also find the score = kl of the article. A score is calculated | required by said Formula (1) etc.

  Based on the previous learning result, a certainty factor that is likely to be correct in the case of kp, kj, and kl at this time is obtained (solution estimation means 27). Here, machine learning (machine learning means) capable of outputting a certainty factor is used.

  This certainty factor is displayed as a correct answer rate corresponding to each article. (Note that this correct answer rate is the correct answer rate of each article. It is different from the accuracy (accuracy rate), recall rate, F value, etc. for the article group up to that article).

  Here, kp, kj, and kl are used as machine learning features, but only some of them may be used as features, and conversely, other features may be added to this feature. A combination of items may be used as a feature.

  As described above, the reliability (accuracy rate) is output by machine learning in the case of assigning the classification and in the case of information retrieval. However, as this machine learning method, a neural network or multiple regression analysis may be used. An explanation of multiple regression analysis may be found in “Multiple regression analysis” in Chapter 3 of “Time Series Analysis and Forecasting with Excel” (Ohm). In the case of multiple regression analysis, “correct answer” may be obtained with value 1 and “incorrect answer” with value 0.

  That is, if there are two types of classification to be obtained, multiple regression analysis can be used. In the case of multiple regression analysis, as many explanatory variables x as the number of features are prepared, and the presence or absence of the features is expressed by 1 and 0 as the value of the explanatory variable x. The objective variable (explained variable) may be obtained with a value of 1 for a certain classification and a value of 0 for another classification.

(Explanation of the use of multiple regression analysis)
In the multiple regression analysis, y is obtained from x1, x2, x3 ,,, when there is a set of data of x1, x2, x3 ,,, y.

y = a0 + a1 * x1 + a2 * x2 + a3 * x3 + ...
The coefficients a0, a1, ... in this equation can be determined appropriately from the data.

(1) Prediction from kp-y pair
y is confidence
x1 = kp
As y = a0 + a1 * kp
Find a0 and a1 from the kp-y data by regression analysis. The formula for obtaining y from kp is obtained.

(2) Prediction from kp-y pairs (secondary use)
y is confidence
As x1 = kp and x2 = kp ^ 2,
y = a0 + a1 * kp + a2 * kp ^ 2
Find a0, a1, a2 from the kp-y set of data by multiple regression analysis. The formula for obtaining y from kp is obtained.

(3) Prediction from kp, kj-y pairs
y is confidence
x1 = kp, x2 = kj,
y = a0 + a1 * kp + a2 * kj
Obtain a0, a1, a2 from the data of kp and kj-y by multiple regression analysis. From kp and kj, an equation for obtaining y is obtained.

(4) Prediction from kp, kj-y pair (second-order use)
y is confidence
x1 = kp, x2 = kj,
y = a0 + a1 * kp + a2 * kj + a3 * kp ^ 2 + a4 * kp * kj + a5 * kj ^ 2
Obtain a0, a1, a2,..., A5 from the data of kp, kj-y pairs by multiple regression analysis. From kp and kj, an equation for obtaining y is obtained.

  These processes can be regarded as a method using multiple regression analysis as a machine learning method, and can also be regarded as using multiple regression analysis as an interpolation method.

(Explanation of the use of machine learning and multiple regression analysis)
When using machine learning and multiple regression analysis, kp-y set data, kp, kj-y set data, and the like are used.

  At this time, for the same kp, the average of y may be obtained, and data having only one y value for each kp may be created and used as the data of the set kp−y. In this case, the data is an average set of kp-y.

  As another method, it is also possible to use all the original data itself for the same kp, and use it as the data of the kp-y pair. That is, the number of data in the kp-y set may be as many as the original number of problems without performing the operation of averaging the certainty factors.

(Explanation of confidence)
As the certainty factor used in the above description, a deviation value of precision, a deviation value of recall, a deviation value of F value, and a deviation value of accuracy rate may be used. Moreover, you may be similar to these. Anything other than these may be used as long as it is calculated numerically.

  For taking out items with large values, “take out items with a value greater than or equal to the threshold”, “take out items with a larger value than the number of predetermined values in descending order”, “maximum value of the taken out values The value obtained by multiplying a predetermined ratio with respect to the value obtained by taking out a value having a value equal to or greater than the obtained value can be used. These threshold values and predetermined values can be set in advance, or the user can change and set the values as appropriate.

  In addition, when an inputted problem is solved and a plurality of answers are extracted and the extracted answer and a predetermined value are output, the predetermined value is the value of the answer like kp, kj, kl described above. The order is the same as the ordering (or the reverse order) (except when a predetermined value from a plurality of viewpoints is used).

§5: Description of experimental results Next, the results of actual experiments will be described. Data from NTCIR-5 Patent classification task was used. Here, there were 1201 patent documents to be classified. And this patent document was divided as follows.

close… 600
open… 601
Here, using the close data, a correspondence table is obtained and the certainty level is predicted. Then, using open data, the validity of the predicted confidence level is confirmed. A table of experimental results is shown in FIG.

  FIG. 34 is an explanatory diagram of the experimental results. In FIG. 34, the values in the table indicate the absolute value (absolute error) of the difference between the true value and the value predicted by the present invention. In FIG. 34, four methods are tried. The certainty factor (reproduction rate, relevance rate, F value) here is a certainty factor (reproduction rate, relevance rate, F value) when up to kjth F terms are output. In other words, it is not the reliability of the kj-th F-term but the reliability of the F-term up to the kj-th.

  (1) base0.5 --- Method to set all the confidence levels to 0.5.

  (2) kp --- A method of predicting by obtaining a correspondence table between kp and certainty factor (here, a correspondence table in the case of values of kp = 0, 0.1, 0.2, ..., 1.0 is obtained).

  (3) kj --- A method of predicting by obtaining a correspondence table between kj and confidence (here, a correspondence table for kj = 1, 2, 3, ..., 200 is obtained).

  (4) kp, kj --- A method for predicting by obtaining a correspondence table of kp, kj and certainty (here, kp = 0, 0.1, 0.2, ..., 1.0 and kj = 1, 2, The correspondence table was obtained for all combinations of the values 3, ..., 200).

  No interpolation processing is necessary for kj. Interpolation processing was performed for kp. This interpolation processing was performed using the following formula already described.

[(Kp -kp3) pr2 + (kp2-kp) pr3] / [(kp2-kp) + (kp -kp3)]
In the table of FIG. 34, kj represents how many F-terms of the system output indicate the result. For example, if kj = 1, the result for the first F-term of the system output is shown. The values in the table of FIG. 34 are written as the absolute value (absolute error) of the difference between the true value and the value predicted by the present invention. To be precise, for each article, the value is the kjth F-term. The absolute value (absolute error) of the difference between the true value (confidence) and the value predicted by the present invention (confidence) was obtained, added, and divided by the total number of articles. That is, the values in the table are the average of absolute errors.

  In the table of FIG. 34, the errors of the other methods are generally much smaller than base0.5. For this reason, the effectiveness of the present invention is understood. Also, compared to using kp and kj alone, the one using both kp and kj has a small error. In addition, kp, kj, kp, and kj methods have low kj, and the upper output has a large error of precision of around 0.2, but excluding it, the error is around 0.1, which is a fairly good prediction. It can be seen that is realized.

§6: Description of program installation Input unit (input unit) 1, document extraction unit (document extraction unit) 2, KDOC extraction unit (KDOC extraction unit) 2, document similarity calculation unit (document similarity calculation unit) 3, score Calculation unit (score calculation unit) 4, score (score _M1 (x)) calculation unit 4, classification set extraction unit (classification set extraction unit) 5, F-term x set extraction unit (F-term x set extraction unit) ) 5, output unit (output unit) 6, document classification device (document classification unit) 10, correspondence table creation unit (correspondence relationship creation unit) 11, confidence factor assignment unit (confidence factor provision unit 12, storage unit (correspondence table)) 13, teacher data storage means 21, solution-feature pair extraction means 22, machine learning means 23, learning result storage means 24, expression pair extraction means 25, feature extraction means 26, solution estimation means 27, output means 28, problem solving means Information retrieval system (apparatus) is a program. This program is executed by the main control unit (CPU) and stored in the main memory, and is processed by a general computer (information processing apparatus). Is composed of hardware such as an input device which is an input means such as a main control unit, main memory, file device, display device, and keyboard.

  The program of the present invention is installed on this computer. In this installation, these programs are stored in a portable recording (storage) medium such as a floppy disk or a magneto-optical disk, and a drive device for accessing the recording medium provided in the computer is used. Alternatively, it is installed in a file device provided in the computer via a network such as a LAN. Then, the program steps necessary for processing are read from the file device into the main memory and executed by the main control unit.

It is explanatory drawing of the document classification device of this invention. It is explanatory drawing of the patent document classification | category apparatus of this invention. It is a classification process flowchart of the patent document of this invention. It is a processing flowchart which calculates | requires the similarity between the input patent document of this invention, and the selected patent document. It is explanatory drawing of a response | compatibility of kp and F value of this invention. It is explanatory drawing of a response | compatibility of kp of this invention, recall, and precision. It is explanatory drawing of the reliability provision apparatus of this invention. It is a correspondence table creation process flowchart of this invention. It is a certainty degree provision process flowchart of this invention. It is a correspondence table creation process flowchart of this invention. It is a certainty degree provision process flowchart of this invention. It is a correspondence table creation process flowchart of this invention. It is a certainty degree provision process flowchart of this invention. It is a correspondence table creation process flowchart of this invention. It is a certainty degree provision process flowchart of this invention. It is a correspondence table creation process flowchart of this invention. It is a certainty degree provision process flowchart of this invention. It is a correspondence table creation process flowchart of this invention. It is a certainty degree provision process flowchart of this invention. It is a correspondence table creation process flowchart of this invention. It is a certainty degree provision process flowchart of this invention. It is a correspondence table creation process flowchart of this invention. It is a certainty degree provision process flowchart of this invention. It is a correspondence table creation process flowchart of this invention. It is a certainty degree provision process flowchart of this invention. It is a correspondence table creation process flowchart of this invention. It is a certainty degree provision process flowchart of this invention. It is a correspondence table creation process flowchart of this invention. It is a certainty degree provision process flowchart of this invention. It is a correspondence table creation process flowchart of this invention. It is a certainty degree provision process flowchart of this invention. It is explanatory drawing of the machine learning method of this invention. It is a conceptual diagram of margin maximization of the support vector machine method of this invention. It is explanatory drawing of the experimental result of this invention.

Explanation of symbols

1 Input section (input means)
6 Output section (output means)
10 Document classification device (problem solving means)
11 Correspondence table creation part (correspondence creation means)
12 certainty degree granting part (confidence degree giving means)
13 Storage (correspondence table)

Claims (15)

An input means for entering the problem;
Problem solving means for solving the inputted problem and extracting a plurality of the answers, and outputting the extracted answer and a predetermined value;
Preparing a plurality of questions to which an answer has been given in advance, and inputting each of the questions to the problem solving means and outputting each answer, obtaining the predetermined value, the answer, and the certainty of the answer, A correspondence creation means for creating a correspondence between a predetermined value and a certainty factor;
When a new question is input from the input means and the answers are ordered by the problem solving means and output, the predetermined value from which a certain answer is output is obtained, and the certainty of the certain answer is given from the correspondence relationship and output. A certainty degree granting device characterized by comprising a certainty degree assigning means.   2. The certainty factor imparting apparatus according to claim 1, wherein an accuracy rate that is a ratio of correct outputs among all outputs is used as the certainty factor.   2. The certainty factor assigning apparatus according to claim 1, wherein a recall rate, which is a ratio of correct answer among the number of correct answers, is used as the certainty factor.   2. The certainty factor imparting apparatus according to claim 1, wherein an F value that is the inverse of the average of the reciprocal of the recall and the reciprocal of the precision is used as the certainty.   The certainty degree granting apparatus according to any one of claims 1 to 4, wherein the number that is given and output by the certainty degree assigning means is a number that maximizes the F value.   2. The certainty factor assigning apparatus according to claim 1, wherein a correct answer rate of each answer is used as the certainty factor. When a plurality of questions to which an answer has been assigned in advance are prepared, and each of the questions is input to the problem solving means and each answer is output, the predetermined value from which the answer is output is obtained, and the limit output is performed. A machine learning means for investigating whether the answer to be done is correct and obtaining the correct answer rate at the predetermined value, and storing a learning result by machine learning a case of a correct answer or an incorrect answer if it is a predetermined value,
7. The certainty factor assigning apparatus according to claim 6, wherein the certainty factor assigning unit uses the learning result as the correspondence relationship.   The predetermined value of a plurality of viewpoints is used as the predetermined value, and the correspondence between the predetermined values of the plurality of viewpoints and the certainty factor is created by the correspondence creation unit. Confidence giving device.   9. The certainty provision according to claim 1, wherein the problem solving means is a document classification device, the problem is a document to which classification is given, and the answer is classification of the document. apparatus.   The said problem solving means is an information search device, the said problem is a document of a question, and the said answer is a document searched from the document of the said question, The claim 1 characterized by the above-mentioned. Confidence giving device.   11. When the problem solving means obtains a score and outputs an answer, a value (kp) obtained by dividing the score of a certain answer by the score of the first answer is used as the predetermined value. The certainty degree grant apparatus in any one of.   11. The certainty degree granting apparatus according to claim 1, wherein an output rank (kj) is used as the predetermined value when the answer is output by the problem solving means.   11. The certainty imparting apparatus according to claim 1, wherein when the problem solving means obtains a score and outputs an answer, a score (kl) is used as the predetermined value. Enter the problem from the input means,
The problem solving means solves the inputted problem and extracts a plurality of answers, and outputs the extracted answer and a predetermined value for ordering the answers,
When preparing a plurality of questions to which answers are assigned in advance by the correspondence creating means, and inputting each question to the problem solving means and outputting each answer, the predetermined value and the answer are output, and the same An average of the certainty values of the predetermined values and the output answers, and a correspondence relationship creating means for creating a correspondence relationship between the predetermined value and the certainty factors,
When a new question is input from the input means by the certainty degree giving means and the answers are ordered by the problem solving means and outputted, the predetermined value from which a certain answer is output is obtained, and the certainty of the certain answer from the correspondence relation A certainty factor assigning method characterized by giving and outputting. An input means for entering the problem;
Problem solving means for solving the input problem, extracting a plurality of the answers, and outputting the extracted answers and a predetermined value for ordering the answers;
When preparing a plurality of questions to which an answer has been assigned in advance and inputting each of the questions to the problem solving means and outputting each answer, the predetermined value and the answer are output, and the same predetermined value and the An average of the certainty of each output answer, a correspondence creating means for creating a correspondence between the predetermined value and the certainty,
When a new question is input from the input means and the answers are ordered by the problem solving means and output, the predetermined value from which a certain answer is output is obtained, and the certainty of the certain answer is given from the correspondence relationship and output. As a means to give confidence
A program that allows a computer to function.

Images