JP7620866B2 - Algorithm determination device, learning device production device, patent classification device, classification information determination device, algorithm determination method, learning device production method, classification information determination method, and program - Google Patents

Description

The present invention relates to a patent classification device that classifies patent information.

There has been a patent classification device that reduces the burden on users when classifying documents such as patents (see Patent Document 1). This patent classification device is characterized by having a classification registration unit that stores training data in a storage unit that associates a document classification with the document, and a classification assignment unit that determines a classification to be assigned to a target document based on a classification of similar documents obtained based on the similarity between the training data stored in the storage unit and the target document to be assigned the classification.

JP 2019-87258 A

However, in conventional technology, the same algorithm was used to classify patents, so patents could not be classified with high accuracy.

In addition, in the conventional technology, the classification of each document to be annotated was determined based on the classification of similar documents obtained based on the similarity between the training data and the document to be annotated, so patents could not be classified with high accuracy.

In addition, in the prior art, patents were classified using the entire document of the patent specification, which contains a lot of information other than the key points of the invention, such as well-known technical configurations and prior art, so patents could not be classified with a high degree of accuracy.

The algorithm determination device of the first invention includes a program storage unit that stores one or more learning device generation programs for generating a learning device for classifying patent information relating to a patent using learning source information having two or more patent classification information having patent information relating to a patent and classification information indicating a classification of the patent information, and a prediction program that predicts the classification of the patent classification information using the learning device and obtains the predicted classification information, a learning source information receiving unit that receives the learning source information, a learning unit that executes each of the one or more learning device generation programs using at least a portion of the patent classification information of the learning source information received by the learning source information receiving unit, and obtains one or more learning devices, and a prediction program that predicts the classification of the patent classification information using the learning device and obtains the predicted classification information. The algorithm determination device includes a prediction unit that applies a prediction program to each of at least a portion of the patent classification information in the learning source information using one or more learning devices generated by the learning unit to obtain two or more predicted classification information for each of the one or more patent classification information; an evaluation unit that uses each of the two or more predicted classification information obtained by the prediction unit and classification information contained in the learning source information to obtain accuracy information regarding the accuracy of each of the two or more predicted classification information, and determines the predicted classification information with the highest accuracy specified by the accuracy information; an algorithm information acquisition unit that acquires algorithm information regarding an algorithm corresponding to the predicted classification information determined by the evaluation unit; and an output unit that outputs the algorithm information.

This configuration makes it possible to determine the appropriate processing for performing prediction processing.

The algorithm determination device of the second invention further includes a learning source information acquisition unit that acquires two or more pieces of learning source information each having two or more pieces of patent classification information having different information from the patent information contained in the two or more pieces of patent classification information contained in the learning source information accepted by the learning source information acceptance unit, the learning unit executes one or more learning device generation programs for each of the two or more pieces of learning source information acquired by the learning source information acquisition unit to acquire two or more learning devices, the prediction unit applies a prediction program to at least some of the patent classification information of the learning source information accepted by the learning source information acceptance unit using each of the two or more learning devices generated by the learning unit to acquire two or more pieces of predicted classification information for each of the one or more pieces of patent classification information, and the algorithm information acquisition unit acquires algorithm information including information identifying information identifying information contained in the learning source information corresponding to the predicted classification information determined by the evaluation unit.

With this configuration, the data used when performing the prediction process can be changed to determine the appropriate process for performing the prediction process.

The algorithm determination device of the third invention is an algorithm determination device according to the first or second invention, in which the program storage unit stores two or more learning device generation programs and two or more prediction programs, the learning unit executes each of the two or more learning device generation programs using at least a portion of the patent classification information of the learning source information accepted by the learning source information acceptance unit to obtain two or more learning devices, the prediction unit applies the prediction program to each of at least a portion of the patent classification information of the learning source information accepted by the learning source information acceptance unit using each of the two or more learning devices generated by the learning unit to obtain two or more predicted classification information for each of the one or more patent classification information, and the algorithm information acquisition unit acquires algorithm information including program identification information that identifies the learning device generation program corresponding to the predicted classification information determined by the evaluation unit.

With this configuration, multiple learning device generation programs can be used to determine the appropriate process for performing prediction processing.

The algorithm determination device of the fourth invention further includes a statistical processing unit that performs one or more types of statistical processing using two or more pieces of prediction classification information acquired by the prediction unit and acquires one or more pieces of prediction classification information that are the results of the statistical processing, and the evaluation unit acquires accuracy information regarding the accuracy of each of the two or more pieces of prediction classification information using two or more pieces of prediction classification information including one or more pieces of prediction classification information acquired by the statistical processing unit and classification information contained in the learning source information, instead of or in addition to the two or more pieces of prediction classification information acquired by the prediction unit, and determines the prediction classification information with the highest accuracy specified by the accuracy information, and the algorithm information acquisition unit acquires algorithm information including statistical processing specification information that specifies the method of statistical processing corresponding to the prediction classification information determined by the evaluation unit.

With this configuration, it is possible to determine an appropriate process for performing prediction processing by using prediction results obtained by performing statistical processing on prediction results obtained using two or more learning devices.

The algorithm determination device of the fifth invention is an algorithm determination device according to the fourth invention, in which the statistical processing unit performs two or more types of statistical processing, obtains two or more pieces of predicted classification information that are the results of the statistical processing, and the two or more types of statistical processing are two or more types of statistical processing that are either a logical sum, a logical product, or a majority vote of the predicted classification information.

With this configuration, by using two or more prediction results obtained by performing two or more statistical processes on prediction results obtained using two or more learning devices, it is possible to determine processing including appropriate statistical processing for performing prediction processing.

The algorithm determination device of the sixth invention further includes, for any one of the first to fifth inventions, an accuracy specification information storage unit that stores accuracy specification information that specifies which of two or more of the four accuracies, i.e., recall rate, precision rate, F-value, or accuracy rate, the accuracy information to be obtained, and the evaluation unit uses each of the two or more prediction classification information acquired by the prediction unit and the classification information contained in the learning source information to obtain accuracy information specified by the accuracy specification information for each of the two or more prediction classification information, and determines the prediction classification information with the highest accuracy specified by the accuracy information.

This configuration allows the user to determine the process that has the highest score desired, among recall, precision, F-measure, or accuracy rate.

The learning device production device of the seventh invention is a learning device production device that includes a learning source information receiving unit that receives learning source information, a learning unit that performs a learning process on the learning source information received by the learning source information receiving unit using an algorithm according to the algorithm information output by the algorithm determination device of any one of the first to sixth inventions to obtain a learning device, and a storage unit that stores the learning device in association with the algorithm information.

This configuration allows for the production of appropriate learning machines.

The patent classification device of the eighth invention is a patent classification device that includes a learning device storage unit in which the learning devices accumulated by the production device of the seventh invention are stored, a reception unit that receives classification target information including a patent identifier that identifies the patent to be classified, a prediction unit that uses the learning device to perform a prediction process on the classification target information using an algorithm according to the algorithm information associated with the learning device and obtains classification result information, and an output unit that outputs the classification result information obtained by the prediction unit in association with the patent identifier included in the classification target information received by the reception unit.

This configuration allows for highly accurate prediction processing using an appropriate learning device.

The patent classification device of the ninth invention is the patent classification device of the eighth invention, further comprising an important information acquisition unit that acquires important information that satisfies a predetermined condition using a patent specification that corresponds to a patent identifier included in the classification target information accepted by the acceptance unit, and a classification target processed information acquisition unit that acquires classification target processed information that corresponds to the patent identifier included in the classification target information accepted by the acceptance unit and includes the important information acquired by the important information acquisition unit, and the prediction unit applies a learning device to the classification target processed information acquired by the classification target processed information acquisition unit, predicts the classification of the classification target processed information using a machine learning algorithm, and acquires classification result information.

This configuration allows for accurate classification of patents by using important information about patent specifications.

The patent classification device of the tenth invention is a patent classification device in which, compared to the ninth invention, the patent information also includes one or more types of information from the scope of claims or abstract corresponding to the patent identifier, and the processed information to be classified includes the important information acquired by the important information acquisition unit and one or more types of information from the scope of claims or abstract corresponding to the patent identifier.

This configuration allows patents to be classified with high accuracy by using the claims and/or abstracts.

The patent classification device of the eleventh invention is a patent classification device according to the ninth or tenth invention, in which the important information acquisition unit determines whether or not each of one or more sentences included in a patent specification corresponding to a patent identifier included in the classification target information received by the reception unit is an effect-solution-means sentence including both an effect expression and a solution expression, and the important information includes one or more effect-solution-means sentences acquired by the effect-solution-means sentence acquisition means, or effect terms and solution terms that are terms acquired from one or more effect-solution-means sentences.

With this configuration, patents can be classified with high accuracy by using effect/solution sentences or effect terms or solution terms.

The patent classification device of the twelfth invention is a patent classification device according to any one of the ninth to eleventh inventions, in which the important information acquisition unit is provided with an effect sentence acquisition means for determining whether or not each of one or more sentences included in a patent specification corresponding to a patent identifier included in the classification target information received by the reception unit is an effect sentence including an expression of an effect, and acquiring one or more effect sentences determined to be effect sentences, and the important information includes one or more effect sentences acquired by the effect sentence acquisition means, or effect terms that are terms acquired from one or more effect sentences.

With this configuration, patents can be classified with high accuracy by using effect sentences or effect terms.

The patent classification device of the thirteenth invention is a patent classification device according to any one of the ninth to twelfth inventions, in which the important information acquisition unit determines whether or not each of one or more sentences included in a patent specification corresponding to a patent identifier included in the classification target information received by the reception unit is a solution sentence including both a solution expression and a solution term, and acquires one or more solution sentences determined to be solution sentences, and the important information includes one or more solution sentences acquired by the solution sentence acquisition means, or solution terms that are terms acquired from one or more solution sentences.

This configuration allows patents to be classified with high accuracy by using solution sentences or solution terms.

The patent classification device of the fourteenth invention is a patent classification device according to any one of the ninth to thirteenth inventions, in which the important information acquisition unit is provided with an object sentence acquisition means for determining whether or not each of one or more sentences included in a patent specification corresponding to a patent identifier included in the classification target information received by the reception unit is an object sentence including an objective expression, and acquiring one or more object sentences determined to be object sentences, and the important information includes one or more object sentences acquired by the object sentence acquisition means, or object terms that are terms acquired from one or more object sentences.

With this configuration, patents can be classified with high accuracy by using the object sentence or object term.

The patent classification device of the fifteenth invention is a patent classification device according to any one of the ninth to fourteenth inventions, in which the important information acquisition unit acquires the frequency of occurrence of one or more terms included in a patent specification corresponding to a patent identifier included in the classification target information received by the reception unit, calculates a score for each of the one or more terms using the frequency of occurrence, and acquires one or more characteristic words whose scores are large enough to satisfy a predetermined condition, and the important information includes one or more characteristic words acquired by the characteristic word acquisition means.

With this configuration, patents can be classified with high accuracy by using characteristic words.

The patent classification device of the sixteenth invention is a patent classification device according to any one of the ninth to fourteenth inventions, the important information acquisition unit being equipped with a characteristic word acquisition means for acquiring the frequency of occurrence of one or more terms included in a patent specification corresponding to a patent identifier included in the classification target information received by the reception unit, calculating a score for each of the one or more terms using the frequency of occurrence, and acquiring one or more characteristic words whose score is large enough to satisfy a predetermined condition, and a related word acquisition means for acquiring one or more related words that are hypernyms, hyponyms, or synonyms corresponding to each of the one or more characteristic words acquired by the characteristic word acquisition means from a related word dictionary in which one or more related words among hypernyms, hyponyms, or synonyms for each of the one or more terms are stored, and the important information includes one or more related words acquired by the related word acquisition means.

With this configuration, patents can be classified with high accuracy by using related words of characteristic words.

The patent classification device of the seventeenth invention is a patent classification device according to any one of the ninth to sixteenth inventions, wherein the important information acquisition unit is equipped with an important tag corresponding sentence acquisition means for acquiring one or more important tag corresponding sentences which are sentences included in a patent specification corresponding to a patent identifier included in the classification target information accepted by the acceptance unit and correspond to a predetermined tag, and the important information includes one or more important tag corresponding sentences acquired by the important tag corresponding sentence acquisition means, or includes important tag corresponding terms which are terms acquired from the one or more important tag corresponding sentences.

With this configuration, patents can be classified with high accuracy by using important tag-corresponding sentences or important tag-corresponding terms.

The patent classification device of the eighteenth invention is a patent classification device according to any one of the ninth to seventeenth inventions, wherein the important information acquisition unit includes a numeric range string acquisition means for acquiring one or more numeric range strings, which are strings included in a patent specification corresponding to a patent identifier included in the classification target information accepted by the acceptance unit and indicate a numeric range, and the important information includes the one or more numeric range strings acquired by the numeric range string acquisition means.

With this configuration, patents can be classified with high accuracy by using numeric range strings.

The patent classification device of the 19th invention is a patent classification device according to any one of the 9th to 18th inventions, wherein the important information acquisition unit includes a numeric range string acquisition means for acquiring one or more numeric range strings, which are strings included in a patent specification corresponding to a patent identifier included in the classification target information received by the reception unit and indicate a numeric range, and a number acquisition means for acquiring three or more numbers included in the numeric range indicated by the numeric range string acquired by the numeric range string acquisition means, and the important information includes the three or more numbers acquired by the number acquisition means.

With this configuration, patents can be classified with high accuracy by using three or more numbers included in the numerical range indicated by the numerical range string.

The patent classification device of the twentieth invention, for any one of the ninth to nineteenth inventions, further includes a patent specification acquisition unit that acquires from an external server device a patent specification corresponding to a patent identifier included in the classification target information accepted by the acceptance unit, and the important information acquisition unit is a patent classification device that acquires important information that satisfies a predetermined condition using the patent specification acquired by the patent specification acquisition unit.

With this configuration, patents can be classified with high accuracy by using important information obtained from patent specifications obtained from an external server device.

The classification information determination device of the twenty-first invention is a classification information determination device that performs machine learning learning processing using learning source information having two or more patent classification information including patent information and classification information of the patent information, and is equipped with a learning device storage unit in which the acquired learning device is stored, a condition information storage unit in which one or more condition information, which is a pair of a condition related to a patent and classification information of the patent information, a patent information reception unit that receives one or more patent information, a prediction unit that acquires predicted classification information that identifies classification information of one or more pieces of patent information using one or more condition information, and an output unit that outputs the predicted classification information, and the prediction unit is equipped with a judgment means that judges whether each of the one or more pieces of patent information matches any of the conditions held by the one or more pieces of condition information, a first decision means that decides classification information that pairs with the condition that the judgment means judges to match, and a second decision means that uses the learning device to acquire classification information of patent information that the judgment means judges not to match any of the conditions through machine learning prediction processing.

This configuration allows patents to be classified with high accuracy.

The classification information determination device of the twenty-second invention is a classification information determination device according to the twenty-first invention, in which one condition is that the patent information constitutes any one of two or more patent classification information contained in the learning source information.

With this configuration, patents from the training data used to obtain the machine learning learning device can be classified with high accuracy, and other patents can also be classified with high accuracy through prediction processing using the learning device.

The classification information determination device of the twenty-third invention is different from the twenty-first invention in that one condition is a search formula.

With this configuration, patents that match the search query can be classified with high accuracy, and even patents that do not match the search query can be classified with high accuracy through prediction processing using a machine learning learner.

The patent classification device of the present invention is a patent classification device that includes: a learning device storage unit that uses a machine learning algorithm to learn learning source information having two or more patent records that are patent records that do not have the full text of the patent specification, and that have important information acquired using a patent specification corresponding to a patent identifier that identifies a patent, and classification result information that identifies a manual classification result of the patent; a reception unit that receives classification target information including a patent identifier that identifies the patent to be classified; an important information acquisition unit that acquires important information that satisfies a predetermined condition using a patent specification that corresponds to the patent identifier included in the classification target information accepted by the reception unit; a classification target processing information acquisition unit that acquires classification target processing information that is information corresponding to the patent identifier included in the classification target information accepted by the reception unit and that includes the important information acquired by the important information acquisition unit; a prediction unit that applies a learning device to the classification target processing information acquired by the classification target processing information acquisition unit, predicts the classification of the classification target processing information using a machine learning algorithm, and acquires classification result information; and an output unit that outputs the classification result information acquired by the prediction unit in association with the patent identifier included in the classification target information accepted by the reception unit.

This configuration (first configuration) allows patents to be classified with high accuracy by using important information about patent specifications.

The patent classification device of the present invention is a patent classification device in which, compared to the first configuration, the patent record also includes one or more types of information of the claims or abstract corresponding to the patent identifier, and the processed information to be classified includes the important information acquired by the important information acquisition unit and one or more types of information of the claims or abstract corresponding to the patent identifier.

This configuration (second configuration) allows patents to be classified with high accuracy by using the claims and/or abstracts.

In addition, in the patent classification device of the present invention, in the first or second configuration, the important information acquisition unit is equipped with an effect-solution-means sentence acquisition means for determining whether or not each of one or more sentences included in a patent specification corresponding to a patent identifier included in the classification target information received by the reception unit is an effect-solution-means sentence including both an expression of an effect and an expression of a solution, and acquiring one or more effect-solution-means sentences determined to be effect-solution-means sentences, and the important information includes one or more effect-solution-means sentences acquired by the effect-solution-means sentence acquisition means, or effect terms and solution terms that are terms acquired from one or more effect-solution-means sentences.

With this configuration (third configuration), patents can be classified with high accuracy by using effect/solution sentences or effect terms or solution terms.

In addition, the patent classification device of the present invention, in any one of the first to third configurations, is a patent classification device in which the important information acquisition unit is equipped with an effect sentence acquisition means that determines whether or not each of one or more sentences included in a patent specification corresponding to a patent identifier included in the classification target information received by the reception unit is an effect sentence including an expression of an effect, and acquires one or more effect sentences that are determined to be effect sentences, and the important information includes one or more effect sentences acquired by the effect sentence acquisition means, or effect terms that are terms acquired from one or more effect sentences.

This configuration (fourth configuration) allows patents to be classified with high accuracy by using effect statements or effect terms.

In addition, the patent classification device of the present invention, in any one of the first to fourth configurations, is a patent classification device in which the important information acquisition unit is equipped with a solution sentence acquisition means for determining whether or not each of one or more sentences included in a patent specification corresponding to a patent identifier included in the classification target information received by the reception unit is a solution sentence including both a solution expression and a solution term, and acquiring one or more solution sentences determined to be solution sentences, and the important information includes one or more solution sentences acquired by the solution sentence acquisition means, or solution terms that are terms acquired from one or more solution sentences.

With this configuration (fifth configuration), patents can be classified with high accuracy by using solution sentences or solution terms.

In addition, the patent classification device of the present invention, in any one of the first to fifth configurations, is a patent classification device in which the important information acquisition unit is equipped with an object sentence acquisition means for determining whether or not each of one or more sentences included in a patent specification corresponding to a patent identifier included in the classification target information received by the reception unit is an object sentence including a target expression, and acquiring one or more object sentences determined to be object sentences, and the important information includes one or more object sentences acquired by the object sentence acquisition means, or object terms that are terms acquired from one or more object sentences.

This configuration (sixth configuration) allows patents to be classified with high accuracy by using object sentences or object terms.

The patent classification device of the present invention, in any one of the first to sixth configurations, is a patent classification device, in which the important information acquisition unit acquires the frequency of occurrence of one or more terms included in a patent specification corresponding to a patent identifier included in the classification target information received by the reception unit, calculates a score for each of the one or more terms using the frequency of occurrence, and includes a characteristic word acquisition means for acquiring one or more characteristic words whose scores are large enough to satisfy a predetermined condition, and the important information includes one or more characteristic words acquired by the characteristic word acquisition means.

With this configuration (seventh configuration), patents can be classified with high accuracy by using characteristic words.

In addition, the patent classification device of the present invention, in any one of the first to sixth configurations, is a patent classification device, in which the important information acquisition unit acquires the frequency of occurrence of one or more terms included in the patent specification corresponding to the patent identifier included in the classification target information accepted by the acceptance unit, calculates a score for each of the one or more terms using the frequency of occurrence, and acquires one or more characteristic words whose score is large enough to satisfy a predetermined condition, and related word acquisition means acquires one or more related words that are hypernyms, hyponyms, or synonyms corresponding to each of the one or more characteristic words acquired by the characteristic word acquisition means from a related word dictionary in which one or more related words among hypernyms, hyponyms, or synonyms for each of the one or more terms are stored, and the important information includes one or more related words acquired by the related word acquisition means.

With this configuration (the eighth configuration), patents can be classified with high accuracy by using related words of characteristic words.

The patent classification device of the present invention is a patent classification device in which, for any one of the first to eighth configurations, the important information acquisition unit is equipped with an important tag corresponding sentence acquisition means for acquiring one or more important tag corresponding sentences which are sentences included in a patent specification corresponding to a patent identifier included in the classification target information accepted by the acceptance unit and correspond to a predetermined tag, and the important information includes one or more important tag corresponding sentences acquired by the important tag corresponding sentence acquisition means, or includes important tag corresponding terms which are terms acquired from one or more important tag corresponding sentences.

With this configuration (ninth configuration), patents can be classified with high accuracy by using important tag-corresponding sentences or important tag-corresponding terms.

The patent classification device of the present invention is a patent classification device in which, for any one of the first to ninth configurations, the important information acquisition unit includes a numeric range string acquisition means for acquiring one or more numeric range strings, which are strings included in a patent specification corresponding to a patent identifier included in the classification target information accepted by the acceptance unit and indicate a numeric range, and the important information includes one or more numeric range strings acquired by the numeric range string acquisition means.

With this configuration (tenth configuration), patents can be classified with high accuracy by using numeric range strings.

The patent classification device of the present invention is a patent classification device in which, for any one of the first to ninth configurations, the important information acquisition unit is equipped with a numeric range string acquisition means for acquiring one or more numeric range strings, which are strings included in a patent specification corresponding to a patent identifier included in the classification target information accepted by the acceptance unit and indicate a numeric range, and a number acquisition means for acquiring three or more numbers included in the numeric range indicated by the numeric range string acquired by the numeric range string acquisition means, and the important information includes the three or more numbers acquired by the number acquisition means.

With this configuration (eleventh configuration), patents can be classified with high accuracy by using three or more numbers included in the numerical range indicated by the numerical range string.

The patent classification device of the present invention, in any one of the first to eleventh configurations, further includes a patent specification acquisition unit that acquires from an external server device a patent specification corresponding to a patent identifier included in the classification target information accepted by the acceptance unit, and the important information acquisition unit is a patent classification device that acquires important information that satisfies a predetermined condition using the patent specification acquired by the patent specification acquisition unit.

With this configuration (twelfth configuration), patents can be classified with high accuracy by using important information obtained from patent specifications obtained from an external server device.

The learning device production device of the present invention is a learning device production device that includes a manual evaluation information storage unit that stores manual evaluation information having two or more manual evaluation records each having a patent identifier that identifies a patent and classification result information that identifies a manual classification result of the patent corresponding to the patent identifier; an important information acquisition unit that acquires important information that satisfies a predetermined condition using a patent specification corresponding to the patent identifier of each of the two or more manual evaluation records in the manual evaluation information stored in the manual evaluation information storage unit; a learning source information acquisition unit that acquires a patent record having important information acquired by the important information acquisition unit and classification result information that is paired with a patent identifier corresponding to the patent specification that is the source of the important information for each of the two or more manual evaluation records, and acquires learning source information having two or more patent records; a learning unit that learns the learning source information acquired by the learning source information acquisition unit using a machine learning algorithm to acquire a learning device; and a storage unit that accumulates the learning devices acquired by the learning unit.

This configuration (the thirteenth configuration) makes it possible to produce a learning machine for classifying patents with high accuracy.

According to the present invention, by performing predictive processing using an appropriate algorithm, patents can be classified with high accuracy.

In addition, according to the present invention, patents can be classified with high accuracy by obtaining predicted classification information using different decision methods for patent information that meets predetermined conditions and patent information that does not.

In addition, the present invention allows for accurate classification of patents by using important information about patent specifications.

Conceptual diagram of patent classification system A in embodiment 1 Block diagram of the patent classification system A Block diagram of the learning device production device 1 that constitutes the patent classification system A A flowchart illustrating an example of the operation of the learning device production device 1. A flowchart for explaining an example of the important information acquisition process A flowchart for explaining an example of the effect solution means statement acquisition process. A flowchart for explaining an example of the effect term acquisition process. Flowchart for explaining an example of the numerical value range character string acquisition process Flowchart for an example of the number acquisition process A flowchart explaining an example of the operation of the patent classification device 2 Block diagram of patent classification system B in embodiment 2 Block diagram of the patent classification device 8 constituting the patent classification system B A flowchart for explaining an example of the operation of the algorithm determination device 5 constituting the patent classification system B A flowchart for explaining the evaluation process example A flowchart for explaining the K-fold cross-validation example A flowchart for explaining the statistical processing evaluation example A flowchart explaining an example of the operation of the learning device production device constituting the patent classification system B A flowchart explaining an example of the operation of the patent classification device Data structure diagram of algorithm information candidates Data structure of the accuracy information Block diagram of patent classification system C in embodiment 3 Block diagram of patent classification system D in embodiment 4 A flowchart explaining the operation of the classification information determination device 7 constituting the patent classification system D. A flowchart explaining the "kth predicted classification information acquisition process" Data structure of the same condition information Block diagram of patent classification system E in embodiment 5 Data structure of the same condition information Overview of a computer system in each embodiment Block diagram of the computer system

Below, an embodiment of an algorithm determination device and the like will be described with reference to the drawings. Note that components with the same reference numerals in the embodiments perform similar operations, and therefore repeated explanations may be omitted.

(Embodiment 1)
In this embodiment, we will describe a patent classification system that includes a patent classification device that acquires important information from patent specifications, does not include the full text of the patent specifications, and classifies patent information using a learning machine obtained by a machine learning algorithm to acquire learning source information having one or more patent records that have important information and manual classification.

In addition, in this embodiment, a patent classification system including a patent classification device that classifies patent information using a learning device obtained by a machine learning algorithm from learning source information having patent records including claims and/or abstracts is described.

In this embodiment, the important information is effect/solution means sentences, effect sentences, solution means sentences, objective sentences, effect terms, solution means terms, objective terms, characteristic words, related words of characteristic words (superordinate words, subordinate words, synonyms, sentences corresponding to predetermined tags, terms contained in the sentences, character strings indicating a numerical range, numbers contained in the range, etc.).

In addition, in this embodiment, a patent classification system including a patent classification device that acquires patent specifications from an external server device and acquires important information from the patent specifications is described.

Furthermore, in this embodiment, we will explain a patent classification system that includes a learning device production device.

Figure 1 is a conceptual diagram of a patent classification system A in this embodiment. The patent classification system A includes a learning device production device 1, a patent classification device 2, a server device 3, and one or more user terminals 4. The learning device production device 1 is a device that produces learning devices. The patent classification device 2 is a device that predicts patent classifications. The server device 3 is a device that stores patent documents including patent specifications, etc., and is a well-known device. The server device 3 is, for example, a commercial patent database server (for example, see HYPAT-i2 (URL: https://portal.hypatweb.jp/)). The user terminal 4 is a terminal used by a user. The user terminal 4 may be, for example, a personal computer, a smartphone, a tablet terminal, etc.

Figure 2 is a block diagram of the patent classification system A in this embodiment. Figure 3 is a block diagram of the learning device production device 1 that constitutes the patent classification system A.

The learning device production device 1 constituting the patent classification system A includes a learning storage unit 11, a learning reception unit 12, a learning processing unit 13, and a learning output unit 14. The learning storage unit 11 includes a manual evaluation information storage unit 111. The learning processing unit 13 includes a patent specification acquisition unit 131, an important information acquisition unit 132, a learning source information acquisition unit 133, and a learning unit 134. The important information acquisition unit 132 includes an effect solution means sentence acquisition unit 1321, an effect sentence acquisition unit 1322, a solution means sentence acquisition unit 1323, an objective sentence acquisition unit 1324, a feature word acquisition unit 1325, a related word acquisition unit 1326, an important tag corresponding sentence acquisition unit 1327, a numeric range character string acquisition unit 1328, and a number acquisition unit 1329. The learning output unit 14 includes an accumulation unit 141.

The patent classification device 2 includes a storage unit 21, a reception unit 22, a processing unit 23, and an output unit 24. The storage unit 21 includes a learning device storage unit 211. The processing unit 23 includes a patent specification acquisition unit 131, an important information acquisition unit 132, a classification target processing information acquisition unit 233, and a prediction unit 234.

Various types of information are stored in the learning storage unit 11 that constitutes the learning module production device 1. The various types of information include, for example, manual evaluation information (described later), patent specifications associated with patent identifiers, a related word dictionary, various patterns such as effect-solution-means sentence patterns (described later), and various important information learning modules.

A related word dictionary is a dictionary of related words. For example, a related word dictionary has many pairs of a term and related words. A related word is a superordinate word, a hyponym, a synonym, or a similar word. A superordinate word is a term of a higher concept. A hyponym is a term of a lower concept. The structure of a related word dictionary does not matter. For example, a related word dictionary has two or more related word information. Related word information has, for example, a term and a related word of the term. Related word information is, for example, a term, a related word of the term, and a flag. The flag is information that specifies the relationship between a term and a related word of the term (for example, one of a superordinate word, a hyponym, a synonym, or a similar word).

An important information learner is a learner that judges whether information is important or not. Examples of important information learners include an effect/solution means sentence learner (described later), an effect term learner (described later), a solution means term learner (described later), an effect sentence learner (described later), a solution means sentence learner (described later), an objective sentence learner (described later), and an objective term learner (described later).

The manual evaluation information storage unit 111 stores manual evaluation information having two or more manual evaluation records. The manual evaluation record is information having classification result information that specifies the manual classification result for one patent. The manual evaluation record has a patent identifier and classification result information. The classification result information of the manual evaluation record is information that specifies the result of classification of the patent identified by the patent identifier of the manual evaluation record. The patent identifier is information that identifies the patent. The patent identifier is, for example, an ID, application number, publication number, registration number, serial number, etc. The classification result information is usually information that specifies the manual classification result for a patent. The classification result information is, for example, information that specifies the technical field of the patent, information that specifies the degree of relevance of the patent, information that specifies the importance of the patent, etc. The classification result information is, for example, "○", "×", "A", "B", "C", "D", "1", "2", "3", ... "9", in-house specific classifications used within the company, etc., and the type of information does not matter. The manual evaluation information may be, for example, a CSV file, Excel file, etc., and the data structure, etc., is not important.

In addition to the patent identifier and classification result information, it is preferable that the manual review record includes, for example, claims, an abstract, and a patent classification code (e.g., IPC, FI, F-term, CPC, etc.). The manual review record may also include a patent specification.

The learning reception unit 12 accepts instructions, information, etc. The instructions, information, etc. are, for example, a learning operation start instruction. Here, the learning operation start instruction is an instruction to start the production of a learning device. The learning operation start instruction is usually an instruction from a user. The learning operation start instruction may have, for example, type-specific information that specifies the type of important information acquired by the important information acquisition unit 132. The type-specific information is information that specifies the type of important information used to create the learning device. The type-specific information may be, for example, information that specifies the means to be operated from among the effect solution means sentence acquisition means 1321, the effect sentence acquisition means 1322, the solution means sentence acquisition means 1323, the objective sentence acquisition means 1324, the feature word acquisition means 1325, the related word acquisition means 1326, the important tag corresponding sentence acquisition means 1327, the numerical range character string acquisition means 1328, and the number acquisition means 1329. The type-specific information may also be information that identifies the important information to be obtained from, for example, an effect-solution means sentence, an effect sentence, a solution means sentence, an objective sentence, an effect term, a solution means term, an objective term, a characteristic word, a related word of a characteristic word (superordinate word, subordinate word, synonym), a tag-corresponding sentence, a tag-corresponding term, a numeric range string, or important information of a number in the range indicated by the numeric range string.

The type specification information is, for example, a vector, and has a structure such as (whether to acquire an effect solution sentence, whether to acquire an effect sentence, whether to acquire a solution sentence, whether to acquire an object sentence, whether to acquire an effect term, whether to acquire a solution term, whether to acquire an object term, whether to acquire a feature word, whether to acquire a related word of the feature word, whether to acquire a tag corresponding sentence, whether to acquire a tag corresponding term, whether to acquire a numeric range string, whether to acquire a number in the range indicated by the numeric range string). In addition, in such a case, when the type specification information is (1, 0, ... 0, 1), the type specification information indicates that only the effect solution sentence and the number in the range indicated by the numeric range string are acquired. Note that in the above vector (1, 0, ... 0, 1), all elements corresponding to "..." are 0. Also, it indicates that important information corresponding to an element "1" is acquired, and important information corresponding to an element "0" is not acquired.

Note that an effect-solution means sentence is a sentence that includes both an expression of the effect and an expression of the solution means. An effect-solution means sentence is a sentence that matches a pattern (which can be called an effect-solution means sentence pattern) such as "<Effect> can be achieved by <Solution means>," "<Effect> becomes possible by providing <Solution means>," "<Effect> can be achieved based on <Solution means>," or "<Effect> is achieved by using <Solution means>." Note that the information indicated by "<" and ">" (for example, <Solution means>, <Effect>) is a variable that can be matched with any character string. Examples of effect-solution means sentences are "By providing a protrusion, it becomes possible to adjust the airflow," and "By using key terms, the accuracy of patent classification is improved."

Moreover, an effect sentence is a sentence that includes an expression of an effect. An effect sentence is a sentence that matches a pattern (which may be called an effect sentence pattern) such as "It is possible to achieve <effect>," "It becomes possible to achieve <effect>," or "It becomes <effect>." An effect sentence is, for example, "According to this embodiment, it becomes possible to adjust the air volume," or "According to this embodiment, the accuracy of patent classification is improved." An effect sentence may be considered to be the same as a problem sentence, which is a sentence that identifies the problem.

A solution sentence is a sentence that includes an expression of a solution. For example, a solution sentence is a sentence that matches a pattern (which can also be called a solution sentence pattern) such as "The key is <solution>," "It is based on <solution>," or "The problem can be solved by <solution>." For example, a solution sentence is "The problem can be solved by the protrusion," or "The above problem can be solved by the key term acquisition unit."

An objective sentence is a sentence that includes an expression of purpose. For example, an objective sentence is a sentence of a pattern (which may be called an objective sentence pattern) such as "In this invention, the objective is to achieve <objective>." or "The objective of the present invention is <objective>." For example, an objective sentence is "In this invention, the objective is to adjust the air volume." or "The objective of the present invention is to improve the accuracy of patent classification." Note that an objective sentence may be considered to be the same as an effect sentence or problem sentence. This is because objectives, effects, and problems are usually corresponding information.

Effect terms are terms that indicate an effect, such as "improved accuracy," "energy saving," "ability to adjust airflow," "resource saving," "improved safety," and "improved processing speed." Note that effect terms can be considered to be the same as problem terms, which are sentences that identify the problem.

Solution terms are terms that indicate the solution, such as "protrusion," "key term," "important information acquisition unit," "processing unit," etc.

Target terms are terms that indicate a purpose, such as "improved accuracy," "energy saving," "ability to adjust airflow," "resource saving," "improved safety," and "improved processing speed." Note that target terms may be considered to be the same as effect terms or task terms.

A characteristic word is a term that indicates the characteristics of a patent specification. A characteristic word is a term that appears frequently in the patent specification of interest. It is preferable that a characteristic word is a term that appears frequently in the patent specification of interest and that appears infrequently in other patent specifications. A characteristic word is a term in a patent specification, for example, a term whose TF/IDF value is large enough to satisfy a predetermined condition. Note that the method of calculating the TF/IDF value is a publicly known technology, so a detailed explanation is omitted.

A related word of a characteristic word is a superordinate word of the characteristic word, a subordinate word of the characteristic word, or a synonym of the characteristic word. A related word of a characteristic word is, for example, a term obtained by searching a related word dictionary using the characteristic word as a key.

A sentence corresponding to an important tag is a sentence that corresponds to a predetermined tag. A predetermined tag is a tag that is predetermined in a patent specification. For example, a predetermined tag is a tag such as "Effects of the invention" or "Problems that the invention aims to solve" or "Industrial applicability".

Tag-enabled terms are terms that are included in important tag-enabled sentences.

A numeric range string is a string included in a patent specification, and indicates a numeric range. A numeric range string has, for example, a unit. A numeric range string is a string that matches the following patterns (which may be called a numeric range string pattern): "<number> <unit> to <number> <unit>", "<number> to <number> <unit>", "<number> to <number>", "<number> <unit> to <number>", "<number> <unit> - <number> <unit>", "<number> - <number> <unit>", "<number> - <number>"). In addition, the positive or negative sign of the numbers, the number of significant digits, etc., do not matter. In addition, the unit may be, for example, a unit of length, a unit of weight, a unit of ratio, etc., and the type does not matter. A numeric range string is, for example, "1 to 10 mm", "2.5 mg - 5.2 mg", "0.8% to 1.9%", etc.

If the numeric range string is "2.5mg-5.2mg", the numbers included in the numeric range indicated by the numeric range string are, for example, "2.5mg", "2.6mg", "2.7mg", ... "5.1mg", "5.2mg".

Here, acceptance usually means receiving information from the user terminal 4, but it may also be considered a concept that includes the acceptance of information input from an input device such as a keyboard, mouse, or touch panel, and the acceptance of information read from a recording medium such as an optical disk, magnetic disk, or semiconductor memory.

The learning processing unit 13 performs various types of processing. The various types of processing are, for example, processing performed by the patent specification acquisition unit 131, the important information acquisition unit 132, the learning source information acquisition unit 133, the learning unit 134, the effect/solution means sentence acquisition unit 1321, the effect sentence acquisition unit 1322, the solution means sentence acquisition unit 1323, the objective sentence acquisition unit 1324, the feature word acquisition unit 1325, the related word acquisition unit 1326, the important tag corresponding sentence acquisition unit 1327, the numeric range character string acquisition unit 1328, and the number acquisition unit 1329.

The learning processing unit 13, for example, learns important information using a machine learning algorithm, acquires a learning device that judges whether the information is important or not, and stores the information in the learning storage unit 11. For machine learning, for example, deep learning, SVM, random forest, decision tree, etc. can be used. In machine learning, to configure a learning device, for example, a learning device is obtained by providing a group of input information (for example, a vector composed of sentences, a distributed representation of terms, etc.) and information to be output (for example, judgment information on whether the information is important or not) as arguments to a machine learning function (for example, fastText, TinySVM, various RandomForest functions, TensorFlow library, etc.). In machine learning, when making predictions, a learning device and a group of input information (for example, a vector composed of sentences, a distributed representation of terms, etc.) are provided as arguments to a machine learning function, and predicted information (for example, judgment information on whether the information is important or not) is obtained.

The learning processing unit 13 acquires an effect-solution-means sentence learner, which is a learner for predicting whether a sentence to be judged is an effect-solution-means sentence or not, using, for example, one or more effect-solution-means sentences and one or more non-effect-solution-means sentences, by a machine learning algorithm, and stores the effect-solution-means sentence learner in the learning storage unit 11. Note that a non-effect-solution-means sentence is a sentence that is not an effect-solution-means sentence.

More specifically, the learning processing unit 13, for example, acquires one or more effective solution means sentences from the learning storage unit 11, performs morphological analysis on each of the acquired one or more effective solution means sentences, constructs a vector from the acquired one or more morphemes, and uses the vector as a positive example; and acquires one or more non-effective solution means sentences from the learning storage unit 11, performs morphological analysis on each of the acquired one or more non-effective solution means sentences, constructs a vector from the acquired one or more morphemes, and uses the vector as a negative example to provide the machine learning function, and performs learning processing by machine learning to acquire an effective solution means sentence learner and store it in the learning storage unit 11. Note that there is no restriction on the method of constructing a vector from a sentence. In other words, there is no restriction on what information is used as a feature when constructing a vector from a sentence. It goes without saying that not only the acquired one or more morphemes but also n-grams may be used as features.

The learning processing unit 13, for example, uses one or more effective terms and one or more non-effective terms to acquire an effective term learner, which is a learner for predicting whether a term to be judged is an effective term or not, through a machine learning algorithm, and stores the effective term learner in the learning storage unit 11. Note that a non-effective term is a term that is not an effective term.

More specifically, the learning processing unit 13, for example, acquires one or more effect terms from the learning storage unit 11, acquires a distributed representation (vector) of each of the acquired one or more effect terms (for example, using word2vec), and uses the distributed representation as a positive example; it also acquires one or more non-effect terms from the learning storage unit 11, acquires a distributed representation (vector) of each of the acquired one or more non-effect terms, and provides the distributed representation as a negative example to a machine learning function; it performs a learning process by machine learning, thereby acquiring an effect term learner, and stores it in the learning storage unit 11.

The learning processing unit 13, for example, uses one or more solution means terms and one or more non-solution means terms to acquire a solution means term learner, which is a learner for predicting whether a term to be judged is a solution means term or not, through a machine learning algorithm, and stores the solution means term learner in the learning storage unit 11. Note that a non-solution means term is a term that is not a solution means term.

More specifically, the learning processing unit 13, for example, acquires one or more solution means terms from the learning storage unit 11, acquires a distributed representation (vector) of each of the acquired one or more solution means terms, and uses the distributed representation as a positive example; it also acquires one or more non-solution means terms from the learning storage unit 11, acquires a distributed representation (vector) of each of the acquired one or more non-solution means terms, and uses the distributed representation as a negative example to provide the machine learning function; it performs a learning process by machine learning, thereby acquiring a solution means term learner, and stores it in the learning storage unit 11.

The learning processing unit 13, for example, uses one or more solution sentences and one or more non-solution sentences to acquire a solution sentence learner, which is a learner for predicting whether a sentence to be judged is a solution sentence or not, through a machine learning algorithm, and stores the solution sentence learner in the learning storage unit 11. Note that a non-solution sentence is a sentence that is not a solution sentence.

More specifically, the learning processing unit 13, for example, acquires one or more solution sentences from the learning storage unit 11, performs morphological analysis on each of the acquired solution sentences, constructs a vector from the acquired morphemes, and treats the vector as a positive example; and acquires one or more non-solution sentences from the learning storage unit 11, performs morphological analysis on each of the acquired non-solution sentences, constructs a vector from the acquired morphemes, and treats the vector as a negative example, and provides the vector to a machine learning function to perform learning processing by machine learning, thereby acquiring a solution sentence learner and storing it in the learning storage unit 11.

The learning processing unit 13 acquires an effect sentence learning device, which is a learning device for predicting whether a sentence to be judged is an effect sentence or not, using, for example, one or more effect sentences and one or more non-effect sentences, by a machine learning algorithm, and stores the effect sentence learning device in the learning storage unit 11. Note that a non-effect sentence is a sentence that is not an effect sentence.

More specifically, the learning processing unit 13, for example, acquires one or more effective sentences from the learning storage unit 11, performs morphological analysis on each of the acquired one or more effective sentences, constructs a vector from the acquired one or more morphemes, and treats the vector as a positive example; and acquires one or more non-effective sentences from the learning storage unit 11, performs morphological analysis on each of the acquired one or more non-effective sentences, constructs a vector from the acquired one or more morphemes, and treats the vector as a negative example, and provides the vector to a machine learning function to perform learning processing by machine learning, thereby acquiring an effective sentence learner and storing it in the learning storage unit 11.

The learning processing unit 13 acquires a target sentence learner, which is a learner for predicting whether a sentence to be judged is a target sentence or not, using, for example, one or more target sentences and one or more non-target sentences, by a machine learning algorithm, and stores the target sentence learner in the learning storage unit 11. Note that a non-target sentence is a sentence that is not a target sentence.

More specifically, the learning processing unit 13, for example, acquires one or more target sentences from the learning storage unit 11, performs morphological analysis on each of the acquired one or more target sentences, constructs a vector from the acquired one or more morphemes, and treats the vector as a positive example; and acquires one or more non-target sentences from the learning storage unit 11, performs morphological analysis on each of the acquired one or more non-target sentences, constructs a vector from the acquired one or more morphemes, and treats the vector as a negative example and provides it to a machine learning function, thereby performing learning processing by machine learning, and acquires a target sentence learner and stores it in the learning storage unit 11.

The learning processing unit 13, for example, uses one or more target terms and one or more non-target terms to acquire a target term learner, which is a learner for predicting whether a term to be judged is a target term or not, by a machine learning algorithm, and stores the target term learner in the learning storage unit 11. Note that a non-target term is a term that is not a target term.

More specifically, the learning processing unit 13, for example, acquires one or more target terms from the learning storage unit 11, acquires a distributed representation (vector) of each of the acquired one or more target terms, and uses the distributed representation as a positive example; it also acquires one or more non-target terms from the learning storage unit 11, acquires a distributed representation (vector) of each of the acquired one or more non-target terms, and uses the distributed representation as a negative example to provide the machine learning function; it performs a learning process by machine learning, thereby acquiring a target term learner, and stores it in the learning storage unit 11.

It goes without saying that the method used by the learning processing unit 13 to obtain vectors from sentences and the method used to obtain vectors from terms are not important. The learning processing unit 13 can use various methods for obtaining vectors from sentences and various methods for obtaining vectors from terms.

The patent specification acquisition unit 131 acquires a patent specification corresponding to a patent identifier included in the classification target information accepted by the reception unit 22. For example, the patent specification acquisition unit 131 acquires a patent specification corresponding to a patent identifier included in the classification target information accepted by the reception unit 22 from an external server device 3. For example, the patent specification acquisition unit 131 acquires a patent specification corresponding to a patent identifier included in the classification target information accepted by the reception unit 22 from the learning storage unit 11. For example, the patent specification acquisition unit 131 may acquire a patent specification included in a manual evaluation record.

The important information acquisition unit 132 acquires important information that satisfies predetermined conditions using patent specifications that correspond to patent identifiers included in the classification target information accepted by the acceptance unit 22 described below.

The important information acquisition unit 132 acquires important information that satisfies predetermined conditions using the patent specification acquired by the patent specification acquisition unit 131.

The important information acquisition unit 132 acquires important information that satisfies predetermined conditions using patent specifications that correspond to the patent identifiers of two or more manual evaluation records contained in the manual evaluation information stored in the manual evaluation information storage unit 111. Important information is a sentence that satisfies the predetermined conditions, or a term that satisfies the predetermined conditions.

The predetermined conditions are, for example, matching an effect solution sentence, matching an effect term, matching a solution term, matching an effect sentence, matching a solution sentence, matching an objective sentence, matching an objective term, matching a feature word, matching a related word, matching an important tag corresponding sentence, matching an important tag corresponding term, matching a numeric range string, matching a numeric value that can be obtained from a numeric range string, etc.

The important information includes, for example, one or more effect solution means sentences acquired by the effect solution means sentence acquisition means 1321, or effect terms and solution means terms that are terms acquired from one or more effect solution means sentences.

The important information includes, for example, one or more effect sentences acquired by the effect sentence acquisition means 1322, or effect terms that are terms acquired from one or more effect sentences.

The important information includes, for example, one or more solution sentences acquired by the solution sentence acquisition means 1323, or solution terms, which are terms acquired from one or more solution sentences.

The important information includes, for example, one or more object sentences acquired by the object sentence acquisition means 1324, or object terms that are terms acquired from one or more object sentences.

The important information includes, for example, one or more characteristic words acquired by the characteristic word acquisition means 1325.

The important information includes, for example, one or more related words acquired by the related word acquisition means 1326.

The important information includes, for example, one or more important tag corresponding sentences acquired by the important tag corresponding sentence acquisition means 1327, or includes important tag corresponding terms, which are terms acquired from one or more important tag corresponding sentences.

The important information includes, for example, one or more numeric range strings acquired by the numeric range string acquisition means 1328.

The important information includes, for example, three or more numbers acquired by the number acquisition means 1329. If the numeric range string is "2.5mg-5.2mg", the numbers three or more included in the numeric range indicated by the numeric range string are, for example, "2.5mg", "2.6mg", "2.7mg", ... "5.1mg", and "5.2mg".

The effect/solution sentence acquisition means 1321, effect sentence acquisition means 1322, solution sentence acquisition means 1323, and objective sentence acquisition means 1324 constituting the important information acquisition unit 132 acquire important information using, for example, the pattern matching method (1) or the machine learning method (2), as described below. Note that it is preferable for the feature word acquisition means 1325, related word acquisition means 1326, important tag corresponding sentence acquisition means 1327, numeric range character string acquisition means 1328, and number acquisition means 1329 to acquire important information by adopting a method other than the above (1) and (2). Note that the following method of acquiring important information is one example, and it goes without saying that important information may be acquired by other methods.

The important information acquisition unit 132 acquires sentences that include specific expressions using the patent specification. Sentences that include specific expressions are, for example, effect-solution means sentences, effect sentences, solution means sentences, or objective sentences.

The important information acquisition unit 132 uses the patent specification to acquire terms contained in sentences that include specific expressions. The terms contained in sentences that include specific expressions are, for example, effect terms, solution terms, or objective terms.

The important information acquisition unit 132 uses the patent specification to acquire characteristic words. The important information acquisition unit 132 uses the characteristic words to acquire related words.

The important information acquisition unit 132 uses the patent specification to acquire important tag-corresponding sentences or important tag-corresponding terms that correspond to a specific tag.

Below, examples of how important information can be obtained will be explained for each means.
(A) Effect solution means statement acquisition means 1321

The effect solution means sentence acquisition means 1321 determines whether or not each of one or more sentences included in a patent specification corresponding to a patent identifier included in the classification target information received by the receiving unit 22 is an effect solution means sentence, and acquires one or more effect solution means sentences that are determined to be effect solution means sentences.
(1) Pattern matching method

The effect solution means sentence acquisition means 1321 acquires one or more sentences from the patent specification to be inspected, and determines whether each sentence matches a predetermined effect solution means sentence pattern. The effect solution means sentence acquisition means 1321 acquires a sentence that matches one or more effect solution means sentence patterns as an effect solution means sentence.

The effect-solution-means sentence pattern is information that specifies the pattern of the effect-solution-means sentence. One or more effect-solution-means sentence patterns are stored, for example, in the learning storage unit 11. The effect-solution-means sentence pattern is, for example, "The <effect> can be achieved by the <solution>," "The <effect> can be achieved by providing the <solution>," and "The <effect> can be achieved based on the <solution>." Note that the pattern matching technology is a publicly known technology, so a detailed explanation will be omitted.

In addition, it is preferable to increase patterns such as effect solution sentence patterns by using a bootstrap algorithm.
(2) Machine learning method

The effect-solving means sentence acquisition means 1321 acquires one or more sentences from the patent specification to be inspected, applies each sentence to an effect-solving means sentence learner that determines whether or not it is an effect-solving means sentence, makes a prediction using a machine learning algorithm, and acquires judgment information that is the result of the prediction. Here, the judgment information is information that indicates whether or not a sentence is an effect-solving means sentence.

As with the learning process, machine learning can be used for prediction processing, such as deep learning, SVM, random forest, and decision trees. When making predictions in machine learning, a learning device and a set of input information (e.g., vectors made up of sentences, distributed representations of terms, etc.) are given as arguments to a machine learning function (e.g., fastText, TinySVM, various RandomForest functions, TensorFlow libraries, etc.), and predicted information (e.g., information on whether the information is important or not) is obtained.

Then, if the judgment information acquired by the machine learning function is "an effect solution means sentence," the effect solution means sentence acquisition means 1321 acquires the sentence as an effect solution means sentence.
(3) Acquisition of effect terms and solution terms

The effect solution means sentence acquisition means 1321 acquires effect terms (terms corresponding to <effect>) and solution means terms (terms corresponding to <solution>) from the acquired effect solution means sentence, for example, by using an effect solution means sentence pattern corresponding to the acquired effect solution means sentence (for example, "<effect> can be achieved by <solution means>.").

The effect/solution sentence acquisition means 1321 also acquires effect terms and solution terms, for example, using a machine learning algorithm.

For example, the effect solution means sentence acquisition means 1321 performs morphological analysis on the acquired effect solution means sentence, applies an effect term learner to each of the one or more acquired terms, and determines whether or not it is an effect term using a machine learning algorithm. Also, for example, the effect solution means sentence acquisition means 1321 performs morphological analysis on the acquired effect solution means sentence, applies a solution term learner to each of the one or more acquired terms, and determines whether or not it is a solution term using a machine learning algorithm.

The effect/solution sentence acquisition means 1321 may perform morphological analysis on each sentence of the patent specification, apply an effect term learner to each of the one or more acquired terms, and determine whether or not the term is an effect term using a machine learning algorithm. The effect/solution sentence acquisition means 1321 may perform morphological analysis on each sentence of the patent specification, apply a solution term learner to each of the one or more acquired terms, and determine whether or not the term is an effect term using a machine learning algorithm.
(B) Effect statement acquisition means 1322

The effect sentence acquisition means 1322 determines whether or not each of one or more sentences included in a patent specification corresponding to a patent identifier included in the classification target information accepted by the accepting unit 22 is an effect sentence, and acquires one or more effect sentences that are determined to be effect sentences.
(1) Pattern matching method

The effect sentence acquisition means 1322 acquires one or more sentences from the patent specification to be inspected, and determines whether each sentence matches a predetermined effect sentence pattern. The effect sentence acquisition means 1322 acquires a sentence that matches one or more effect sentence patterns as an effect sentence.

An effect sentence pattern is information that specifies a pattern of an effect sentence. One or more effect sentence patterns are stored, for example, in the learning storage unit 11. Moreover, an effect sentence pattern is, for example, "Can achieve <effect>." or "Has <effect>."

In addition, it is preferable to increase the number of patterns such as effect sentence patterns by using a bootstrap algorithm.
(2) Machine learning method

The effect sentence acquisition means 1322 acquires one or more sentences from the patent specification to be inspected, applies to each sentence an effect sentence learning device that determines whether or not it is an effect sentence, makes a prediction using a machine learning algorithm, and acquires judgment information that is the result of the prediction. Here, the judgment information is information that indicates whether or not a sentence is an effect sentence.

Then, if the judgment information acquired by the machine learning function is that the sentence is an "effect sentence," the effect sentence acquisition means 1322 acquires the sentence as an effect sentence.
(3) Acquisition of effect terms

The effect sentence acquisition means 1322 acquires an effect term (a term corresponding to <effect>) from the acquired effect sentence, for example, by using an effect sentence pattern corresponding to the acquired effect sentence (for example, "<effect> can be done.").

For example, the effect sentence acquisition means 1322 performs morphological analysis on the acquired effect sentence, applies an effect term learning device to each of one or more acquired terms, and determines whether or not the term is an effect term using a machine learning algorithm.

In addition, the effect sentence acquisition means 1322 may perform morphological analysis of each sentence in the patent specification, apply an effect term learning device to each of one or more acquired terms, and determine whether or not the term is an effect term using a machine learning algorithm.
(C) Solution sentence acquisition means 1323

The solution sentence acquisition means 1323 determines whether or not each of one or more sentences included in a patent specification corresponding to a patent identifier included in the classification target information accepted by the accepting unit 22 is a solution sentence, and acquires one or more solution sentences that are determined to be solution sentences.
(1) Pattern matching method

The solution sentence acquisition means 1323 acquires one or more sentences from the patent specification to be inspected, and determines whether each sentence matches a predetermined solution sentence pattern. The solution sentence acquisition means 1323 acquires a sentence that matches one or more solution sentence patterns as a solution sentence.

The solution sentence pattern is information that identifies a pattern of a solution sentence. One or more solution sentence patterns are stored, for example, in the learning storage unit 11. The solution sentence pattern is, for example, "The point is <solution>." or "Based on <solution>."

It is also preferable to increase the number of patterns such as solution sentence patterns using a bootstrap algorithm.
(2) Machine learning method

The solution sentence acquisition means 1323 acquires one or more sentences from the patent specification to be inspected, applies each sentence to a solution sentence learner that determines whether it is a solution sentence or not, makes a prediction using a machine learning algorithm, and acquires judgment information that is the result of the prediction. Here, the judgment information is information that indicates whether a sentence is a solution sentence or not.

Then, if the judgment information acquired by the machine learning function is "a solution sentence", the solution sentence acquisition unit 1323 acquires the sentence as a solution sentence.
(3) Acquisition of solution terms

The solution sentence acquisition means 1323 acquires a solution term (a term corresponding to <solution>) from the acquired solution sentence, for example, by using a solution sentence pattern corresponding to the acquired solution sentence (for example, "The point is <solution>.").

The solution sentence acquisition unit 1323 also acquires solution terms, for example, using a machine learning algorithm.

For example, the solution sentence acquisition means 1323 performs morphological analysis on the acquired solution sentence, applies a solution terminology learner to each of the one or more acquired terms, and determines whether or not it is a solution term using a machine learning algorithm. Also, for example, the effect solution sentence acquisition means 1321 performs morphological analysis on the acquired effect solution sentence, applies a solution terminology learner to each of the one or more acquired terms, and determines whether or not it is a solution term using a machine learning algorithm.

In addition, the solution means sentence acquisition means 1323 may perform morphological analysis of each sentence in the patent specification, apply a solution means term learning device to each of one or more acquired terms, and determine whether or not the term is a solution means term using a machine learning algorithm.
(D) Objective sentence acquisition means 1324

The object sentence acquisition means 1324 determines whether or not each of one or more sentences included in a patent specification corresponding to a patent identifier included in the classification target information accepted by the acceptance unit 22 is an object sentence, and acquires one or more object sentences that are determined to be object sentences.
(1) Pattern matching method

The object sentence acquisition means 1324 acquires one or more sentences from the patent specification to be inspected, and determines whether each sentence matches a predetermined object sentence pattern. The object sentence acquisition means 1324 acquires a sentence that matches one or more object sentence patterns as an object sentence.

The objective sentence pattern is information that specifies the pattern of the objective sentence. One or more objective sentence patterns are stored, for example, in the learning storage unit 11. An objective sentence pattern is, for example, "The objective of the present invention is to achieve <objective>." or "The objective of the present invention is <objective>."

In addition, it is preferable to increase the number of patterns such as objective sentence patterns by using a bootstrap algorithm.
(2) Machine learning method

The objective sentence acquisition means 1324 acquires one or more sentences from the patent specification to be inspected, applies an objective sentence learner to each sentence to determine whether it is an objective sentence or not, makes a prediction using a machine learning algorithm, and acquires judgment information that is the result of the prediction. Here, the judgment information is information indicating whether a sentence is an objective sentence or not.

Then, if the judgment information acquired by the machine learning function is that the sentence is a “object sentence”, the object sentence acquisition means 1324 acquires the sentence as an object sentence.
(3) Acquisition of target terms

The objective sentence acquisition means 1324 acquires an objective term (a term corresponding to <objective>) from the acquired objective sentence, for example, by using an objective sentence pattern corresponding to the acquired objective sentence (for example, "The objective of the present invention is to achieve <objective>.").

The objective sentence acquisition means 1324 also acquires objective terms, for example, using a machine learning algorithm.

For example, the objective sentence acquisition means 1324 performs morphological analysis on the acquired objective sentence, applies an objective term learner to each of the one or more acquired terms, and determines whether or not each is an objective term using a machine learning algorithm. Also, for example, the objective sentence acquisition means 1324 performs morphological analysis on the acquired objective sentence, applies an objective term learner to each of the one or more acquired terms, and determines whether or not each is an objective term using a machine learning algorithm.

In addition, the target sentence acquisition means 1324 may perform morphological analysis of each sentence in the patent specification, apply a target term learner to each of one or more acquired terms, and determine whether or not the term is a target term using a machine learning algorithm.
(E) characteristic word acquisition means 1325

The characteristic word acquisition means 1325 acquires the frequency of occurrence of one or more terms included in the patent specification corresponding to the patent identifier included in the classification target information accepted by the acceptance unit 22, calculates a score for each of the one or more terms using the frequency of occurrence, and acquires one or more characteristic words whose score is large enough to satisfy a predetermined condition.

The characteristic word acquisition means 1325 acquires, for example, the number of occurrences of one or more terms included in a patent specification corresponding to a patent identifier included in the classification target information accepted by the accepting unit 22. The characteristic word acquisition means 1325 also acquires, for example, the number of occurrences of one or more terms in a set of target patent specifications. Then, the characteristic word acquisition means 1325 calculates the TF/IDF value (score) of one or more terms included in a patent specification corresponding to a patent identifier included in the classification target information. Next, the characteristic word acquisition means 1325 acquires, as characteristic words of the patent specification, one or more terms whose scores are high enough to satisfy a predetermined condition. Note that any technology may be used to extract characteristic words from a document. The characteristic word acquisition means 1325 can use various technologies.
(F) Related word acquisition unit 1326

The related word acquisition means 1326 acquires, from the related word dictionary, one or more related words corresponding to each of the one or more characteristic words acquired by the characteristic word acquisition means 1325. Such related words are an example of important information. The related word dictionary used by the related word acquisition means 1326 may be an internal dictionary of the learning device production device 1 or the patent classification device 2, or may be a dictionary of a server (not shown).
(F) Important tag corresponding sentence acquisition unit 1327

The important tag corresponding sentence acquisition means 1327 acquires one or more important tag corresponding sentences, which are sentences included in the patent specification corresponding to the patent identifier included in the classification target information accepted by the acceptance unit 22.

The important tag corresponding sentence acquisition means 1327, for example, reads important tag identification information that identifies a predetermined tag from the learning storage unit 11, and acquires one or more sentences in the tag identified by the important tag identification information from the patent specification to be processed. Note that the important tag identification information is, for example, the [effects of the invention], the [problems to be solved by the invention], or the [industrial applicability]. For example, the technology for acquiring one or more sentences corresponding to the [effects of the invention] from a patent specification is a publicly known technology.

In addition, the important tag corresponding sentence acquisition unit 1327 may acquire important tag terms from one or more sentences corresponding to the important tag identification information. For example, the important tag corresponding sentence acquisition unit 1327 performs morphological analysis on one or more sentences and acquires one or more independent words as important tag terms.
(G) Numeric range character string acquisition unit 1328

The numeric range string acquisition means 1328 acquires one or more numeric range strings, which are strings included in a patent specification corresponding to a patent identifier included in the classification target information accepted by the acceptance unit 22 and indicate a numeric range.

The numeric range string acquisition means 1328 acquires, for example, a string that matches a numeric range string pattern from a patent specification. The numeric range string pattern is stored in the learning storage unit 11. The numeric range string patterns are, for example, "<number> <unit> to <number> <unit>", "<number> to <number> <unit>", "<number> <unit> to <number> <unit>", "<number> to <number>", "<number> <unit> - <number> <unit>", "<number> - <number> <unit>", and "<number> - <number>".

It is preferable that the numerical value range character string acquisition means 1328 acquires one or more numerical value range character strings from one or more sentences in an area corresponding to a specific tag in a patent specification corresponding to a patent identifier included in the classification target information accepted by the acceptance unit 22. The area corresponding to the specific tag is, for example, [Form for carrying out the invention], [Example], [Experiment], etc.
(H) Number acquisition means 1329

The number acquisition means 1329 acquires 3 or more numbers included in the numerical range indicated by the numerical range string acquired by the numerical range string acquisition means 1328. The number acquisition means 1329 acquires at least both ends of the numerical range indicated by the numerical range string acquired by the numerical range string acquisition means 1328 and one or more numerical values within the range. The number acquisition means 1329 usually acquires the upper and lower limit values of the numerical range, but it is not necessary to acquire the upper and lower limit values. The number acquisition means 1329 is a numerical value within the numerical range indicated by the numerical range string, and it is preferable to acquire the numerical value of a string that does not appear in the numerical range string.

When the acquired numeric range string is "2.5mg-5.2mg", it is preferable for the number acquisition means 1329 to acquire a number included in the numeric range indicated by the numeric range string, for example, "2.5mg", "2.6mg", "2.7mg", ... "5.1mg", "5.2mg". In other words, the number acquisition means 1329 generates a numeric value in the range "2.5-5.2" indicated by the acquired numeric range string, which is a numeric value with the number of digits indicated by the number of significant digits of the numeric range string (here, for example, one decimal place), and acquires a numeric value of 3 or more. Note that the numeric value acquired by the number acquisition means 1329 may or may not have a unit.

The learning source information acquisition unit 133 acquires a patent record having important information acquired by the important information acquisition unit 132 and classification result information for each of the two or more manual evaluation records, and acquires learning source information having two or more patent records. The classification result information is information that pairs with a patent identifier corresponding to the patent specification that is the source of the important information, and is usually information that indicates the result of manual classification of the patent.

The patent record preferably also includes one or more of the following types of information: claims or abstract corresponding to the patent identifier. The patent record preferably also includes one or more patent classification codes.

The learning unit 134 learns the learning source information acquired by the learning source information acquisition unit 133 using a machine learning algorithm and acquires a learning device.

The learning unit 134 uses information other than the classification result information among the learning source information acquired by the learning source information acquisition unit 133 as input, and configures a learning device that outputs the classification result information using a machine learning algorithm. Note that for machine learning, for example, deep learning, SVM, random forest, decision tree, etc. can be used. In machine learning, to configure a learning device, for example, a learning device is obtained by providing a group of input information (for example, information other than the classification result information among the learning source information) and information to be output (for example, classification result information) as arguments to a machine learning function (for example, fastText, TinySVM, various RandomForest functions, TensorFlow library, etc.). Note that the group of input information and the information to be output constitute a patent record.

The learning output unit 14 outputs various information. Examples of the various information include the learning device and information that the learning process has been completed.

Here, output is a concept that includes displaying on a display, projecting using a projector, printing on a printer, outputting sound, sending to an external device, storing on a recording medium, and passing on the processing results to other processing devices or other programs, etc.

The accumulation unit 141 accumulates the learning device acquired by the learning unit 134. The accumulation unit 141 accumulates the learning device, for example, in the learning storage unit 11. It is preferable that the accumulation unit 141 accumulates the learning device acquired by the learning unit 134 by associating the type-specific information contained in the instruction to start the learning operation with the learning device.

The storage unit 21 that constitutes the patent classification device 2 stores various types of information. The various types of information are, for example, learning devices.

The learning device storage unit 211 stores one or more learning devices. The learning devices in the learning device storage unit 211 are, for example, learning devices acquired by the learning device production device 1. The learning devices in the learning device storage unit 211 correspond to, for example, type-specific information.

The learning device is a learning device obtained by learning source information having two or more patent records using a machine learning algorithm. The patent records do not contain the full text of the patent specifications. The patent records contain important information obtained using the patent specifications corresponding to the patent identifiers that identify the patents, and classification result information that identifies the manual classification results of the patents.

The reception unit 22 receives, for example, a prediction start instruction. The prediction start instruction is an instruction to start prediction. The prediction start instruction includes, for example, classification target information of the prediction target, information specifying the learning device to be used, etc.

The reception unit 22 receives, for example, one or more pieces of classification target information. The classification target information includes a patent identifier that identifies the patent to be classified. The classification target information includes, for example, one or more pieces of information from among the scope of claims and abstract. The classification target information includes, for example, one or more patent classification codes. The classification target information includes, for example, a patent specification.

Here, acceptance usually means receiving information from the user terminal 4, but it may also be considered a concept that includes the acceptance of information input from an input device such as a keyboard, mouse, or touch panel, and the acceptance of information read from a recording medium such as an optical disk, magnetic disk, or semiconductor memory.

The processing unit 23 performs various types of processing. For example, the various types of processing are performed by the patent specification acquisition unit 131, the important information acquisition unit 132, the classification target processing information acquisition unit 233, the prediction unit 234, etc.

The classification target processing information acquisition unit 233 acquires classification target processing information, which is information corresponding to the patent identifier included in the classification target information accepted by the acceptance unit 22 and includes important information acquired by the important information acquisition unit 132.

The processed information to be classified includes, for example, important information acquired by the important information acquisition unit 132 and one or more types of information of the claims or abstract corresponding to the patent identifier.

When the classification target information accepted by the reception unit 22 does not have a patent specification, the classification target processed information acquisition unit 233 acquires classification target processed information consisting of, for example, part or all of the classification target information accepted by the reception unit 22 and important information acquired by the important information acquisition unit 132.

When the classification target information accepted by the reception unit 22 includes a patent specification, the classification target processed information acquisition unit 233, for example, removes some information including the patent specification from the classification target information accepted by the reception unit 22, and acquires classification target processed information consisting of the information after removing the patent specification, etc., and the important information acquired by the important information acquisition unit 132.

In addition, it is preferable for the important information acquisition unit 132 here to use type-specific information associated with the learning device being used.

The prediction unit 234 applies the learning device to the classification target processing information acquired by the classification target processing information acquisition unit 233, predicts the classification of the classification target processing information by a machine learning algorithm, and acquires classification result information. For machine learning, for example, deep learning, SVM, random forest, decision tree, etc. can be used. The prediction unit 234 provides the learning device and the classification target processing information as arguments to the machine learning function, and acquires classification result information. Note that the learning device is a learning device in the learning device storage unit 211.

The output unit 24 outputs the classification result information acquired by the prediction unit 234 in association with the patent identifier included in the classification target information accepted by the acceptance unit 22.

Here, output means, for example, transmission to the user terminal 4, but it is also a concept that includes display on a display, projection using a projector, printing on a printer, sound output, storage on a recording medium, and delivery of the processing results to other processing devices or other programs, etc.

The server device 3 is a server that stores a large number of patent specifications corresponding to patent identifiers. When the server device 3 receives a patent acquisition instruction having a patent identifier from the learning device production device 1 or the patent classification device 2, it acquires the patent specification corresponding to the patent identifier from the patent database in which it is stored, and transmits the patent specification to the learning device production device 1 or the patent classification device 2.

The user terminal 4 receives a learning operation start instruction from the user and transmits the learning operation start instruction to the learning device production device 1. The user terminal 4 also receives a prediction operation start instruction from the user and transmits the prediction operation start instruction to the patent classification device 2. It is preferable that the user terminal 4 receives and outputs classification result information, which is the result of the prediction, in response to the transmission of the prediction operation start instruction.

The learning storage unit 11, the manual evaluation information storage unit 111, the storage unit 21, and the learning device storage unit 211 are preferably non-volatile recording media, but can also be realized using volatile recording media.

The process by which information is stored in the learning storage unit 11, etc. is not important. For example, information may be stored in the learning storage unit 11, etc. via a recording medium, information transmitted via a communication line, etc. may be stored in the learning storage unit 11, etc., or information inputted via an input device may be stored in the learning storage unit 11, etc.

The learning reception unit 12, patent specification acquisition unit 131, learning output unit 14, and output unit 24 can typically be realized using wireless or wired communication means, etc.

The learning processing unit 13, important information acquisition unit 132, learning source information acquisition unit 133, learning unit 134, effect solution means sentence acquisition means 1321, effect sentence acquisition means 1322, solution means sentence acquisition means 1323, objective sentence acquisition means 1324, feature word acquisition means 1325, related word acquisition means 1326, important tag corresponding sentence acquisition means 1327, numerical range character string acquisition means 1328, number acquisition means 1329, storage unit 141, processing unit 23, classification target processing information acquisition unit 233, and prediction unit 234 can usually be realized by an MPU, memory, etc. The processing procedure of the learning processing unit 13, etc. is usually realized by software, and the software is recorded on a recording medium such as a ROM. However, it may also be realized by hardware (dedicated circuit).

Next, an example of the operation of the patent classification system A will be described. First, an example of the operation of the learning module production device 1 will be described using the flowchart in FIG. 4.

(Step S401) The learning reception unit 12 determines whether or not a learning operation start instruction has been received. If a learning operation start instruction has been received, the process proceeds to step S402, and if a learning operation start instruction has not been received, the process returns to step S401.

(Step S402) The learning processing unit 13 assigns 1 to counter i.

(Step S403) The learning processing unit 13 judges whether or not the i-th manual evaluation record exists in the manual evaluation information of the learning target stored in the manual evaluation information storage unit 111.
If the i-th manual evaluation record exists, the process proceeds to step S404; if not, the process proceeds to step S408.

(Step S404) If a patent specification exists in the i-th manual evaluation record, the patent specification acquisition unit 131 acquires the patent specification from the i-th manual evaluation record. Also, if a patent specification does not exist in the i-th manual evaluation record, the patent specification acquisition unit 131 acquires the patent identifier held by the i-th manual evaluation record, and receives the patent specification from the server device 3 using the patent identifier as a key.

(Step S405) The important information acquisition unit 132 acquires one or more types of important information from the patent specification acquired in step S404. An example of such an important information acquisition process is described using the flowchart in FIG. 5.

(Step S406) The learning source information acquisition unit 133 uses the i-th manual evaluation record and the important information acquired in step S405 to acquire the i-th patent record that has at least the classification result information and important information in the i-th manual evaluation record, and temporarily stores it in a buffer not shown.

(Step S407) The learning processing unit 13 increments the counter i by 1. Return to step S403.

(Step S408) The learning source information acquisition unit 133 constructs learning source information containing all the patent records temporarily stored in step S406.

(Step S409) The learning unit 134 performs a learning process using the learning source information acquired in step S408 to acquire a learning device.

(Step S410) The accumulation unit 141 associates the type-specific information contained in the learning operation start instruction with the learning device, and accumulates them in the learning storage unit 11. Return to step S401.

In the flowchart in Figure 4, processing ends when the power is turned off or an interrupt occurs to end processing.

Next, an example of the important information acquisition process in step S405 will be described using the flowchart in FIG.

(Step S501) The important information acquisition unit 132 acquires type-specific information contained in the learning operation start instruction. In the prediction process, the important information acquisition unit 132 acquires type-specific information associated with the learning device used in the prediction process.

(Step S502) The important information acquisition unit 132 checks the type-specific information acquired in step S501 and determines whether to acquire an effect solution means sentence as important information. If an effect solution means sentence is to be acquired, the process proceeds to step S503; if an effect solution means sentence is not to be acquired, the process proceeds to step S504.

(Step S503) The important information acquisition unit 132 performs an effect solution means statement acquisition process. An example of the effect solution means statement acquisition process is described using the flowchart in FIG. 6.

(Step S504) The important information acquisition unit 132 checks the type-specific information acquired in step S501 and determines whether or not to acquire an effect sentence as important information. If an effect sentence is to be acquired, the process proceeds to step S505; if an effect sentence is not to be acquired, the process proceeds to step S506.

(Step S505) The important information acquisition unit 132 performs effect acquisition processing. An example of the effect statement acquisition processing is described using the flowchart in FIG. 6.

(Step S506) The important information acquisition unit 132 checks the type-specific information acquired in step S501 and determines whether or not to acquire a solution statement as important information. If a solution statement is to be acquired, the process proceeds to step S507; if a solution statement is not to be acquired, the process proceeds to step S508.

(Step S507) The important information acquisition unit 132 performs a solution statement acquisition process. An example of the solution statement acquisition process is described using the flowchart in FIG. 6.

(Step S508) The important information acquisition unit 132 checks the type-specific information acquired in step S501 and determines whether to acquire the object sentence as important information. If the object sentence is to be acquired, the process proceeds to step S509; if the object sentence is not to be acquired, the process proceeds to step S510.

(Step S509) The important information acquisition unit 132 performs a purpose sentence acquisition process. An example of the purpose sentence acquisition process is described using the flowchart in FIG. 6.

(Step S510) The important information acquisition unit 132 checks the type-specific information acquired in step S501 and determines whether to acquire effect terms as important information. If effect terms are to be acquired, the process proceeds to step S511; if effect terms are not to be acquired, the process proceeds to step S512.

(Step S511) The important information acquisition unit 132 performs effect term acquisition processing. An example of the effect term acquisition processing is described using the flowchart in FIG. 7.

(Step S512) The important information acquisition unit 132 checks the type-specific information acquired in step S501 and determines whether or not to acquire solution means terms as important information. If solution means terms are to be acquired, the process proceeds to step S513; if solution means terms are not to be acquired, the process proceeds to step S514.

(Step S513) The important information acquisition unit 132 performs a solution term acquisition process. An example of the solution term acquisition process is described using the flowchart in FIG. 7.

(Step S514) The important information acquisition unit 132 checks the type-specific information acquired in step S501 and determines whether to acquire the target term as important information. If the target term is acquired, the process proceeds to step S515; if the target term is not acquired, the process proceeds to step S516.

(Step S515) The important information acquisition unit 132 performs a target term acquisition process. An example of the target term acquisition process is described using the flowchart in FIG. 7.

(Step S516) The important information acquisition unit 132 examines the type-specific information acquired in step S501 and determines whether to acquire a characteristic word as important information. If a characteristic word is to be acquired, the process proceeds to step S517; if a characteristic word is not to be acquired, the process proceeds to step S518.

(Step S517) The important information acquisition unit 132 performs a feature word acquisition process. Note that, as an example of the feature word acquisition process, the TF/IDF algorithm described above is used.

(Step S518) The important information acquisition unit 132 checks the type-specific information acquired in step S501 and determines whether or not to acquire related words as important information. If related words are to be acquired, the process proceeds to step S519; if related words are not to be acquired, the process proceeds to step S520.

(Step S519) The important information acquisition unit 132 performs a related word acquisition process. Note that as an example of the related word acquisition process, for example, as described above, a related word dictionary is searched using each of the one or more characteristic words acquired in step S517 as a key, and one or more related words that are paired with each characteristic word are acquired. Note that it is assumed that the characteristic word acquisition process is performed as a preprocessing for acquiring related words.

(Step S520) The important information acquisition unit 132 checks the type-specific information acquired in step S501 and determines whether to acquire an important tag-corresponding sentence as important information. If an important tag-corresponding sentence is to be acquired, the process proceeds to step S521. If an important tag-corresponding sentence is not to be acquired, the process proceeds to step S523.

(Step S521) The important information acquisition unit 132 performs an important tag corresponding sentence acquisition process. Note that an example of the important tag corresponding sentence acquisition process has been described above, so a description thereof will be omitted here.

(Step S522) The important information acquisition unit 132 checks the type-specific information acquired in step S501 and determines whether to acquire important tag terms as important information. If important tag terms are to be acquired, the process proceeds to step S523; if important tag terms are not to be acquired, the process proceeds to step S524.

(Step S523) The important information acquisition unit 132 performs important tag term acquisition processing. Note that an example of the important tag term acquisition processing has been described above, so a description thereof will be omitted here.

(Step S524) The important information acquisition unit 132 checks the type-specific information acquired in step S501 and determines whether or not to acquire a numeric range string as important information. If a numeric range string is acquired, the process proceeds to step S525; if a numeric range string is not acquired, the process returns to the upper level process.

(Step S525) The important information acquisition unit 132 performs a process to acquire a numeric range string. An example of the process is described with reference to the flowchart in FIG. 8.

(Step S526) The important information acquisition unit 132 performs a number acquisition process using one or more numeric range character strings acquired in step S523. An example of the number acquisition process is described using the flowchart in FIG. 9.

Next, examples of sentence acquisition processing, such as an example of the effect solution means sentence acquisition processing in step S503, an example of the effect acquisition processing in step S505, an example of the solution means sentence acquisition processing in step S507, and an example of the objective sentence acquisition processing in step S509, will be described using the flowchart in FIG. 6.

(Step S601) The important information acquisition unit 132 acquires one or more sentences within a predetermined range of sentence extraction targets from the patent specification from which sentences are to be extracted. Note that the predetermined range of sentence extraction targets may differ depending on the type of important information. The predetermined range of sentence extraction targets is, for example, the entire patent specification, the "effects of the invention," and the "form for implementing the invention."

(Step S602) The important information acquisition unit 132 assigns 1 to counter i.

(Step S603) The important information acquisition unit 132 determines whether the i-th sentence is present among the one or more sentences acquired in step S601. If the i-th sentence is present, the process proceeds to step S604, and if the i-th sentence is not present, the process returns to the upper process.

(Step S604) The important information acquisition unit 132 determines whether the i-th sentence acquired in step S603 corresponds to a sentence of important information of the corresponding type. An example of such a determination process has been described above. Such a determination process may be, for example, a method based on pattern matching or a method based on machine learning.

(Step S605) If the result of the determination in step S604 is that the sentence is of important information of the corresponding type, the important information acquisition unit 132 proceeds to step S606, and if the result is not that the sentence is of important information of the corresponding type, the important information acquisition unit 132 proceeds to step S607.

(Step S606) The important information acquisition unit 132 acquires the i-th sentence as important information and temporarily stores it in a buffer (not shown).

(Step S607) The important information acquisition unit 132 increments the counter i by 1. Return to step S603.

Next, examples of term acquisition processing, such as an example of effect term acquisition processing in step S511, an example of solution term acquisition processing in step S513, and an example of target term acquisition processing in step S515, will be described using the flowchart in FIG. 7.

(Step S701) The important information acquisition unit 132 performs a process of acquiring a sentence corresponding to the term. The sentence acquisition process is the process described using the flowchart in FIG. 6. Note that if the term is an effect term, the sentence to be acquired is one or more types of sentences, either an effect solution means sentence or an effect sentence. Also, if the term is a solution means term, the sentence to be acquired is one or more types of sentences, either an effect solution means sentence or a solution means sentence. Also, if the term is an object term, the sentence to be acquired is an object sentence.

(Step S702) The important information acquisition unit 132 assigns 1 to counter i.

(Step S703) The important information acquisition unit 132 determines whether the i-th sentence is present among the one or more sentences acquired in step S701. If the i-th sentence is present, the process proceeds to step S704; if the i-th sentence is not present, the process returns to the upper level process.

(Step S704) The important information acquisition unit 132 performs morphological analysis on the i-th sentence to acquire one or more terms. Note that the terms acquired here may be independent words only. The terms acquired here are terms acquired using the patterns of each sentence, and are terms that correspond to variables in the patterns. If the term corresponds to a variable in the pattern, then if the term is an effect term, the variable in the pattern is <effect>.

(Step S705) The important information acquisition unit 132 acquires one or more terms acquired in step S704 as important information and temporarily stores them in a buffer (not shown).

(Step S706) The important information acquisition unit 132 increments the counter i by 1. Return to step S703.

Next, an example of the numeric range character string acquisition process in step S523 will be described using the flowchart in FIG. 8. In the flowchart in FIG. 8, the description of the same steps as in the flowchart in FIG. 6 will be omitted.

(Step S801) The important information acquisition unit 132 assigns 1 to counter j.

(Step S802) The important information acquisition unit 132 determines whether the jth numeric range character string acquisition pattern is stored. If the jth numeric range character string acquisition pattern is stored, the process proceeds to step S803. If not, the process proceeds to step S607.

(Step S803) The important information acquisition unit 132 checks whether or not a string that matches the jth numeric range string acquisition pattern exists in the i-th sentence. If a matching string exists, the process proceeds to step S804; if a matching string does not exist, the process proceeds to step S805.

(Step S804) The important information acquisition unit 132 acquires one or more strings that match the jth numeric range string acquisition pattern in the ith sentence, and stores them in a buffer (not shown).

(Step S805) The important information acquisition unit 132 increments the counter j by 1. Return to step S802.

Next, an example of the number acquisition process in step S526 will be explained using the flowchart in Figure 9.

(Step S901) The important information acquisition unit 132 assigns 1 to counter i.

(Step S902) The important information acquisition unit 132 determines whether the i-th numeric range character string acquired in step S804 exists. If it exists, the process proceeds to step S903; if it does not exist, the process returns to the upper level process.

(Step S903) The important information acquisition unit 132 acquires the lower limit numerical value of the i-th numerical range string.

(Step S904) The important information acquisition unit 132 acquires the upper limit numerical value of the i-th numerical range string.

(Step S905) The important information acquisition unit 132 acquires the units from the i-th numeric range string.

(Step S906) The important information acquisition unit 132 acquires the number of significant digits from the numeric value of the i-th numeric range string.

(Step S907) The important information acquisition unit 132 acquires a numeric value that is greater than the lower limit and less than the upper limit, and has a number of digits equal to the number of significant digits, and is equal to or greater than 1.

(Step S908) The important information acquisition unit 132 adds the unit acquired in step S905 after each of the numerical values (1 or more) acquired in step S907.

(Step S909) The important information acquisition unit 132 increments the counter i by 1. Return to step S902.

Next, an example of the operation of the patent classification device 2 will be explained using the flowchart in Figure 10.

(Step S1001) The reception unit 22 determines whether or not a prediction start instruction including one or more pieces of classification target information has been received. If a prediction start instruction has been received, the process proceeds to step S1002, and if a prediction start instruction has not been received, the process returns to step S1001. Note that a prediction start instruction including one or more pieces of classification target information may be a prediction start instruction including information for acquiring one or more pieces of classification target information (e.g., a file name, a URL, etc.).

(Step S1002) The processing unit 23 assigns 1 to the counter i.

(Step S1003) The processing unit 23 determines whether or not there is i-th classification target information corresponding to the prediction start instruction received in step S1001. If there is i-th classification target information, the process proceeds to step S1004, and if there is no i-th classification target information, the process returns to step S1001.

(Step S1004) The patent specification acquisition unit 131 determines whether or not a patent specification exists in the i-th classification target information. If a patent specification exists, the process proceeds to step S1005. If a patent specification does not exist, the process proceeds to step S1006.

(Step S1005) The patent specification acquisition unit 131 acquires the patent specification from the i-th classification target information. Go to step S1008.

(Step S1006) The patent specification acquisition unit 131 acquires a patent identifier from the i-th classification target information.

(Step S1007) The patent specification acquisition unit 131 receives the patent specification from the server device 3 using the patent identifier acquired in step S1006 as a key.

(Step S1008) The important information acquisition unit 132 acquires the i-th important information using the acquired patent specification. An example of the important information acquisition process has been described using the flowchart in FIG. 5. Note that here, it is preferable for the important information acquisition unit 132 to acquire the important information using type-specific information associated with the learning device used in the prediction process.

(Step S1009) The classification target processing information acquisition unit 233 acquires the i-th classification target processing information that contains the i-th important information acquired in step S1008 and does not contain the full text of the patent specification.

(Step S1010) The prediction unit 234 obtains the learning device to be used in the prediction process from the learning device storage unit 211.

(Step S1011) The prediction unit 234 performs prediction processing using the i-th classification target processing information and the learning device acquired in step S1010, according to a machine learning algorithm, and acquires classification result information.

(Step S1012) The output unit 24 stores the classification result information acquired in step S1011 in association with the i-th classification target information.

(Step S1013) The processing unit 23 increments the counter i by 1. Return to step S1003.

In the flowchart in Figure 10, processing ends when the power is turned off or an interrupt occurs to end processing.

As described above, according to this embodiment, patents can be classified with high accuracy by using important information about patent specifications.

In addition, this embodiment allows for accurate classification of patents by using the claims and/or abstracts.

In addition, according to this embodiment, patents can be classified with high accuracy by using effect/solution sentences or effect terms or solution terms.

In addition, according to this embodiment, patents can be classified with high accuracy by using effect statements or effect terms.

In addition, according to this embodiment, patents can be classified with high accuracy by using solution sentences or solution terms.

In addition, according to this embodiment, patents can be classified with high accuracy by using object sentences or object terms.

In addition, this embodiment allows patents to be classified with high accuracy by using feature words.

In addition, according to this embodiment, patents can be classified with high accuracy by using related words of characteristic words.

In addition, according to this embodiment, patents can be classified with high accuracy by using important tag-corresponding sentences or important tag-corresponding terms.

In addition, according to this embodiment, patents can be classified with high accuracy by using numeric range strings.

In addition, according to this embodiment, patents can be classified with high accuracy by using numbers of 3 or more included in the numerical range indicated by the numerical range string.

In addition, according to this embodiment, patents can be classified with high accuracy by using important information obtained from patent specifications obtained from an external server device.

Furthermore, according to this embodiment, by using type-specific information to obtain only the appropriate important information from multiple types of important information, patents can be classified with higher accuracy. For example, an effect/solution statement is considered to be usable in classifying patents in all technical fields, while a numeric range string and numbers included in the numeric range indicated by the numeric range string are considered to be usable in classifying patents in some technical fields, such as the chemical field.

Furthermore, this embodiment makes it possible to produce a learning machine for classifying patents with high accuracy.

The processing in this embodiment may be realized by software. This software may be distributed by software download or the like. This software may also be recorded on a recording medium such as a CD-ROM and distributed. This also applies to the other embodiments in this specification. The software that realizes the patent classification device 2 in this embodiment is a program such as the following. In other words, this program is a program for making a computer that can access a learning source information having two or more patent records that are patent records that do not have the full text of the patent specification and that have important information acquired using a patent specification corresponding to a patent identifier that identifies a patent and classification result information that identifies a manual classification result of the patent, using a machine learning algorithm, and storing the acquired learning device, function as a reception unit that accepts classification target information including a patent identifier that identifies a patent to be classified, an important information acquisition unit that acquires important information that satisfies a predetermined condition using a patent specification that corresponds to a patent identifier included in the classification target information accepted by the reception unit, a classification target processed information acquisition unit that acquires classification target processed information that is information corresponding to a patent identifier included in the classification target information accepted by the reception unit and includes the important information acquired by the important information acquisition unit, a prediction unit that applies the learning device to the classification target processed information acquired by the classification target processed information acquisition unit, predicts the classification of the classification target processed information using a machine learning algorithm, and acquires classification result information, and an output unit that outputs the classification result information acquired by the prediction unit in association with the patent identifier included in the classification target information accepted by the reception unit.

The software for realizing the learning device production device 1 is a program for making a computer that can access a manual evaluation information storage unit in which manual evaluation information having two or more manual evaluation records each having a patent identifier that identifies a patent and classification result information that identifies the manual classification result of the patent corresponding to the patent identifier behave as follows: an important information acquisition unit that acquires important information that satisfies a predetermined condition using a patent specification corresponding to the patent identifier of each of the two or more manual evaluation records in the manual evaluation information stored in the manual evaluation information storage unit; a learning source information acquisition unit that acquires, for each of the two or more manual evaluation records, a patent record having important information acquired by the important information acquisition unit and classification result information that is paired with a patent identifier corresponding to the patent specification that is the source of the important information, and acquires learning source information having the two or more patent records; a learning unit that learns the learning source information acquired by the learning source information acquisition unit using a machine learning algorithm and acquires a learning device; and a storage unit that accumulates the learning devices acquired by the learning unit.

(Embodiment 2)
In this embodiment, a patent classification system B that uses a different algorithm to classify patents with high accuracy will be described.

In this embodiment, we will also explain patent classification system B, which uses different data.

In this embodiment, we will also explain patent classification system B when the learning module generation program is different.

In addition, in this embodiment, we will explain patent classification system B, which uses different statistical processing.

Furthermore, in this embodiment, a patent classification system B is described in which the two or more statistical processing methods include two or more of the OR, AND, or majority vote methods of the prediction results of each learning device.

In addition, in this embodiment, we will explain patent classification system B, in which accuracy is the recall rate, precision rate, F-measure, or accuracy rate.

In this embodiment, we also describe an algorithm determination device 5 that can determine an appropriate process for performing prediction processing.

In this embodiment, we also describe a learning module production device that produces an optimal learning module using the algorithm determined by the algorithm determination device 5.

In this embodiment, we also describe a patent classification device that uses an optimal learning device produced by a learning device production device to classify patents with high accuracy.

Figure 11 is a block diagram of a patent classification system B in this embodiment. The patent classification system B includes an algorithm determination device 5, a learning device production device 6, a patent classification device 8, a server device 3, and one or more user terminals 4.

The algorithm determination device 5 includes a storage unit 51, a reception unit 52, a processing unit 53, and an output unit 54. The storage unit 51 includes a program storage unit 511 and an accuracy specification information storage unit 512. The reception unit 52 includes a learning source information reception unit 521. The processing unit 53 includes a learning unit 531, a prediction unit 532, an evaluation unit 533, an algorithm information acquisition unit 534, a learning source information acquisition unit 535, and a statistical processing unit 536.

The learning device production device 6 includes a learning storage unit 61, a learning acceptance unit 62, a learning processing unit 13, and a learning output unit 64. The learning acceptance unit 62 includes a learning source information acceptance unit 621. The learning processing unit 13 includes a patent specification acquisition unit 131, an important information acquisition unit 132, a learning source information acquisition unit 133, and a learning unit 134. The important information acquisition unit 132 includes an effect solution means sentence acquisition means 1321, an effect sentence acquisition means 1322, a solution means sentence acquisition means 1323, an objective sentence acquisition means 1324, a feature word acquisition means 1325, a related word acquisition means 1326, an important tag corresponding sentence acquisition means 1327, a numeric range character string acquisition means 1328, and a number acquisition means 1329. The learning output unit 64 includes an accumulation unit 641.

Figure 12 is a block diagram of the patent classification device 8. The patent classification device 8 includes a storage unit 21, a reception unit 81, a processing unit 82, and an output unit 24. The processing unit 82 includes a patent specification acquisition unit 131, an important information acquisition unit 132, a classification target processing information acquisition unit 233, a statistical processing unit 536, and a prediction unit 821.

The server device 3 and the user terminal 4 are the same elements as those in embodiment 1.

Various types of information are stored in the storage unit 51 constituting the algorithm determination device 5. Examples of the various types of information include programs described below and accuracy specification information described below. The storage unit 51 may also store, for example, learning source information described below.

The program storage unit 511 stores one or more learning device generation programs and a prediction program. Here, the prediction program corresponds to each of the one or more learning device generation programs. In other words, the learning device generation program and the prediction program usually correspond one-to-one, but may also correspond many-to-one.

The learning module generation program is a program that uses source information to generate a learning module for classifying patent information. The source information is source information for generating a learning module. The source information has, for example, two or more pieces of patent classification information. The patent classification information is information related to the classification of patents. The patent classification information has, for example, patent information and classification information.

Patent information is information about patents. For example, it is preferable that patent information is a part of a patent specification, but it may be the entire text of a patent specification. A patent specification may be, for example, a patent gazette, a published patent gazette, or the like, and the type does not matter. Patent information usually includes a patent identifier. As mentioned above, a patent identifier is, for example, an ID, an application number, or the like. In addition, patent information includes, for example, one or more pieces of information such as an abstract, claims, specification, and patent classification code. It is preferable that patent information includes the important information described above. However, the type of information included in the patent information does not matter.

Classification information is information that indicates the classification of patent information. Classification information is usually information that indicates a manual classification of patents. Classification information can be anything, such as "○, ×", "A, B, C", or a technical field ID.

The patent classification information may be, for example, the patent record in embodiment 1. As described above, the patent record has important information and manual classification result information. The classification information may be the classification result information in embodiment 1. However, the classification information is not limited to manual information, and may be information with a high score among predicted classification information (for example, classification information in which the primitive score acquired by the prediction unit 532 described later is higher than or equal to a predetermined threshold value).

However, the structure of the learning source information does not matter. In addition, the learning source information may be accepted by the learning source information accepting unit 521 described later, or may be acquired by the learning source information acquiring unit 535 described later, for example.

The learning machine generation program is a program that performs learning processing using a machine learning algorithm. For machine learning, for example, deep learning, SVM, random forest, decision tree, etc. can be used. In machine learning, to configure a learning machine, for example, a learning machine can be obtained by providing the learning source information to a machine learning function (for example, fastText, TinySVM, various RandomForest functions, TensorFlow library, etc.).

The prediction program is a program that uses a learning device to predict the classification of patent classification information and obtain predicted classification information. The learning device is obtained by the learning unit 531, which will be described later.

The program storage unit 511 may store, for example, two or more learning device generation programs and two or more prediction programs.

The prediction program is a program that performs prediction processing using a machine learning algorithm. The prediction program has, for example, the machine learning function described above. When the prediction program is given a learning device and patent classification information of the classification target and executed, predicted classification information, which is the prediction result, is obtained.

Here, the two or more prediction programs correspond to the two or more learning device generation programs, respectively. In other words, there is usually a one-to-one correspondence between the learning device generation programs and the prediction programs.

The accuracy specification information storage unit 512 stores accuracy specification information. The accuracy specification information is information that specifies which of two or more predetermined types of accuracy the accuracy information is to be obtained for. The two or more predetermined types of accuracy are preferably two or more of the four types of accuracy, for example, recall rate, precision rate, F-value, or accuracy rate.

The recall rate, precision rate, and F-measure are well-known techniques. The accuracy rate is, for example, the percentage of all predicted classification information acquired by the prediction unit 532 that matches the classification information included in the learning source information.

Accuracy information is information related to accuracy. Accuracy information is, for example, information related to recall, information related to precision, information related to F value, information related to accuracy rate, etc., but any information related to accuracy will do. Information related to recall is, for example, recall, but may be information obtained based on recall. Information related to precision is, for example, precision, but may be information obtained based on precision. Information related to F value is, for example, F value, but may be information obtained based on F value. Information related to accuracy rate is, for example, accuracy rate, but may be information obtained based on accuracy rate.

The accuracy specification information is, for example, received by the reception unit 52 and accumulated in the accuracy specification information storage unit 512, but may also be stored in the accuracy specification information storage unit 512 in advance.

The reception unit 52 receives various types of information. For example, various types of information are learning source information.

The reception unit 52 also receives, for example, the accuracy specification information described above.

As mentioned above, acceptance usually means receiving information from the user terminal 4, but it may also be considered a concept that includes the acceptance of information input from an input device such as a keyboard, and the acceptance of information read from a recording medium such as a disk or semiconductor memory.

The learning source information receiving unit 521 receives the learning source information described above.

The learning source information receiving unit 521 receives the learning source information from, for example, the user terminal 4. However, the learning source information may be stored in a recording medium such as a disk or semiconductor memory, and the learning source information receiving unit 521 may read the learning source information from such a recording medium. In addition, the patent information included in the learning source information received by the learning source information receiving unit 521 may be a patent identifier. In that case, the learning source information receiving unit 521 may receive patent information corresponding to the patent identifier included in the learning source information received from the user terminal 4, for example, from the server device 3. Note that the above items also apply to the learning source information learning receiving unit 621 constituting the learning device production device 6.

The processing unit 53 performs various types of processing. The various types of processing are, for example, the processing of the learning unit 531, the prediction unit 532, the evaluation unit 533, the algorithm information acquisition unit 534, the learning source information acquisition unit 535, and the statistical processing unit 536. The processing unit 53 also performs, for example, various judgments and controls of various counters that are explained in the flowcharts.

The processing unit 53 may perform the processing performed by the important information acquisition unit 132. In other words, the processing unit 53 may have the important information acquisition unit 132.

The learning unit 531 constituting the processing unit 53 executes one or more learning device generation programs using, for example, at least a portion of the patent classification information of the learning source information accepted by the learning source information accepting unit 521, and obtains one or more learning devices.

Here, the "patent classification information of at least a part of the learning source information accepted by the learning source information accepting unit 521" used by the learning device 531 is, for example, N percent of the patent classification information of the learning source information. N percent is, for example, 90%, but it could also be 80% or 95%, and the numerical value is not important.

The learning unit 531 may, for example, execute one or more learning device generation programs stored in the program storage unit 511 for each of two or more pieces of learning source information acquired by the learning source information acquisition unit 535, and acquire two or more learning devices.

The learning unit 531 may, for example, execute two or more learning device generation programs using at least a portion of the patent classification information of the learning source information accepted by the learning source information accepting unit 521, and obtain two or more learning devices.

The prediction unit 532 may, for example, apply a prediction program to each of at least a portion of the patent classification information among the learning source information accepted by the learning source information accepting unit 521 using one or more learning devices generated by the learning unit 531, and obtain two or more predicted classification information for each of the one or more patent classification information.

Here, the "at least some of the patent classification information among the learning source information accepted by the learning source information accepting unit 521" that is the prediction target of the prediction unit 532 is usually one or more pieces of patent classification information that are not used by the learning unit 134. For example, when the learning device 531 uses N percent (e.g., 90%) of the learning source information, it is preferable that the prediction target of the prediction unit 532 is "10-N" percent (e.g., 10%) of the learning source information. However, the learning source information may include, for example, patent classification information that is not used for learning and is not the subject of prediction, or it may include patent classification information that is used for learning and is also the subject of prediction.

The prediction unit 532 may, for example, apply a prediction program to each of at least a portion of the patent classification information among the learning source information accepted by the learning source information accepting unit 521 using two or more learning devices generated by the learning unit 531, and obtain two or more predicted classification information for each of one or more patent classification information.

The prediction unit 532 may, for example, apply a prediction program using two or more learning devices obtained for each combination of algorithm type (e.g., deep learning 1, random forest, deep learning 2, etc.) and information type used (e.g., abstract and classification code, abstract, claims and classification code, etc.), obtain two or more primitive scores corresponding to two or more pieces of classification information such as "A, B, C" or "○, ×", and obtain the classification information with the highest primitive score as the predicted classification information. Note that the primitive score is the score output by the prediction program.

As a result, for example, a primitive score (=S1) for classification information "A", a primitive score (=S2) for classification information "B", and a primitive score (=S3) for classification information "C" are obtained. Then, for example, if S1>S2>S3, the classification information "A" corresponding to the highest score S1 may be obtained as the predicted classification information.

The evaluation unit 533 uses each of the two or more pieces of predicted classification information acquired by the prediction unit 532 and the classification information contained in the learning source information to acquire accuracy information regarding the accuracy of each of the two or more pieces of predicted classification information, and determines the predicted classification information with the highest accuracy specified by the accuracy information. The accuracy is, for example, the recall rate, the precision rate, the F-measure, or the accuracy rate.

The evaluation unit 533 may, for example, use two or more pieces of prediction classification information including one or more pieces of prediction classification information acquired by the statistical processing unit 536 instead of or in addition to the two or more pieces of prediction classification information acquired by the prediction unit 532, and classification information contained in the learning source information, to acquire accuracy information regarding the accuracy of each of the two or more pieces of prediction classification information, and determine the prediction classification information with the highest accuracy as identified by the accuracy information.

The evaluation unit 533 may, for example, use each of the two or more pieces of prediction classification information acquired by the prediction unit 532 and the classification information contained in the learning source information to acquire accuracy information for each of the two or more pieces of prediction classification information that is identified by the accuracy identification information, and determine the prediction classification information with the highest accuracy identified by the accuracy information.

The evaluation unit 533, for example, obtains accuracy information corresponding to the accuracy specification information stored in the accuracy specification information storage unit 512.

The evaluation unit 533 obtains accuracy information, for example, by using K-fold cross-validation on the learning source information.

The algorithm information acquisition unit 534 acquires algorithm information. The algorithm information is information about an algorithm corresponding to the predicted classification information determined by the evaluation unit 533. As described above, the predicted classification information determined by the evaluation unit 533 is the predicted classification information with the highest accuracy.

The algorithm information includes one or more of the following information: information that specifies the type of information to be used (information specification information described later), information that specifies the learning device generation program to be used (program specification information described later), and information that specifies the statistical processing (statistical processing specification information described later).

The algorithm information is, for example, information that can identify the learning source information corresponding to the most accurate predicted classification information among two or more pieces of learning source information acquired by the learning source information acquisition unit 535 described later. The information that can identify the learning source information may be, for example, information identifying information that identifies information contained in the learning source information. The information identifying information is usually information that identifies the type of information included in the patent information that constitutes the learning source information (for example, "abstract", "claims", "abstract + claims", etc.).

That is, the algorithm information acquisition unit 534 may acquire algorithm information including, for example, information identification information that identifies information contained in the learning source information corresponding to the predicted classification information determined by the evaluation unit 533.

Alternatively, the algorithm information may be, for example, information that identifies the learning device generation program that corresponds to the most accurate prediction classification information among two or more learning device generation programs stored in the program storage unit 511.

That is, the algorithm information acquisition unit 534 may acquire algorithm information including program identification information that identifies a learning device generation program corresponding to the prediction classification information determined by the evaluation unit 533, for example.

Alternatively, the algorithm information may be, for example, information that identifies the statistical processing corresponding to the most accurate predicted classification information among two or more statistical processing operations performed by the statistical processing unit 536 described below.

That is, the algorithm information acquisition unit 534 may acquire algorithm information including, for example, statistical processing specification information that specifies the statistical processing method corresponding to the predicted classification information determined by the evaluation unit 533.

Alternatively, the algorithm information may be information that specifies the accuracy desired by the user, such as recall, precision, F-measure, or accuracy rate.

That is, the algorithm information acquisition unit 534 may acquire algorithm information including the accuracy specification information accepted by the acceptance unit 52, for example.

The learning source information acquisition unit 535 acquires two or more pieces of learning source information each having two or more pieces of patent classification information having different information from the patent information contained in each of the two or more pieces of patent classification information contained in the learning source information accepted by the learning source information acceptance unit 521.

Here, each of the two or more pieces of learning source information having two or more pieces of patent classification information with different information is, for example, two or more pieces of learning source information in which the information contained in the patent information constituting each of the two or more pieces of patent classification information held by one piece of learning source information is different.

The learning source information acquisition unit 535 may acquire one or more types of important information from the specification in the patent information contained in each of two or more pieces of patent classification information contained in the learning source information accepted by the learning source information acceptance unit 521. In such a case, the learning source information acquisition unit 535 has an important information acquisition unit 132.

Specifically, for example, the patent information contained in each of the two or more patent classification information contained in the learning source information accepted by the learning source information acceptance unit 521 includes an abstract, a patent claim, and a specification, and the learning source information acquisition unit 535 may use the patent information contained in each of the two or more patent classification information contained in the accepted learning source information to acquire, for example, a first learning source information having two or more patent classification information each of which includes a summary in the patent information, a second learning source information having two or more patent classification information each of which includes a summary and a patent claim, a third learning source information having two or more patent classification information each of which includes a summary in the patent information, a third learning source information having two or more patent classification information each of which includes a summary and an effect solution statement, and a fourth learning source information having two or more patent classification information each of which includes a summary, a patent claim, and an effect solution statement.

The two or more pieces of learning source information mentioned above are, for example, "learning source information having two or more patent classification information having patent information and classification information consisting of an abstract and claims", "learning source information having two or more patent classification information having patent information and classification information consisting of an abstract", "learning source information having two or more patent classification information having patent information and classification information consisting of claims", "learning source information having two or more patent classification information having patent information and classification information consisting of an abstract and important information", and "learning source information having two or more patent classification information having patent information and classification information consisting of claims and important information".

Alternatively, the two or more pieces of learning source information each having two or more pieces of patent classification information with different information may be, for example, two or more pieces of patent classification information in which the combination of one or more pieces of patent classification information belonging to the learning part and one or more pieces of patent classification information belonging to the prediction part is different when the patent classification information of at least a part of the learning source information (hereinafter, the learning part) is used in the learning process of the learning unit 531 and each piece of patent classification information of at least a part of the learning source information (hereinafter, the prediction part) is the subject of prediction by the prediction unit 532.

Specifically, for example, if the learning source information receiving unit 521 receives learning source information containing patent classification information for 100 items, the learning source information acquiring unit 535 may divide the 100 items into 10 groups, numbered first through ten (generally, into K groups), and assign 9 of the 10 groups ((K-1) groups) to the learning part and the remaining group to the prediction part, repeating this process 10 times while changing the group combinations (K-fold cross-validation).

This allows 10 pieces of learning source information to be obtained, for example: first learning source information in which 90 items belonging to the first to ninth nine groups are the learning portion and 10 items belonging to the tenth group are the prediction portion; second learning source information in which 90 items belonging to the first to eighth and tenth nine groups are the learning portion and 10 items belonging to the ninth group are the prediction portion; third learning source information in which 90 items belonging to the first to seventh, ninth and tenth nine groups are the learning portion and 10 items belonging to the eighth group are the prediction portion; and tenth learning source information in which 90 items belonging to the second to tenth nine groups are the learning portion and 10 items belonging to the first group are the prediction portion.

The statistical processing unit 536 performs one or more types of statistical processing using two or more pieces of predicted classification information acquired by the prediction unit 532, and acquires one or more pieces of predicted classification information that are the results of the statistical processing. The statistical processing is processing related to statistics. The statistical processing is, for example, a logical sum (OR) or logical product (AND) of the predicted classification information, or a majority vote, but any processing related to statistics is acceptable.

The statistical processing unit 536 may, for example, perform two or more types of statistical processing and obtain two or more pieces of predicted classification information that are the results of the statistical processing. Two or more types of statistical processing are, for example, two or more types of statistical processing of the logical sum, logical product, or majority vote of the above-mentioned predicted classification information.

Here, the score for taking the logical sum, logical product, or majority vote is obtained, for example, using two or more operation results of a primitive algorithm (i.e., the primitive scores obtained for each of two or more pieces of classification information before the prediction unit 532 obtains the predicted classification information, which is the final result).

The statistical processing unit 536 preferably compares the results of each of the multiple operations of the primitive algorithm with the final result, and obtains a higher score the greater the number or percentage of matches.

When the scores contained in the multiple operation results of the primitive algorithm are all the same, it is preferable for the statistical processing unit 536 to compare the multiple operation results of the primitive algorithm with the final result, and obtain a higher score the greater the number or percentage of matches.

For example, the statistical processing unit 536 calculates the score in the case of taking a logical sum, a logical product, or a majority vote, for example, as follows.
(1) If all of the multiple operation results of the primitive algorithm match the final result (i.e., the predicted classification information obtained by the prediction unit 532), the statistical processing unit 536 assigns the highest score among the two or more operation results of the primitive algorithm.
(2) If more than 50% of the results of multiple operations of a primitive algorithm differ from the final result, the lowest score (0.5 in the case of binary classification) is assigned.
(3) In cases other than (1) and (2) above (i.e., when the percentage is less than 100% and more than 50% is the same as the final result), the statistical processing unit 536 assigns the lowest score among the multiple scores of the primitive algorithms.

The output unit 54 outputs various types of information. Examples of the various types of information include algorithm information. As mentioned above, output is, for example, transmission to the user terminal 4, but it may also be considered as a concept that includes display on a display, printing on a printer, sound output from a speaker, storage on a recording medium, and delivery of the processing results to other processing devices or other programs.

The output unit 54 stores the algorithm information acquired by the algorithm information acquisition unit 534, for example, in association with the learning source information.

The output unit 54 transmits or delivers the algorithm information acquired by the algorithm information acquisition unit 534, for example, to the learning device production device 6. However, the algorithm information may be transmitted to the user terminal 4, displayed on a display, or stored in a recording medium, for example, and the output form and destination are not important.

The learning storage unit 61 constituting the learning device production device 6 may store various types of information. Examples of various types of information include learning devices, learning source information, and algorithm information. The learning storage unit 61 may also store, for example, the aforementioned effective solution sentences, the aforementioned non-effective solution sentences, the aforementioned solution terms, the aforementioned non-solution terms, the aforementioned effect terms, the aforementioned solution sentences, the aforementioned non-solution sentences, the aforementioned effect sentences, the aforementioned non-effect sentences, the aforementioned objective sentences, the aforementioned non-objective sentences, the aforementioned objective terms, and the aforementioned non-objective terms. Other information may be explained at appropriate times.

The learning reception unit 62 receives various types of information. For example, various types of information are the learning source information described above. As described above, reception usually means reception from the user terminal 4, but it may also be considered to be a concept that includes reception of information input from an input device such as a keyboard, reception of information read from a recording medium such as a disk or semiconductor memory, etc.

The learning source information receiving unit 621 receives the learning source information. The learning source information receiving unit 621 typically receives the learning source information from the user terminal 4. However, the learning source information may be received via an input device such as a keyboard, or may be read from a recording medium such as a disk, and the manner of reception is not important.

The learning processing unit 13 includes a patent specification acquisition unit 131, an important information acquisition unit 132, a learning source information acquisition unit 133, and a learning unit 134. The important information acquisition unit 132 includes an effect solution means sentence acquisition unit 1321, an effect sentence acquisition unit 1322, a solution means sentence acquisition unit 1323, an objective sentence acquisition unit 1324, a feature word acquisition unit 1325, a related word acquisition unit 1326, an important tag corresponding sentence acquisition unit 1327, a numeric range character string acquisition unit 1328, and a numeric acquisition unit 1329, which basically perform the same processing as in the first embodiment.

However, the learning unit 134 performs a learning process on the learning source information accepted by the learning source information acceptance unit 621 or the learning source information acquired by the learning source information acquisition unit 133 using an algorithm according to the algorithm information output by the algorithm determination device 5, and acquires a learning device.

The learning unit 134, for example, acquires information identified by information identifying information contained in the algorithm information from the learning source information accepted by the learning source information accepting unit 621, and acquires learning source information having two or more patent classification information that includes the information and the classification information contained in the learning source information. In addition, the learning unit 134 provides the acquired learning source information to a program identified by program identifying information contained in the algorithm information, for example, and performs a learning process to configure a learning device. Note that if the information identifying information includes information that identifies important information, the learning unit 134 performs important information acquisition as performed by the important information acquisition unit 132.

The learning unit 134 may, for example, acquire information identified by information identifying information contained in the algorithm information from the learning source information accepted by the learning source information accepting unit 621, and acquire two or more pieces of learning source information having two or more pieces of patent classification information containing the information and the classification information contained in the learning source information. The learning unit 134 may also provide each of the two or more pieces of acquired learning source information to one or more programs identified by program identifying information contained in the algorithm information, perform a learning process, and configure two or more learners. Note that if the information identifying information includes information identifying important information, the learning unit 134 performs the important information acquisition performed by the important information acquisition unit 132.

For example, if the algorithm information output by the algorithm determination device 5 is (a) information that can identify the learning source information corresponding to the most accurate predicted classification information among two or more pieces of learning source information, the learning unit 134 acquires a learning device using the learning source information identified by the algorithm information.

Alternatively, if the algorithm information is (b) information that identifies a learning device generation program that corresponds to the most accurate predicted classification information among two or more learning device generation programs stored in the program storage unit 511, the learning unit 134 obtains a learning device using the learning device generation program identified by the algorithm information.

Alternatively, if the algorithm information is (c) information that identifies a statistical process that corresponds to the prediction classification information with the highest accuracy among two or more statistical processes, the learning unit 134 obtains two or more learning devices using two or more pieces of learning source information corresponding to the information that identifies the statistical process, two or more learning device generation programs corresponding to the information that identifies the statistical process, two or more pieces of learning source information and one or more learning device generation programs corresponding to the information that identifies the statistical process, or one or more pieces of learning source information and two or more learning device generation programs corresponding to the information that identifies the statistical process.

For example, when the algorithm information corresponds to information that specifies the type of accuracy desired by the user, among the recall rate, precision rate, F-value, or accuracy rate (i.e., the accuracy specification information received by the reception unit 52), the learning unit 134 acquires a learning device for increasing the accuracy specified by the algorithm information. Note that the above-mentioned effect can be achieved if the learning unit 134 generates a learning device according to the algorithm information.

The storage unit 641 stores the learning devices acquired by the learning unit 134. The storage unit 641 usually stores the learning devices acquired by the learning unit 134 in association with the algorithm information output by the algorithm determination device 5. The learning devices are usually stored in the learning device storage unit 211 constituting the patent classification device 2. However, the learning devices may also be stored in the learning storage unit 61, for example, and the storage destination is not important.

The learning module storage unit 211 constituting the patent classification device 8 stores one or more learning modules acquired by the learning module production device 6. A learning module is usually associated with algorithm information.

The reception unit 81 receives classification target information including a patent identifier that identifies the patent to be classified.

The prediction unit 821 performs prediction processing using a learning device according to an algorithm in accordance with the algorithm information associated with the learning device in the learning device storage unit 211 for the classification target information received by the reception unit 81, and obtains classification result information.

The prediction unit 821 acquires information to be used in the prediction process, which is, for example, information corresponding to the classification target information accepted by the acceptance unit 81 and information specified by the information specifying information contained in the algorithm information. Next, the prediction unit 821 provides the information to be used in the prediction process to, for example, one or more prediction programs specified by one or more program specifying information contained in the algorithm information, executes the one or more prediction programs, and acquires classification result information for each prediction program.

In addition, if the algorithm information includes statistical processing specific information, the prediction unit 821 statistically processes the classification result information for each of two or more prediction programs to obtain final classification result information.

The prediction unit 72 may apply a learning device to the classification target processing information acquired by the classification target processing information acquisition unit 233, predict the classification of the classification target processing information using a machine learning algorithm, and acquire classification result information.

The output unit 73 outputs the classification result information acquired by the prediction unit 72 in association with the patent identifier included in the classification target information accepted by the acceptance unit 71.

The storage unit 51, the program storage unit 511, the accuracy specification information storage unit 512, and the learning storage unit 61 are preferably non-volatile recording media, but can also be realized using volatile recording media.

The process by which information is stored in the storage unit 51, etc. is not important. For example, information may be stored in the storage unit 51, etc. via a recording medium, information transmitted via a communication line, etc. may be stored in the storage unit 51, etc., or information inputted via an input device may be stored in the storage unit 51, etc.

The reception unit 52, the learning source information reception unit 521, the output unit 54, the learning reception unit 62, the learning source information reception unit 621, the learning output unit 64, the storage unit 641, and the reception unit 81 can typically be realized by wireless or wired communication means, etc.

The processing unit 53, learning unit 531, prediction unit 532, evaluation unit 533, algorithm information acquisition unit 534, learning source information acquisition unit 535, statistical processing unit 536, processing unit 82, prediction unit 821, etc. can usually be realized by a processor, memory, etc. The processing procedure of the processing unit 53, etc. is usually realized by software, and the software is recorded in a recording medium such as a ROM. However, it may also be realized by hardware (dedicated circuit). The processor may be, for example, a CPU, MPU, GPU, etc.

Next, an example of the operation of the patent classification system B will be described. First, an example of the operation of the algorithm determination device 5 will be described using the flowchart in FIG. 13.

(Step S1301) The processing unit 53 obtains all candidates for algorithm information.

For example, when the storage unit 51 stores one or more pieces of information specifying information (e.g., three pieces of "abstract", "abstract, patent classification code", and "abstract, effect solution statement"), one or more pieces of program specifying information (e.g., two pieces of "deep learning" and "random forest"), and one or more pieces of statistical processing specifying information (e.g., "AND" and "OR"), the processing unit 53 acquires six algorithm information candidates that are combinations of any one of the three pieces of information specifying information and any one of the two pieces of program specifying information. The processing unit 53 also acquires six algorithm information candidates that are combinations of any one of the three pieces of information specifying information and two types of statistical processing specifying information of the predicted results of the programs specified by each of the two pieces of program specifying information. That is, in this example, the processing unit 53 acquires 12 algorithm information candidates.

(Step S1302) The processing unit 53 assigns 1 to the counter i.

(Step S1303) The processing unit 53 determines whether or not a candidate for the i-th algorithm information exists among the candidates for algorithm information acquired in step S1301. If a candidate for the i-th algorithm information exists, the process proceeds to step S1304, and if a candidate for the i-th algorithm information does not exist, the process proceeds to step S1306.

(Step S1304) The evaluation unit 533 evaluates the accuracy (e.g., recall, F-value, etc.) specified by the accuracy specification information stored in the storage unit 51 when the information and algorithm corresponding to the i-th algorithm information candidate are used, obtains the accuracy information, and temporarily stores the accuracy information in association with the i-th algorithm information candidate. This evaluation process will be described using the flowchart in FIG. 14.

(Step S1305) The processing unit 53 increments the counter i by 1. Return to step S1303.

(Step S1306) The algorithm information acquisition unit 534 acquires the accuracy information of the best value.

(Step S1307) The algorithm information acquisition unit 534 acquires, as algorithm information, candidates for algorithm information that are paired with the accuracy information of the best value acquired in step S1306.

(Step S1308) The algorithm information acquisition unit 534 accumulates algorithm information.

In the flowchart in Figure 13, processing ends when the power is turned off or an interrupt occurs to end processing.

Next, the evaluation process in step S1304 will be explained using the flowchart in FIG.

(Step S1401) The learning source information acquisition unit 535 acquires information identification information corresponding to candidate algorithm information.

(Step S1402) The learning source information acquisition unit 535 assigns 1 to counter i.

(Step S1403) The learning source information acquisition unit 535 determines whether the i-th patent information is present in the learning source information to be processed. If the i-th patent information is present, the process proceeds to step S1404, and if the i-th patent information is not present, the process proceeds to step S1406. Note that when proceeding to step S1406, the learning source information to be evaluated has been completed. In addition, the learning source information to be processed is stored, for example, in the storage unit 51.

(Step S1404) The learning source information acquisition unit 535 acquires information corresponding to the i-th patent information and identified by the information identification information acquired in step S1401. The evaluation unit 533 acquires patent classification information consisting of the classification information possessed by the learning source information to be processed and the acquired information, and temporarily stores the information in a buffer not shown.

(Step S1405) The learning source information acquisition unit 535 increments the counter i by 1. Return to step S1403.

(Step S1406) The evaluation unit 533 determines whether or not there is statistical processing specific information corresponding to the candidate algorithm information. If there is statistical processing specific information, the process proceeds to step S1409. If there is no statistical processing specific information, the process proceeds to step S1407.

(Step S1407) The evaluation unit 533 obtains program-specific information corresponding to the candidate algorithm information.

(Step S1408) The evaluation unit 533 performs K-fold cross-validation using the learning program and the prediction program identified by the program identification information acquired in step S1407. An example of the K-fold cross-validation process is described with reference to the flowchart in FIG. 15.

(Step S1409) The evaluation unit 533 performs an evaluation when statistical processing specific information corresponding to the candidate algorithm information exists. An example of such a statistical processing evaluation process is described using the flowchart in FIG. 15.

(Step S1410) The evaluation unit 533 temporarily stores the acquired accuracy information in association with the candidate algorithm information. It then returns to the upper-level process.

Next, an example of the K-fold cross-validation process in step S1408 will be described using the flowchart in FIG. 15.

(Step S1501) The evaluation unit 533 divides the learning source information into k (k is a natural number greater than or equal to 2).

(Step S1502) The evaluation unit 533 assigns 1 to counter i.

(Step S1503) The evaluation unit 533 determines whether or not the i-th split pattern exists in the learning source information divided into k. If the i-th split pattern exists, the process proceeds to step S1504. If the i-th split pattern does not exist, the process proceeds to step S1513. Note that when divided into k, k split patterns usually exist. This is a well-known technique in K-fold cross-validation.

(Step S1504) The learning unit 531 acquires information to be learned (patent classification information to be used for the learning process in the k division pattern), which is part of the learning source information.

(Step S1505) The learning unit 531 provides the learning program identified by the program identification information with the information on the learning subject acquired in step S1504, executes the learning program, and acquires a learning device.

(Step S1506) The prediction unit 532 assigns 1 to counter j.

(Step S1507) The prediction unit 532 determines whether or not the jth piece of information to be predicted, which is part of the learning source information, exists. If the jth piece of information to be predicted exists, the process proceeds to step S1508; if the jth piece of information to be predicted does not exist, the process proceeds to step S1510.

(Step S1508) The prediction unit 532 provides the jth information of the prediction target to the prediction program identified by the program identification information, executes the prediction program, and obtains the prediction result.

(Step S1509) The prediction unit 532 increments the counter j by 1. Return to step S1507.

(Step S1510) The evaluation unit 533 obtains accuracy specification information corresponding to the candidate algorithm information.

(Step S1511) The evaluation unit 533 compares the set of prediction results acquired in step S1508 with the set of classification information contained in the patent classification information of the prediction target in the learning source information, and acquires accuracy information of the accuracy (e.g., recall, F-value) specified by the accuracy specification information acquired in step S1510. Note that such accuracy information is primary accuracy information.

(Step S1512) The evaluation unit 533 increments the counter i by 1. Return to step S1503.

(Step S1513) The evaluation unit 533 acquires accuracy information that is a representative value (e.g., average, median) of the k pieces of primary accuracy information acquired in step S1511, and stores the accuracy information in association with the candidate algorithm information. It then returns to the upper-level process.

Next, an example of the statistical processing evaluation process in step S1409 will be described using the flowchart in FIG. 16. Note that in the flowchart in FIG. 16, the description of the same steps as in the flowchart in FIG. 15 will be omitted.

(Step S1601) The evaluation unit 533 assigns 1 to counter j.

(Step S1602) The evaluation unit 533 determines whether the jth program-specific information exists. If the jth program-specific information exists, the process proceeds to step S1603. If the jth program-specific information does not exist, the process proceeds to step S1609.

(Step S1603) The learning unit 531 provides the learning program identified by the jth program identification information with the information on the learning target acquired in step S1504, executes the learning program, and acquires a learning device.

(Step S1604) The prediction unit 532 assigns 1 to counter l.

(Step S1605) The prediction unit 532 determines whether or not the lth piece of information to be predicted, which is part of the learning source information, exists. If the lth piece of information to be predicted exists, the process proceeds to step S1606, and if the lth piece of information to be predicted does not exist, the process proceeds to step S1608.

(Step S1606) The prediction unit 532 provides the lth information of the prediction target to the prediction program identified by the program identification information, executes the prediction program, and obtains the prediction result.

(Step S1607) The prediction unit 532 increments counter l by 1. Return to step S1605.

(Step S1608) The evaluation unit 533 increments the counter j by 1. Return to step S1602.

(Step S1609) The statistical processing unit 536 assigns 1 to counter l.

(Step S1610) The statistical processing unit 536 determines whether or not information on the lth prediction target exists. If information on the lth prediction target exists, the process proceeds to step S1611. If information on the lth prediction target does not exist, the process proceeds to step S1613.

(Step S1611) The statistical processing unit 536 performs statistical processing (e.g., "AND", "OR", "majority vote") specified by the statistical processing specific information possessed by the candidate algorithm information on two or more prediction results for the information of the lth prediction target, and obtains the statistical processing result.

(Step S1612) The statistical processing unit 536 increments counter l by 1. Return to step S1610.

(Step S1613) The evaluation unit 533 obtains accuracy specification information corresponding to the candidate algorithm information.

(Step S1614) The evaluation unit 533 compares the set of statistical processing results obtained in step S1611 with the set of classification information contained in the patent classification information of the prediction target in the learning source information, and obtains accuracy information of the accuracy (e.g., recall, F-value) specified by the accuracy specification information obtained in step S1613. Note that such accuracy information is primary accuracy information.

(Step S1615) The evaluation unit 533 increments the counter i by 1. Return to step S1503.

(Step S1616) The evaluation unit 533 acquires accuracy information that is a representative value (e.g., average, median) of the k pieces of primary accuracy information acquired in step S1614, and stores the accuracy information in association with the candidate algorithm information. It then returns to the upper-level process.

Next, an example of the operation of the learning module production device 6 will be explained using the flowchart in Figure 17.

(Step S1701) The learning acceptance unit 62 acquires learning source information.

(Step S1702) The learning processing unit 13 acquires algorithm information corresponding to the learning source information acquired in step S1701.

(Step S1703) The learning processing unit 13 acquires the information-specific information contained in the algorithm information.

(Step S1704) The learning processing unit 13 acquires information that corresponds to each patent classification information contained in the learning source information, which is information identified by the information identification information acquired in step S1703, and acquires information consisting of that information and the classification information contained in each patent classification information. The learning processing unit 13 then acquires the learning source information to be used in the learning process, which is learning source information that has the acquired information.

(Step S1705) The learning processing unit 13 assigns 1 to counter i.

(Step S1706) The learning processing unit 13 determines whether or not the i-th program-specific information exists in the algorithm information acquired in step S1702. If the i-th program-specific information exists, the process proceeds to step S1707, and if the i-th program-specific information does not exist, the process ends.

(Step S1707) The learning processing unit 13 retrieves the learning program identified by the program identification information retrieved in step S1707 from the learning storage unit 61.

(Step S1708) The learning processing unit 13 provides the learning source information acquired in step S1704 to the learning program acquired in step S1707, executes the learning program, and acquires a learning device. Then, the learning processing unit 13 stores the learning device in a pair with the algorithm information acquired in step S1702.

(Step S1709) The learning processing unit 13 increments the counter i by 1. Return to step S1706.

Next, an example of the operation of the patent classification device 8 will be explained using the flowchart in Figure 18.

(Step S1801) The reception unit 81 determines whether or not classification target information has been received. If classification target information has been received, the process proceeds to step S1802, and if classification target information has not been received, the process returns to step S1801.

(Step S1802) The processing unit 82 acquires the algorithm information stored in the storage unit 21.

(Step S1803) The processing unit 82 acquires the information-specific information contained in the algorithm information acquired in step S1802.

(Step S1804) The processing unit 82 acquires information that is identified by the information identification information acquired in step S1803 and corresponds to the information to be classified.

(Step S1805) The processing unit 82 acquires the program-specific information contained in the algorithm information acquired in step S1802.

(Step S1806) The processing unit 82 assigns 1 to counter i.

(Step S1807) The processing unit 82 determines whether the i-th program-specific information is present in the algorithm information acquired in step S1802. If the i-th program-specific information is present, the process proceeds to step S1808. If the i-th program-specific information is not present, the process proceeds to step S1811.

(Step S1808) The prediction unit 821 obtains from the storage unit 21 the prediction program identified by the program identification information obtained in step S1805.

(Step S1809) The prediction unit 821 provides the prediction program acquired in step S1808 with the learning device paired with the algorithm information acquired in step S1802 and the information acquired in step S1804, executes the prediction program, acquires a prediction result, and stores the prediction result in association with the classification target information. It is preferable that the prediction unit 821 also acquires a score.

(Step S1810) The processing unit 82 increments the counter i by 1. Return to step S1807.

(Step S1811) The prediction unit 821 determines whether or not statistical processing specific information is present in the algorithm information acquired in step S1802. If statistical processing specific information is present, the process proceeds to step S1812; if statistical processing specific information is not present, the process proceeds to step S1814.

(Step S1812) The statistical processing unit 536 performs statistical processing specified by the statistical processing specification information on the prediction results of the two or more prediction processes in step S1809, and obtains the statistical processing result. Note that the statistical processing result is the final prediction result.

(Step S1813) The statistical processing unit 536 uses the score obtained by the prediction unit 821 to obtain a score corresponding to the statistical processing result.

(Step S1814) The output unit outputs the final prediction result obtained in step S1812 and the score obtained in step S1813.

Below, we will explain a specific example of the operation of patent classification system B. Note that the following explanation can be modified in various ways and does not limit the scope of the present invention in any way.

In this example, the information specific information is either "Abstract, IPC" or "Abstract, Effect Solution Means Statement, IPC". The program specific information is either "Deep Learning.exe" or "Random Forest.exe". The statistical information specific information is either "AND" or "OR".

The storage unit 51 of the algorithm determination device 5 stores eight algorithm information candidates 1 to 8, for example, as shown in FIG. 19, which will be described later. The algorithm information candidates are candidates for algorithm information. The algorithm information candidates 1 to 4 are four algorithm information candidates that are combinations of one of the two types of information specifying information and one of the two types of program specifying information. The algorithm information candidates 5 to 8 are four algorithm information candidates that are combinations of one of the two types of information specifying information and two types of statistical processing specific information of the predicted results of the program specified by each of the two types of program specifying information. In this example, the eight algorithm information candidates 1 to 8 are stored in the storage unit 51 in advance, but may be acquired by the processing unit 53 in response to a user's instruction input through a setting screen, which will be described later, and stored in the storage unit 51.

In addition, the storage unit 51 may store eight pieces of accuracy information 1 to 8 corresponding to the eight algorithm information candidates 1 to 8, as shown in FIG. 20, which will be described later. The eight pieces of accuracy information 1 to 8 are acquired by the algorithm information acquisition unit 534 and stored in the storage unit 51 in association with the eight algorithm information candidates 1 to 8.

Figure 19 is a data structure diagram of an algorithm information candidate. An algorithm information candidate has information specific information, program specific information, and statistical information specific information. Algorithm information candidate 1 associated with ID "1" has information specific information "abstract, IPC", program specific information "deep learning.exe", and statistical information specific information "-". Note that "-" is information indicating that no statistical processing will be performed.

Algorithm information candidate 2 associated with ID "2" has information specific information "abstract, effect solution statement, IPC", program specific information "deep learning.exe", and statistical information specific information "-". Similarly, algorithm information candidate 3 has information specific information "abstract, IPC", program specific information "random forest.exe", and statistical information specific information "-". Algorithm information candidate 4 has information specific information "abstract, effect solution statement, IPC", program specific information "random forest.exe", and statistical information specific information "-".

Algorithm information candidate 5 has information specific information "abstract, IPC", program specific information "deep learning.exe, random forest.exe", and statistical information specific information "OR". Algorithm information candidate 6 has information specific information "abstract, effect solution means statement, IPC", program specific information "deep learning.exe, random forest.exe", and statistical information specific information "OR". Algorithm information candidate 7 has information specific information "abstract, IPC", program specific information "deep learning.exe, random forest.exe", and statistical information specific information "AND". Algorithm information candidate 8 has information specific information "abstract, effect solution means statement, IPC", program specific information "deep learning.exe, random forest.exe", and statistical information specific information "AND".

20 is a data structure diagram of accuracy information. The accuracy information has a recall rate, a precision rate, an F value, and a correct answer rate. Accuracy information 1 associated with ID "1" has a recall rate " _a1 ", a precision rate " _b1 ", an F value "( _a1 + _b1 )/2", and a correct answer rate " _c1 ". Accuracy information 2 associated with ID "2" has a recall rate " _a2 ", a precision rate " _b2 ", an F value "( _a2 + _b2 )/2", and a correct answer rate " _c2 ".

Similarly, the accuracy information 3 has a recall rate " _a3 ", a precision rate "b3", an F value "( _a3 + _b3 )/2", and a correct answer rate " _c3 ". The accuracy information 4 has a recall rate " _a4 ", a precision rate " _b4 ", an F value "( _a4 + _b4 )/2", and a correct answer rate " _c4 ". The accuracy information 5 has a recall rate " _a5 ", a precision rate " _b5 ", an F value "( _a5 + _b5 )/2", and a correct answer rate " _c5 ". The accuracy information 6 has a recall rate " _a6 ", a precision rate " _b6 ", an F value "( _a6 + _b6 )/2", and a correct answer rate " _c6 ". The accuracy information 7 has a recall rate "a ₇ ", a precision rate "b ₇ ", an F-measure "(a ₇ +b ₇ /2", and an accuracy rate "c ₇ ". The accuracy information 8 has a recall rate "a ₈ ", a precision rate "b ₈ ", an F-measure "(a ₈ +b ₈ )/2", and an accuracy rate "c ₈ ".

The user inputs learning source information having 100 pieces of patent classification information, each composed of a patent identifier and classification information, into the user terminal 4. The user terminal 4 accepts the input of the learning source information and transmits the accepted learning source information to the algorithm determination device 5.

The program storage unit 511 stores two learning device generation programs corresponding to the two types of program specific information described above, and two prediction programs corresponding to the two learning device generation programs.

The accuracy specification information storage unit 512 stores one of the four pieces of accuracy specification information, "recall rate," "precision rate," "F-value," and "accuracy rate" (here, "accuracy rate"). Note that the stored accuracy specification information is, for example, acquired by the processing unit 53 in response to a user instruction input via a setting screen described below and accumulated in the storage unit 51, but may be default information or information acquired automatically.

In the algorithm determination device 5, the learning source information receiving unit 521 receives the learning source information, and the processing unit 53 performs the following processing using the received learning source information and the eight algorithm information candidates 1 to 8 stored in the storage unit 51.

In detail, the learning source information acquisition unit 535 constituting the processing unit 53 acquires, for each of the two or more (here, 100) pieces of patent classification information contained in the received learning source information, two or more pieces of patent classification information each having two or more pieces of patent classification information that have information identified by the two types of information identification information constituting the algorithm information candidate from among the patent information contained in the patent classification information. As a result, for one accepted patent classification information, two or more pieces of patent classification information having different information (here, two patent classification information pieces: patent classification information having "abstract, IPC" and patent classification information having "abstract, effect solution statement, IPC") are acquired.

Next, for each of the two or more pieces of acquired learning source information, the learning unit 531 executes one of the two learning device generation programs stored in the program storage unit 511 that is identified by the program identification information of the algorithm information candidate, using at least a portion (e.g., 90 pieces) of patent classification information of the learning source information, to acquire two learning devices. The prediction unit 532 uses each of the two acquired learning devices to apply a prediction program corresponding to the learning device generation program used to acquire the learning device to at least a portion (e.g., 10 pieces not used to generate the learning device) of the patent classification information of the learning source information, and acquires two or more pieces (here, four pieces) of predicted classification information for each of the two or more pieces (here, the above 10 pieces) of patent classification information.

The statistical processing unit 535 uses the two or more pieces of predicted classification information acquired to perform one of the two statistical processes (AND, OR) described above that is specified by the statistical information specification information, and acquires two or more pieces of predicted classification information (here, four pieces) that are the results of the statistical processing.

The evaluation unit 533 uses the four or more pieces of prediction classification information thus acquired (here, four pieces corresponding to the combination of any one of the two types of information identification information and any one of the two types of program identification information, and four pieces corresponding to the combination of any one of the two types of information identification information and any one of the two types of statistical processing identification information of the predicted results of the program identified by each of the two types of program identification information, for a total of eight pieces) and the classification information contained in the learning source information to acquire accuracy information for each piece of prediction classification information, and stores it in the storage unit 51 in association with the algorithm information candidates.

Of the eight pieces of predicted classification information, for the first four pieces of predicted classification information, the accuracy information corresponding to the predicted classification information may be obtained, for example, by the K-fold cross-validation described in the flowchart of FIG. 15, and for the latter four pieces of predicted classification information, the accuracy information corresponding to the predicted classification information may be obtained, for example, by the statistical processing evaluation described in the flowchart of FIG. 16.

As a result, the eight pieces of accuracy information 1 to 8 shown in FIG. 20 are stored in the storage unit 51 in correspondence with the eight algorithm information candidates 1 to 8 shown in FIG. 19.

The algorithm information acquisition unit 534 acquires the best value (e.g., maximum value) of the accuracy information 1 to 8 stored in the storage unit 51 for the accuracy (here, accuracy rate) specified by the accuracy specification information stored in the accuracy specification information storage unit 512. For example, when _c3 is the best value among the accuracy rates _c1 to c8 corresponding to the accuracy information 1 to ₈ , the accuracy information 3 corresponding to the best value is acquired.

For example, when the stored accuracy specification information is information indicating that all four types of accuracy are used (for example, for each of the four types of accuracy, x points (for example, 1 point) are added to the accuracy information corresponding to the best value to obtain the accuracy information corresponding to the highest score), the best value among the recall rates _a1 to _a8 is _a2 , the best value among the precision rates _b1 to _b8 is _b3 , the best value among the F-values "( _a1 + _b1 ) / 2" to "( _a8 + _b8 ) / 2" is "( _a2 + _b2 ) / 2", and the best value among the accuracy rates c1 to _c8 is _c4 , " ₁ point + 1 point = 2 points" is obtained for the accuracy information 2, "1 point" is obtained for the accuracy information 3, "1 point" is obtained for the accuracy information 4, and "0 point" is obtained for each of the other accuracy information 1, 5 to 8, and the accuracy information 2 corresponding to the highest point of 2 points among the 8 points thus obtained may be obtained.

Also, for example, the stored accuracy specification information is information indicating that one of four types of accuracy (for example, precision) is emphasized (for example, for each of the four types of accuracy, x points (for example, 1 point) are added to the accuracy information corresponding to the best value, and if the accuracy information to which x points have been added includes the accuracy information to be emphasized, y points (for example, 2 points) are further added to the accuracy information to be emphasized), the best value of the recall rates _a1 to _a8 is _a2 , the best value of the precision rates _b1 to _b8 is _b3 , the best value of the F-values "( _a1 + _b1 )/2" to "( _a8 + _b8 )/2" is "( _a2 + _b2 )/2", and the best value of the accuracy rates _c1 to _c8 is "(a3 + b3)/2". If ₃ is the best value, then "1 point + 1 point = 2 points" is obtained for accuracy information 2, and "(1 point + 2 points) + 1 point = 4 points" is obtained for accuracy information 3, and "0 points" is obtained for each of the other accuracy information 1, 4 to 8, and accuracy information 3 corresponding to 4 points, the highest point of the 8 points obtained in this way, may be obtained.

The algorithm information acquisition unit 534 acquires algorithm information that pairs with the accuracy information of the best value (for example, when accuracy information 3 is acquired, algorithm information candidate 3 that pairs with the accuracy information 3) from the storage unit 51 as algorithm information. The algorithm information acquisition unit 534 stores the acquired algorithm information in the storage unit 51 in association with the received learning source information.

In this way, the output unit 54 outputs the algorithm information acquired by the algorithm information acquisition unit 534 in response to the learning source information being received by the learning source information acceptance unit 521. The output destination of the algorithm information is usually the learning machine production device 6.

In the learning machine production device 6, the learning processing unit 13 accumulates the algorithm information output from the algorithm determination device 5 in the learning storage unit 61, and the learning unit 134 uses the algorithm information stored in the learning storage unit 61 to acquire a learning machine, for example, according to the procedure shown in the flowchart of FIG. 17. The acquired learning machine is handed over to the patent classification device 8.

The algorithm information acquired by the learning machine production device 6 is also output to the patent classification device 8. In the patent classification device 8, the processing unit 82 accumulates the learning machine delivered from the learning machine production device 6 and the algorithm information output from the learning machine production device 6 in the storage unit 21. The reception unit 81 then receives classification target information from the user terminal 4, and the processing unit 82 executes, for example, the process shown in the flowchart of FIG. 18 for the received classification target information using the learning machine and algorithm information stored in the storage unit 21, and acquires predicted classification information and a score corresponding thereto. The output unit 24 transmits the acquired patent classification information and score to the user terminal 4, which is the sender of the classification target information. As a result, the patent classification information and score are displayed on the display of the user terminal 4.

The algorithm information acquired by the learning machine production device 6 may also be transmitted to the user terminal 4, which is the transmission source of the learning source information. The user terminal 4 outputs the algorithm information transmitted from the algorithm determination device 5 via an output device such as a display. As a result, the algorithm information is displayed on the display of the user terminal 4.

As described above, according to this embodiment, the algorithm determination device 5 can determine an appropriate process for performing prediction processing.

In addition, the algorithm determination device 5 can determine an appropriate process for performing the prediction process by changing the data used when performing the prediction process.

In addition, the algorithm determination device 5 can determine an appropriate process for performing prediction processing by using multiple learning device generation programs.

The algorithm determination device 5 can also determine an appropriate process for performing prediction processing by using prediction results obtained by performing statistical processing on prediction results obtained using two or more learning devices.

The algorithm determination device 5 can also determine processing including appropriate statistical processing for performing prediction processing by using two or more prediction results obtained by performing two or more statistical processing on prediction results obtained using two or more learning devices.

The algorithm determination device 5 can also determine the process that has the highest score desired by the user (the accuracy information identified by the accuracy identification information accepted by the acceptance unit 52) among the recall rate, precision rate, F-value, or accuracy rate.

In addition, the learning module production device 6 can produce an appropriate learning module by performing a learning process using an algorithm according to the algorithm information output by the algorithm determination device 5.

In addition, the patent classification device 2 can classify patents with high accuracy by performing prediction processing using an appropriate learning device produced by the learning device production device 6.

The patent classification device 2 can classify patents with higher accuracy by using the appropriate learning machine produced by the learning machine production device 6 and important information about the patent specification.

The software for realizing the algorithm determination device 5 in this embodiment is a program as follows. In other words, this program includes a computer that can access a program storage unit 511 that stores one or more learning device generation programs for generating a learning device for classifying patent information relating to a patent using learning source information having two or more patent classification information having patent information relating to a patent and classification information indicating a classification for the patent information, and a prediction program for predicting the classification of the patent classification information using a learning device and acquiring the predicted classification information, a learning source information receiving unit 521 that receives the learning source information, a learning unit 531 that executes each of the one or more learning device generation programs and acquires one or more learning devices using at least a part of the patent classification information of the learning source information received by the learning source information receiving unit 521, and a program storage unit 511 that stores a prediction program for predicting the classification of the patent classification information using a learning device and acquiring the predicted classification information. A prediction unit 532 applies a prediction program to at least some of the patent classification information of the attached learning source information using one or more learning devices generated by the learning unit 531, and acquires two or more predicted classification information for each of the one or more patent classification information; an evaluation unit acquires accuracy information regarding the accuracy of each of the two or more predicted classification information using each of the two or more predicted classification information acquired by the prediction unit 532 and the classification information contained in the learning source information, and determines the predicted classification information with the highest accuracy specified by the accuracy information; an algorithm information acquisition unit 534 acquires algorithm information regarding the algorithm corresponding to the predicted classification information determined by the evaluation unit; and a program for functioning as an output unit that outputs the algorithm information.

The software for realizing the learning device production device 6 in this embodiment is the following program. In other words, this program causes a computer to function as a learning source information receiving unit 621 that receives learning source information, a learning unit 134 that performs a learning process on the learning source information received by the learning source information receiving unit 621 using an algorithm according to the algorithm information output by the algorithm determination device 5 to obtain a learning device, and a storage unit 641 that stores the learning device in association with the algorithm information.

Furthermore, the software for realizing the patent classification device 2 in this embodiment is the following program. In other words, this program causes a computer that can access a learning device storage unit in which learning devices accumulated by the learning device production device 6 are stored to function as a reception unit that receives classification target information including a patent identifier that identifies the patent to be classified, a prediction unit 234 that performs a prediction process on the classification target information using the learning device according to an algorithm in accordance with the algorithm information associated with the learning device and obtains classification result information, and an output unit that outputs the classification result information obtained by the prediction unit 234 in association with the patent identifier included in the classification target information received by the reception unit.

(Embodiment 3)
FIG. 21 is a block diagram of a patent classification system C in this embodiment.

The patent classification system C includes an algorithm determination device 5, a learning device production device 1, a patent classification device 2, a server device 3, and one or more user terminals 4.

The algorithm determination device 5 is the same element as in embodiment 2. The learning device production device 1, the patent classification device 2, the server device 3, and the user terminal 4 are the same elements as in embodiment 1.

However, the algorithm used by the learning processing unit 13 constituting the learning device production device 1 is the algorithm determined by the algorithm determination device 5.

The important information acquisition unit 132 constituting the patent classification device 2 includes an effect/solution means sentence acquisition means 1321, an effect sentence acquisition means 1322, a solution means sentence acquisition means 1323, an objective sentence acquisition means 1324, a feature word acquisition means 1325, a related word acquisition means 1326, an important tag corresponding sentence acquisition means 1327, a numeric range string acquisition means 1328, and a number acquisition means 1329.

Next, the operation of the patent classification system C will be described. The flowcharts of the algorithm determination device 5 constituting the patent classification system C are shown in Figs. 13 to 16. The flowcharts of the learning module production device 1 are shown in Figs. 4 to 9. The flowchart of the patent classification device 2 is shown in Fig. 10.

As described above, according to this embodiment, in addition to the effects described in embodiment 1, the effects described in embodiment 2 can be obtained.

(Embodiment 4)
In this embodiment, a patent classification system D is described that is equipped with a classification information determination device in which condition information, which is a pair of a condition and a class (classification information), is stored, and for a prediction target (patent information) that matches the one condition, the class that pairs with the one condition is determined, and when the one condition is not matched, the class is determined using a machine learning algorithm.

In addition, in this embodiment, the machine learning algorithm is, for example, the same as that described in the first to third embodiments, but the type is not important. We will explain the patent classification system D equipped with the classification information determination device.

In addition, in this embodiment, we will explain a patent classification system D that is equipped with a classification information determination device, where one condition is that the patent is the training data used when creating a machine learning learning device.

Figure 22 is a block diagram of a patent classification system D in this embodiment. The patent classification system D includes a learning module production device 1, a classification information determination device 7, a server device 3, and a user terminal 4. In the patent classification system D, a learning module production device 6 may be used instead of the learning module production device 1.

The learning module production device 1 has the same configuration as that in the first embodiment and performs the same operation. That is, the learning module production device 1 uses the learning source information to perform a machine learning learning process and obtain a learning module.

The classification information determination device 7 includes a storage unit 71, a reception unit 72, a processing unit 73, and an output unit 74. The storage unit 71 includes a learning device storage unit 711, and a condition information storage unit 712. The reception unit 72 includes a patent information reception unit 721. The processing unit 73 includes a prediction unit 731. The prediction unit 731 includes a judgment means 7311, a first decision means 7312, and a second decision means 7313.

The storage unit 71 constituting the classification information determination device 7 stores various types of information. The various types of information include, for example, the learning device described above and condition information described below. The storage unit 71 may also store, for example, one or more pieces of patent information, or patent classification information.

The learning device storage unit 711 stores learning devices. In this embodiment, the learning device production device 1 performs a machine learning learning process using the learning source information to obtain a learning device. As described above, the learning source information has two or more pieces of patent classification information, and the patent classification information has patent information and classification information of the patent information. The obtained learning device is handed over to the classification information determination device 7 and is accumulated in the learning device storage unit 711, for example, by the processing unit 73 described later.

However, the learning device may be stored in advance in the learning device storage unit 711, or may be acquired within the classification information determination device 7. In the latter case, for example, learning source information is stored in the storage unit 71, and the classification information determination device 7 may include a learning unit (not shown) that performs machine learning learning processing using the learning source information and acquires the learning device.

The condition information storage unit 712 stores one or more pieces of condition information. The condition information is a combination of a condition related to a patent and classification information of the patent information. In this embodiment, the condition is, for example, that the patent information to be predicted is patent information that constitutes any one of two or more pieces of patent classification information contained in the learning source information. The condition is, for example, a search formula for a patent. For example, the search formula is "Name of invention includes 'machine learning'" and "Applicant=ABC & IPC=G06F*".

As mentioned above, patent information is information about patents, such as all or part of a patent specification, such as a patent gazette or published patent gazette. As mentioned above, classification information is information that indicates the classification of the patent information, and can be anything, such as "○, ×" or "A, B, C."

The condition information includes, for example, a patent identifier (e.g., publication number, patent number, etc.) corresponding to one of the two or more pieces of patent classification information contained in the learning source information, and classification information corresponding to the one piece of patent classification information. For example, if the learning device is obtained using a portion of the two or more pieces of patent classification information contained in the learning source information, the condition information includes a patent identifier corresponding to one of the two or more pieces of patent classification information used when obtaining the learning device, and classification information corresponding to the one piece of patent classification information.

However, the condition information may be, for example, one of two or more pieces of patent classification information contained in the learning source information. For example, if the learning device is obtained using a portion of two or more pieces of patent classification information contained in the learning source information, the condition information may be one of the two or more pieces of patent classification information used when obtaining the learning device.

The learning module production device 1 may, for example, deliver to the classification information determination device 7, together with the learning module, the learning source information used to obtain the learning module.

In the classification information determination device 7, for example, the processing unit 73 receives the learning source information, and uses the learning source information to acquire two or more pieces of condition information having a patent identifier and classification information, and stores the information in the learning device storage unit 711. However, the acquisition of the two or more pieces of condition information may be performed on the learning device production device 1 side, and the acquired two or more pieces of condition information may be handed over to the classification information determination device 7.

As a result, the condition information storage unit 712 stores two or more pieces of condition information (e.g., 90 pieces of condition information 1 to 90 shown in FIG. 25 described later) corresponding to two or more pieces of patent classification information (e.g., 90 pieces of patent classification information 11 to 100 out of the 100 patent classifications 1 to 100 constituting the aforementioned patent classification information) used when the learning device production device 1 obtained the learning device.

The reception unit 72 receives various types of information. For example, various types of information are patent information. As mentioned above, reception usually means reception from the user terminal 4, but it may also be considered to be a concept that includes, for example, reception of information input from an input device such as a keyboard, and reception of information read from a recording medium such as a disk or semiconductor memory.

The patent information receiving unit 721 receives one or more pieces of patent information. The patent information receiving unit 721 typically receives one or more pieces of patent information from the user terminal 4. However, for example, one or more pieces of patent information may be stored on a recording medium such as a disk, and the patent information receiving unit 721 may read one or more pieces of patent information from such a recording medium.

The processing unit 73 performs various types of processing. The various types of processing are, for example, the processing of the prediction unit 731, the judgment means 7311, the first decision means 7312, and the second decision means 7313. The processing unit 73 also performs, for example, various judgments that are explained in the flowcharts. Note that other processing performed by the processing unit 73 may be explained at appropriate times.

The prediction unit 731 uses one or more pieces of condition information to obtain predicted classification information that identifies classification information for one or more pieces of patent information.

The prediction unit 731 executes a predicted classification information acquisition process for each of one or more pieces of patent information accepted by the patent information acceptance unit 721. The predicted classification information acquisition process is a process for acquiring predicted classification information. The predicted classification information is information that identifies the classification information of the patent information.

The determination means 7311 determines whether or not each of the one or more pieces of patent information matches any of the conditions contained in each of the one or more pieces of condition information.

The judgment means 7311 judges, for example, for each of one or more pieces of patent information accepted by the patent information accepting unit 721, whether or not the patent information satisfies any of one or more conditions corresponding to one or more pieces of condition information stored in the condition information storage unit 712.

In this embodiment, for example, the condition information is a pair of a patent identifier and classification information, and the condition information storage unit 712 stores one or more patent identifiers in association with the classification information, and the judgment means 7311 may judge, for each of the one or more accepted patent information, whether or not the patent identifier held by the patent information matches any of the one or more patent identifiers stored in the condition information storage unit 712.

The first determination means 7312 obtains classification information that pairs with the condition that the determination means 7311 determines to be met.

The first determination means 7312, for example, for each of one or more pieces of patent information, obtains classification information that pairs with a patent identifier that is determined to match the patent identifier possessed by the patent information.

The second determination means 7313 uses a learning device stored in the learning device storage unit 711 to obtain classification information of the patent information through machine learning prediction processing. For patent information that the judgment means 7311 judges not to match any of the conditions, the second determination means 7313 uses a learning device stored in the learning device storage unit 711 to obtain classification information through machine learning prediction processing.

In this embodiment, the second determination means 7313 obtains, for example, classification information of patent information corresponding to a patent identifier that is determined not to match any of one or more patent identifiers stored in the condition information storage unit 712, by a machine learning prediction process.

The output unit 74 outputs various information. The various information is, for example, predicted classification information. For each of one or more pieces of accepted patent information, the output unit 74 outputs, as predicted classification information, the classification information determined by the first determination means 7312 or the classification information acquired by the second determination means 7313, paired with a patent identifier corresponding to the patent information.

As mentioned above, output means, for example, transmission to the user terminal 4, but it may also be considered as a concept including display on a display, printing on a printer, sound output from a speaker, storage on a recording medium, and delivery of the processing results to other processing devices or other programs.

The output unit 74 outputs the predicted classification information acquired by the prediction unit 731. The output unit 74 typically transmits the acquired predicted classification information to the user terminal 4. However, the predicted classification information may be, for example, displayed on a display, stored in a recording medium, or handed over to another processing device or another program, and the output form and destination are not important.

The server device 3 operates in the same manner as in embodiment 1.

User terminal 4 operates in the same manner as in embodiment 1.

The storage unit 71, the learning unit storage unit 711, and the condition information storage unit 712 are preferably non-volatile recording media, but can also be realized using volatile recording media.

The process by which information is stored in the storage unit 71, etc. is not important. For example, information may be stored in the storage unit 71, etc. via a recording medium, information transmitted via a communication line, etc. may be stored in the storage unit 1, etc., or information inputted via an input device may be stored in the storage unit 1, etc.

The reception unit 72, the patent information reception unit 721, and the output unit 74 can typically be realized using wireless or wired communication means, etc.

The processing unit 73, prediction unit 731, judgment means 7311, first decision means 7312, and second decision means 7313 can usually be realized by an MPU, memory, etc. The processing procedures of the processing unit 73, etc. are usually realized by software, and the software is recorded in a recording medium such as a ROM. However, they may also be realized by hardware (dedicated circuitry).

Next, the operation of the patent classification system D will be explained using the flowcharts in Figures 23 and 24. Figure 23 is a flowchart explaining the operation of the classification information determination device 7.

(Step S2301) The processing unit 73 judges whether the patent information receiving unit 721 has received one or more pieces of patent information. If it is judged that the patent information receiving unit 721 has received one or more pieces of patent information, the process proceeds to step S2302, and if it is judged that the patent information receiving unit 721 has not received any pieces of patent information, the process returns to step S2501.

(Step S2302) The processing unit 73 assigns 1 to the counter k.

(Step S2303) The processing unit 73 determines whether the kth patent information exists. If it is determined that the kth patent information exists, the process proceeds to step S2303, and if it is determined that the kth patent information does not exist, the process returns to step S2301.

(Step S2304) The prediction unit 731 executes a process for acquiring predicted classification information k that identifies the classification information of the kth patent information (hereinafter, the "kth predicted classification information acquisition process"). The kth predicted classification information acquisition process will be described with reference to the flowchart in FIG. 24.

(Step S2305) The processing unit 73 increments the counter k by 1. Return to step S2303.

In the flowchart of FIG. 23, the process ends when the classification information determination device 7 is powered off or an interrupt occurs to end the process.

Figure 24 is a flowchart explaining the "kth predicted classification information acquisition process" in step S2304 above.

(Step S2401) The prediction unit 731 assigns 1 to counter n.

(Step S2402) The prediction unit 731 determines whether the nth condition exists. If it is determined that the nth condition exists, the process proceeds to step S2403. If it is determined that the nth condition does not exist, the process proceeds to step S2407.

(Step S2403) The judgment means 7311 constituting the prediction unit 731 judges whether the kth patent information matches the nth condition. If it is judged that the kth patent information matches the nth condition, the process proceeds to step S2605, and if it is judged that it does not match, the process proceeds to step S2404.

(Step S2404) The prediction unit 731 increments the counter n by 1. Return to step S2402.

(Step S2405) The first determination means 7312 determines classification information that pairs with the nth condition among one or more conditions corresponding to one or more pieces of condition information stored in the condition information storage unit 712.

(Step S2406) The prediction unit 731 sets the classification information determined in step S2405 to "predicted classification information k". Then it returns to the upper level process.

(Step S2407) The second determination means 7313 uses the learning device stored in the learning device storage unit 711 to obtain classification information for the k-th patent information through machine learning prediction processing.

(Step S2408) The prediction unit 731 sets the classification information acquired in step S2407 to "predicted classification information k". Then it returns to the upper level process.

Below, we will explain a specific example of the operation of patent classification system D in this embodiment.

As described in the first embodiment, the learning device production device 1 performs a machine learning learning process using the learning source information to acquire a learning device. The learning device production device 1 passes the acquired learning device and the learning source information used when acquiring the learning device to the classification information determination device 7. In this example, it is assumed that a learning device is acquired using learning source information having 100 pieces of patent classification information 1 to 100, and that the learning device is handed over together with the learning source information.

In the classification information determination device 7, the processing unit 73 receives the learning device and stores it in the learning device storage unit 711. The processing unit 73 also receives the learning source information and acquires two or more pieces of condition information corresponding to the learning source information (in this example, 90 pieces of condition information corresponding to the 90 pieces of patent classification information 11 to 100 used when acquiring the learning device out of the 100 pieces of patent classification information 1 to 100 contained in the learning source information) and stores them in the condition information storage unit 712.

As a result, the learning device storage unit 711 stores the learning device acquired by the learning device production device 1 using patent classification information 11 to 100 out of the 100 pieces of patent classification information contained in the received learning source information. In addition, the condition information storage unit 712 stores two or more pieces of condition information (in this example, 90 pieces corresponding to the 90 pieces of patent classification information 11 to 100 used when acquiring the learning device) as shown in FIG. 25.

Figure 25 is a data structure diagram of condition information in this embodiment. Condition information has a patent identifier and classification information. Two or more pieces of condition information stored in the condition information storage unit 712 are associated with an ID (e.g., "1", "2", etc.). For example, condition information associated with ID "1" (hereinafter, condition information 1) has a patent identifier "JP Patent Publication 2017-dddd" and classification information "B". Condition information 2 associated with ID "2" has a patent identifier "JP Patent Publication 2017-eeee" and classification information "A". Similarly, condition information 100 has a patent identifier "JP Patent Publication 2018-ffff" and classification information "C".

In this example, it is assumed that 100 pieces of patent information corresponding to the 100 pieces of patent classification information 1 to 100 contained in the received learning source information are read from a previously obtained medium such as a disk.

In the classification information determination device 7, the patent information receiving unit 721 receives the 100 pieces of patent information read out as described above, and the prediction unit 731 executes a predicted classification information acquisition process to acquire predicted classification information that identifies the classification information of each of the received 100 pieces of patent information.

That is, the judgment means 7311 constituting the prediction unit 731 first judges whether the first patent information corresponding to the patent classification information 1 matches any of the 10 conditions corresponding to the 10 condition information 1 to 10 stored in the condition information storage unit 712. The patent information is judged to match none of the conditions.

In response to the above determination result that none of the conditions are met, the second determination means 7313 uses the learning device stored in the learning device storage unit 711 to obtain classification information for the first patent information through machine learning prediction processing. In this example, it is assumed that classification information "A" has been obtained. As a result, classification information "A" by the learning device is obtained as predicted classification information for the first patent information.

Similarly, for patent classification information 2 to 10, it is determined that none of the conditions are met, and predicted classification information by the learning device is obtained (for example, predicted classification information "A" for patent classification information 2, predicted classification information "B" for patent classification information 10, etc.). As a result, predicted classification information "A"..."B" by the learning device is obtained as predicted classification information for each of the second to tenth patent information.

Next, the judgment means 7311 judges whether the 11th patent information corresponding to the patent classification information 11 matches any of the 90 conditions corresponding to the 90 condition information 1 to 90 stored in the condition information storage unit 712. The condition information is judged to match the first condition of the 90 conditions.

In response to the above determination result that the first condition is met, the first determination means 7312 determines classification information "B" that pairs with the patent identifier "JP Patent Publication 2017-dddd", which is the first condition out of the 90 conditions corresponding to the 90 pieces of condition information stored in the condition information storage unit 712. As a result, classification information "B", which constitutes patent classification information 11, is obtained as predicted classification information for the 11th patent information.

Similarly, patent classification information 12 to 100 is determined to match the 12th to 90th conditions of the 90 conditions, and classification information "B" to "C" that pairs with the patent identifiers "JP Patent Publication 2017-eeee" to "JP Patent Publication 2018-ffff", which are the 12th to 100th conditions, is determined. As a result, classification information "A" to "C" that constitutes each of patent classification information 12 to 100 is obtained as predicted classification information for each of the 12th to 100th patent information.

The output unit 74 stores the classification information thus obtained for each of the 100 pieces of patent information on a recording medium such as a disk, in association with the patent information.

As a result, for example, 100 pairs of patent information and classification information for that patent information are stored on the recording medium.

As described above, according to this embodiment, the learning device production device 1 performs a machine learning learning process using learning source information having two or more pieces of patent classification information each having patent information and classification information, acquires a learning device, and the acquired learning device is stored in the learning device storage unit 711. The condition information storage unit 712 stores one or more pieces of condition information that are a pair of a condition related to a patent and classification information. The classification information determination device 7 accepts one or more pieces of patent information, acquires predicted classification information that identifies classification information for each of the one or more pieces of patent information using one or more pieces of condition information, and outputs the predicted classification information.

When obtaining predicted classification information, the classification information determination device 7 determines whether or not one or more pieces of patent information match any of the conditions contained in one or more pieces of condition information, and determines classification information that pairs with the condition that is determined to match, while using a learning device to obtain classification information for patent information that is determined not to match any of the conditions through machine learning prediction processing, thereby enabling patents to be classified with high accuracy.

In addition, in the above configuration, one condition is that the patent information constitutes any one of two or more patent classification information contained in the learning source information, so that patents in the training data used when the machine learning learning device was obtained can be classified with high accuracy, and other patents can also be classified with high accuracy by prediction processing using the learning device.

The software for realizing the classification information determination device 7 in this embodiment is, for example, a program as follows. In other words, this program performs machine learning learning processing using learning source information having two or more patent classification information each having patent information and classification information of the patent information, and includes a computer that can access a learning device storage unit 711 in which the acquired learning device is stored, and a condition information storage unit 712 in which one or more condition information, which is a pair of a condition related to a patent and classification information of the patent information, a patent information receiving unit 721 that receives one or more patent information, and a prediction unit 731 that uses the one or more condition information to acquire predicted classification information that identifies classification information of each of the one or more patent information. and functions as an output unit 74 that outputs the predicted classification information, and the prediction unit 731 is a program that includes a judgment means 7311 that judges whether or not each of the one or more pieces of patent information matches any of the conditions contained in each of the one or more pieces of condition information, a first decision means 7312 that decides classification information that pairs with the condition that the judgment means 7311 judges to match, and a second decision means 7313 that uses the learning device to obtain classification information of patent information that the judgment means 7311 judges not to match any of the conditions through machine learning prediction processing.

(Embodiment 5)
In this embodiment, a patent classification system E is described that is equipped with a classification information determination device in which condition information, which is a pair of a condition and a class (classification information), is stored, and for a prediction target (patent information) that matches the one condition, the class that pairs with the one condition is determined, and when the prediction target does not match the one condition, the class is determined using a machine learning algorithm.

In addition, in this embodiment, the machine learning algorithm is, for example, similar to that described in the first to third embodiments, but the type is not important. We will explain the patent classification system E equipped with the classification information determination device.

In addition, in this embodiment, a patent classification system E equipped with a classification information determination device will be described, in which one condition is a patent search formula (e.g., "the name of the invention contains 'machine learning'").

Figure 26 is a block diagram of a patent classification system E in this embodiment. The patent classification system E includes an algorithm determination device 5, a learning device production device 6, a classification information determination device 7a, a server device 3, and a user terminal 4.

The classification information determination device 7a includes a storage unit 71, a reception unit 72a, a processing unit 73a, and an output unit 74. The reception unit 72a includes a patent information reception unit 721, and a search expression reception unit 722. The processing unit 73a includes a patent specification acquisition unit 131, an important information acquisition unit 132, a classification target processing information acquisition unit 233, a prediction unit 731, and a search unit 732. The prediction unit 731 includes a judgment means 7311, a first decision means 7312, and a second decision means 7313.

The user terminal 4 includes a user reception unit 41, a user processing unit 42, a user transmission unit 43, a user reception unit 44, and a user output unit 45.

The algorithm determination device 5 has the same configuration as that in embodiment 2 and performs the same operations.

The learning device production device 6 has the same configuration as that in embodiment 2 and performs the same operation.

The classification information determination device 7a has the same configuration and operates in the same manner as the classification information determination device 7 in embodiment 4, except for the points described below.

That is, one condition constituting each of the one or more pieces of condition information stored in the condition information storage unit 712 is a search formula. A search formula is information indicating the conditions when searching patent information (hereinafter, may be referred to as search conditions). Search conditions are, for example, keywords such as "machine learning" and patent classification codes such as IPC, F-term, and FI, but they can also be period information that specifies the period to which the application date or publication date belongs, and the type is not important.

The search criteria may also include a combination of item information and a keyword, such as "Name of invention contains machine learning." Item information is information that identifies an item in patent information. Item information may be, for example, "Name of invention," "Abstract," "Effects of invention," etc., but may also be "Full text," and the type of information is not important.

The search formula may be, for example, two or more conditions such as keywords or patent classification codes combined with a logical operator (for example, AND or OR). The search formula may also be a combination of multiple search formulas (combined with logical operators), and the format is not important.

The reception unit 72a, for example, receives search queries in addition to patent information.

The search formula receiving unit 722 receives a search formula. The search formula receiving unit 722 typically receives the search formula from the user terminal 4. However, the search formula may be received, for example, via an input device such as a keyboard in the classification information determination device 7a, and the manner of reception is not important.

In addition to the processing of the prediction unit 731 and the like, the processing unit 73a further performs the processing of, for example, the patent specification acquisition unit 131, the important information acquisition unit 132, the classification target processing information acquisition unit 233, and the search unit 732. Note that the patent specification acquisition unit 131, the important information acquisition unit 132, and the classification target processing information acquisition unit 233 perform the same operations as those in the first embodiment.

The search unit 732 performs a search using the search formula received by the search formula receiving unit 722, and obtains search results. The search target is, for example, one or more pieces of patent information in the storage unit 71, but it may also be one or more pieces of patent information in the server device 3. The search result is one or more pieces of patent information that satisfy the search conditions indicated by the search formula, among the one or more pieces of patent information that are the search target. Note that the search result may be one or more patent identifiers corresponding to the one or more pieces of patent information.

The user reception unit 41 constituting the user terminal 4 receives various types of information. For example, the various types of information are the search expressions described above. The reception by the user reception unit 41 is usually via an input device such as a keyboard within the user terminal 4. However, reception may also be, for example, reading from a recording medium or receiving from an external device, and the manner is not important. The same points also apply to reception of search expressions by the search expression reception unit 411.

The search query reception unit 411 receives a search query.

The user processing unit 42 performs various types of processing. For example, various types of processing include determining whether the user reception unit 41 has received information such as a search query, and converting the received information into information to be transmitted, but the type of processing is not important.

The user transmission unit 43 transmits various types of information. The various types of information are, for example, search expressions. The user transmission unit 43 transmits, for example, the search expressions accepted by the user acceptance unit 41 to the classification information determination device 7a.

The user receiving unit 44 receives various types of information. The various types of information are, for example, search expressions. The user receiving unit 44 receives search results from the classification information determination device 7a, for example, in response to the transmission of the search expression by the user transmitting unit 43.

The user output unit 45 outputs various information. For example, the various information is search results. The user output unit 45 displays the search results and other information on a display or stores them in a recording medium, for example. However, the search results and other information may be handed over to another program or sent to another device, and the output format is not important.

The search expression receiving unit 411 and the user receiving unit 41 may or may not include an input device such as a keyboard or a touch panel. The search expression receiving unit 411 and the like may be realized by the driver software of the input device, or by the input device and its driver software.

The user transmitting unit 43 and the user receiving unit 44 can typically be realized using wireless or wired communication means, etc.

The search unit 732 and the user processing unit 42, etc., can usually be realized by an MPU, memory, etc. The processing procedure of the processing unit 73a, etc. is usually realized by software, and the software is recorded in a recording medium such as a ROM. However, they may also be realized by hardware (dedicated circuitry).

The operation of patent classification system E is the same as that described in embodiment 4 using the flowcharts in Figures 23 and 24.

Below, we will explain a specific example of the operation of patent classification system E in this embodiment.

As described in the second embodiment, the algorithm determination device 5 determines an appropriate process for performing the prediction process. As described in the second embodiment, the learning device production device 6 uses the learning source information to perform a learning process that is a machine learning learning process and follows the decision of the algorithm determination device 5, and acquires a learning device. The learning device production device 6 delivers the acquired learning device and the learning source information used when acquiring the learning device to the classification information determination device 7a. In this example, it is assumed that a learning device is acquired using learning source information having 100 pieces of patent classification information, and the learning device is handed over together with the learning source information.

In the classification information determination device 7a, the processing unit 73a receives the learning device and stores it in the learning device storage unit 711. The processing unit 73a also receives the learning source information, acquires two or more pieces of condition information corresponding to the learning source information (in this example, 90 pieces of condition information corresponding to the 90 pieces of patent classification information 11 to 100 used when acquiring the learning device), and stores them in the condition information storage unit 712.

As a result, the learning module storage unit 711 stores the learning modules acquired by the learning module production device 6 using 90 pieces of patent classification information 11 to 100 from the above learning source information. In addition, the condition information storage unit 712 stores, for example, two or more pieces of condition information (condition information 1 to 3, etc.) as shown in FIG. 27.

Figure 27 is a data structure diagram of condition information in this embodiment. Condition information has a search formula and classification information. Two or more pieces of condition information stored in the condition information storage unit 712 are associated with an ID (e.g., "1", "2", etc.). For example, condition information 1 associated with ID "1" has a search formula "'Name of invention' contains 'machine learning'" and classification information "A". Condition information 2 associated with ID "2" has a search formula "'machine learning' and 'natural language processing'" and classification information "B". Similarly, condition information 3 has a search formula "'H06 aa/aa' or 'G06 bb/bb' or ..." and classification information "C".

The user inputs one or more pieces of patent information via the input device of the user terminal 4. The patent information input here is a patent identifier such as a publication number. For example, assume that 10 patent identifiers corresponding to 10 pieces of patent classification information 1 to 10 out of the 100 pieces of patent classification information 1 to 100 contained in the learning source information described above are input.

In the user terminal 4, the user reception unit 41 receives the 10 patent identifiers entered above, and the user transmission unit 43 transmits the received 10 patent identifiers to the classification information determination device 7a.

In the classification information determination device 7a, the patent information receiving unit 721 receives the 10 patent identifiers transmitted above, the output unit 74 transmits a request including the received 10 patent information identifiers to the server device 3, and the patent information receiving unit 721 receives 10 pieces of patent information corresponding to the transmitted 10 patent information identifiers from the server device 3. The prediction unit 731 executes a predicted classification information acquisition process for acquiring predicted classification information that specifies the classification information of the patent information for each of the received 10 pieces of patent information.

That is, the judgment means 7311 constituting the prediction unit 731 first judges whether the first patent information corresponding to the patent classification information 1 matches any of the two or more conditions corresponding to the two or more condition information 1 to 3, etc. stored in the condition information storage unit 712. Here, it is assumed that the patent information is judged to match none of the conditions.

In response to the above determination result that none of the conditions are met, the second determination means 7313 uses the learning device stored in the learning device storage unit 711 to obtain classification information for the first patent information through machine learning prediction processing. In this example, it is assumed that classification information "A" has been obtained. As a result, classification information "A" by the learning device is obtained as predicted classification information for the first patent information.

Similarly, patent classification information 2 to 9 is determined to not match any of the conditions, and predicted classification information by the learning device (for example, predicted classification information "A" for patent classification information 2) is obtained. As a result, predicted classification information "A", "A", ... by the learning device is obtained as predicted classification information for each of the second to ninth patent information.

Next, the judgment means 7311 judges whether the tenth patent information corresponding to patent classification information 10 satisfies any of the two or more conditions described above. It is assumed that the condition information is judged to satisfy the third condition corresponding to condition information 3.

In response to the above judgment result that the third condition is met, the first determination means 7312 acquires classification information "C" that pairs with the search expression "H06 aa/aa" or "G06 bb/bb" or...", which is the third condition out of the three or more conditions corresponding to the ten pieces of condition information stored in the condition information storage unit 712. As a result, classification information "C", which constitutes patent classification information 10, is acquired as predicted classification information for the tenth patent information.

The output unit 74 thus associates the classification information acquired for each of the 10 pieces of patent information with the patent information and transmits it to the user terminal 4. As a result, 10 sets of information, each of which is a pair of patent information and classification information, are transmitted.

In the user terminal 4, the user receiving unit 44 receives the 10 pieces of group information transmitted above, and the user output unit 45 outputs the received 10 pieces of group information via a display. Note that the output group information does not have to be the same as the received group information. For example, the patent information constituting the output group information may be a part of the patent information constituting the received group information (e.g., a patent identifier and abstract, etc.).

As a result, 10 pairs of patent information and classification information are displayed on the display of the user terminal 4.

As described above, according to this embodiment, the learning module production device 6 performs a machine learning learning process using learning source information having two or more pieces of patent classification information each having patent information and classification information, acquires a learning module, and the acquired learning module is stored in the learning module storage unit 711. The condition information storage unit 712 stores one or more pieces of condition information that are a pair of a condition related to a patent and classification information. The classification information determination device 7a accepts one or more pieces of patent information, acquires predicted classification information that identifies classification information for each of the one or more pieces of patent information using one or more pieces of condition information, and outputs the predicted classification information.

When obtaining predicted classification information, the classification information determination device 7a determines whether or not one or more pieces of patent information match any of the conditions contained in one or more pieces of condition information, and determines classification information that pairs with the condition that is determined to match, while using a learning device to obtain classification information for patent information that is determined not to match any of the conditions through machine learning prediction processing, thereby enabling patents to be classified with high accuracy.

In addition, in the above configuration, one condition is a search formula, so that patents that match the search formula can be classified with high accuracy, and patents that do not match the search formula can also be classified with high accuracy through prediction processing using a machine learning learner.

In addition, the algorithm determination device 5 determines an appropriate process for performing the prediction process, and the learning module production device 6 performs the learning process using an algorithm according to that determination, thereby producing an appropriate learning module, which enables patents to be classified with high accuracy.

The software for realizing the classification information determination device 7a in this embodiment is, for example, a program as follows. That is, this program performs machine learning learning processing using learning source information having two or more patent classification information having patent information and classification information of the patent information, and includes a computer that can access a learning device storage unit 711 in which the acquired learning device is stored, and a condition information storage unit 712 in which one or more condition information, which is a pair of a condition related to a patent and classification information of the patent information, a patent information receiving unit 721 that receives one or more patent information, and a prediction unit 73 that acquires predicted classification information that identifies classification information of each of the one or more patent information using the one or more condition information. 1 and functions as an output unit that outputs the predicted classification information, and the prediction unit 731 includes a judgment means 7311 that judges whether or not each of the one or more pieces of patent information matches any of the conditions contained in each of the one or more pieces of condition information, a first decision means 7312 that decides classification information that pairs with the condition that the judgment means 7311 judges to match, and a second decision means 7313 that uses the learning device to obtain classification information of patent information that the judgment means 7311 judges not to match any of the conditions through machine learning prediction processing.

Figure 28 also shows the external appearance of a computer that executes the program described in this specification to realize the patent classification device 2 and the like of the various embodiments described above. The above-mentioned embodiments can be realized by computer hardware and a computer program executed thereon. Figure 28 is an overview of this computer system 300, and Figure 29 is a block diagram of system 300. Figures 28 and 29 are diagrams showing the external appearance of a computer that realizes an engagement system.

In FIG. 28, computer system 300 includes computer 301 including CD-ROM drive 3012, keyboard 302, mouse 303, and monitor 304.

In FIG. 29, in addition to a CD-ROM drive 3012, the computer 301 includes an MPU 3013, a bus 3014 connected to the CD-ROM drive 3012 etc., a ROM 3015 for storing programs such as a boot-up program, a RAM 3016 connected to the MPU 3013 for temporarily storing instructions for application programs and providing temporary storage space, and a hard disk 3017 for storing application programs, system programs, and data. Although not shown here, the computer 301 may further include a network card that provides a connection to a LAN.

Programs that cause the computer system 300 to execute the functions of the learning device production device 1, patent classification device 2, algorithm determination device 5, classification information determination device 7, etc., of each of the above-mentioned embodiments may be stored on a CD-ROM 3101, inserted into the CD-ROM drive 3012, and then transferred to the hard disk 3017. Alternatively, the programs may be transmitted to the computer 301 via a network (not shown) and stored on the hard disk 3017. The programs are loaded into the RAM 3016 when executed. The programs may be loaded directly from the CD-ROM 3101 or the network.

The program does not necessarily have to include an operating system (OS) or third-party programs that cause the computer 301 to execute the functions of the learning device production device 1, patent classification device 2, algorithm determination device 5, etc., of the above-mentioned embodiments. The program only needs to include an instruction portion that calls appropriate functions (modules) in a controlled manner to obtain the desired results. How the computer system 300 operates is well known, and a detailed description will be omitted.

In the above program, the steps of transmitting information and receiving information do not include processing performed by hardware, such as processing performed by a modem or interface card in the transmission step (processing that is performed only by hardware).

The program may be executed by a single computer or multiple computers. In other words, the program may be executed by centralized processing or distributed processing.

Furthermore, in each of the above embodiments, it goes without saying that two or more communication means present in one device may be realized physically by one medium.

In addition, in each of the above embodiments, each process may be realized by centralized processing by a single device, or may be realized by distributed processing by multiple devices. In other words, the learning device production device 1, the patent classification device 2, and the algorithm determination device 5 may operate in a standalone manner. When the learning device production device 1, the patent classification device 2, and the algorithm determination device 5 operate in a standalone manner, the reception unit receives instructions, information, etc. from a user, etc. Furthermore, the output unit displays information, etc., outputs sound, or transmits information, etc. to a display device.

The present invention is not limited to the above-described embodiment, and various modifications are possible, and it goes without saying that these are also included within the scope of the present invention.

As described above, the patent classification system of the present invention has the effect of classifying patents with high accuracy by performing predictive processing using an appropriate algorithm, and is useful as a patent classification system, etc.

In addition, the patent classification system of the present invention has the effect of being able to classify patents with high accuracy by using important information related to patent specifications, and is useful as a patent classification system, etc.

The classification information determination device of the present invention has the effect of being able to classify patents with high accuracy by obtaining predicted classification information using different determination methods for patent information that meets predetermined conditions and patent information that does not, and is useful as a classification information determination device, etc.

1, 6 Learning device production device 2, 8 Patent classification device 3 Server device 4 User terminal 5 Algorithm determination device 7, 7a Classification information determination device 11, 61 Learning storage unit 12, 62 Learning reception unit 13 Learning processing unit 14, 64 Learning output unit 21, 51, 71 Storage unit 22, 52, 72, 81 Reception unit 23, 53, 73, 73a, 82 Processing unit 24, 54, 74 Output unit 41 User reception unit 42 User processing unit 43 User transmission unit 44 User reception unit 45 User output unit 111 Manual evaluation information storage unit 131 Patent specification acquisition unit 132 Important information acquisition unit 133, 535 Learning source information acquisition unit 134, 531 Learning unit 141, 641 Accumulation unit 211, 711 Learning device storage unit 233 Classification target processing information acquisition unit 234, 532, 731, 821 Prediction unit 421, 732 Search unit 511 Program storage unit 512 Precision specification information storage unit 521, 621 Learning source information reception unit 531 Learning device 533 Evaluation unit 534 Algorithm information acquisition unit 536 Statistical processing unit 621 Learning source information learning reception unit 712 Condition information storage unit 721 Patent information reception unit 1321 Effect solution means sentence acquisition means 1322 Effect sentence acquisition means 1323 Solution means sentence acquisition means 1324 Object sentence acquisition means 1325 Feature word acquisition means 1326 Related word acquisition means 1327 Important tag corresponding sentence acquisition means 1328 Numeric range character string acquisition means 1329 Number acquisition means 7311 Judgment means 7312 First determination means 7313 Second determining means

Claims (15)

a program storage unit storing one or more learning device generation programs for generating a learning device for classifying patent information relating to a patent using learning source information having two or more patent classification information including patent information relating to a patent and classification information indicating a classification for the patent information, and one or more prediction programs for predicting a classification of the patent classification information using the learning device and acquiring predicted classification information;
a learning source information receiving unit that receives one piece of learning source information;
a learning source information acquisition unit that acquires two or more pieces of learning source information each having two or more pieces of patent classification information having different information from the patent information included in the two or more pieces of patent classification information included in the learning source information accepted by the learning source information acceptance unit;
a learning unit that executes the one or more learning device generation programs for each of the two or more pieces of learning device information acquired by the learning device information acquisition unit , and acquires two or more learning devices;
a prediction unit that applies one or more prediction programs to at least a portion of each of the patent classification information of the learning source information received by the learning source information receiving unit, using two or more learning devices generated by the learning unit, to obtain two or more predicted classification information for each of the one or more patent classification information;
an evaluation unit that uses each of the two or more pieces of predicted classification information acquired by the prediction unit and classification information included in the learning source information to acquire accuracy information regarding the accuracy of each of the two or more pieces of predicted classification information, and determines the predicted classification information having the highest accuracy specified by the accuracy information;
an algorithm information acquisition unit that acquires algorithm information , which is information about an algorithm, including information specifying information that specifies information included in the learning source information corresponding to the predicted classification information determined by the evaluation unit;
and an output unit that outputs the algorithm information. The method further includes a statistical processing unit that performs one or more types of statistical processing using the two or more pieces of predicted classification information acquired by the prediction unit, and acquires one or more pieces of predicted classification information as a result of the statistical processing,
The evaluation unit is
acquiring accuracy information regarding accuracy of each of the two or more pieces of predicted classification information using, instead of or in addition to the two or more pieces of predicted classification information acquired by the prediction unit, each of the two or more pieces of predicted classification information including the one or more pieces of predicted classification information acquired by the statistical processing unit and classification information included in the learning source information, and determining the predicted classification information having the highest accuracy specified by the accuracy information;
The algorithm information acquisition unit
The algorithm determination device according to claim 1 , further comprising: acquiring algorithm information including statistical processing specification information that specifies a statistical processing method corresponding to the predicted classification information determined by the evaluation unit. The method further includes an accuracy specification information storage unit in which accuracy specification information is stored, the accuracy specification information specifying which of two or more of the four accuracies, namely, recall rate, precision rate, F-value, or accuracy rate, accuracy information is to be acquired;
The evaluation unit is
3. The algorithm determination device according to claim 1, further comprising: a prediction unit that uses each of two or more pieces of predicted classification information acquired by the prediction unit and classification information contained in the learning source information to obtain accuracy information for each of the two or more pieces of predicted classification information, the accuracy information being specified by accuracy specifying information; and a prediction classification information having the highest accuracy specified by the accuracy information is determined. An algorithm determination device according to any one of claims 1 to 3;
A learning source information receiving unit that receives learning source information;
a learning unit that performs a learning process on the learning source information received by the learning source information receiving unit using an algorithm according to the algorithm information acquired by the algorithm determination device to acquire a learning device;
and a storage unit that stores the learning device in association with the algorithm information. The production device according to claim 4 ,
A receiving unit that receives classification target information including a patent identifier that identifies a patent to be classified;
a prediction unit that performs a prediction process on the classification target information using a learning device according to an algorithm in accordance with algorithm information associated with a learning device stored by the production device , and acquires classification result information;
A patent classification device comprising: an output unit that outputs the classification result information acquired by the prediction unit in correspondence with the patent identifier contained in the classification target information accepted by the acceptance unit. an important information acquisition unit that acquires important information that satisfies a predetermined condition by using a patent specification corresponding to a patent identifier included in the classification target information received by the reception unit;
a classification target processing information acquisition unit that acquires classification target processing information, which is information corresponding to a patent identifier included in the classification target information received by the reception unit and includes the important information acquired by the important information acquisition unit;
The prediction unit is
A patent classification device as described in claim 5, wherein the learning device is applied to the classification target processed information acquired by the classification target processed information acquisition unit, the classification of the classification target processed information is predicted using a machine learning algorithm, and classification result information is acquired. a learning module storage unit that performs a machine learning learning process using learning source information having two or more pieces of patent classification information each having patent information and classification information of the patent information, and stores the acquired learning module;
a condition information storage unit for storing one or more pieces of condition information each of which is a combination of a condition related to a patent and classification information of patent information;
a patent information receiving unit that receives one or more pieces of patent information;
a prediction unit that obtains predicted classification information that identifies classification information of each of the one or more pieces of patent information using the one or more pieces of condition information;
an output unit that outputs the predicted classification information,
The prediction unit is
A determination means for determining whether or not each of the one or more patent information matches any of the conditions contained in each of the one or more condition information;
a first determination means for determining classification information that is paired with the condition that is determined to be met by the determination means;
A classification information determination device comprising: a second determination means for obtaining classification information of patent information that the judgment means determines does not meet any of the conditions using the learning device, through machine learning prediction processing. An algorithm determination method implemented by the algorithm determination device according to any one of claims 1 to 3, comprising :
An algorithm determination method comprising the steps performed by the algorithm determination device . A method for producing a learning device implemented by the production device according to claim 4, comprising the steps of :
A method for producing a learning device comprising the steps performed by the production device . A patent classification method implemented by the patent classification device according to claim 5 or 6, comprising :
A patent classification method comprising the steps performed by the patent classification device . A classification information determination method that performs machine learning learning processing using learning source information having two or more patent classification information having patent information and classification information of the patent information, and is realized by a learning device storage unit in which an acquired learning device is stored, a condition information storage unit in which one or more condition information, which is a combination of one condition related to a patent and classification information of the patent information, a patent information receiving unit, a prediction unit, and an output unit,
a patent information receiving step in which the patent information receiving unit receives one or more pieces of patent information;
a prediction step in which the prediction unit obtains predicted classification information that identifies classification information of each of the one or more pieces of patent information by using the one or more pieces of condition information;
an output step in which the output unit outputs the predicted classification information;
The prediction unit is
A determination means for determining whether or not each of the one or more patent information matches any of the conditions contained in each of the one or more condition information;
a first determination means for determining classification information that is paired with the condition that is determined to be met by the determination means;
A classification information determination method comprising: a second determination means for obtaining classification information of patent information that the judgment means determines does not meet any of the conditions using the learning device, through machine learning prediction processing. On the computer,
A program for executing all the processes performed by the algorithm determination device according to any one of claims 1 to 3 . On the computer,
A program for executing all the processes performed by the learning device production device according to claim 4 . On the computer,
A program for executing all the processes performed by the patent classification device according to claim 5 or 6 . A computer that performs machine learning learning processing using learning source information having two or more patent classification information having patent information and classification information of the patent information, and that can access a learning device storage unit in which the acquired learning device is stored, and a condition information storage unit in which one or more condition information, which is a combination of one condition related to a patent and classification information of the patent information,
a patent information receiving unit that receives one or more pieces of patent information;
a prediction unit that obtains predicted classification information that identifies classification information of each of the one or more pieces of patent information using the one or more pieces of condition information;
an output unit that outputs the predicted classification information;
The prediction unit is
A determination means for determining whether or not each of the one or more patent information matches any of the conditions contained in each of the one or more condition information;
a first determination means for determining classification information that is paired with the condition that is determined to be met by the determination means;
A program comprising: a second determination means for using the learning device to obtain classification information of patent information that the judgment means has determined does not meet any of the conditions through machine learning prediction processing.

Images