JP7226783B2 - Information processing system, information processing method and program - Google Patents

Description

The present invention relates to an information processing system, an information processing method, and a program.

Conventionally, in researching documents such as patent documents, papers, and court cases, a keyword is selected, a database is searched, and each hit document is opened to check the contents and confirm the relevance to the keyword. Was.
For this reason, for example, if the number of hits is about 1,000, the number of pages in one document is about tens of pages, so it is necessary to check a huge amount of sentences of tens of thousands of pages in total. (see (1) in FIG. 1).
In addition, since the operator who conducts the research (hereinafter referred to as "searcher") opens the documents one by one and reads and confirms the sentences, it is possible to lose concentration due to long hours of confirmation work and miss relevant documents. However, there were also adverse effects such as accumulation of eye fatigue.

Therefore, for example, as in Patent Document 1, the search probability, which is the probability that the relevance is considered to be high, and the noise probability, which is the probability that the relevance is low and is considered to be noise, are used to evaluate patents. A system has been developed that classifies documents and makes document searches for patents and the like more efficient.

Further, for example, in Patent Document 2, first, in advance, a substantial amount of documents ranging from hundreds to thousands of documents are manually investigated by a searcher (hereinafter referred to as "pre-survey"). We have prepared a certain number of patent documents that are related to the technology and patent documents that are not related to the technology. Then, related patent feature vectors are defined for related patent documents, and non-related patent feature vectors are defined for less related patent documents.
Then, the average vector of each feature vector is calculated (a kind of clustering), and the stored A search patent feature vector is calculated for the patent document to be searched, and it is stored separately according to the COS scale (cosine similarity) between the related patent feature vector (or unrelated patent feature vector) and the search patent feature vector. Techniques for extracting highly relevant patents from a huge amount of patent documents have been developed.

Japanese Patent No. 6453502 JP 2014-112283 A

However, even with the system of Patent Literature 1, the set theme is evaluated on two axes: the search probability, which is the probability that it is considered to be highly relevant, and the noise probability, which is the probability that it is considered to be noise with low relevance. It is a process that makes it easier to select a target case by axis, and it is only possible to extract related patents by roughly classifying each theme. For this reason, it is suitable for applications such as formulating new keywords for uninvestigated or unknown themes, searching based on the keywords, and extracting documents that are likely to be related to the hit documents. was not

In the technique of Patent Document 2, based on related patent documents that have been picked up in advance, a search formula is automatically generated for searching, or a related patent feature vector is generated to determine the degree of similarity. .
For this reason, although preliminary research is required in advance, it is used to extract patent documents that are likely to be related without omission from the huge number of patent documents registered in the database, ranging from tens of thousands to millions. was suitable for
However, it is thought that about 20 to 30 related patent documents should be picked up in advance. It was necessary to manually check the patent documents related to the research target technology one by one from among the hundreds to thousands of patent documents hit by the predetermined search conditions.

In addition, we assumed that hundreds to thousands of patent documents would need to be confirmed because, in general, only a few percent of the patent documents hit by the search conditions are recognized as relevant. Therefore, in order to pick up dozens of related patent documents in advance, it is presumed that a searcher needs to check hundreds to thousands of patent documents.

In this way, conventionally, a process for determining whether a given patent document is included in a category of a certain theme (the type of Patent Document 1), or a searcher conducting a pre-search based on a predetermined keyword and obtaining a patent to some extent After manually conducting such searches, a considerable number of patent documents, etc. are picked up, and a system that searches a large database for patent documents that are similar to the multiple picked-up patent documents without omission (Patent document 2 type ) were known.
However, there is a problem that any of the prior patent documents cannot be applied to determine whether or not there is a high relevance to the technology to be searched in a situation where a document search is performed from scratch without a preliminary search. .

In addition, none of the prior patent documents describes or suggests what kind of algorithm should be used as information processing by analyzing the thoughts of searchers when manually extracting highly relevant patent documents. It wasn't.
Specifically, in any of the prior arts, we analyze what kind of relationship there is between a set keyword and a patent document so that it can be determined that they are highly related, and what kind of feature vector is generated for machine learning. There was no description or suggestion as to whether an efficient literature search would be possible if

Therefore, in the present invention, the searcher manually extracts patent documents that are likely to be related, that is, analyzes the thoughts of people in document research such as patents, and in addition to the features that people are aware of, By vectorizing including features that you are not aware of and machine learning with a large amount of data, you can build a system that refines the method of document research work, improve the accuracy of document research, and eventually document research. The purpose is to dramatically reduce the man-hours of
In addition, the purpose is to increase the time it takes for a searcher to read one document, etc., and to improve the accuracy of selecting documents that are judged to be particularly relevant among related documents. do.

Further, according to the technology of the present invention, it can be applied as it is to search not only for patent documents but also for papers and judicial precedents.
In addition, it is intended to be applicable not only to Japanese documents but also to documents written in foreign languages.

The present invention solves the above problems by means of the following solutions.
The first invention is
A system for determining the relevance between a set of search keywords and a document,
a means for obtaining text data of a document;
means for obtaining a set of keywords from the first stage to the n-th stage as a set of search keywords;
KW existence location analysis means for analyzing the location of keywords included in the text data of the document;
a KW space distribution vector generation means for generating a KW space distribution vector indicating how the locations of the keywords analyzed by the KW existence location analysis means are distributed in text data of the document;
The KW space distribution vector generating means generates, for each document, based on the location of each keyword in the group of keywords in the first to nth stages included in the text data of the document, the first to nth stages and generate KW spatial distribution vectors of
Based on the KW space distribution vectors of the 1st to nth stages generated for each document, the positional relationship between the keywords of the 1st to nth stage keywords and the characteristics of the distribution are evaluated, and the search keywords provided with classification means for determining the relevance between the set of
An information processing system characterized by
With this configuration, the mutual positional relationship and distribution features of the keywords of the keyword group of the first stage and the keywords of the keyword group of the nth stage are local mutual positional relationships and distribution features, and local It is possible to comprehensively evaluate the features of high-order positional relationships that combine various features in a complex manner.

The second invention is
In the information processing system according to the first invention, further,
KW dependent word existence location analysis means for analyzing the existence location of the dependent word of the keyword existing in the vicinity of the existence location of the keyword included in the text data of the document;
KW for generating a KW related word space distribution vector indicating how the locations of dependent words near the keyword analyzed by the KW related word presence location analyzing means are distributed in the text data of the document; Dependent word space distribution vector generation means,
The KW dependent word space distribution vector generation means generates for each document based on the presence location of the dependent word corresponding to each keyword of the first to n-th keyword groups included in the text data of the document. , respectively generate the KW-related word space distribution vectors of the 1st to nth stages corresponding to the keyword groups of the 1st to nth stages,
Based on the KW space distribution vectors of the first to nth stages and the KW related word space distribution vectors of the first to nth stages generated for each document,
Features of positional relationships and distributions among keywords in the first to n-th keyword group, and mutual relationships between each keyword in the first to n-th keyword group and their corresponding dependent words Equipped with classification means for evaluating positional relationships and distribution characteristics to determine relevance between a set of search keywords and documents;
An information processing system characterized by
With this configuration, in addition to the characteristics of the distribution of keywords, when there is a dependent word that denies the keyword, the location of the keyword and the characteristics of the distribution of the keyword act in the direction of lowering the evaluation as a characteristic, and the keyword is affirmed or not. When there is a subject word to be emphasized, the feature of the location and distribution of the keyword can act in the direction of strongly evaluating the feature.

The third invention is
In the information processing system according to the first invention, further,
technical terminology analysis means for analyzing technical terms included in text data of documents in order of frequency;
a technical term existence location analysis means for analyzing the location of the technical terms from the top 1 to the top m of the technical terms analyzed by the technical term analysis means;
a technical term space distribution vector generation means for generating a technical term space distribution vector indicating how the locations of the technical terms analyzed by the technical term presence location analysis means are distributed in the text data of the document; prepared,
The technical term space distribution vector generating means, based on the locations of the top 1 to top m technical terms included in the text data of the document, the top 1 to top m th technology for each document Generate a term space distribution vector, respectively,
Based on the 1st to n-th KW space distribution vectors and the 1st to m-th technical term space distribution vectors generated for each document,
Features of mutual positional relationships and distributions of keywords in the first to n-th keyword groups, and keywords in the first to n-th keyword groups and the top first to top m-th technical terms A classifying means for evaluating the mutual positional relationship and distribution characteristics of and determining the relevance between the set of search keywords and the document;
An information processing system characterized by comprising
With this configuration, in addition to the characteristics of the distribution of keywords, the relationship between the set of keywords and the literature is further evaluated by further evaluating the technical terms that are often present in the literature and the positional relationship with the keywords and the characteristics of the distribution. be able to judge accurately.

A fourth invention is an example of a detailed configuration of the classification means of the first invention,
In the information processing system according to the first invention,
The classification means for determining the relevance between a set of search keywords and documents based on the first to n-th KW space distribution vectors generated for each document is composed of a neural network of at least three layers. if
In the first layer of the neural network,
convolution means for performing convolution processing on the KW space distribution vectors of the first stage to the nth stage using the corresponding convolution filters;
synthesizing means for synthesizing the results of the convolution processing to generate a feature map that extracts the mutual positional relationship and distribution features of the keywords in the first to n-th keyword groups,
In the second and subsequent layers of the neural network,
Convolution means for performing convolution processing using a plurality of convolution filters on a plurality of feature maps generated in the previous layer;
synthesizing means for synthesizing the results of the convolution processing to generate a feature map that extracts high-order features of mutual positional relationships and distributions of keywords in the first to n-th keyword groups;
An information processing system characterized by comprising
With this configuration, the mutual positional relationship and distribution features of the keywords of the keyword group of the first stage and the keywords of the keyword group of the nth stage are local mutual positional relationships and distribution feature maps, local By generating a feature map for evaluating the features of high-order positional relationships, which are composite features combined, it becomes possible to perform multifaceted and comprehensive evaluation.
A neural network with two or more intermediate layers and three or more layers as a whole is generally called a deep neural network. .

A fifth invention is an example of a detailed configuration of the classification means of the second invention,
In the information processing system according to the second invention,
1st to nth KW space distribution vectors generated for each document, and 1st to nth KW-related word space distribution vectors corresponding to the 1st to nth keywords And, when the classification means for determining the relevance between the set of search keywords and the document based on is composed of a neural network of at least three layers,
In the first layer of the neural network,
Corresponding to the KW space distribution vectors of the first to nth stages and the KW-related word space distribution vectors of the first to nth stages corresponding to the keyword groups of the first to nth stages, respectively Convolution means for performing convolution processing using a convolution filter that
synthesizing means for synthesizing the results of the convolution process to generate a feature map that extracts the mutual positional relationships and distribution features of the keywords of the first to nth stage keywords and their dependent words; with
In the second and subsequent layers of the neural network,
Convolution means for performing convolution processing using a plurality of convolution filters on a plurality of feature maps generated in the previous layer;
Combining the results of the convolution process to generate a feature map that extracts the mutual positional relationships and high-order features of the distribution of the keywords in the group of keywords from the first stage to the n-th stage and their dependent words. means and
An information processing system characterized by comprising
With this configuration, in addition to the feature map of the distribution of keywords, when there is a dependent word that denies the keyword, the feature of the location and distribution of the keyword acts in the direction of low evaluation as a feature, and the keyword is affirmed. Or, if there is a catchword to be emphasized, the feature of the location and distribution of the keyword generates a feature map that can act in the direction of strongly evaluating it as a feature, so that more multifaceted and comprehensive evaluation can be performed. become.

A sixth invention is an example of a detailed configuration of the classification means of the third invention,
In the information processing system according to the third invention,
Based on the 1st to n-th KW space distribution vectors and the 1st to m-th technical term space distribution vectors generated for each document, the set of search keywords and the relationship between the documents When the classification means for determining gender is composed of a neural network of at least three layers,
In the first layer of the neural network,
Convolution means for performing convolution processing on the first to n-th KW space distribution vectors and the first to top m-th technical term space distribution vectors using convolution filters corresponding to them, respectively ,
By synthesizing the results of the convolution process, generate a feature map that extracts the mutual positional relationships and distribution features of the keywords in the 1st to n-th keyword groups and the top 1 to top m-th technical terms. and a synthesizing means for
In the second and subsequent layers of the neural network,
Convolution means for performing convolution processing using a plurality of convolution filters on a plurality of feature maps generated in the previous layer;
By synthesizing the results of the convolution process, the mutual positional relationship and high-order features of the distribution of the keywords in the 1st to nth stage keywords and the top 1st to top mth technical terms are extracted. a compositing means for generating a map;
An information processing system characterized by comprising
With this configuration, in addition to the feature map of the distribution of keywords, a feature map that can further evaluate the positional relationship with keywords and the characteristics of the distribution of technical terms that are often found in the literature is generated. evaluation becomes possible.

A seventh invention is an example of a learning process of a classification means using a KW space distribution vector,
In the information processing system according to any one of the first invention or the fourth invention,
means for acquiring a learning data set in which a set of search keywords, text data of a document, and a label value indicating the relationship between the set of search keywords and the document are associated;
Learning processing means for updating the parameters of the classification means based on the error between the score value of the relevance between the search keyword set and the document output by the classification means and the obtained label value,
inputting the KW spatial distribution vectors of the first stage to the nth stage generated for each document based on the learning data set into the classification means;
The learning processing means learns by updating the parameters of the classification means based on the error between the score value of the relevance between the search keyword set and the document output by the classification means and the label value. matter,
An information processing system characterized by
With this configuration, the mutual positional relationship and distribution features of the keywords of the keyword group of the first stage and the keywords of the keyword group of the nth stage, the local mutual positional relationship and distribution characteristics, and the local It is possible to generate a trained machine learning model that can evaluate features of high-order positional relationships that combine various features in a complex manner.

An eighth invention is an example of learning processing of a classifying means using a KW space distribution vector and a KW dependent word space distribution vector,
In the information processing system according to any one of claims 2 and 5,
means for acquiring a learning data set in which a set of search keywords, text data of a document, and a label value indicating the relationship between the set of search keywords and the document are associated;
The 1st to n-th KW space distribution vectors generated for each document, and the 1st to n-th KW related word space distribution vectors corresponding to the 1st to n-th keywords A classification means for determining the relevance between a set of search keywords and documents based on
Learning processing means for learning parameters of the classification means based on the error between the score value of the relevance between the set of search keywords and the document output by the classification means and the label value,
Based on the learning data set, the 1st to nth KW spatial distribution vectors generated for each document, and the 1st to nth stages corresponding to the keyword groups of the 1st to nth stages inputting the KW related word space distribution vector into the classification means;
Based on the error between the score value of the relevance between the search keyword set and the document output by the classification means and the label value, the learning processing means updates the parameters of the classification means to perform learning processing. to do
An information processing system characterized by
With this configuration, in addition to the characteristics of the distribution of keywords, when there is a dependent word that denies the keyword, the location of the keyword and the characteristics of the distribution of the keyword act in the direction of lowering the evaluation as a characteristic, and the keyword is affirmed or not. When there is a subject word to be emphasized, it is possible to generate a trained machine learning model that can act in the direction of strongly evaluating the location and distribution of the keyword as a feature.

A ninth invention is an example of a learning process of a classification means using a KW space distribution vector and a technical term space distribution vector,
A third invention is the information processing system according to any one of the sixth inventions,
means for acquiring a learning data set in which a set of search keywords, text data of a document, and a label value indicating the relationship between the set of search keywords and the document are associated;
Learning processing means for learning parameters of the classification means based on the error between the score value of the relevance between the set of search keywords and the document output by the classification means and the label value,
Based on the learning data set, the 1st to n-th KW space distribution vectors generated for each document and the top 1 to top m-th technical term space distribution vectors are input to the classification means. death,
Based on the error between the score value of the relevance between the search keyword set and the document output by the classification means and the label value, the learning processing means updates the parameters of the classification means to perform learning processing. to do
An information processing system characterized by
With this configuration, in addition to the characteristics of the distribution of keywords, the relationship between the set of keywords and the literature can be further evaluated by further evaluating the technical terms that are often present in the literature and the positional relationship with the keywords and the characteristics of the distribution. A trained machine learning model that can be accurately determined can be generated.

A tenth invention is
Learning data containing KW spatial distribution vectors, which is a so-called data structure invention, specifically,
Learning vector data used for learning processing of the classification means according to the seventh invention,
The KW space distribution vector generating means generates the 1st to n-th columns for each document based on the location of each keyword in the group of keywords in the 1st to n-th columns included in the text data of the document. the KW spatial distribution vector of the stages,
It is vector data for learning including
With this configuration, the mutual positional relationship and distribution features of the keywords of the keyword group of the first stage and the keywords of the keyword group of the nth stage, the local mutual positional relationship and distribution characteristics, and the local It is possible to generate a machine learning model for comprehensively evaluating features of high-order positional relationships that combine various features.

The eleventh invention is
Learning data containing a KW space distribution vector and a KW related word space distribution vector, which is a so-called data structure invention, specifically,
Learning vector data used for learning processing of the classification means according to the eighth invention,
The KW space distribution vector generating means divides into the first to n-th keyword groups based on the location of each of the first to n-th keywords contained in the document text data. and the KW spatial distribution vector of the first stage to the nth stage generated for each document by
The KW dependent word space distribution vector generating means generates for each document based on the location of the dependent word corresponding to each keyword of the first to n-th keyword groups included in the text data of the document. 1st stage to nth stage KW-related word space distribution vector corresponding to the 1st stage to nth stage keyword group generated in
It is vector data for learning including
With this configuration, in addition to the characteristics of the distribution of keywords, when there is a dependent word that denies the keyword, the location of the keyword and the characteristics of the distribution of the keyword act in the direction of lowering the evaluation as a characteristic, and the keyword is affirmed or not. When there is a subject word to be emphasized, it is possible to generate a machine learning model that acts in the direction of strongly evaluating the presence location and distribution of the keyword as a feature.

A twelfth invention is
Learning data containing the KW spatial distribution vector and the technical term spatial distribution vector, which is a so-called data structure invention, specifically:
Learning vector data used for learning processing of the classification means according to the ninth invention,
The KW space distribution vector generating means generates the first to n-th keyword groups based on the location of each keyword in the first to n-th keyword groups included in the text data of the document. Separated into each, the KW spatial distribution vector of the 1st stage to the nth stage generated for each document,
The top 1 to top m-th technologies generated by the technical term space distribution vector generation means for each document based on the locations of the top 1 to top m-th technical terms included in the text data of the document. a term spatial distribution vector;
It is vector data for learning including
With this configuration, in addition to the characteristics of the distribution of keywords, the relationship between the set of keywords and the literature is further evaluated by further evaluating the technical terms that are often present in the literature and the positional relationship with the keywords and the characteristics of the distribution. It can generate machine learning models that can make accurate decisions.

A thirteenth invention is
The invention of the trained model after the learning process has progressed in the fourth invention or the seventh invention, specifically:
In the information processing system according to the fourth invention or the seventh invention, a learned model learned by the learning processing means for determining the relevance between a set of search keywords and a document,
Learning processing means for updating the parameters of the classification means based on the error between the score value of the relevance between the search keyword set and the document output by the classification means and the obtained label value,
inputting the KW spatial distribution vectors of the first stage to the nth stage generated for each document based on the learning data set into the classification means;
Based on the error between the score value of the relevance between the search keyword set and the document output by the classification means and the label value, the learning processing means obtains by learning processing for updating the parameters of the classification means. It is a trained model that has been
With this configuration, the mutual positional relationship and distribution features of the keywords of the keyword group of the first stage and the keywords of the keyword group of the nth stage are local mutual positional relationships and distribution features, and local It is possible to provide a learning model that comprehensively evaluates features of high-order positional relationships in which various features are combined.

A fourteenth invention is
The invention of the trained model after the learning process has progressed in the fifth invention or the eighth invention, specifically,
In the information processing system according to the fifth invention or the eighth invention, a learned model learned by the learning processing means for determining the relevance between a set of search keywords and a document,
Learning processing means for learning parameters of the classification means based on the error between the score value of the relevance between the set of search keywords and the document output by the classification means and the label value,
KW spatial distribution vectors of the 1st to nth stages generated for each document based on the learning data set, and KWs of the 1st to nth stages corresponding to the keyword groups of the 1st to nth stages inputting the dependent word space distribution vector into the classification means;
Based on the error between the score value of the relevance between the search keyword set and the document output by the classification means and the label value, the learning processing means obtains by learning processing for updating the parameters of the classification means. It is a trained model that has been
With this configuration, in addition to the characteristics of the distribution of keywords, when there is a dependent word that denies the keyword, the location of the keyword and the characteristics of the distribution of the keyword act in the direction of lowering the evaluation as a characteristic, and the keyword is affirmed or not. When there is a subject word to be emphasized, it is possible to provide a learning model that works in the direction of strongly evaluating the presence location and distribution of the keyword as a feature.

A fifteenth invention is
The invention of the trained model after the learning process has progressed in the sixth invention or the ninth invention, specifically,
In the information processing system according to the sixth or ninth invention, a learned model learned by the learning processing means for determining the relevance between a set of search keywords and a document,
Learning processing means for learning parameters of the classification means based on the error between the score value of the relevance between the set of search keywords and the document output by the classification means and the label value,
The first to n-th KW space distribution vectors generated for each document based on the learning data set and the top 1 to top m-th technical term space distribution vectors are input to the classification means. ,
Based on the error between the score value of the relevance between the search keyword set and the document output by the classification means and the label value, the learning processing means obtains by learning processing for updating the parameters of the classification means. It is a trained model that has been trained.
With this configuration, in addition to the characteristics of the distribution of keywords, the relationship between the set of keywords and the literature is further evaluated by further evaluating the technical terms that are often present in the literature and the positional relationship with the keywords and the characteristics of the distribution. It is possible to provide a learning model capable of making an accurate determination.

A sixteenth invention is
A parameter in a trained model obtained by learning processing in the information processing system according to any one of the seventh to ninth inventions,
Based on the error between the score value of the relevance between the search keyword set and the document output by the classification means and the label value, the learning processing means updates the parameters of the classification means. It is a parameter after learning.
With this configuration, only the parameters are provided, and the mutual positional relationship and distribution characteristics of the keywords of the keyword group of the first stage and the keywords of the keyword group of the nth stage are local mutual positional relationships It is possible to generate a learning model that comprehensively evaluates features of high-order positional relationships that are combined with features of distribution and local features.

A seventeenth invention is
An invention of a judgment processing system using a trained model,
An information processing system that determines the relevance between a set of search keywords and documents by using a trained model obtained by learning processing in the information processing system according to any one of the seventh to ninth inventions be.

The eighteenth invention is
In an information processing system that determines the relevance between a set of search keywords and a document by using a trained model obtained by learning processing in the information processing system according to any one of the seventh to ninth inventions ,
means for outputting determination results in the order in which the set of search keywords and the document are determined to be highly relevant;
A means for distinguishingly displaying a range that is likely to include the top p% of documents that the searcher actually judges to be highly relevant;
An information processing system characterized by comprising
With this configuration, even if a machine learning model with a black box inside is adopted, it is possible to predict the reliability of the output result.

A nineteenth invention is an invention of a method corresponding to the first invention,
A method for determining the relevance between a set of search keywords and a document,
obtaining text data of the literature;
a step of obtaining a set of keywords from the 1st stage to the nth stage as a set of search keywords;
a KW presence location analysis step of analyzing the location of keywords included in the text data of the document;
a KW space distribution vector generation step of generating a KW space distribution vector indicating how the presence locations of the keywords analyzed by the KW presence location analysis means are distributed in the text data of the document;
The KW space distribution vector generating means generates the KW space distribution of the first to n-th stages based on the location of each keyword in the first to n-th stage keywords included in the text data of the document. generate vectors respectively,
Based on the KW space distribution vectors of the 1st to nth stages generated for each document, the positional relationship between the keywords of the 1st to nth stage keywords and the characteristics of the distribution are evaluated, and the search keywords determining the relevance of the set of
An information processing method characterized by
With this configuration, the mutual positional relationship and distribution features of the keywords of the keyword group of the first stage and the keywords of the keyword group of the nth stage are local mutual positional relationships and distribution features, and local It is possible to comprehensively evaluate the features of high-order positional relationships that combine various features in a complex manner.

A twentieth invention is an invention of a method corresponding to the second invention,
The information processing method according to the nineteenth invention, further comprising:
a KW dependent word existence location analysis step of analyzing the existence location of a dependent word of a keyword existing in the vicinity of the location of the keyword included in the text data of the document;
KW for generating a KW-related word space distribution vector showing how the locations of dependent words near the keyword analyzed by the KW-dependent word presence location analyzing step are distributed in the text data of the document; a dependent word space distribution vector generation step;
According to the KW dependent word space distribution vector generation step, the first Generating KW related word space distribution vectors of the 1st to n-th stages corresponding to the keyword groups of the 1st to n-th stages,
The 1st to n-th KW space distribution vectors generated for each document, and the 1st to n-th KW related word space distribution vectors corresponding to the 1st to n-th keywords ,On the basis of the,
Features of positional relationships or distributions among keywords in the first to n-th keyword group, and mutual relationships between each keyword in the first to n-th keyword group and their corresponding dependent words Evaluating the positional relationship and distribution characteristics to determine the relevance between the set of search keywords and the document,
An information processing method characterized by
With this configuration, in addition to the characteristics of the distribution of keywords, when there is a dependent word that denies the keyword, the location of the keyword and the characteristics of the distribution of the keyword act in the direction of lowering the evaluation as a characteristic, and the keyword is affirmed or not. When there is a subject word to be emphasized, the feature of the location and distribution of the keyword can act in the direction of strongly evaluating the feature.

A twenty-first invention is an invention of a method corresponding to the third invention,
The information processing method according to the nineteenth invention, further comprising:
A technical term analysis step of analyzing technical terms included in text data of documents in order of frequency;
A technical term presence location analysis step of analyzing the locations of the technical terms from the top 1 to the top m of the technical terms analyzed by the technical term analysis step;
a technical term space distribution vector generation step of generating a technical term space distribution vector indicating how the locations of the technical terms analyzed by the technical term presence location analysis step are distributed in the text data of the document; prepared,
By the technical term space distribution vector generation step, the top 1 to top m th technical term space distribution vectors based on the locations of the top 1 to top m th technical terms included in the text data of the document respectively, and
Based on the first to n-th KW spatial distribution vectors generated for each document and the top 1 to top m-th technical term space distribution vectors,
Features of mutual positional relationships and distributions of keywords in the first to n-th keyword groups, and keywords in the first to n-th keyword groups and the top first to top m-th technical terms Evaluating the mutual positional relationship and distribution characteristics with, to determine the relevance between the set of search keywords and the document,
An information processing method characterized by
With this configuration, in addition to the characteristics of the distribution of keywords, the relationship between the set of keywords and the literature is further evaluated by further evaluating the technical terms that are often present in the literature and the positional relationship with the keywords and the characteristics of the distribution. be able to judge accurately.

A twenty-second invention is a program for executing the information processing method according to any one of the nineteenth invention to the twenty-first invention in a computer system.

According to the present invention, it is possible to provide an information processing system, an information processing method, and a program for determining the relevance between a set of search keywords and a document.

It is a figure which shows the difference between the document search schemes, such as the conventional patent, and the document search scheme of this invention. It is a figure which shows an example of the system configuration|structure of this invention. It is a figure which shows an example of the whole structure of this invention. It is a figure which shows an example of the whole structure in the case of implement|achieving by a dictionary and an algorithm. FIG. 4 is a diagram showing an overview of characteristics of the distribution of keywords in documents hit by predetermined search conditions. Of these, (1) shows the characteristics of the distribution of keywords in patent documents with low relevance to the keywords, and (2) shows the characteristics of the distribution of keywords in patent documents with high relevance to the keywords. is shown. In the present invention, based on the set of the first to n-th keywords and the text data of the literature, the KW existence location analysis means analyzes the existence locations of the keywords contained in the literature, and generates a KW space distribution vector. The means is based on the location of each keyword in the first to n-th keyword groups contained in the text data of the document, a feature vector expressing the feature of the location (first to n-th KW space distribution vector, etc.) is a diagram showing an example of an outline of processing for generating each keyword group in each stage. It is a figure which shows an example of the outline|summary of a structure of the classification means (machine-learning model) in this invention. FIG. 5 is a diagram showing an example of convolution processing of the first layer and the second layer when the classification means (machine learning model) in the present invention is configured by a neural network; FIG. 10 is a flow chart showing the entire process of determining the relevance between a set of search keywords and a document in the present invention; FIG. FIG. 10 is a flowchart of processing in the classification means in the present invention, and is a diagram showing an example of a flowchart of convolution processing of the first layer, the second layer and subsequent layers when the classification means (machine learning model) is composed of a neural network. . It is a figure which shows an example of the data for learning (for training/testing) in this invention. It is an example of setting values of the learning process and setting values of the configuration of the classification means 500 in the present invention. It is an example of setting values of the learning process and setting values of the configuration of the classification means 500 in the present invention. It is an example of setting values of the learning process and setting values of the configuration of the classification means 500 in the present invention. It is an example of performance evaluation of processing for determining the relevance between a set of search keywords and documents in the present invention. FIG. 10 is a diagram showing the range of scores in which the top 3% of documents with the highest number are included, and the man-hour reduction rate. It is an example of the display screen or the output result of the evaluation result of the process of determining the relevance between the search keyword set and the document in the present invention.

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings.

In the embodiment of the present invention, an information processing apparatus for document research will be described.

1. Overall Overview of the Present Invention FIG. 2 is an example of a system configuration including an information processing apparatus 1000 according to an embodiment of the present invention. In FIG. 2, an information processing device 1000 and one or more user terminals 3000 and 3100 are connected via a communication network 2000 .

In this embodiment, the information processing terminal 1000 performs determination processing of the relevance between the set of search keywords and the document, and the user terminals 3000 and 3100 perform input of keywords and output such as screen display of determination results.
The communication network 2000 is a wired or wireless communication line, such as the Internet such as WAN (Wide Area Network), LAN (Local Area Network), satellite line, public telephone line, and the like.

The information processing apparatus 1000 may be a server, a cloud, or a computer with a certain processing capability, such as a desktop personal computer or a notebook personal computer.
Also, although not shown, the information processing apparatus 1000 includes control means including a central processing unit (CPU) for processing data and performing various kinds of processing; It has storage means for storing, input means for acquiring document data, search keywords, and various setting values, output means for printing or displaying determination results, and communication means for communicating with a communication network.

Then, the program stored in the storage means is loaded into the temporary storage means or the like, and the central processing unit executes each step of the program to configure each means such as the analysis means 300 and the classification means 500 which will be described later.
In addition, although the monitor device is provided in FIG. 2, the monitor device may or may not be provided.

Also, the user terminals 3000 and 3100 may be terminals that can be connected to the communication network 2000 . For example, the user terminals 3000 and 3100 are various mobile information terminals such as a desktop computer, a notebook computer, a smart phone, a goggle-type head-up display worn on the head such as a smart watch, a VR headset, and a glasses-type terminal. It may be a PDA (Personal Digital Assistant).

In addition, the user terminals 3000 and 3100 are selectively provided with control means, storage means, input means, output means for displaying and printing determination results, and the like, as required.
In the above description, a system configured by a network has been described as an example, but the present invention is not limited to this. Terminals 3000 and 3100 may be in the form of a so-called stand-alone device in which one information processing device 1000 is integrated.

With reference to FIG. 3, an overview of determination processing and learning processing for determining the relevance between a set of search keywords and documents according to the present invention will be described.
FIG. 3 is a diagram showing an example of the overall configuration of the information processing apparatus 1000 according to the embodiment of the present invention, showing functional blocks of each process.
Of the configuration of this block diagram, the functional blocks from symbols 100 to 500 are common to the determination process and the learning process, and in the determination process, the text data of the document and the search keyword are acquired and the score is output. On the other hand, the learning process is different in that a process of updating the parameters of the classifier (machine learning model) 500 based on the error between the score output and the label (teacher data) is added.

In addition, when developing a machine learning model, (1) what kind of features of the data, which is the material, are extracted and how, and how to convert them into feature vectors so that the features of the material can be expressed well. (also called feature expression engineering)”, (2) arrangement of convolution layers and pooling layers, configuration of convolution filters, number of channels (number of types) and arrangement, number of neural network stages, feedforward network or recursive type (3) What kind of training data and teacher data Prepare a set of, what unit (mini-batch size, number of epochs, learning rate, etc.) and what algorithm (gradient descent, error backpropagation, dropout, etc.) to use for learning There is a scene of "learning processing".

<Determination process>
The information processing apparatus 1000 includes means for acquiring text data 100 of documents such as patent documents, papers, and court cases, and search keywords 200 obtained by searching the document data. Literature data stored in various servers and clouds on a network, or storage means of the information processing system 1000 and user terminals 3000 and 3100 can be used.
The search keywords are, for example, the first-level keyword group (e.g., mail or correspondence), the second-level keyword group (e.g., nuisance or spam), and the third-level keyword group (e.g., filter or separation Or detection) and 4th stage ..., etc., as a combination of multiple stages of keyword groups in the form of multiplying the keyword groups of the 1st to nth stages that group together synonyms and similar concepts can be obtained.

Next, the analysis means 300 performs a morphological analysis that decomposes the document text data 100 into morphemes such as words, and the KW existence location analysis means (not shown) of the analysis means 300 analyzes the acquired first to second Locations where each keyword of the n-level keyword group exists in the document are examined, the document is divided into the first to n-th keyword groups, and the keyword distribution analysis results are temporarily saved.
Then, the feature vector generating means (spatial distribution vector generating means) 400, based on the keyword distribution analysis results recorded separately for each keyword group of the 1st to nth stages, is divided into each keyword group, and a spatial distribution vector of the keyword in each stage is generated by vectorizing the location and distribution characteristics of the keyword, separately for each stage.
Note that the feature vector (spatial distribution vector) is a vectorized index that indicates the location and distribution of words such as (search) keywords in documents, dependent words of keywords, and technical terms.
For this reason, it is called a "KW spatial distribution vector" when indicating the location and distribution characteristics of a (search) keyword in a document, is called a "technical term spatial distribution vector".
Similarly, the feature vector generation means (spatial distribution vector generation means) 400 includes "KW space distribution vector generation means" for keywords, "KW dependent word space distribution vector generation means" for dependent words of keywords, The case of technical terms is called "technical term space distribution vector generating means".

Specifically, the feature vector generation means (KW space distribution vector generation means) 400 first uses A KW space distribution vector is generated that indicates how the location of each keyword of the keyword group in the first row of is distributed in the text data of the document.

Then, based on the distribution analysis result of each keyword of the second-stage keyword group, the presence location of each keyword of the second-stage keyword group (for example, "nuisance or spam") in the document is the text data of the document Generate a KW spatial distribution vector that indicates how it is distributed in
Thereafter, similarly, the keyword group is divided for each n-th stage keyword group, and a KW space distribution vector for each stage is generated.

Here, the keyword space distribution vector for each stage is generated separately for each keyword group from the 1st stage to the n-th stage because the literature vector containing the hit points of the keywords for all stages is generated as in the prior art. is created, it is not possible to pick up the positional relationship and distribution characteristics of the neighboring keywords in each row. . Note that when keywords are in a positional relationship close to each other, for example, when the keywords are separated by several words in the same paragraph, or when the paragraphs are close to each other, such as the paragraphs before and after the keywords. Say.

More specifically, the KW spatial distribution vector in the first stage indicating the distribution of each keyword in the keyword group in the first stage (e.g., "mail or correspondence"), the keyword group in the second stage (e.g., "nuisance or spam ”), the second KW space distribution vector showing the distribution of each keyword, . A keyword space distribution vector for each stage is generated by dividing the keyword groups for each stage, such as KW space distribution vector.

After that, by convolving the keyword space distribution vector of each stage with a predetermined convolution filter and synthesizing the convolution results, the positional relationship and distribution characteristics of the keywords of the first to n-th stage keywords, and , a classification means 500 composed of a neural network that can sequentially evaluate higher-order features that are multifaceted combinations of these features can be constructed.
Based on the KW space distribution vector generated for each document based on the text data of the document and the search keyword, the classification means 500 identifies the positional relationships and distribution features of the keywords in the first to n-th level keywords. to determine the relevance of the set of search keywords to the document.

<Learning processing>
Since the classification means 500 before learning progresses cannot determine with sufficient accuracy, learning processing (machine learning) is performed by the learning processing means 600 .
Note that machine learning means obtaining lines for classification by mathematical processing according to changes in data. The line for classification is determined (learned) so that the error with the data is minimized, and the function used at this time is the mean square, etc. In machine learning, it is called the error function or loss function. When a (deep) neural network is selected as a machine learning model, machine learning (training) is the process of updating parameters such as the weights of convolution filters so that the error function is minimized.

The learning processing means 600 performs gradient descent and error backpropagation based on the error between the score value of the association between the search keyword set and the document output by the classification means 500 and the label value that is the acquired teacher data. The learning process is performed by updating the parameters of the classification means 500 based on a technique such as a method.
More specifically, in the learning process, a set of data for learning as shown in FIG. , and updates the parameters of the classification means 500 based on the error between the score value output by the classification means 500 and the label value (teacher data).
As shown in FIGS. 12 to 14, the learning process is performed a predetermined number of times (the number of epochs) by determining the degree of parameter update (learning rate) in a fixed group of units (mini-batch size, etc.).
In the learning process, the dropout method is appropriately used in order to improve the generalization performance and robust performance of the classifying means 500 and to prevent biased over-learning.
Classification means 500 that has sufficiently advanced learning, based on the text data of the document and the KW spatial distribution vector generated for each document based on the search keyword, By evaluating the positional relationships and distribution features with high accuracy, it becomes possible to determine the relevance between the search keyword set and the document with a certain degree of accuracy.
Note that the classification means (machine learning model) 500 that has undergone sufficient learning (training) is called a "learned machine learning model".

Note that the process of determining the relevance between the set of search keywords and the document is executed by the machine learning-based classification means 500 and the learning processing means 600. For example, as shown in FIG. , can also be implemented by an algorithm-based evaluation processing means 800 .
In this case, based on the text data 100 of the document and the search keyword 200, the analysis means 300 performs morphological analysis and keyword distribution analysis, and the feature vector generation means 400 generates the KW space distribution vector. do
After that, the evaluation processing means 700 evaluates the similarity between the KW space distribution vectors of the 1st to nth stages based on the mathematical distance (Euclidean distance and inner product), cosine similarity, etc. Based on the search keyword set and the document It is possible to evaluate the relevance to

In this aspect as well, unlike the conventional method of generating a feature vector of the entire document including the distribution of a plurality of keywords and comparing the similarity with the feature vectors of other documents, the relationship between the first keyword and the third keyword is It is possible to evaluate the similarity of the feature vectors of the document, including the distribution of local keywords, such as the evaluation of the second keyword and the third keyword, such as the evaluation of the connection between the second keyword and the third keyword. becomes possible.

Also, in the evaluation dictionary 800, the mathematical distance between the first keyword and the second keyword, the mathematical distance between the second keyword and the third keyword, etc. are stored in what kind of case is the relevance to the technology to be researched? Indices for determining whether the degree of similarity is high, indices for determining whether the similarity is high when the distribution of keywords is similar, and which items (problems, solutions, etc.) in patent documents, etc. means, examples, effects, etc.) are recorded.

However, these indicators registered in the evaluation dictionary are manually analyzed in large quantities from various aspects of the survey results, and are finally registered in the dictionary as indicators that have been arrived at through trial and error. Besides, it requires a lot of know-how.
On the other hand, if a machine learning model is used as the classification means 500, it is only necessary to learn a large amount of correct values (labels) related to relevance among the results of researching documents. There is an advantage that it can be reduced.

2. Characteristic Analysis of Keyword Distribution as a Prerequisite for Building a Literature Search System FIG. It is a diagram analyzing the thought process in the brain of a searcher in .
Of these, (1) shows the characteristics of the distribution of keywords in patent documents with low relevance to keywords, and (2) shows the characteristics of the distribution of keywords in patent documents with high relevance to keywords. is shown.
If possible, it would be more accurate to explain using the results of analysis with three or more keywords, but for the sake of simplicity, the results of analysis with two keywords will be used.

First, looking at FIG. 5(1), as a feature of the keyword distribution of the patent documents that the searcher confirmed and judged to have low relevance to the keyword, in the same item (for example, "claims"), There is a feature that the keyword in the first stage (for example, "○△ calculation") and the keyword in the second stage (for example, "employee") do not hit as a set.
In addition, there is a feature that the keywords hit only a part of the entire literature, and only a few hits at a time.
In addition, there is a characteristic that keywords are not hit in items such as relatively important problems and solutions.
Furthermore, it can be seen that there is a feature that another technical term (expense settlement), which is different from the technical term 〇△ calculation, appears frequently throughout.

On the other hand, looking at FIG. 5(2), as a feature of the keyword distribution of the patent documents that the searcher confirmed and judged to be highly relevant to the keyword, in the same item (for example, "means for solving the problem"), There is a feature that a keyword in the first row (for example, “○△ calculation”) and a keyword in the second row (for example, “employee”) are hit as a set.

In addition, it can be seen that there is also a feature that multiple hits are made for the other items as a set of the first-level keyword and the second-level keyword.
In addition, there is also a characteristic that the distances between the hits of the first-stage keyword and the second-stage keyword as a set are close to each other.
In addition, it can be seen that each keyword is a set and has a feature that it exists widely in the entire document.
It should be noted that, although not shown, if a certain keyword has many hits, but the other keyword has few hits, it is a feature that the relevance can be evaluated as low.

Furthermore, although not shown, when there are many keywords such as the third keyword, the fourth keyword, etc., there are many possible combinations of keywords in each row, and each row and the keyword in each row can be combined. A lot of mutual positional relationships and distribution characteristics can be grasped.
More specifically, for example, a set of keyword pairs of a certain column and a certain column exists locally in a predetermined range of the document, or a pair of keywords of a certain column and a certain column exists throughout the document. By grasping characteristics such as existence in a set, it is possible to evaluate the relevance between the set of keywords and the document, that is, the relevance between the technology to be researched and the document.
In addition, when it is done manually, it is usually not possible to recognize and analyze the positional relationship and distribution characteristics of keywords that are located far away from each other. According to this method, relevance can be evaluated by analyzing multifaceted positional relationships and distribution features such as local or wide range distribution features between keywords.

As described above, in order to evaluate the relationship between the set of keywords and the literature (that is, the relationship between the research target technology and the literature), the locations and distribution characteristics of the keywords at each level must be evaluated. It can be seen that it is effective to divide and vectorize each. This is because vectorizing them separately makes it possible to multilaterally analyze and evaluate the local positional relationships and distribution characteristics of the feature vectors in each row, as well as the overall distribution characteristics. .
Based on the results of analysis of the searcher's brain's thinking process in such a literature survey, we will consider what kind of feature vector should be established.

3. Regarding the relationship between the feature vector generation of the present invention and the subsequent machine learning model In formulating the feature vector, it is rational to consider the structure of the classification means (machine learning model) that performs classification by inputting the feature vector. .
Therefore, as a machine learning model, for example, as an example, the structure of a neural network (including a deep neural network) will be considered.
Neural networks focus on extracting the internal features of the feature vector and the features of multiple feature vectors and creating a feature map by convolution processing with a convolution filter for multiple feature vector inputs. can.

Specifically, in a layer close to the input layer, a local feature map is generated for single feature vectors or between feature vectors. It can be noted that the next map is gradually formed, and finally, the process of outputting the determination result based on the combination of local features, the features of a slightly wider range, and the overall feature that integrates them is performed.

In addition, it can be noted that neural networks are known to have good compatibility with image data as input vectors, as evidenced by achievements in object recognition and the like.
And in this point, even in literature research, there is an aspect of reading sentences by making use of the logical grasping function of the left brain, but rather, if you become a veteran, you will consciously or unconsciously use the right brain and use a kind of image processing. Since the relationship between keywords and documents is evaluated by grasping the positional relationships and distribution characteristics of keywords (a kind of image recognition), the positional relations and distribution characteristics of keywords can be evaluated using image processing. is assumed to be effective.

In the discussion of the structure of the machine learning model, a neural network or deep neural network was mainly used as an example, but it is not limited to this, and any machine learning model that handles vector data can be used. , SVM (support vector machine), logistic regression model, and Bayes as machine learning models for learning with supervised data, and clustering, K-means, etc. as machine learning models without supervised data.

Here, the analysis of the thinking process of the searcher's brain in the literature survey, which was examined in the analysis of the characteristics of the keyword distribution in FIG. According to this, when determining the relevance between the set of search keywords of the present invention and the document, while well expressing the location and distribution characteristics of the keywords in each stage in the document, the later machine learning model In order to be able to evaluate the positional relationship and distribution characteristics between keywords in a given row and keywords in another row, it is reasonable to separate the keywords in each row and vectorize them individually. I came to think.

More specifically, vectorize the locations where each keyword of the first stage keyword group (eg, 〇△ process, 〇〇 process) exists in the document, and vectorize the second stage keyword group (eg, employee, employee , employees) are vectorized in the document, the keyword group in the third stage (e.g., assessment evaluation, performance evaluation) is vectorized in the document, and the fourth stage It is rational to separate the keywords in each row like . . . of the keyword group and vectorize them individually.

Also, although it is not essential, for example, when using a neural network as a machine learning model, when vectorizing, the location and distribution characteristics of keywords can be used as if they were grasped by the right brain of a searcher as described above. It is useful to treat it as if it were image data, and vector data such as vertical and horizontal image data may be generated.

4. About feature vector generation processing 4-1. Generation of KW Space Distribution Vector Next, processing for generating a feature vector (KW space distribution vector) will be described with reference to FIG.
First, as a premise, the keywords are a group of keywords in the first stage (eg, "○△ processing, ...") and a group of keywords in the second stage (eg, "employee, employee, employee, ...") , 3rd row keyword group (e.g., “assessment evaluation, performance evaluation, …”), 4th row . . . It is not limited to this.
It should be noted that the tier can be arbitrarily selected, and the determination performance is not affected by which keyword group is in which tier.

Then, the search conditions are formulated as follows: (keyword group in the first stage) AND (keyword group in the second stage) AND (keyword group in the third stage) AND . . . is stored in the storage unit.
Also, for example, it is desirable that the group of keywords in the second row (e.g., employee, employee, employee) is composed of synonyms of similar concepts, but is not limited to this.
Here, (employee, employee, employee) indicates a set of OR conditions such as (employee or employee or employee or).
Incidentally, a dictionary of synonyms may be prepared separately, and synonyms may be presented to the keywords set by the searcher for selection, or the synonyms may be automatically added to the keyword group in a predetermined row.

It is assumed that the information processing system of the present application is often used to determine the relevance of documents hit by a certain search condition (keyword set) under the same search condition (keyword set). It is desirable that the set of keywords for the search condition for collecting documents and the set of search keywords for determining the relevance between the set of search keywords and the documents are the same, but are not limited to this. It doesn't matter if it's different.
According to the present invention, the relevance to a document is determined based on how the keywords in multiple columns are distributed in the document. This is because it is possible to set a set and determine the relevance between the set of keywords and the document.

First, the input means of the information processing system acquires the text data 100 of the document and the search keyword 200, the analysis means 300 performs morphological analysis, analyzes the place where the set keyword exists in the document, Based on the item name and paragraph information, the keyword distribution analysis result is temporarily recorded for each item or paragraph (→process 1 of balloon in FIG. 6).

Next, the feature vector generating means 400 vectorizes the distribution of keywords for each paragraph while considering the number of the word on the basis of the keyword distribution analysis results summarized for each item or paragraph. (→ Processing 2 of balloon in FIG. 6).
More specifically, as shown from the center to the right side of FIG. Since it exists, from the top of CH2 (second KW space distribution vector) on the right side of FIG. Vectorization is done by setting the bit to 1.

Similarly, for example, in the "fourth paragraph", the keyword "employee" of the keyword group in the second row exists in the "ninth word" from the beginning of the paragraph, so CH2 ( 2nd KW spatial distribution vector) Vectorization is performed by setting the bit in the ``4th row (corresponding to the 4th paragraph)'' to the ``9th column (corresponding to the 9th word)'' to 1. .

In addition, in the second level keyword group, two different keywords "employee" and "employee" are hit, but both are keywords in the second level, so they are the same KW space in the second level. It is treated as an element of the distribution vector.
Thus, a second KW spatial distribution vector for the CH2 (second row) keyword can be generated.
Similarly, the KW space distribution vectors of the 1st stage to the nth stage are generated respectively (→process 3 of balloon in FIG. 6).

Documents such as patents, papers, and judicial precedents have different lengths of sentences and different numbers of words. Therefore, it is preferable to normalize the vectors appropriately so that the vectors have the same number of dimensions.
In FIG. 6, a 2000-dimensional vector of horizontal 20 (corresponding to 20 words)×vertical 100 (corresponding to 100 paragraphs) is used for convenience of drawing, but this is only an example and various Dimension can be set.
In the case of patent documents, there are many sentences of about 10,000 to 30,000 characters, so for example, the number of words in each paragraph is 60 words (about 3 characters per word on average x 60 words = about 200 characters), and the number of paragraphs is 100. It can be normalized to the extent of paragraphs, but other dimensions may be used.

The KW space distribution vector is used because, for example, as shown on the right side of FIG. This is due to the fact that it appears to be
In this case, when a convolutional neural network is selected as a machine learning model, there is an advantage that it is highly compatible with the convolution processing (matrix operation) of the convolution filter.
In addition, the learning process can be replaced by learning the parameters of the convolution filter, and it becomes a so-called loose connection, which is the opposite concept of a fully connected neural network, reducing the number of training data, robust performance, and generalization performance. It has the advantage that it can be improved.
In addition, by preparing multiple types of convolution filters, there is also the advantage that it is possible to extract and evaluate the positional relationships and distribution features of various keywords.

In other words, if the keywords in each column exist within a paragraph, or if they exist in a paragraph and the paragraphs before and after it, they often have a relationship in a certain context (that is, keywords and (Because it is an index that indicates the high relevance of the document), by vectorizing the vertical and horizontal grids, when convolution processing is performed using a grid-shaped convolution filter, the positions of the keywords It can be expected to efficiently extract features of relationships and distributions.

In addition, the relationship between the KW space distribution vectors in the first and third stages can be evaluated, and the relationship between the KW space distribution vectors in the first and third stages, or between the second and third stages can be evaluated individually. , or a combination of these evaluations, or a comprehensive evaluation of the relationship between the KW space distribution vectors in the 1st to n-th stages.
Then, when combined with a machine learning model such as a convolutional neural network, in a layer close to the input layer, local features such as the distribution features in the vicinity of the KW spatial distribution vectors at each stage are extracted, and the layer proceeds Gradually, it becomes possible to extract comprehensive features that combine them.
Note that the feature vector does not have to be an image data-like vector composed of rows and columns (vertical and horizontal) squares. . . ) and bits of 0 and 1 may be used.
As mentioned above, the degree of importance of the item in which the keyword exists in the document varies depending on the item (for example, in patent documents, the items of solutions and effects are important items, etc.). Vectorization including such information contributes to the improvement of evaluation accuracy.
In this respect, for example, in the case of patent documents, it can be noted that item names and their order are defined. That is, since the feature vector (spatial distribution vector) of the present invention is vectorized in the order of paragraphs, it is vectorized including information such as the positional relationship and order of the items in the patent document, and the evaluation accuracy You can contribute to the improvement.

4-2. Generation of KW Related Word Spatial Distribution Vector Next, a process of generating a feature vector (KW related word space distribution vector) will be described.
The analysis means 300 performs morphological analysis to divide sentences into words, and analyzes nouns, case particles, particles, auxiliary verbs, verbs, adjectives, adjectives, etc., and separates them into independent words (nouns, verbs, adjectives, etc.). It is broken down into units called clauses consisting of adjuncts (particles, auxiliary verbs). Then, the clauses are combined and the relationship (dependency) between the modification and the modification clause is analyzed.
Based on the results of such morphological analysis, syntactic analysis, and dependency analysis, from the text data of the document, not only the KW space distribution vector described above, but also the KW A dependent word space distribution vector can also be generated.
For example, negative dependencies such as ``not'', ``cannot be said'', and ``necessarily'', and positively affirming or emphasizing dependencies such as `` Dependent words such as adjectives and adjective nouns such as "effective", "higher level", "particularly", and "extremely" are recorded in a dictionary, and the analysis means 300 detects these dependent words. Sometimes, the distribution situation may be paired with the keyword and the KW space distribution vector of the n-th stage to generate a KW-related word space distribution vector and use it.
If there is a dependent word that negates the keyword, the location and distribution characteristics of the keyword act in the direction of low evaluation as a feature, and if there is a dependent word that positively affirms or emphasizes the keyword, the keyword can be made to act in the direction of strongly evaluating the location and distribution of .
Therefore, for positive dependent words and negative dependent words, the distribution of the dependent words is generated as a spatial distribution vector, and input to the classification means 500 together with the KW spatial distribution vector. The location and distribution of dependent words and the location and distribution of keywords are evaluated in relation to each other to improve the accuracy of determining the relationship between the set of keywords and the document. . Note that it is not necessary to vectorize both positive and negative dependent words, and for example, only positive dependent words may be spatially distributed vectorized.

4-3. Generation of Technical Term Spatial Distribution Vector Next, the process of generating a feature vector (technical term spatial distribution vector) will be described.
The analysis means 300 performs morphological analysis to separate sentences into words, analyzes nouns, case particles, particles, auxiliary verbs, verbs, adjectives, adjectives, etc., and identifies words (nouns) such as technical terms in the literature. are detected and sorted in order of frequency of occurrence, based on the location and distribution of each word in order of frequency of occurrence, the top 1 to top m-th technical term space distribution vectors are generated and used. You may
For example, as shown in FIG. 5(1), many words such as "expense settlement" and "transfer" are included in the document, and such words are detected and sorted in order of frequency.
Patent documents and papers generally contain many descriptions of technology. Since it includes words such as "terms" and "terms relating to human behavior", "technical terms" may include "legal terms".

In FIG. 5(1), for example, the word "expense settlement", which is a technical term in the literature, has the highest frequency of occurrence, and its number of occurrences is 9 times. It is the second highest, and the number of times is 5 times.
On the other hand, the frequency of appearance of the word “○△ calculation” in the search keywords is 1, and the frequency of appearance of the word “employee” is 1, which is very different. It can be used as information that makes it possible to infer that the relevance to the document is low.
In addition, in FIG. 5 (2), the word “○△ calculation”, which is a technical term in the literature, has the highest appearance frequency, and the number of occurrences is 13 times. is the second highest, and the number of times is 10 times, occupying the top position. On the other hand, the keywords for search include "○△ calculation" and "employee", and the correlation between the frequency of occurrence of technical terms and the frequency of occurrence of the set of keywords is high, and the relevance between the set of keywords and the literature It can be used as information to infer that the
Therefore, by generating the distribution status of technical terms as a spatial distribution vector and inputting it to the classification means 500 together with the KW spatial distribution vector, the location and distribution status of technical terms and the location and distribution of keywords can be obtained. The situation was evaluated in relation to each other to improve the accuracy of determining the relevance of the set of keywords to the document.

5. Configuration of Classification Means (Machine Learning Model) Next, an example of the configuration of the classification means (machine learning model) 500 will be described.
FIG. 7 shows a configuration example when a deep neural network (DNN) with two or more intermediate layers is adopted as a machine learning model.
Classification means (machine learning model) 500 consists of 1st to Jth layers for feature extraction, and an Xth layer for comprehensively classifying (determining) based on the features extracted in the intermediate layer. It is

The first layer and the second layer, the second layer and the third layer, .
The transfer function is also called an activation function, and a nonlinear sigmoid function or a linear ReLU (ramp function) may be used to avoid the vanishing gradient problem in backward propagation learning processing.
Although not shown, a pooling layer may be provided in the convolution layer to perform pooling processing for further abstracting the convolution result.
This is because the pooling process can reduce the influence of deviations in the positional relationship between keywords.

The convolution filter is between the first and second layers, between the second and third layers, between the third and fourth layers, between the (n−1)th layer and the nth layer, Different sizes can be used.
A convolution filter (α×α) between the first layer and the second layer extracts the positional relationship and distribution features between keywords in a slightly wider range with a size of 5×5, for example. The convolution filter (β × β) between the three layers extracts features that are a combination of multiple features within a predetermined range among the features extracted in the first layer. It can be 3×3. Of course, other sizes may be used.
Also, for the convenience of notation, each layer is shown as β×β in order, but each layer may have a different size.
The number of types of convolution filters can also be determined according to the target performance. In the figure, 20 types are used as an example. Not limited.

The final X-th layer is, for example, a convolution layer with a 1×1 size convolution filter, and is a layer that performs comprehensive judgment from the feature maps generated in the previous layers. to output the probability that the document is associated with the set of keywords, ie, the document is associated with the researched technology.
For convenience of explanation, for example, if a keyword consists of three stages, the input consists of 3CHs of KW spatial distribution vectors in the first to third stages.
Assuming that there are 20 types of convolution filters, the results of the convolution processing for 3 channels are combined into one image in the convolution processing of the first layer, and 20 types of it are generated. A convolution process is performed based on the convolution results of the first to third stages, extracting the positional relationships and distribution features of the hit locations near the keywords in the first to third stages, and generating a feature map of the distribution of the neighborhood (Fig. Step 1 to step 2 of the balloon in 7).

In the second layer, for example, when 20 types of convolution filters are provided, the 20 convolution results of the previous stage are each subjected to convolution processing by a set of 20 convolution filters, and the results are combined into one, Since 20 types of it are generated, 20 convolution results are generated in the next stage, and the features generated in the previous stage are combined in a complex manner, such as the combination of the hit points of the keywords that are slightly separated and the distribution features. Generate a feature map (step 3 in callout in FIG. 7).
The same processing is sequentially performed for each layer, and as the layers progress, a more comprehensive high-level feature map is generated that includes local features and features that combine them in a complex manner (Fig. 7). step 4) of the balloon, and output the final judgment result.

6. Convolution Processing Related to Feature Map Formation FIG. 8 shows details of the convolution processing performed in the first layer.
First, forward propagation during determination processing will be mainly described, and reverse propagation during learning will also be described below.
6-1. Judgment result (score) output processing, which is forward propagation processing Convolution filters for 3 CHs are defined for the KW spatial distribution vectors for the first to third 3 CHs, and convolution processing is performed for each CH. (process 1 of balloon in FIG. 8).
Note that the convolution process is performed by performing a matrix operation on the numerical values (parameters) of the convolution filter while appropriately striding the convolution filter for each element of the KW space distribution vector generated from the literature data. During convolution processing, padding processing may be performed as appropriate to fill the outside with 0 so that the number of dimensions in the next stage does not change from that in the previous stage.
Also, the application interval (stride) of the convolution filter can be about 1 to 2, but is not limited to this.

Then, the results of the convolution processing for 3CH are combined into one image. For synthesis processing, simple addition, simple averaging, weighted averaging, or the like of each convolution result can be selected, and the results output via a predetermined transfer function with appropriate bias can be used.
As a result, for example, at the stage where the KW spatial distribution vector of CH1 is subjected to convolution processing with the convolution filter for CH1, the positional relationship between the keywords in the first stage of CH1 and the distribution features are extracted formally. However, by synthesizing the convolution results of the KW spatial distribution vectors of the first to third stages of CH1 to CH3, convolution processing was performed while moving the convolution filters for each CH in conjunction. (process 2 of balloon in FIG. 8).

In other words, the above processing is equivalent to extracting the mutual positional relationships and distribution features of the first to third keywords with a convolution filter (description 1 of the balloon in FIG. 8), and based on the above processing If a learning process as described later is performed, the parameters of the filters of CH1 to CH3 are learned as a set, and as a convolution filter that picks up the characteristics of the mutual (slightly adjacent) positional relationship of the first to third keywords It becomes possible to make it function (description 2 in FIG. 8).
The synthesized result can be regarded as a feature map that extracts the mutual (slightly adjacent) positional relationships and distribution features of the keywords in the first to third stages (description 3 of balloon in FIG. 8).

After that, by changing the type of convolution filter, 20 pieces of synthesis results of 1 set x 20 types of convolution processing for 3 channels are generated, and there are multiple feature maps that can extract various patterns of positional relationships and distribution features between keywords. generated.
In addition, although not shown, the convolution processing of the second and subsequent layers is similarly performed, and a large number of multifaceted high-order feature maps are generated by combining a plurality of local or multifaceted feature maps, and at the output stage of the final layer, A determination result is generated and a score value, which is an index of relevance between the set of keywords and the document, is output.
The above is the judgment processing of the forward propagation until the score value is output.

6-2. In the case of backward propagation learning processing, the learning processing is the same up to the output of the score value, but parameter update processing is added. When the machine learning model of the classifier is a neural network, the parameter update process is the update of parameters such as the weight of the convolution filter.
The learning processing means 600 multiplies a certain number of learning data (literature data) called a batch size by a predetermined learning rate, and calculates the error between the score value of the output result and the label value (teaching data) which is the correct value. Based on, the parameters of the classification means 500 are updated. This single learning process is called a mini-batch, and is repeated until the learning process for all the learning data is completed. The unit of the learning process up to this point is called one epoch, and while the learning rate is appropriately adjusted, the learning process is repeated for several tens to several hundred epochs (see FIGS. 12 to 14).
As a result of such learning processing, it is possible to accurately judge whether the set of keywords in each column and the document are highly related when the keywords in each column are in the positional relationship and distribution. classifier (trained machine learning model) can be generated.

As described above, according to the present invention, regardless of the document category and search conditions, if the set of keywords and the document have high or low relevance, mutual The feature vector (spatial distribution vector) generated by using the property that the positional relationship and distribution features show a certain tendency. Regardless, it is possible to accurately determine whether the set of keywords and the document are highly related.

7. Overall Processing Flow FIG. 9 is an overall processing flow of the present invention.
First, the input means of the information processing system 1000 acquires the contents of each set of keywords in the first to n-th keyword groups (step S1).
Further, the input means reads the text data of one document hit by the retrieval condition of the predetermined retrieval keyword (step S2). The text data of the document may be the result of scanning the document with a scanner or character recognition of image data taken with a photograph.
The analysis means 300 morphologically analyzes the document text data (step S3), and analyzes the location and distribution of each keyword in the acquired first to n-th keywords (step S4).

Based on the analysis result, the feature vector generating means 400 generates feature vectors (KW space distribution vectors) for each of the first to n-th keyword groups (step S5).
The generated feature vector (KW space distribution vector) is input to the classification means (machine learning model) 500 (step S6), and the classification means (machine learning model) 500 determines the relationship between the keyword set and the document, that is, the research object An evaluation result regarding the relevance to technology is output (step S7).
The above is repeated for all documents (NO in step S8), and when all documents are completed (YES in step S8), the evaluation results are sorted in order of score and output (steps S9 and S10).
Although the learning process is not shown, the difference between the score value of the evaluation result output by the classification means (machine learning model) 500 in forward propagation and the label (teacher data) is propagated in the backward direction, and for example, a neural network In the case of , it is performed by updating the parameters such as the weights and biases of the neural network by the error backpropagation method, the gradient effect method, or the like.

FIG. 10 is a flow chart showing the details of convolution processing of the first to second layers in the classifier (machine learning model) 500. FIG.
First, a channel (CH) to be processed and a convolution filter for feature extraction for the CH are set (step P1).
The KW space distribution vector of the CH, which is the input data, is subjected to convolution processing while shifting the convolution filter (step P2) (see the balloon processing 1 in FIG. 8).
Here, the interval (stride) for shifting the convolution filter is normally set to 1, but it can be changed somewhat depending on the amount of learning data. I don't mind.
The convolution result is saved (step P3), and the same convolution processing is performed for all CHs (step P4).

Next, the convolution processing results of CH1 to CHn are combined to create one convolution result (step P5).
The combined value of this convolution result is a feature map of the positional relationship and distribution of the keywords in the 1st to n-th stages.
Then, the type of convolution filter is updated, and similar convolution processing and synthesis processing are performed (steps P6 and P7).
For example, when 20 kinds of convolution filters are prepared, a synthesized result of 20 convolution results is obtained (step P8).
After going through the learning process, the result of synthesizing these 20 convolution results is to prepare 20 types of feature maps of the positional relationships and distributions of the keywords in the 1st to nth stages. It is the basis for multifaceted evaluation of positional relationships and distribution characteristics.

In the above description, the convolution result is created after applying the convolution filter to each CH, and then the convolution results of each CH are synthesized. Each time the filter is applied, the convolution results of CH1 to CHn may be synthesized, and one synthesis result may be obtained while shifting the convolution filter.
Similarly, for the second and subsequent layers, set convolution filters (step P9) → convolution processing (steps P10 to P12) → synthesis of convolution results (step P13) → update convolution filter types (steps P14 to 15). , the same processing is repeated, and when, for example, 20 types of convolution filters are prepared, a synthesized value of 20 convolution results is obtained (step P16).
The same process is repeated for the third and subsequent layers, and for the feature map generated in the first layer, which shows the relatively localized positional relationships and distribution features of keywords, as the layers progress, the features in the previous stage are combined in a complex manner. It also generates higher-order feature maps.

8. Learning Data FIG. 11 is a diagram showing an aspect of learning data.
Learning data includes learning data for training and learning data for testing for testing performance after learning, but both have the same data structure. For example, if there are 100,000 learning data, 70,000 may be allocated for training and 30,000 for testing.
The structure of the learning data consists of the first stage keyword (1st KW) to the nth stage keyword (nth KW), the document number and storage location of the document data, the document determined by the searcher, and the keyword of each stage keyword group. It has a structure for recording in association with a label (teaching data) that is a correct value indicating the degree of relevance of the data.
A label (teaching data) is an indicator that a searcher quantifies the degree of relevance between a set of keywords and a document based on a certain standard. For example, the index may range from 0 to 100 points, such as 100 when the keyword set and the document are highly related, and 0 when there is almost no relationship.

When there are multiple searchers, it is assumed that there will be individual differences and it may not be possible to unify the label values, but we will try to statistically adjust the values based on the values investigated with the same set of documents. Alternatively, by setting a high learning rate (or a large number of epochs) for the learning data of the survey results of veteran searchers and a low learning rate (or a small number of epochs) for other searchers, the veteran searchers It may be compensated by learning with emphasis on the results of the survey.
A plurality of DATA1, DATA2, . Each DATA is usually a collection of several hundred to several thousand documents, and about 100 DATA is equivalent to 50,000 to 100,000 documents.
Note that the learning data may have various configurations such as a file structure in addition to the database structure.
Also, in the case of document data for a new survey, it has the same data structure, and since the content of the document has not yet been confirmed, the same is true except that the label (teaching data) is NULL. is the configuration.

9. Set values for learning processing and set values for the configuration of the classification means. An example of setting values for the learning process and setting values for the configuration of the classification means 500 for testing how the error between the sex index and the label (teacher data), which is the correct value given in advance, changes. be.
As an example of the machine learning model, it is assumed that a deep neural network with 10 layers is selected.
As shown in FIG. 12, when only the spatial distribution vectors of the keywords in the 1st to n-th stages (approximately 1st to 3rd to 5th stages) are used as feature vectors, the number of document data for training is 9. ,000 cases, it is assumed that even if the number of epochs is increased, a considerable amount of error will remain.
In addition, it is assumed that the numerical values of errors during training and during testing are different, the errors during testing do not converge, and the reliability of the numerical values of the judgment results output by the classifier cannot be expected.

On the other hand, as shown in FIG. 13, even when only the KW spatial distribution vectors of the 1st to nth stages (about the 1st to 3rd to 5th stages) are used as feature vectors, the document data for training When the number of cases is about 90,000, the error is reduced to 10% or less if a sufficient number of epochs is taken, and the error values are quite close between training and testing, and the judgment results output by the classifier are similar. Reliability is also expected to be considerably higher.
Furthermore, as shown in FIG. 14, as a feature vector, in addition to the KW spatial distribution vectors (about the first to fifth stages) of the first to nth stages (about the first to third to fifth stages), Classification means for additionally inputting the spatial distribution vectors of the top 1st to nth (about 5th) technical terms in the literature, or the spatial distribution vectors of dependent words for the 1st to nth keywords , it is necessary to increase the number of types of convolution filters and the number of parameters to be learned. It is expected that the results will be more reliable.

10. Example of Performance Evaluation FIG. 15 shows an example of performance evaluation of processing for determining the relevance between a set of search keywords and documents in the present invention. FIG. 10 is a diagram showing an example of a range of scores in which the top 3% of highly relevant documents are included and an example of a man-hour reduction rate, depending on the number of vector types;
As shown in FIG. 15, according to the number of types of convolution filters (corresponding to the number of parameters) and the number of document data for training, or according to the number of types of feature vectors, the accuracy of the score value output by the classification means 500 is improved. can be expected.

For example, as shown in the first line of FIG. 15, when the number of document data is 9,000 (see FIG. 12), the searcher confirms and determines that the set of keywords and the document are actually highly related. The range that can be said to include the top 3% of documents that can be classified is the range of the top 80% of the score values output by the classification means 500 .
In this case, it can be seen that the reliability is such that the top 3% of documents cannot be picked up unless 80% of all the documents hit by the keyword are confirmed. In other words, when 1000 documents are hits, 200 documents need not be checked, so the man-hour reduction effect is only 20%.
On the other hand, as described in the third row of FIG. 15, when the number of document data for training is 90,000 (see FIG. 13), even if the feature vector is only the KW spatial distribution vector, the relevance is The top 30% of output score values are sufficient for the range in which the top 3% of documents that can be judged to be high are included, and the performance is considerably improved. In this case, the man-hour reduction rate is 70%.

For example, the confirmation result of "the range in which the top 3% of documents are included" by the searcher is recorded in the database in association with the information of the technical field (IPC classification etc. in the case of patent documents), and the average value It can be used as a numerical value indicating the reliability of the trained machine learning model after being converted into a numerical value through a statistical method such as , dispersion, etc. (see FIG. 16 described later).

11. Example of output result FIG. 16 is a diagram showing an example of an output result such as a display screen or a printed output of the evaluation result of the association between the set of keywords output by the classification means (machine learning model) 500 and the document. .
The evaluation results are sorted in descending order of the output score value, and in addition to the hit location and number of keywords in the 1st to nth keyword group, the document number and link to the storage location of the document, and other bibliographic items. etc. is output.
The rightmost item distinguishes the range that is likely to include the top 3% or 5% of the most relevant documents. The distinguishing display is indicated by shading in the figure, but the range may be indicated by colors, character strings, arrows, or the like.
In general, machine learning models, especially deep learning models, are said to have unknown reliability of output results because the decision process is a black box.
Therefore, in the information processing system of the present invention, the reliability of the output result of such a model is displayed.

In this way, in the case of the conventional technology, vectorization is performed based on word frequency information, or feature vectorization is performed on the entire document. While it is not possible to evaluate physical positional relationships, distribution features, and high-order features that combine them in a composite manner, the present invention enables them.
Then, even when various keywords are set, regardless of the type of document or category of keywords, if the set of keywords and the document have high or low relevance, the mutual relationship between the keywords in the first to nth stages Since the positional relationships and distribution features of keywords show a certain tendency, a machine learning model that has undergone a certain amount of learning using training data can accurately evaluate the relationship between a set of keywords and documents based on them. A processing system can be provided.
For this reason, when selecting keywords and researching literature, even if it is the first time to select predetermined keywords and research, keywords can be set without pre-searching and struggling to find relevant literature. It is possible to greatly reduce the man-hours of literature research from the initial stage of starting the research.

The present invention can be applied to documents in general written in any language, and can be used for search processing of various domestic and foreign documents. For example, in addition to Japanese domestic patent documents, papers, and judicial precedents, various documents from other countries such as the United States and China can also be used.

100 Document data such as patent documents, papers, court cases 200 Search keyword 300 Analysis means 400 Feature vector generation means 500 Classification means 600 Learning processing means 700 Evaluation processing means 800 Evaluation dictionary 1000 Information processing device 2000 Communication network 3000 User terminal 3100 User terminal

Claims (18)

A system for determining the relevance between a set of search keywords and a document,
a means for obtaining text data of a document;
means for obtaining a set of keywords from the first stage to the n-th stage as a set of search keywords;
KW existence location analysis means for analyzing the location of keywords included in the text data of the document;
a KW space distribution vector generation means for generating a KW space distribution vector indicating how the locations of the keywords analyzed by the KW existence location analysis means are distributed in text data of the document;
The KW space distribution vector generating means generates, for each document, based on the location of each keyword in the group of keywords in the first to nth stages included in the text data of the document, the first to nth stages and generate KW spatial distribution vectors of
Based on the KW space distribution vectors of the 1st to nth stages generated for each document, the positional relationship between the keywords of the 1st to nth stage keywords and the characteristics of the distribution are evaluated, and the search keywords provided with classification means for determining the relevance between the set of
An information processing system characterized by The information processing system according to claim 1, further comprising:
KW dependent word existence location analysis means for analyzing the existence location of the dependent word of the keyword existing in the vicinity of the location of the keyword included in the text data of the document;
KW for generating a KW related word space distribution vector indicating how the locations of dependent words near the keyword analyzed by the KW related word presence location analyzing means are distributed in the text data of the document; Dependent word space distribution vector generation means,
The KW dependent word space distribution vector generating means generates for each document based on the location of the dependent word corresponding to each keyword of the first to n-th keyword groups included in the text data of the document. , respectively generate the KW-related word space distribution vectors of the 1st to nth stages corresponding to the keyword groups of the 1st to nth stages,
Based on the first to n-th KW space distribution vectors generated for each document and the first to n-th KW related word space distribution vectors,
Characteristics of positional relationships and distributions among keywords in the first to n-th keyword groups, and mutual relationships between each keyword in the first to n-th keyword groups and their corresponding dependent words Equipped with classification means for evaluating positional relationships and distribution characteristics to determine relevance between a set of search keywords and documents;
An information processing system characterized by The information processing system according to claim 1, further comprising:
technical terminology analysis means for analyzing technical terms included in text data of documents in order of frequency;
a technical term existence location analysis means for analyzing the location of the technical terms from the top 1 to the top m of the technical terms analyzed by the technical term analysis means;
a technical term space distribution vector generation means for generating a technical term space distribution vector indicating how the locations of the technical terms analyzed by the technical term presence location analysis means are distributed in the text data of the document; prepared,
The technical term space distribution vector generating means, based on the locations of the top 1 to top m technical terms included in the text data of the document, the top 1 to top m th technology for each document Generate a term space distribution vector, respectively,
Based on the first to n-th KW spatial distribution vectors generated for each document and the top 1 to top m-th technical term space distribution vectors,
Features of mutual positional relationships and distributions of keywords in the first to n-th keyword groups, and keywords in the first to n-th keyword groups and the top first to top m-th technical terms A classifying means for evaluating the mutual positional relationship and distribution characteristics of and determining the relevance between the set of search keywords and the document;
An information processing system comprising: In the information processing system according to claim 1,
The classification means for determining the relevance between a set of search keywords and documents based on the first to n-th KW space distribution vectors generated for each document is composed of a neural network of at least three layers. if
In the first layer of the neural network,
convolution means for performing convolution processing on the KW space distribution vectors of the first stage to the nth stage using the corresponding convolution filters;
synthesizing means for synthesizing the results of the convolution processing to generate a feature map that extracts the mutual positional relationship and distribution features of the keywords in the first to n-th keyword groups,
In the second and subsequent layers of the neural network,
Convolution means for performing convolution processing using a plurality of convolution filters on a plurality of feature maps generated in the previous layer;
synthesizing means for synthesizing the results of the convolution processing to generate a feature map that extracts high-order features of mutual positional relationships and distributions of keywords in the first to n-th keyword groups;
An information processing system comprising: In the information processing system according to claim 2,
The 1st to n-th KW space distribution vectors generated for each document, and the 1st to n-th KW related word space distribution vectors corresponding to the 1st to n-th keywords , Based on, when the classification means for determining the relevance between the set of search keywords and the document is composed of a neural network of at least three layers,
In the first layer of the neural network,
Corresponding to the KW space distribution vectors of the first to nth stages and the KW-related word space distribution vectors of the first to nth stages corresponding to the keyword groups of the first to nth stages, respectively Convolution means for performing convolution processing using a convolution filter that
synthesizing means for synthesizing the results of the convolution process to generate a feature map that extracts the mutual positional relationships and distribution features of the keywords of the first to nth stage keywords and their dependent words; with
In the second and subsequent layers of the neural network,
Convolution means for performing convolution processing using a plurality of convolution filters on a plurality of feature maps generated in the previous layer;
Combining the results of the convolution process to generate a feature map that extracts the mutual positional relationships and high-order features of the distribution of the keywords in the group of keywords from the first stage to the n-th stage and their dependent words. means and
An information processing system comprising: In the information processing system according to claim 3,
Based on the 1st to n-th KW space distribution vectors generated for each document and the 1st to top m-th technical term space distribution vectors, the set of search keywords and the relevance of the document When the classification means for determining is composed of a neural network of at least three layers,
In the first layer of the neural network,
Convolution means for performing convolution processing on the first to n-th KW space distribution vectors and the first to top m-th technical term space distribution vectors using convolution filters corresponding to them, respectively ,
By synthesizing the results of the convolution process, generate a feature map that extracts the mutual positional relationships and distribution features of the keywords in the 1st to n-th keyword groups and the top 1 to top m-th technical terms. and a synthesizing means for
In the second and subsequent layers of the neural network,
Convolution means for performing convolution processing using a plurality of convolution filters on a plurality of feature maps generated in the previous layer;
By synthesizing the results of the convolution process, the mutual positional relationship and high-order features of the distribution of the keywords in the 1st to nth stage keywords and the top 1st to top mth technical terms are extracted. a compositing means for generating a map;
An information processing system comprising: In the information processing system according to any one of claims 1 and 4,
means for acquiring a learning data set in which a set of search keywords, text data of a document, and a label value indicating the relationship between the set of search keywords and the document are associated;
Learning processing means for updating the parameters of the classification means based on the error between the score value of the relevance between the search keyword set and the document output by the classification means and the obtained label value,
inputting the KW spatial distribution vectors of the first stage to the nth stage generated for each document based on the learning data set into the classification means;
The learning processing means learns by updating the parameters of the classification means based on the error between the score value of the relevance between the search keyword set and the document output by the classification means and the label value. matter,
An information processing system characterized by In the information processing system according to any one of claims 2 and 5,
means for acquiring a learning data set in which a set of search keywords, text data of a document, and a label value indicating the relationship between the set of search keywords and the document are associated;
The 1st to n-th KW space distribution vectors generated for each document, and the 1st to n-th KW related word space distribution vectors corresponding to the 1st to n-th keywords A classification means for determining the relevance between a set of search keywords and documents based on
Learning processing means for learning parameters of the classification means based on the error between the score value of the relevance between the set of search keywords and the document output by the classification means and the label value,
The KW spatial distribution vectors of the 1st to nth stages generated for each document based on the learning data set, and the 1st to nth stages corresponding to the keyword groups of the 1st to nth stages inputting the KW related word space distribution vector into the classification means;
Based on the error between the score value of the relevance between the search keyword set and the document output by the classification means and the label value, the learning processing means updates the parameters of the classification means to perform learning processing. to do
An information processing system characterized by In the information processing system according to claim 3 or claim 6,
means for acquiring a learning data set in which a set of search keywords, text data of a document, and a label value indicating the relationship between the set of search keywords and the document are associated;
Learning processing means for learning parameters of the classification means based on the error between the score value of the relevance between the set of search keywords and the document output by the classification means and the label value,
Inputting the first to n-th KW space distribution vectors generated for each document based on the learning data set and the top first to top m-th technical term space distribution vectors to the classification means ,
Based on the error between the score value of the relevance between the search keyword set and the document output by the classification means and the label value, the learning processing means updates the parameters of the classification means to perform learning processing. to do
An information processing system characterized by 8. The information processing system according to claim 7, wherein a learned model for causing a computer to determine the relevance between a set of search keywords and documents learned by said learning processing means,
Learning processing means for updating the parameters of the classification means based on the error between the score value of the relevance between the search keyword set and the document output by the classification means and the obtained label value,
inputting the KW spatial distribution vectors of the first stage to the nth stage generated for each document based on the learning data set into the classification means;
Based on the error between the score value of the relevance between the search keyword set and the document output by the classification means and the label value, the learning processing means obtains by learning processing for updating the parameters of the classification means. trained model. 9. The information processing system according to claim 8, wherein a learned model for causing a computer to determine the relevance between a set of search keywords and documents learned by the learning processing means,
Learning processing means for learning parameters of the classification means based on the error between the score value of the relevance between the set of search keywords and the document output by the classification means and the label value,
KW space distribution vectors of the 1st to nth stages generated for each document based on the learning data set, and KWs of the 1st to nth stages corresponding to the keyword groups of the 1st to nth stages inputting the dependent word space distribution vector into the classification means;
Based on the error between the score value of the relevance between the search keyword set and the document output by the classification means and the label value, the learning processing means obtains by learning processing for updating the parameters of the classification means. trained model. 10. The information processing system according to claim 9, wherein a learned model for causing a computer to determine the relevance between a set of search keywords and documents learned by said learning processing means,
Learning processing means for learning parameters of the classification means based on the error between the score value of the relevance between the set of search keywords and the document output by the classification means and the label value,
The first to n-th KW space distribution vectors generated for each document based on the learning data set and the top 1 to top m-th technical term space distribution vectors are input to the classification means. ,
Based on the error between the score value of the relevance between the search keyword set and the document output by the classification means and the label value, the learning processing means obtains by learning processing for updating the parameters of the classification means. trained model. Using the learned model obtained by the learning process in the information processing system according to any one of claims 7 to 9, determining the relevance between the set of search keywords and the document,
An information processing system characterized by An information processing system that determines the relevance between a set of search keywords and a document using a learned model obtained by learning processing in the information processing system according to any one of claims 7 to 9,
means for outputting determination results in the order in which the set of search keywords and the document are determined to be highly relevant;
a means for distinguishingly displaying a range that is highly likely to include the top p% of documents that the searcher actually judges to be highly relevant;
An information processing system comprising: A method for determining the relevance between a set of search keywords and a document,
obtaining text data of the literature;
a step of obtaining a set of keywords from the 1st stage to the nth stage as a set of search keywords;
a KW presence location analysis step of analyzing the location of keywords included in the text data of the document;
a KW space distribution vector generation step for generating a KW space distribution vector indicating how the presence locations of the keywords analyzed by the KW presence location analysis step are distributed in the text data of the document;
By the KW space distribution vector generation step , the KW space distribution of the 1st to nth stages based on the location of each keyword of the 1st to nth stage keywords included in the text data of the document generate vectors respectively,
Based on the KW space distribution vectors of the 1st to nth stages generated for each document, the positional relationship between the keywords of the 1st to nth stage keywords and the characteristics of the distribution are evaluated, and the search keywords determining the relevance of the set of
An information processing method characterized by: The information processing method according to claim 15 , further comprising:
a KW dependent word existence location analysis step of analyzing the existence location of the dependent word of the keyword existing in the vicinity of the location of the keyword included in the text data of the document;
KW for generating a KW-dependent word space distribution vector indicating how the locations of dependent words near the keyword analyzed by the KW-dependent word presence location analyzing step are distributed in the text data of the document; a dependent word space distribution vector generation step;
According to the KW dependent word space distribution vector generation step, the first Generating KW related word space distribution vectors of the 1st to n-th stages corresponding to the keyword groups of the 1st to n-th stages,
The 1st to n-th KW space distribution vectors generated for each document, and the 1st to n-th KW related word space distribution vectors corresponding to the 1st to n-th keywords ,On the basis of the,
features of positional relationships or distributions among keywords in the first to n-th keyword group, and mutual relationships between each keyword in the first to n-th keyword group and their corresponding dependent words; Evaluating the positional relationship and distribution characteristics to determine the relevance between the set of search keywords and the document,
An information processing method characterized by: The information processing method according to claim 15 , further comprising:
A technical term analysis step of analyzing technical terms included in text data of documents in order of frequency;
A technical term existence location analysis step of analyzing the locations of the technical terms from the top 1 to the top m of the technical terms analyzed by the technical term analysis step;
a technical term space distribution vector generation step of generating a technical term space distribution vector indicating how the locations of the technical terms analyzed by the technical term presence location analysis step are distributed in the text data of the document; prepared,
By the technical term space distribution vector generation step, the top 1 to top m-th technical term space distribution vectors based on the locations of the top 1 to top m-th technical terms contained in the text data of the document respectively, and
Based on the 1st to n-th KW spatial distribution vectors generated for each document and the 1st to top m-th technical term spatial distribution vectors,
Features of mutual positional relationships and distributions of keywords in the first to n-th keyword groups, and keywords of the first to n-th keyword groups and the top first to top m-th technical terms Evaluating the mutual positional relationship and distribution characteristics with, to determine the relevance between the set of search keywords and the document,
An information processing method characterized by: A program for causing a computer to execute the information processing method according to any one of claims 15 to 17 .

Images