JP7614738B2 - SEARCH DEVICE AND SEARCH METHOD, LEARNING DEVICE AND LEARNING METHOD, QUESTION ANSWER PREDICTION SYSTEM AND QUESTION ANSWER PREDICTION METHOD, AND PROGRAM - Google Patents

Description

The present invention relates to a search device and a search method, a learning device and a learning method, a question answer prediction system and a question answer prediction method, and a program.

Full-text search technology and machine reading comprehension technology are well known as technologies for obtaining answers to questions. Full-text search technology searches for documents that match a question based on the frequency of occurrence of words in the question within the document. Machine reading comprehension technology learns from a set of questions, answers, and documents as training data, and when a question is input, outputs the section of the document that contains the answer.

A document search device using full-text search technology is described in, for example, Patent Document 1. This document search device performs a full-text search on multiple documents stored in a document database based on search conditions input by a user, and extracts multiple documents that match the search conditions.

An answer learning device using machine reading technology is described in, for example, Patent Document 2. This answer learning device includes a machine reading unit that uses a pre-trained reading model to estimate the range in the text that is the basis for the answer to the question, based on the input text and question, and estimates the start and end of the range.

JP 2019-89634 A JP 2020-061173 A

Full-text search technology is a technology that extracts desired documents from a large number of documents, but even if the answer to a question is contained in the documents extracted using full-text search technology, the reason for extracting the document is simply that the words in the question were present in the document, and no other reason can be presented.

Machine reading comprehension technology learns from questions, documents, and answers as training data, so if the question and document are present in the training data, it is possible to extract the section of the document that corresponds to the answer to the question. However, if there is no document that contains the answer to the user's question, the answer cannot be obtained.

The present invention has been made in consideration of the above problems, and aims to provide a search device and search method, a learning device and learning method, a question answer prediction system and question answer prediction method, and a program that can output answers to questions with high accuracy.

(1) One aspect of the present invention is a system for generating feature words corresponding to training questions by using a set of training questions and feature words that characterize training documents containing training answers to the training questions as training data, the system including a feature word generation unit that generates feature words corresponding to the training questions using the training model; a search unit that executes a process of searching a literature database for documents containing words that match a word included in the input question based on a word included in the input question, and a process of searching the literature database for documents containing words that match the feature words based on the feature words generated by the feature word generation unit; and a search unit that executes a search function that generates answer candidates corresponding to the training questions and the training documents and likelihoods of the answer candidates by using a set of training questions, training documents containing training answers to the training questions, and the training answers as training data. a machine reading comprehension unit that has a machine reading comprehension model trained to output a core, and uses the machine reading comprehension model to output answer candidates corresponding to an input question and documents searched based on words contained in the question and scores indicating the likelihood of the answer candidates, and answer candidates corresponding to the question and documents searched based on feature words corresponding to the question and a score indicating the likelihood of the answer candidates; and a prediction unit that obtains from the machine reading comprehension unit answer candidates corresponding to the input question and documents searched based on words contained in the question and a score indicating the likelihood of the answer candidates, and answer candidates corresponding to the question and documents searched based on feature words corresponding to the question, and a score indicating the likelihood of the answer candidates, and outputs a result of predicting an answer to the question based on a comparison of the scores indicating the likelihood of each answer candidate.

( 2 ) One aspect of the present invention is the above-mentioned search device, wherein when an answer candidate output by the machine reading comprehension model corresponding to the input question and literature searched based on words contained in the question is the same as an answer candidate corresponding to the question and literature searched based on feature words corresponding to the question, and when a score indicating the likelihood of each of the same answer candidates exceeds a threshold, the prediction unit may add up the scores of the same answer candidates.

(3) One aspect of the present invention is a learning device that includes a first learning unit that generates a generative model trained to generate feature words corresponding to a training question using a set of training questions and feature words that are words that characterize training literature including training answers to the training questions as training data, and a second learning unit that generates a machine reading comprehension model trained to output answer candidates corresponding to the training questions and the training literature and scores indicating the likelihood of the answer candidates using the set of training questions, training literature including training answers to the training questions, and the training answers as training data.

(4) One aspect of the present invention provides a learning device including: a first learning unit that uses a set of training questions and feature words that characterize training documents including training answers to the training questions as training data to generate a generative model trained to generate feature words corresponding to the training questions; and a second learning unit that uses a set of training questions, training documents including training answers to the training questions, and the training answers as training data to generate a machine reading comprehension model trained to output answer candidates corresponding to the training questions and the training documents and scores indicating the likelihood of the answer candidates; a feature word generation unit that has the generative model and uses the generative model to generate feature words corresponding to an input question; a process of searching a literature database for documents including words that match words included in the input question based on words included in the input question; and a process of searching a literature database for documents including words that match words included in the input question based on the feature words generated by the feature word generation unit. from the literature database; a machine reading comprehension unit that has the machine reading comprehension model and uses the machine reading comprehension model to output answer candidates corresponding to the input question and documents searched based on words contained in the question and scores indicating the likelihood of the answer candidates, and answer candidates corresponding to the question and documents searched based on feature words corresponding to the question and a score indicating the likelihood of the answer candidates; and a prediction unit that obtains from the machine reading comprehension unit the answer candidates corresponding to the input question and documents searched based on words contained in the question and the answer candidates corresponding to the question and documents searched based on feature words corresponding to the question, and outputs a result of predicting an answer to the question based on a comparison of the scores indicating the likelihood of each answer candidate.

(5) One aspect of the present invention includes a step of a learning device generating a generative model trained to generate feature words corresponding to a learning question using a set of a learning question and feature words that characterize a learning document including a learning answer to the learning question as learning data; a step of the learning device generating a machine reading comprehension model trained to output answer candidates corresponding to the learning question and the learning document and scores indicating the likelihood of the answer candidates using the learning question, the learning document including a learning answer to the learning question, and the set of the learning answers as learning data; a step of a search device generating feature words corresponding to an input question using the generative model; a process of the search device searching a literature database for literature including words that match a word included in the input question based on the word; and a process of searching a literature database for literature including words that match the feature words based on the feature words. a step of the search device using the machine reading comprehension model to output answer candidates corresponding to the input question and documents searched based on words contained in the question and scores indicating the likelihood of the answer candidates, and answer candidates corresponding to the question and documents searched based on feature words corresponding to the question and scores indicating the likelihood of the answer candidates; and a step of the search device acquiring answer candidates corresponding to the input question and documents searched based on words contained in the question and scores indicating the likelihood of the answer candidates, and answer candidates corresponding to the question and documents searched based on feature words corresponding to the question, and outputting a result of predicting an answer to the question based on a comparison of the scores indicating the likelihood of each answer candidate.

(6) One aspect of the present invention is a search device having a generative model trained to generate feature words corresponding to a training question using a set of training questions and feature words that characterize training documents including training answers to the training questions as training data, and generating feature words corresponding to the training questions using the generative model; the search device executing a process of searching a literature database for documents including words that match a word included in the input question based on a word included in the input question, and a process of searching the literature database for documents including words that match the feature words based on the generated feature words; and the search device using the set of training questions, training documents including training answers to the training questions, and the training answers as training data to search for answer candidates corresponding to the training questions and the training documents and the likelihood of the answer candidates. a step of using the machine reading comprehension model to output answer candidates corresponding to the input question and documents searched based on words contained in the question and scores indicating the likelihood of the answer candidates, and answer candidates corresponding to the question and documents searched based on feature words corresponding to the question and a score indicating the likelihood of the answer candidates; and a step of the search device acquiring answer candidates corresponding to the input question and documents searched based on words contained in the question and scores indicating the likelihood of the answer candidates, and answer candidates corresponding to the question and documents searched based on feature words corresponding to the question, and outputting a result of predicting an answer to the question based on a comparison of the scores indicating the likelihood of each answer candidate.

(7) One aspect of the present invention is a search device having a computer having a generative model trained to generate feature words corresponding to training questions using a set of training questions and feature words that characterize training documents containing training answers to the training questions as training data, and performing the steps of: generating feature words corresponding to an input question using the generative model; searching a literature database for documents containing words that match words included in the input question based on the words included in the input question; and searching the literature database for documents containing words that match the generated feature words; and using the set of training questions, training documents containing training answers to the training questions, and the training answers as training data, searching for answer candidates corresponding to the training questions and the training documents and the likelihoods of the answer candidates. The program has a machine reading comprehension model trained to output a score indicating the likelihood of each answer candidate, and uses the machine reading comprehension model to output answer candidates corresponding to the input question and documents searched based on words contained in the question, and answer candidates corresponding to the question and documents searched based on feature words corresponding to the question, and scores indicating the likelihood of the answer candidates; obtain answer candidates corresponding to the input question and documents searched based on words contained in the question, and answer candidates corresponding to the question and documents searched based on feature words corresponding to the question, and output a result of predicting the answer to the question based on a comparison of the scores indicating the likelihood of each answer candidate.

(8) One aspect of the present invention is a learning method comprising the steps of: a learning device generating a generative model trained to generate feature words corresponding to a training question using a set of training questions and feature words that are words that characterize training literature including training answers to the training questions as training data; and a learning device generating a machine reading comprehension model trained to output answer candidates corresponding to the training questions and the training literature and scores indicating the likelihood of the answer candidates using the set of training questions, training literature including training answers to the training questions, and the training answers as training data.

(9) One aspect of the present invention is a program that causes a computer of a learning device to execute the steps of: generating a generative model trained to generate feature words corresponding to a learning question, using as training data a set of learning questions and feature words that are words that characterize training literature including training answers to the training questions; and generating a machine reading comprehension model trained by the learning device to use as training data a set of training questions, training literature including training answers to the training questions , and the training answers, and to output answer candidates corresponding to the training questions and the training literature, and scores indicating the likelihood of the answer candidates.

According to one aspect of the present invention, answers to questions can be output with high accuracy.

1 is a block diagram showing a configuration example of a question answer prediction system according to an embodiment. FIG. 2 is a block diagram for explaining a learning process of a feature word generation model according to an embodiment. FIG. 2 is a block diagram showing an example of the configuration of a feature word generation model according to the embodiment. FIG. 2 is a block diagram for explaining a learning process of a machine reading comprehension model according to an embodiment. FIG. 2 is a block diagram showing an example of a configuration of a machine reading comprehension model according to an embodiment. 1 is a flowchart illustrating an example of a processing procedure in a question answer prediction system according to an embodiment. 1 is a block diagram showing an example of a processing procedure in a question answer prediction system according to an embodiment;

Below, a search device and search method, a learning device and learning method, a question answer prediction system and question answer prediction method, and a program to which the present invention is applied will be described with reference to the drawings.

<Configuration of question answer prediction system>
1 is a block diagram showing a configuration example of a question answer prediction system according to an embodiment. The question answer prediction system 1 according to the embodiment is a Web system or an application system for providing a service that, when an unknown question sentence is received, searches documents using words contained in the question, predicts an answer using the searched documents, and provides the predicted answer.

The question answer prediction system 1 includes, for example, one or more terminal devices 100, a service providing server 200, job queues 230 and 240, a full-text search server 300, a literature database 310, a feature word generation unit 400, a feature word generation model 410, a machine reading unit 500, and a machine reading model 510. The terminal device 100, the service providing server 200, the job queues 230 and 240, the full-text search server 300, the literature database 310, the feature word generation unit 400, the feature word generation model 410, the machine reading unit 500, and the machine reading model 510 in the question answer prediction system 1 are connected to, for example, a communication network and include a communication interface such as a NIC (Network Interface Card) or a wireless communication module (not shown in FIG. 1). The communication network may include, for example, the Internet, a Wide Area Network (WAN), a Local Area Network (LAN), a cellular network, Wi-Fi (registered trademark), etc.

The terminal device 100 is, for example, a computer such as a personal computer or a smartphone equipped with a display unit, an operation unit, and a CPU. The terminal device 100 is equipped with a browser 110 for displaying content, accepting operation contents, and transmitting and receiving information. The terminal device 100 may launch a UA (User Agent) such as an application program, not limited to the browser 110. The UA is, for example, an application for receiving a service provided by the service providing server 200. The terminal device 100 uses the application as the UA to perform display processing and operation acceptance processing using content received from the service providing server 200. The terminal device 100 of the embodiment provides a request including a question based on a user's operation to the service providing server 200, and accepts a response including an answer to the question from the service providing server 200.

The service providing server 200 is a server device that provides a service of providing answers to questions in response to requests from the terminal device 100. The service providing server 200 includes, for example, a job management unit 210 and a result providing unit 220. Functional units such as the job management unit 210 and the result providing unit 220 are realized by a processor, such as a CPU (Central Processing Unit), executing a program stored in a program memory. In addition, some or all of these functional units may be realized by hardware such as a processor capable of high-speed machine learning processing (matrix calculation processing), such as a GPU (Graphics Processing Unit) or a TPU (Tensor Processing Unit), a combination of processors such as a GPU or a TPU, an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable Gate Array), or may be realized by software and hardware working together.

The job management unit 210 manages jobs for the feature word generation process and the machine comprehension process in response to requests from the terminal device 100. When the job management unit 210 receives a question from the terminal device 100, it registers a job for the feature word generation process in the job queue 230. The job management unit 210 registers a document to be subjected to the machine comprehension process in the job queue 240. The result providing unit 220 provides the terminal device 100 with a response including an answer as a processing result. The job queue 230 may be configured in a storage unit in the feature word generation unit 400, or may be configured in a storage unit in the service providing server 200. The job queue 240 may be configured in a storage unit in the machine comprehension unit 500, or may be configured in a storage unit in the service providing server 200.

The feature word generation unit 400 performs a feature word generation process in which it generates multiple question words by dividing the question sentence received from the service providing server 200 into words, and generates feature words using each question word. The feature word generation unit 400 is, for example, a processor such as a CPU, and realizes the feature word generation process by executing the feature word generation model 410. The feature word generation model 410 is expressed by various parameters for realizing the feature word generation process. The processor may include a combination of processors capable of performing high-speed matrix calculation processing, such as a GPU or a TPU.

The full-text search server 300 is a server device that executes full-text search processing in response to a request from the service providing server 200. The full-text search server 300 divides the question received from the service providing server 200 into words (hereinafter also referred to as question terms), and performs full-text search processing using the divided word groups. The full-text search processing is a process of searching whether documents registered in the literature database 310 contain words that match the question terms, and extracting documents that contain words that match the question terms. The literature database 310 is a database in which a large amount of literature data is stored. The literature database 310 is realized, for example, by a processing unit that performs database management, a hard disk drive (HDD), a solid state drive (SSD), a flash memory, an electrically erasable programmable read only memory (EEPROM), a read only memory (ROM), or a random access memory (RAM).

The machine reading unit 500 performs machine comprehension processing using multiple question words and documents that make up the question received from the service providing server 200. The machine reading unit 500 is, for example, a processor such as a CPU, and realizes the machine reading comprehension processing by executing the machine reading comprehension model 510. The machine reading comprehension model 510 is expressed by various parameters for realizing the machine reading comprehension processing. The machine reading unit 500 supplies answer candidates and scores for the question to the service providing server 200 as the processing results of the machine reading comprehension processing. The answer candidates and scores are subjected to a predetermined processing by the result providing unit 220 and provided to the terminal device 100.

Figure 2 is a block diagram for explaining the learning process of the feature word generation model of the embodiment. The feature word generation model 410 is trained using a question sentence and feature words extracted from documents by the learning processing unit 600 described below, for example, from machine reading comprehension learning data used to train the machine reading comprehension model 510. The learning process of the feature word generation model 410 first inputs a question sentence into the feature word generation model 410, and learns processing parameters in the feature word generation model 410 so that the feature word generation model 410 outputs a specified feature word corresponding to the question sentence.

The learning processing unit 600 performs processing in the following steps, for example. (1) The learning processing unit 600 creates an index of words and the number of occurrences for, for example, the question sentence and all documents to be searched. (2) The learning processing unit 600 uses the created index to extract words that characterize documents, as feature words, for documents including the question sentence and the answer that pairs with the question sentence. The feature words that characterize documents are a group of words that increase the BM25 score of the target document compared to all documents. (3) The learning processing unit 600 learns the processing parameters of the feature word generation model 410 so that, for the pair of the question sentence and the feature word corresponding to the question sentence obtained in the above steps, the question sentence is input and, for example, the top 10 words in the BM25 score are output as feature words. The learning processing unit 600 repeats the above (2) and (3) once or multiple times for all pairs of question sentences and documents in the learning data until the error in the output of the feature word generation model 410 converges.

Figure 3 is a block diagram showing an example of the configuration of a feature word generation model according to an embodiment. The feature word generation model 410 is, for example, a multi-layer neural network including an input layer, an intermediate layer, and an output layer, with each layer having multiple nodes. The learning processing unit 600 learns to output feature words corresponding to a question sentence by updating processing parameters such as word embedding, attention, activation function, error function, gradient descent algorithm, weighting coefficient, and weighting matrix in the multi-layer neural network. Note that, although the feature word generation model 410 in the embodiment is a multi-layer neural network, this is not limited thereto, and other machine learning models may be used. Furthermore, the feature word generation model 410 is not limited to a machine learning model as long as it is a model capable of generating feature words from a question sentence.

Figure 4 is a block diagram for explaining the learning process of the machine reading comprehension model of the embodiment. The machine reading comprehension model 510 is trained using learning data for machine reading comprehension processing that includes a question, a document containing an answer to the question, and multiple sets of answers. The answer is, for example, information indicating a section in a document. The learning process of the machine reading comprehension model 510 first inputs a question and a document into the machine reading comprehension model 510, and learns processing parameters in the machine reading comprehension model 510 so that the machine reading comprehension model 510 outputs an answer corresponding to the question and document.

FIG. 5 is a block diagram showing an example of the configuration of a machine reading comprehension model according to an embodiment. The machine reading comprehension model 510 is, for example, a multi-layer neural network having an input layer, an intermediate layer, and an output layer, with each layer having multiple nodes. The learning processing unit 610 learns to output feature words corresponding to a question sentence by updating processing parameters such as word embedding, attention, activation function, error function, gradient descent algorithm, weighting coefficient, and weighting matrix in the multi-layer neural network. Note that, although the machine reading comprehension model 510 in the embodiment is a multi-layer neural network, it is not limited to this and may be another machine learning model. Furthermore, the machine reading comprehension model 510 is not limited to a machine learning model as long as it is a model that can output a part that is an answer to the question sentence that exists in the document from the question sentence and the document.

The output layer of the machine reading comprehension model 510 may be any node capable of outputting a score for evaluating an interval indicating an answer contained in a document. The score for evaluating an interval indicating an answer contained in a document is expressed, for example, as a value in which the values of all intervals fall within a predetermined range of 0 to 1 and the sum with other intervals is not taken into consideration. This score may be interpreted, for example, as a value indicating the absolute likelihood between intervals. The activation function applied to this output layer is called, for example, a Sigmoid function. For example, when multiple intervals are extracted using multiple documents, the machine reading comprehension model 510 outputs the interval and score using the Sigmoid function. When multiple intervals are extracted using multiple documents, for example, interval A is extracted by machine reading comprehension processing from document A obtained by full-text search processing from a question word, and interval B is extracted by machine reading comprehension processing from document B obtained by full-text search processing from a feature word. This allows the service providing server 200 to compare the scores of section A and section B and rank them, even if documents A and B are different.

6 is a flowchart illustrating an example of a processing procedure in the question answer prediction system according to the embodiment. FIG 7 is a block diagram illustrating an example of a processing procedure in the question answer prediction system according to the embodiment.
In the above-described question answer prediction system, first, the service providing server 200 accepts a question sentence from the terminal device 100 (step S100). Next, the service providing server 200 provides the question sentence to the feature word generating unit 400. The feature word generating unit 400 generates question words by dividing the question sentence into words, and generates feature words based on the question words (step S102). The service providing server 200 acquires the generated feature words.

Next, the service providing server 200 provides the question term and the characteristic words to the full-text search server 300. The full-text search server 300 executes a full-text search process using the question term and a full-text search process using the characteristic words (step S104). The service providing server 200 acquires documents as a result of the full-text search process using the question term and documents as a result of the full-text search process using the characteristic words.

Next, the service providing server 200 divides the acquired literature into documents having the length of the processing unit of the machine reading comprehension process (step S106), and the machine reading unit 500 executes the machine reading comprehension process for each document (step S108). The service providing server 200 acquires the answer candidates acquired from the documents corresponding to each question and the scores of the answer candidates.

Next, the service providing server 200 generates an answer as a processing result based on the answer candidates and scores acquired from the machine reading unit 500 (step S110). At this time, the service providing server 200 outputs the answer based on a comparison of the scores between the answer candidates. Specifically, when the answer candidates are the same and the scores of the same answer candidates exceed a threshold, the service providing server 200 adds up the scores of the same answer candidates.

Here, when the same answer is output from different documents, the possibility that it is correct can be increased. However, if the scores are added without taking into account the score values output by the machine reading comprehension model 510, the accuracy of the answer being correct will decrease when a large number of answers with low scores are output for frequently occurring words, etc. Also, if negative values are included in the score range, there is a possibility that the score will decrease due to addition. Therefore, the service providing server 200 sets a threshold value to limit the score addition targets, and adds the scores of answer candidates with scores exceeding the threshold value. When there are multiple answer candidates with scores that are not added, the service providing server 200 determines the maximum score among the scores of the multiple answer candidates as the score of that answer candidate.

The service providing server 200 ranks the multiple answer candidates by score. The service providing server 200 generates result information in which the answer candidates are arranged in descending order from the answer candidate with the highest score (step S110), and provides the generated information to the terminal device 100 (step S112).

In addition, the question answer prediction system 1 of the embodiment may display text information of a document including the answer on the terminal device 100, and perform a learning process for the machine reading comprehension model 510 using the answer based on the user's operation.

Effects of the embodiment
As described above, according to the embodiment of the question answer prediction system 1, a search device (e.g., the service providing server 200, the full-text search server 300, and the machine reading unit 500) can be realized that includes a search unit (e.g., the full-text search server 300) that inputs a question and searches for literature based on words included in the question, and a prediction unit (e.g., the machine reading unit 500) that inputs information based on the literature searched by the search unit and has a prediction model (e.g., the machine reading comprehension model 510) that outputs a result of predicting an answer to the question. In this search device, the prediction model is, for example, a model trained using a set including learning questions, learning answers, and learning literature including the learning answers as learning data.
According to the question answer prediction system 1 of the embodiment, a learning device (for example, a combination of the learning processing unit 600 and the learning processing unit 610) including a first learning unit (for example, the learning processing unit 610) that learns questions, answers to the questions, and documents containing the answers as first learning data, and generates a prediction model (for example, the machine reading comprehension model 510) that outputs an answer when an unknown question and document are input, and a second learning unit (for example, the learning processing unit 600) that learns questions and feature words related to the questions from the first learning data as learning data, and generates a feature word generation model (for example, the feature word generation model 410) that outputs feature words related to the unknown question when an unknown question is input, and It is possible to realize a question answer prediction system comprising a search device (e.g., a combination of the full-text search server 300, the feature word generation unit 400, and the machine reading comprehension unit 500) that includes a feature word generation unit (e.g., the feature word generation unit 400) that generates feature words related to an unknown question when an unknown question is input, a search unit (e.g., the full-text search server 300) that performs a process of searching for literature based on words contained in the unknown question and a process of searching for literature based on the feature words generated by the feature word generation unit, and a prediction unit (e.g., the machine reading comprehension unit 500) that has a prediction model, inputs information based on the literature searched by the search unit, and outputs a result of predicting an answer to the unknown question.
According to the embodiment, the question answer prediction system 1 can realize a question answer prediction method including the steps of: learning questions, answers to the questions, and literature containing the answers as first learning data, and generating a prediction model that outputs an answer when an unknown question and literature are input; learning questions and feature words related to the questions from the first learning data as learning data, and generating a feature word generation model that outputs feature words related to the unknown question when an unknown question is input; generating feature words related to the unknown question using the feature word generation model when an unknown question is input; performing a process of searching for literature based on words included in the unknown question and a process of searching for literature based on the feature words generated by the feature word generation unit; and using the prediction model to input information based on the searched literature, and outputting a result of predicting the answer to the unknown question.
According to the question answer prediction system 1, it is possible to output an answer to a question with high accuracy. That is, according to the question answer prediction system 1, it is possible to provide information that has a high probability of being an answer among character strings contained in documents searched from a question as an answer result, rather than a document search result based on a simple match of character strings, etc., to a user's question. As a result, according to the question answer prediction system 1, it is possible to provide an answer that takes into account the context of the question.

According to the embodiment of the question answer prediction system 1, a feature word generation unit (e.g., the feature word generation unit 400) having a generative model that receives input of words contained in a question and generates feature words related to the question is provided, and a search unit can execute a process of searching for documents based on the words contained in the question and a process of searching for documents based on the feature words generated by the feature word generation unit. The generative model is, for example, a model trained to receive input of a learning question contained in the learning data and output feature words related to the input learning question.
According to this question answer prediction system 1, it is possible to output an answer to a question with even higher accuracy. That is, according to the question answer prediction system 1, even if there is a document searched from a word included in a question but the document does not contain the answer, the answer can be obtained from the document searched from the characteristic word. Furthermore, according to the question answer prediction system 1, even if the document searched from the user's question does not contain the correct answer, if the document searched from the characteristic word contains the correct answer, the correct answer to the question can be provided.

According to the embodiment of the question answer prediction system 1, the prediction model includes an output layer that outputs one or more answer candidates, which are information on which the results are based, and the scores of the one or more answer candidates, and the prediction unit outputs answer candidates and scores in documents based on words included in the question, and answer candidates and scores in documents based on characteristic words, and the result can be output based on a comparison of the scores between the answer candidates. Also, according to the embodiment of the question answer prediction system 1, the prediction model includes an output layer that outputs one or more answer candidates, which are information on which the results are based, and scores of the one or more answer candidates, in which upper and lower limits for one or more scores are set and the sum is not adjusted to a predetermined value, and the prediction unit outputs answer candidates and scores in documents based on words included in the question, and answer candidates and scores in documents based on characteristic words, and the result can be output based on a comparison of the scores between the answer candidates.
According to this question answer prediction system 1, it is possible to output answers to questions with even higher accuracy. That is, according to the question answer prediction system 1, by comparing the scores of multiple answer candidates extracted from documents based on the question and feature words, it is possible to provide information with a high probability of being the answer from a large number of answer candidates.

According to the question answer prediction system 1, when the answer candidates output by the prediction model are the same, the prediction unit can add up the scores of the same answer candidates if the scores of the same answer candidates exceed a threshold. As a result, according to the question answer prediction system 1, even if a large number of answer candidates with low answer probability are output from multiple documents, it is possible to prevent the accuracy of the answer provided to the user from decreasing. Specifically, it is possible to prevent information that appears in many documents but does not match the context of the question from being provided as an answer.

Note that although each embodiment and each modified example have been described, these are merely examples and are not intended to be limiting. For example, any of the embodiments and modified examples, or a part of each embodiment or a part of each modified example, may be combined with one or more other embodiments or one or more other modified examples to realize one aspect of the present invention.

In addition, the programs for executing the various processes of the individual estimation device 100 in this embodiment may be recorded on a computer-readable recording medium, and the programs recorded on the recording medium may be read into and executed by a computer system to perform the various processes described above relating to the terminal device 100, the service providing server 200, the full-text search server 300, the feature word generation unit 400, and the machine reading unit 500.

Note that the term "computer system" here may include hardware such as the OS and peripheral devices. In addition, if a WWW system is used, the term "computer system" also includes the homepage provision environment (or display environment). Furthermore, "computer-readable recording media" refers to storage devices such as flexible disks, magneto-optical disks, ROMs, writable non-volatile memories such as flash memory, portable media such as CD-ROMs, and hard disks built into computer systems.

Furthermore, the term "computer-readable recording medium" refers to a volatile memory (e.g., DRAM (Dynamic Random Access Memory)) within a computer system that is a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line.
This also includes those that hold a program for a certain period of time, such as a random access memory (RDMA). The program may be transmitted from a computer system in which the program is stored in a storage device to another computer system via a transmission medium, or by a transmission wave in the transmission medium.

Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The above program may also be one that realizes part of the above-mentioned functions. Furthermore, it may be one that can realize the above-mentioned functions in combination with a program already recorded in the computer system, a so-called differential file (differential program).

The above describes an embodiment of the present invention in detail with reference to the drawings, but the specific configuration is not limited to this embodiment, and includes designs that do not deviate from the gist of the invention.

1 Question answer prediction system 100 Terminal device 110 Browser 200 Service providing server 210 Job management unit 220 Result providing unit 230, 240 Job queue 300 Full-text search server 310 Literature database 400 Feature word generation unit 410 Feature word generation model 500 Machine reading comprehension unit 510 Machine reading comprehension model 600, 610 Learning processing unit

Claims (9)

a feature word generation unit that has a generation model trained to generate feature words corresponding to the training questions using a set of training questions and feature words that characterize training documents including training answers to the training questions as training data, and that uses the generation model to generate feature words corresponding to the input questions;
a search unit that executes a process of searching a literature database for documents containing words that match a word included in the input question, based on the word included in the question, and a process of searching the literature database for documents containing words that match the feature word, based on the feature word generated by the feature word generation unit;
a machine reading comprehension unit that has a machine reading comprehension model trained to output answer candidates corresponding to the learning questions and the learning documents and scores indicating the likelihood of the answer candidates, using a set of learning questions, learning documents including learning answers to the learning questions, and the learning answers as learning data, and that uses the machine reading comprehension model to output answer candidates corresponding to the input question and documents searched based on words included in the question and scores indicating the likelihood of the answer candidates, and answer candidates corresponding to the question and documents searched based on feature words corresponding to the question and scores indicating the likelihood of the answer candidates;
a prediction unit that obtains from the machine comprehension unit answer candidates corresponding to the input question and documents searched based on words included in the question and scores indicating the likelihood of the answer candidates, and answer candidates corresponding to the question and documents searched based on feature words corresponding to the question and scores indicating the likelihood of the answer candidates, and outputs a result of predicting an answer to the question based on a comparison of the scores indicating the likelihood of each answer candidate;
A search device comprising: the prediction unit, when an answer candidate output by the machine reading comprehension model, which corresponds to the input question and a document searched based on a word included in the question, is the same as an answer candidate corresponding to the question and a document searched based on a feature word corresponding to the question, and when a score indicating the likelihood of each of the same answer candidates exceeds a threshold, adds up the scores of the same answer candidates;
The search device according to claim 1 . a first learning unit that uses a set of training questions and feature words that characterize training documents including training answers to the training questions as training data to generate a generative model trained to generate feature words corresponding to the training questions;
a second learning unit that uses a set of learning questions, learning documents including learning answers to the learning questions, and the learning answers as learning data to generate a machine reading comprehension model that has been trained to output answer candidates corresponding to the learning questions and the learning documents and scores indicating the likelihoods of the answer candidates;
Learning device. a learning device including: a first learning unit that uses a set of learning questions and feature words that characterize learning documents including learning answers to the learning questions as learning data to generate a generation model trained to generate feature words corresponding to the learning questions; and a second learning unit that uses a set of learning questions, learning documents including learning answers to the learning questions, and the learning answers as learning data to generate a machine reading comprehension model trained to output answer candidates corresponding to the learning questions and the learning documents and scores indicating the likelihood of the answer candidates;
a feature word generation unit having the generative model and using the generative model to generate feature words corresponding to an input question; a search unit executing a process of searching a literature database for documents containing words matching a word included in the input question based on the word included in the question, and a process of searching the literature database for documents containing words matching the feature words based on the feature words generated by the feature word generation unit; a machine reading comprehension unit having the machine reading comprehension model and using the machine reading comprehension model to output answer candidates corresponding to the input question and documents searched based on the words included in the question and scores indicating the likelihood of the answer candidates, and answer candidates corresponding to the question and documents searched based on the feature words corresponding to the question; and a prediction unit obtaining from the machine reading comprehension unit answer candidates corresponding to the input question and documents searched based on the words included in the question and scores indicating the likelihood of the answer candidates, and answer candidates corresponding to the question and documents searched based on the feature words corresponding to the question, and scores indicating the likelihood of the answer candidates from the machine reading comprehension unit, and outputting a result of predicting an answer to the question based on a comparison of the scores indicating the likelihood of each answer candidate.
A question answer prediction system comprising: a step in which a learning device uses a set of training questions and feature words that characterize training documents including training answers to the training questions as training data to generate a generative model trained to generate feature words corresponding to the training questions;
a step of generating a machine reading comprehension model trained by the learning device to use a set of learning questions, learning documents including learning answers to the learning questions, and the learning answers as learning data, and to output answer candidates corresponding to the learning questions and the learning documents and scores indicating the likelihoods of the answer candidates;
A step in which the search device generates feature words corresponding to an input question using the generation model;
the search device executes a process of searching a literature database for documents including words that match a word included in the input question based on the word, and a process of searching the literature database for documents including words that match the characteristic word based on the characteristic word;
a step of the search device using the machine reading comprehension model to output answer candidates corresponding to the input question and documents searched based on words included in the question and scores indicating the likelihood of the answer candidates, and outputting answer candidates corresponding to the question and documents searched based on feature words corresponding to the question and scores indicating the likelihood of the answer candidates;
the search device acquires answer candidates corresponding to the input question and documents searched based on words included in the question, and scores indicating the likelihood of the answer candidates, and answer candidates corresponding to the question and documents searched based on feature words corresponding to the question, and scores indicating the likelihood of the answer candidates, and outputs a result of predicting an answer to the question based on a comparison of the scores indicating the likelihood of each answer candidate;
The method for predicting an answer to a question includes the steps of: a step in which the search device has a generation model trained to generate feature words corresponding to the training questions using a set of training questions and feature words that characterize training documents including training answers to the training questions as training data, and generates feature words corresponding to the input question using the generation model;
The search device executes a process of searching a literature database for documents containing words that match a word included in the input question based on the word, and a process of searching the literature database for documents containing words that match the generated characteristic word based on the generated characteristic word;
the search device has a machine reading comprehension model trained to output answer candidates corresponding to the learning questions and the learning documents and scores indicating the likelihood of the answer candidates, using a set of learning questions , learning documents including learning answers to the learning questions, and the learning answers as learning data, and uses the machine reading comprehension model to output answer candidates corresponding to the input question and documents searched based on words contained in the question and scores indicating the likelihood of the answer candidates, and answer candidates corresponding to the question and documents searched based on feature words corresponding to the question and scores indicating the likelihood of the answer candidates;
the search device acquires answer candidates corresponding to the input question and documents searched based on words included in the question, and scores indicating the likelihood of the answer candidates, and answer candidates corresponding to the question and documents searched based on feature words corresponding to the question, and scores indicating the likelihood of the answer candidates, and outputs a result of predicting an answer to the question based on a comparison of the scores indicating the likelihood of each answer candidate;
A search method comprising: The computer of the search device
a step of having a generative model trained to generate feature words corresponding to training questions using a set of training questions and feature words that characterize training documents including training answers to the training questions as training data, and generating feature words corresponding to an input question using the generative model;
A step of executing a process of searching a literature database for documents containing words that match a word included in the input question based on the word, and a process of searching the literature database for documents containing words that match the generated characteristic word based on the generated characteristic word;
a step of having a machine reading comprehension model trained to output answer candidates corresponding to the learning questions and the learning documents and scores indicating the likelihood of the answer candidates, using a set of learning questions, learning documents including learning answers to the learning questions, and the learning answers as learning data, and using the machine reading comprehension model to output answer candidates corresponding to the input question and documents searched based on words contained in the question and scores indicating the likelihood of the answer candidates, and answer candidates corresponding to the question and documents searched based on feature words corresponding to the question and scores indicating the likelihood of the answer candidates;
obtaining answer candidates corresponding to the input question and documents searched based on words included in the question, and scores indicating the likelihood of the answer candidates, and obtaining answer candidates corresponding to the question and documents searched based on feature words corresponding to the question, and scores indicating the likelihood of the answer candidates, and outputting a result of predicting an answer to the question based on a comparison of the scores indicating the likelihood of each answer candidate;
A program to execute. a step in which a learning device uses a set of training questions and feature words that characterize training documents including training answers to the training questions as training data to generate a generative model trained to generate feature words corresponding to the training questions;
a step of generating a machine reading comprehension model trained by the learning device to use a set of learning questions, learning documents including learning answers to the learning questions, and the learning answers as learning data, and to output answer candidates corresponding to the learning questions and the learning documents and scores indicating the likelihoods of the answer candidates;
A learning method having: The learning device computer
generating a generative model trained to generate feature words corresponding to the training questions, using a set of training questions and feature words that characterize training documents including training answers to the training questions as training data;
a step of generating a machine reading comprehension model trained by the learning device to use a set of learning questions, learning documents including learning answers to the learning questions, and the learning answers as learning data, and to output answer candidates corresponding to the learning questions and the learning documents and scores indicating the likelihoods of the answer candidates;
A program to execute.

Images