JP7687589B2 - Question creation device, question creation method, and program - Google Patents

Description

This disclosure relates to a question creation device, a question creation method, and a program.

With the recent development of information technology, information technology is being applied to various fields. For example, various learning support tools that utilize information technology have been developed. One example of such a learning support tool is a learning support app that allows a learner to operate a smartphone, tablet, personal computer, etc. and learn English by solving problems displayed on the learning support app.

One example of the question format provided by such learning support apps is fill-in-the-blank questions, and a method for creating questions to automatically generate such fill-in-the-blank questions has been proposed. Fill-in-the-blank questions are questions that require learners to guess the word or term that should fill in the blank from the context, or to select it from a list of options.

JP 2018-17904 A

In recent years, artificial intelligence has been used in fields such as natural language processing, and various natural language processing models such as BERT (Bidirectional Encoder Representations from Transformers) by Google have been developed, and the high performance of these natural language processing models in natural language processing has attracted attention.

The objective of this disclosure is to provide a technique for creating questions for fill-in-the-blank questions using a natural language processing model.

In order to solve the above problem, one aspect of the present invention relates to a question creation device having a sentence acquisition unit that acquires a first sentence including a word to be learned from a dataset, a confidence calculation unit that masks the word in the first sentence and calculates the confidence of the word in the masked portion of the first sentence and the confidence of a candidate word that is another candidate that can be placed in the masked portion using a prediction model, and a question output unit that outputs the masked sentence as a fill-in-the-blank question if the confidence of the word is greater than the confidence of the candidate word by a predetermined value or more.

The present disclosure provides a technique for creating questions for fill-in-the-blank questions that utilizes a natural language processing model.

1 is a schematic diagram showing a question sentence creation device according to an embodiment of the present disclosure. 1 is a schematic diagram showing a learner's learning using a learning support system according to an embodiment of the present disclosure. 1 is a schematic diagram showing a question sentence creation process according to an embodiment of the present disclosure. 1 is a block diagram showing a hardware configuration of a question sentence creation device according to an embodiment of the present disclosure. 1 is a block diagram showing a functional configuration of a question sentence creation device according to an embodiment of the present disclosure. FIG. 11 is a schematic diagram showing a question sentence creation process according to another embodiment of the present disclosure. FIG. 11 is a schematic diagram showing a question sentence creation process according to another embodiment of the present disclosure. FIG. 13 is a block diagram showing a functional configuration of a question sentence creation device according to another embodiment of the present disclosure. 13 is a flowchart illustrating a question sentence creation process according to an embodiment of the present disclosure.

<Terminology>
In this specification, the confidence level is the correct (predicted) probability of a mask prediction task (word fill-in-the-blank task), and the confidence level of a word is the likelihood (confidence level) that the word will be placed in the corresponding position in the sentence. The "confidence level of a word" is calculated based on one or more other words in the sentence. The "confidence level of a word" may also be calculated based on one or more words in the sentences before and after the corresponding sentence. The higher the value (i.e., the confidence level value), the more frequently the corresponding word is used by analogy with other words. Note that it is sufficient for the confidence level to be calculated, and there is no need for a specific calculation method. In this embodiment, the confidence level calculated by BERT is used for explanation.

In the following embodiment, a question creation device for fill-in-the-blank questions is disclosed.

As shown in FIG. 1, a question creation device 100 according to an embodiment of the present disclosure receives as input a word to be learned and a sentence containing the word, and outputs fill-in-the-blank questions for the input word using any appropriate natural language processing model such as BERT. For example, as shown in FIG. 2, when learner A learns English using a learning app operated by a learning support system, the learning support system obtains information about English words that learner A has learned from learner A's learning history, etc., and creates fill-in-the-blank questions for English words that learner A has not learned from English content obtained from a database or the web.

Specifically, as shown in FIG. 3, the question creation device 100 masks (S13 in FIG. 3, S33 in FIG. 7) the English words to be learned in sentences (S12 in FIG. 3, S32 in FIG. 7) acquired from a dataset (S11 in FIG. 3, S31 in FIG. 7 described later) or the like, inputs the sentences including the masked portions into a natural language processing model such as BERT (S14 in FIG. 3, S34 in FIG. 7), and estimates candidates for the English words to be placed in the masked portions. According to the present disclosure, the question creation device 100 estimates the confidence of each candidate English word estimated from the natural language processing model, and when the difference between the highest first confidence, i.e., the confidence of the masked English word to be learned and the second highest second confidence, is equal to or greater than a predetermined value, outputs the sentence including the masked portion as a fill-in-the-blank question for the English word having the first confidence (S15 in FIG. 3, S35 in FIG. 7). This makes it possible to output as a question a sentence that contains an English word with a high degree of certainty that has a significant difference from other English word candidates as the English word to be filled in the masked portion, and to create fill-in-the-blank questions in which only the English word being studied is the correct answer.

The question creation device 100 may be a server that provides questions to a learner's smartphone, tablet, personal computer, etc. via a learning app, as described below. For example, the question creation device 100 may be a server of a company, vendor, etc. that operates a learning app, and obtains the sentence to be processed from a database that stores sentences held by the company, vendor, etc. as a dataset, and creates a question as described below. Preferably, the dataset is composed of copyright-managed sentences.

The question creation device 100 may have a hardware configuration, for example as shown in FIG. 4, including a processor 101 such as a CPU (Central Processing Unit), a memory 102 such as a RAM (Random Access Memory) or a flash memory, a storage 103 such as a hard disk, and an input/output (I/O) interface 104.

The processor 101 executes various processes of the question creation device 100, which will be described later.

The memory 102 stores various data and programs in the question creation device 100, and functions in particular as a working memory for working data, programs currently being executed, and the like. Specifically, the memory 102 stores programs for executing and controlling various processes described below that are loaded from the storage 103, and functions as a working memory while the processor 101 is executing the programs.

Storage 103 stores various data and programs in the question creation device 100.

The I/O interface 104 is an interface for receiving commands and input data from a user, displaying and playing back output results, and inputting and outputting data between external devices. For example, the I/O interface 104 may be a device for inputting and outputting various types of data, such as a USB (Universal Serial Bus), a communication line, a keyboard, a mouse, a display, a microphone, and a speaker.

However, the question creation device 100 according to the present disclosure is not limited to the above-mentioned hardware configuration, and may have any other appropriate hardware configuration. For example, one or more of the various processes performed by the question creation device 100 may be realized by a processing circuit or electronic circuit that is hardwired to realize the process.

Next, the question creation device 100 according to an embodiment of the present disclosure will be described in more detail with reference to Figures 5 to 9. The question creation device 100 receives as input a word to be learned (e.g., an English word, term, person's name, place, etc.) and a sentence, and outputs a fill-in-the-blank question with the word left blank. In the following embodiment, an English word is applied as the word to be learned, and a fill-in-the-blank question in which the English word is masked is output, but the present disclosure is not limited thereto and may be applied to the learning of terms in any subject other than English language learning.

FIG. 5 is a block diagram showing the functional configuration of a question sentence creation device 100 according to an embodiment of the present disclosure. As shown in FIG. 5, the question sentence creation device 100 has a sentence acquisition unit 110, a confidence factor calculation unit 120, and a question sentence output unit 130.

The sentence acquisition unit 110 acquires sentences including the words to be learned from the dataset. In this specification, a sentence is one or more words ending with a period "." or a full stop ".", and a sentence is a series of multiple consecutive sentences having a coherent meaning. The sentence or sentences to be acquired may be acquired, for example, from a database or the web. Preferably, the sentence or sentences to be acquired may be extracted from a dataset in a database where copyright is managed.

The confidence calculation unit 120 masks the word to be learned in the acquired sentence, and uses a prediction model to calculate the confidence of the word to be learned in the masked portion of the sentence and the confidence of other candidate words that can be placed in the masked portion. Specifically, the confidence calculation unit 120 converts a sentence or text containing the word to be learned into a sentence or text in which the masked portion is left blank by masking the word. The confidence calculation unit 120 then inputs the sentence or text containing the masked portion into any appropriate natural language processing model such as BERT. BERT outputs the candidate words to be placed in the masked portion and the confidence of each candidate word, as shown in FIG. 3.

In the illustrated specific example, for the masked word "continue", BERT outputs candidate words and confidence levels for the masked part "continue 0.9373...", "cease 0.0506...", and "begin 0.0025..." from an input sentence containing the masked part "This city will cease to be and will ( ) to grow!". Here, the confidence level indicates the probability that the word will be included in the masked part, and for example, the confidence level of the masked word is the highest. Note that there may be cases where the confidence level of the masked word is not the highest.

For the masked word "hurt," BERT outputs candidate words and confidence scores for the masked part "hurt 0.9465...," "wounded 0.0167...," and "offended 0.0062..." from an input sentence that includes the masked part "I apologize if my actions ( ) your pride."

If the masked word (for example, "continue" or "hurt" in the above-mentioned embodiment) is not predicted as a possible candidate for the masked portion, the question creation device 100 will handle this as an error and obtain another sentence.

In the above-described embodiment, one sentence is input to the BERT, but the present disclosure is not limited thereto, and in one embodiment, a sentence consisting of multiple consecutive sentences may be input to the BERT. For example, as shown in FIG. 6, the certainty calculation unit 120 may mask (S23 in FIG. 6) the word to be learned from a sentence consisting of a sentence including the word to be learned and sentences before and after the sentence (S21, S22 in FIG. 6), and input the sentence including the masked portion to the BERT (S24 in FIG. 6). In a similar manner, the BERT outputs candidate words and certainty for the masked portion from the input sentence including the masked portion (S25 in FIG. 6).

If the difference between the confidence of the word and the confidence of the candidate word is equal to or greater than a predetermined value, the question output unit 130 outputs the masked sentence as a fill-in-the-blank question. Specifically, the question output unit 130 compares the confidence of each candidate word in the masked portion and calculates the difference between the highest confidence (for example, the confidence of the masked word) and the next highest confidence. If the difference between the highest confidence and the next highest confidence is equal to or greater than a predetermined threshold, the question output unit 130 determines that the input sentence is suitable as a fill-in-the-blank question for the word, and outputs the masked sentence as a fill-in-the-blank question for the word. On the other hand, if the difference between the highest confidence and the next highest confidence is less than a predetermined threshold, the question output unit 130 determines that the input sentence is not suitable as a fill-in-the-blank question for the word, and does not output the masked sentence as a fill-in-the-blank question for the word. In other words, if there is no significant difference between these confidences, it is difficult to uniquely identify the correct answer to the fill-in-the-blank question for the input sentence, and it is considered that the input sentence is not suitable as a fill-in-the-blank question. Similarly, if the highest confidence level is not the confidence level of the masked word, the input sentence is also considered to be unsuitable as a fill-in-the-blank question. In other words, the question sentence output unit 130 outputs the masked sentence as a fill-in-the-blank question if the highest confidence level is the confidence level of the masked word and the difference between the confidence level of the masked word and the confidence level of the candidate word is equal to or greater than a predetermined value.

In one embodiment, if the difference between the confidence of the word and the confidence of the candidate word is less than a predetermined value, the sentence acquisition unit 110 acquires a sentence adjacent to the acquired sentence (i.e., a sentence including the learning target word from the dataset), the confidence calculation unit 120 uses a prediction model to calculate the confidence of the learning target word and the confidence of the candidate word in the masked portion of the sentence composed of the acquired sentence and the sentence adjacent to the acquired sentence, and the question sentence output unit 130 may output the masked sentence as a fill-in-the-blank question for the word if the difference between the confidence of the learning target word and the confidence of each candidate word is equal to or greater than a predetermined value. For example, if the input sentence to the BERT is not suitable as a fill-in-the-blank question, the sentences before and after the input sentence may be searched, and the sentence composed of the input sentence and the sentences before and after the input sentence may be input to the BERT. In general, it is considered that as the range of the context expands, the candidate words to be included in the masked portion are narrowed down, and it is assumed that the difference between the confidence of the learning target word and the next highest confidence becomes large. Therefore, when the input sentence is not suitable as a fill-in-the-blank question, the sentence acquisition unit 110 may extract sentences before and after the input sentence from the dataset, and the certainty calculation unit 120 may mask the words to be learned in the extracted sentences, input the sentences including the masked parts to BERT, and calculate the candidate words and the certainty of each candidate word. Then, when the difference between the highest certainty of the word to be learned and the next highest certainty is equal to or greater than a predetermined value, the question sentence creation device 100 may output the sentence as a fill-in-the-blank question. That is, if the difference between the confidence of the word and the confidence of the candidate word in the nth sentence (n is an integer equal to or greater than 1) is less than a predetermined value, the sentence acquisition unit 110 acquires the n+1th sentence adjacent to the nth sentence, the confidence calculation unit 120 uses the prediction model to calculate the confidence of the word to be studied and the confidence of the candidate word in the masked portion of the sentence composed of the acquired sentence (nth sentence) and the sentence adjacent to the acquired sentence (n+1th sentence), and the question sentence output unit 130 can output the masked sentence as a fill-in-the-blank question for the word if the difference between the confidence of the word to be studied and the confidence of each candidate word is equal to or greater than a predetermined value (for example, acquisition of the adjacent sentence may be repeated up to a specified number of times, or until the difference between the confidence of the word to be studied and the confidence of each candidate word is equal to or greater than a predetermined value).

In addition, in one embodiment, the question output unit 130 may output a word and a selection candidate word associated with the word as options in a fill-in-the-blank question. That is, the question output unit 130 may automatically generate options for a fill-in-the-blank question. For example, the question output unit 130 may select the word to be studied and a derivative of the word or a word of a different part of speech as options for the blank in a fill-in-the-blank question. Specifically, when the word to be studied is a verb, the question output unit 130 may refer to a dictionary database or the like to determine the corresponding noun, adjective, adverb, etc. of the verb as options.

In one embodiment, the question output unit 130 may display one or more letters of a word in a fill-in-the-blank question. For example, the question output unit 130 may display the first letter of the correct word (for example, if the correct word is "continue," then the first letter "c") as a hint in the blank of the fill-in-the-blank question.

In one embodiment, the question output unit 130 may display a sentence and a sentence accompanying the sentence in a fill-in-the-blank question. For example, as shown in FIG. 7, the question output unit 130 may display an English sentence and a translation of the English sentence in an English fill-in-the-blank question.

In one embodiment, the question output unit 130 may display a sentence and an image associated with the sentence in a fill-in-the-blank question. For example, in a fill-in-the-blank question such as "Carbon dioxide is a bond between one carbon atom and two ( )," a diagram of a carbon atom and two oxygen atoms bonded together, "O=C=O," may be displayed together.

In one embodiment, the question creation device 100 may further include a word providing unit 140 that provides words to be learned, as shown in FIG. 8. For example, the word providing unit 140 may determine words to be learned based on the learning history and learning level of the learner, and provide the words to be learned to the sentence acquisition unit 110, the confidence calculation unit 120, and/or the question output unit 130.

The above-mentioned embodiments may be combined as appropriate.

Figure 9 is a flowchart showing a question creation process according to one embodiment of the present disclosure. The question creation process is executed by the above-mentioned question creation device 100, and in particular can be realized by the processor of the question creation device 100 executing a program.

As shown in FIG. 9, in step S101, the question creation device 100 acquires sentences including the words to be studied from the dataset. For example, the question creation device 100 extracts sentences or paragraphs including the words to be studied from a copyright-managed database. Specifically, if the words to be studied are English words for high school entrance exams, the question creation device 100 extracts English sentences including these English words from the database.

In step S102, the question creation device 100 masks the word to be learned in the acquired sentence. For example, when a sentence containing the word to be learned is acquired, the question creation device 100 leaves the location of the word in the acquired sentence blank.

In step S103, the question creation device 100 uses the inference model to calculate the confidence of the word in the masked portion of the masked sentence and the confidence of the candidate words. For example, the question creation device 100 inputs a sentence including a masked portion into BERT, and obtains candidate words that may be included in the masked portion and the confidence of each candidate word. For example, the confidence of the masked word is the highest.

In step S104, if the difference between the certainty of the masked word and the certainty of the other candidate words is equal to or greater than a predetermined value, the question creation device 100 outputs the masked sentence as a fill-in-the-blank question. Note that the predetermined value may be set to any appropriate value that can significantly distinguish the masked word from the candidate words.

Although the examples of the present invention have been described in detail above, the present invention is not limited to the specific embodiments described above, and various modifications and variations are possible within the scope of the gist of the present invention as described in the claims.

100 Question sentence creation device 110 Sentence acquisition unit 120 Confidence factor calculation unit 130 Question sentence output unit 140 Word provision unit

Claims (10)

a sentence acquisition unit that acquires a first sentence including a word to be learned from the dataset;
a confidence calculation unit that masks the word in the first sentence and calculates the confidence of the word in the masked portion of the first sentence and the confidence of other candidate words that may be included in the masked portion by using an inference model;
a question output unit that sets a predetermined value for distinguishing the word from the candidate word, and outputs the masked sentence as a fill-in-the-blank question when the certainty of the word is greater than the certainty of the candidate word by at least the predetermined value;
A problem creation device having the above structure. if the difference between the confidence of the word and the confidence of the candidate word is less than the predetermined value,
the sentence acquisition unit acquires a second sentence adjacent to the first sentence in a sentence in the dataset ;
the certainty calculation unit uses the inference model to calculate certainty of the word and certainty of the candidate word in a masked portion of the sentence composed of the first sentence and the second sentence;
2 . The question creation device according to claim 1 , wherein the question output unit outputs the masked sentence as a fill-in-the-blank question for the word when the certainty of the word is greater than the certainty of the candidate word by at least the predetermined value. if the difference between the confidence of the word and the confidence of the candidate word is less than the predetermined value,
the sentence acquisition unit acquires an n+1-th sentence adjacent to an n-th sentence in the sentences in the dataset , where n is an integer equal to or greater than 1;
the certainty calculation unit uses the inference model to calculate certainty of the word and certainty of the candidate word in a masked portion of the sentence composed of the nth sentence and the n+1th sentence;
2 . The question creation device according to claim 1 , wherein the question output unit outputs the masked sentence as a fill-in-the-blank question for the word when the certainty of the word is greater than the certainty of the candidate word by at least the predetermined value. The question creation device according to any one of claims 1 to 3, wherein the question output unit outputs the word and a selection candidate word associated with the word as options in the fill-in-the-blank question. The question creation device according to any one of claims 1 to 4, wherein the question output unit displays one or more characters of the word in the fill-in-the-blank question. The question creation device according to any one of claims 1 to 5, wherein the question output unit displays the masked sentence and a sentence associated with the masked sentence in the fill-in-the-blank question. The question creation device according to any one of claims 1 to 5, wherein the question output unit displays the masked sentence and an image associated with the sentence in the fill-in-the-blank question. The question creation device according to any one of claims 1 to 7, further comprising a word providing unit that provides the words to be studied. obtaining a first sentence from the dataset that includes a word to be trained;
masking the word in the first sentence and calculating the confidence of the word in the masked portion of the first sentence and the confidence of other candidate words that may be included in the masked portion using an inference model;
setting a predetermined value for distinguishing the word from the candidate word, and outputting the masked sentence as a fill-in-the-blank question if the certainty of the word is greater than the certainty of the candidate word by at least the predetermined value;
A method of creating questions that is carried out by a computer. obtaining a first sentence including a word to be trained from the dataset;
A process of masking the word in the first sentence and calculating the confidence of the word in the masked portion of the first sentence and the confidence of other candidate words that may be included in the masked portion using a prediction model;
a process of setting a predetermined value for distinguishing the word from the candidate word, and outputting the masked sentence as a fill-in-the-blank question if the certainty of the word is greater than the certainty of the candidate word by at least the predetermined value;
A program that causes a computer to execute the following.

Images