JP7729291B2 - Answer generation device, machine learning method, and program - Google Patents

Description

Article 30, paragraph 2 of the Patent Act applies. Published at the 28th Annual Meeting of the Association for Natural Language Processing (Proceedings) on March 7, 2022. Published at the International Conference on Document Understanding (ICDAR 2021) and on the YouTube website on September 17, 2021.

This disclosure relates to a technology that uses question text indicating a question sentence and a document image containing the text as input and generates answer information to the question based on the content of the document image.

Question-answering technologies exist that take question text indicating a question and a document image containing the text as input, and generate answer text to the question based on the contents of the document image (Non-Patent Document 1, Non-Patent Document 2). In these conventional systems, training data consists of a triplet of {document image, question, correct answer information}. Conventional systems first extract text and document regions (title, paragraph, caption, etc.) from the document image, acquiring visual information such as text, coordinates of text regions in the document, and icons. Next, conventional systems train to generate answers using the information acquired from the document image and a given question as input.

Powalski Rafal, Borchmann Lukasz, Jurkiewicz Dawid, Dwojak Tomasz, Pietruszka Michal, Palka Gabriela: Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformer. ICDAR21 Ryota Tanaka, Kyosuke Nishida, Sen Yoshida: VisualMRC: Machine Reading Comprehension on Document Images. AAAI21

However, conventional technologies do not learn how objects and text are arranged within document images, and there remain challenges in understanding document images that include not only text but also objects.

The present invention has been made in consideration of the above points, and aims to generate more accurate answer information by taking into account the positional relationship between objects and text in a document image.

In order to solve the above problem, the invention of claim 1 is an answer generation device that generates answer information to a question based on the content of a document image, and includes an encoding unit that outputs encoded features based on text information in the document image, object information indicating an object in the document image, and the question, a decoding unit that outputs output probability distribution information of the answer information based on the encoded features, and a decoder unit that derives first rectangular selection area information indicating a rectangular area of a predetermined document text selected from rectangular area information indicating rectangular areas of multiple document texts in the document image, and a decoder unit that derives first rectangular selection area information indicating a rectangular area of a predetermined object selected from rectangular area information indicating rectangular areas of objects in the document image. The answer generation device includes: a correct layout relation derivation unit that derives second rectangular selection area information that indicates the layout relationship of the specified document text relative to the specified object, and derives correct layout relation information that indicates the layout relationship of the specified document text relative to the specified object; a layout relation prediction unit that predicts the layout relationship between the specified document text related to the first rectangular selection area information and the specified object related to the second rectangular selection area information based on the encoded features, and outputs predicted layout relation distribution information; and a parameter learning unit that learns model parameters of a neural network based on output probability distribution information of the answer information, correct answer information, the predicted layout relation distribution information, and the correct layout relation information.

As described above, the present invention has the effect of improving the ability to understand the positional relationship between objects and text, thereby enabling the generation of more accurate answer information.

1 is a diagram illustrating the overall configuration of a communication system according to an embodiment of the present invention. 1 is a diagram illustrating an electrical hardware configuration of an answer generation device according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an electrical hardware configuration of a communication terminal according to the present embodiment. FIG. 2 is a functional configuration diagram of the answer generation device in a learning phase according to the first embodiment. 10A and 10B are diagrams illustrating examples of a question sentence, a document image, and correct answer information. FIG. 3 is a detailed configuration diagram of a sequence conversion unit according to the first embodiment. FIG. 2 is a functional configuration diagram of the answer generation device in the inference phase according to the first embodiment. 5 is a flowchart showing a process or operation executed by the answer generation device in a learning phase according to the first embodiment. 5 is a flowchart showing processing or operations executed by the answer generation device in an inference phase according to the first embodiment. FIG. 11 is a functional configuration diagram of the answer generation device in a learning phase according to the second embodiment. FIG. 11 is a conceptual diagram of processing by a correct layout relationship deriving unit according to the second embodiment. FIG. 11 is a detailed configuration diagram of a sequence conversion unit according to the second embodiment. FIG. 11 is a functional configuration diagram of the answer generation device in the inference phase according to the second embodiment. 10 is a flowchart showing a process or operation executed by the answer generation device in a learning phase according to the second embodiment. 10 is a flowchart showing a process or operation executed by the answer generation device in a learning phase according to the second embodiment. 10 is a table showing experimental results according to the first and second embodiments.

The following describes an embodiment of the present invention based on the drawings.

First Embodiment First, a first embodiment of the present invention will be described with reference to Figures 1 to 9. Note that the answer generation device 3a according to the first embodiment provides a specific improvement over the conventional technology that receives question text indicating a question sentence and a document image including the text as input and generates answer information to a question based on the contents of the document image, as described above, and represents an advancement in the technical field related to generation of answer information using a neural network.

[System configuration of the embodiment]
First, the overall configuration of the communication system according to this embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing the overall configuration of the communication system according to this embodiment.

As shown in FIG. 1, the communication system 1 of this embodiment is constructed with an answer generation device 3 and a communication terminal 5. The communication terminal 5 is managed and used by a user. The user refers to the output results of the answer generation device and decides on the next course of action.

In addition, the answer generation device 3 and the communication terminal 5 can communicate via a communication network 100 such as the Internet. The communication network 100 may be connected wirelessly or via a wired connection.

The answer generation device 3 is composed of one or more computers. If the answer generation device 3 is composed of multiple computers, it may be referred to as an "answer generation device" or an "answer generation system."

As shown in Figure 5, even if the text image does not contain correct answer information related to direct numerical information (e.g., "40%), the answer generation device 3 automatically generates arithmetic operation process information showing the process of arithmetic operations using a template such as "100% - *% =" (where * is a variable), thereby improving the ability to understand numerical information. For example, all numerical values extracted from the text image are substituted for "*" in the template.

The communication terminal 5 is a computer, and FIG. 1 shows a laptop computer as an example. In FIG. 1, a user operates the communication terminal 5. Note that the answer generation device 3 may perform processing independently without using the communication terminal 5.

[Hardware configuration]
<Hardware configuration of answer generation device>
Next, the electrical hardware configuration of the answer generation device 3 will be described with reference to Fig. 2. Fig. 2 is a diagram showing the electrical hardware configuration of the answer generation device.

As shown in FIG. 2, the answer generation device 3 is a computer that includes a processor such as a CPU (Central Processing Unit) 301, a ROM (Read Only Memory) 302, a RAM (Random Access Memory) 303, an SSD (Solid State Drive) 304, an external device connection I/F (Interface) 305, a network I/F 306, a media I/F 309, and a bus line 310.

Of these, CPU 301 controls the overall operation of the answer generation device 3. ROM 302 stores programs used to drive CPU 301, such as IPL (Initial Program Loader). RAM 303 is used as a work area for CPU 301.

SSD304 reads and writes various data under the control of CPU301. Note that a HDD (Hard Disk Drive) may be used instead of SSD304.

The external device connection I/F 305 is an interface for connecting various external devices. In this case, external devices include a display, speakers, keyboard, mouse, USB (Universal Serial Bus) memory, and printer.

Network I/F 306 is an interface for data communication via communication network 100.

The media I/F 309 controls the reading and writing (storage) of data from and to a recording medium 309m such as a flash memory. Recording media 309m also includes DVDs (Digital Versatile Discs) and Blu-ray Discs (registered trademarks).

Bus line 310 is an address bus, data bus, etc. that electrically connects each component, such as CPU 301, shown in Figure 2.

<Hardware configuration of communication terminal>
Next, the electrical hardware configuration of the communication terminal 5 will be described with reference to Fig. 3. Fig. 3 is a diagram showing the electrical hardware configuration of the communication terminal.

As shown in FIG. 3, the communication terminal 5 is a computer and includes a CPU 501, ROM 502, RAM 503, SSD 504, an external device connection I/F (Interface) 505, a network I/F 506, a display 507, a pointing device 508, a media I/F 509, and a bus line 510.

Of these, the CPU 501 controls the overall operation of the communication terminal 5. The ROM 502 stores programs used to drive the CPU 501, such as IPL. The RAM 503 is used as a work area for the CPU 501.

The SSD 504 reads and writes various data under the control of the CPU 501. Note that a hard disk drive (HDD) may be used instead of the SSD 504.

The external device connection I/F 505 is an interface for connecting various external devices. In this case, external devices include a display, speakers, keyboard, mouse, USB memory, and printer.

The network I/F 506 is an interface for data communication via the communication network 100.

Display 507 is a type of display means, such as liquid crystal or organic EL (Electro Luminescence), that displays various images.

The pointing device 508 is a type of input means used to select and execute various instructions, select processing targets, move the cursor, etc. If the user uses a keyboard, the function of the pointing device 508 may be turned off.

The media I/F 509 controls the reading and writing (storage) of data from and to a recording medium 509m such as a flash memory. Recording media 509m also includes DVDs and Blu-ray Discs (registered trademarks).

The bus line 510 is an address bus, data bus, etc. that electrically connects the various components, such as the CPU 501 shown in Figure 3.

[Functional configuration of the answer generation device]
The functional configuration of the answer generation device 3a according to this embodiment in the learning phase and inference (prediction) phase will be described. Note that although the functional configuration of the learning phase is shown in Fig. 4 and the functional configuration of the inference phase is shown in Fig. 7, the answer generation device 3a may have all the components of Fig. 4 and Fig. 7 collectively.

<Learning phase functional configuration>
First, the functions of the answer generation device 3a in the learning phase will be described with reference to Fig. 4. Fig. 4 is a functional configuration diagram of the answer generation device in the learning phase according to the first embodiment. Note that the answer generation device 3a is an example of the answer generation device 3.

As shown in FIG. 4, the answer generation device 3a has an input unit 30, a text extraction unit 31a, an object extraction unit 32a, a data expansion unit 33, an encoding unit 35a, a decoding unit 36a, and a parameter learning unit 38a. Each of these units has a function realized by instructions from the CPU 301 in FIG. 2 based on a program. In addition, a machine learning model 41a is constructed in the RAM 303 or SSD 304. The processing of the encoding unit and decoding unit is based on the model parameters of the neural network.

The input unit 30 inputs learning data (question sentence, correct answer information, and document image data) to the answer generation device 3a via the communication terminal 7. Alternatively, the input unit 30 inputs learning data directly to the answer generation device 3a. Figure 5 is a diagram showing examples of question sentences, document images, and correct answer information. For example, as shown in Figure 5, the question sentence is "What is the percentage of cases in which diabetes was prevented?" The document image is the image on the right side of Figure 5. The correct answer information is "40%."

The text extraction unit 31a inputs the document image input by the input unit 30 and outputs the text information in the document image (document text sequence and information on document text rectangular areas).

Therefore, the text extraction unit 31a detects text regions included in the document image, recognizes the text in the detected regions using OCR (Optical Character Recognition), or the like, and outputs text information from the document image. The text information from the document image includes at least a document text sequence and a document text rectangular region. The document text sequence output here is sorted left-to-right and top-to-down according to the text coordinates. Furthermore, the document text rectangular region is represented by, for example, the top-left and bottom-right coordinates of a rectangular region in the document image that indicates "60%." Any process can be used as long as it can output a text sequence and a text rectangular region from the document image. In this embodiment, Tesseract, as shown in Reference 1, is used as an example of the process.
<Reference 1> Google: Tesseract Manual. 2018.(https://github.com/tesseract-ocr/tesseract/blob/master/doc/tesseract.1.asc)
The object extraction unit 32a receives the document image input by the input unit 30 and outputs object information in the document image (object region feature information, object region semantic label, and object rectangular region information).

Therefore, the object extraction unit 32a extracts object information contained in the document image and outputs it as object information in the document image. The object information in the document image includes at least the region feature v ^obj of the object (such as a graph or icon), an object region semantic label which is a label indicating the meaning of the object region, and object rectangular region information which indicates the coordinates of the object region. Any process may be used for these processes as long as the object region feature information, object region semantic label, and object rectangular region information can be acquired from the document image. In this embodiment, Faster-RCNN in Reference 2 is used as an example of the process.
<Reference 2> Shaoqing Ren, Kaiming He, Ross B. Girshick, Jian Sun: Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. NIPS 2015: 91-99
In this embodiment, the region feature v ^obj is represented by a 2048-dimensional vector sequence, the object region semantic labels are represented by 391 types of semantic labels (such as graph or dog) defined in Reference 3, and the rectangular region information is represented by the upper left and lower right coordinates of the region on the image information.
<Reference 3> Spandan Mada, Zoya Bylinskii, Mathew Tanik, Adria Recasens, Kimberli Zhong, Sami Alesheikh, Hanspeter Pfister, Aude Oliva, Fredo Durand: Synthetically Trained Icon Proposals for Parsing and Summarizing Inforgraphics. arXiv:1807.10441
The data expansion unit 33 receives the correct answer information input by the input unit 30 and the document text sequence extracted by the text extraction unit 31a, and outputs arithmetic operation process information indicating the arithmetic operation process.

Therefore, the data expansion unit 33 outputs an arithmetic operation process (e.g., "100% - 60%=") using the correct answer information and the text in the document image.

The processing performed by the data expansion unit 33 is as follows:

S111: The data expansion unit 33 extracts a predetermined number of items of numerical data from the document text sequence. In this embodiment, seven items are extracted as an example. Numerical values written in text (one, two, etc.) are converted into numerical data by the data expansion unit 33. In the example of Figure 5, the following numerical data are extracted: "1.7, 1, 10, 4.3, 3.6, 6, 60."

S112: The data extension unit 33 stores templates indicating three different types of arithmetic operation processes, for example, "A+B," "A-B," and "100-A." Templates indicating arithmetic operation processes other than these can also be used. The data extension unit 33 takes arbitrary numerical values from the numerical data extracted in S111 as A and B and substitutes them into the template indicating the arithmetic operation process. In the example of Figure 5, when A = 60 and B = 6, 66, 54, and 40 are output as the substitution results.

S113: The data expansion unit 33 performs a calculation using the specified template substituted in S112 above, and compares the calculation result (substitution result) with the correct answer information. If the comparison shows that the calculation result and the correct answer information have the same text (value), the template substituted in S112 above is identified as an arithmetic operation process, and arithmetic operation process information indicating this arithmetic operation process is generated and output. In the example of Figure 5, the substitution result of S2 above, "40%", matches the correct answer information, so the arithmetic operation process information is "100% - 60%".

The encoding unit 35a and decoding unit 36a constitute the sequence conversion unit 20a. Figure 6 is a detailed configuration diagram of the sequence conversion unit according to the first embodiment. The sequence conversion unit 20a uses neural network model parameters to generate output sequence information based on input information. The model configuration employs a general encoder-decoder model format that takes a sequence as input and generates a sequence.

As shown in Figure 6, the encoding unit 35a has a text information in document image receiving unit 351, an object information in document image receiving unit 352, and a question sentence receiving unit 353. These constitute the input layer of the neural network.

The text information in document image receiving unit 351 receives text information in the document image (text sequence, rectangular area information of the text). The object information in document image receiving unit 352 receives object information in the document image (object area features, object area semantic labels, rectangular area information of the object area). The question sentence receiving unit 353 receives a question sentence.

Furthermore, the encoding unit 35 a has a plurality of transformer layers 1 (Transformer) 355 ₁ to 355 _L. The plurality of transformer layers 1 (Transformer) 355 ₁ to 355 _L receive the above information from the input layer and sequentially perform the calculations described below.

The decoding unit 36a also has a plurality of transformer layers 1 (Transformer) _365-1 through 365 _-L . The plurality of transformer layers 1 (Transformer) _365-1 through 365- _L receive encoding features from the transformer layer 355- _L of the encoding unit 35a and sequentially perform the calculations described below. The encoding features indicate vectors of a predetermined dimension obtained by converting the document image and question sentence information input to the encoding unit 35a based on the model parameters of the neural network.

The plurality of transformer layers 1 (Transformer) 355 ₁ to 355 _L and the plurality of transformer layers 1 (Transformer) 365 ₁ to 365 _L form intermediate layers of a neural network.

Furthermore, the decoding unit 36a has an answer information output unit 357 and an arithmetic operation process information output unit 358. These constitute the output layer of the neural network. The answer information output unit 357 outputs output probability distribution information of the answer information obtained by the intermediate layer. The arithmetic operation process information output unit 358 outputs output probability distribution information of the arithmetic operation process obtained by the intermediate layer.

As described above, the sequence conversion unit 20a inputs the question sentence, text information in the document image, object information in the document image, answer information, and arithmetic operation process information, and outputs output sequence information. The answer information uses correct answer information in the learning phase, and answer information inferred by the text generation unit 39 in the inference phase. The arithmetic operation process information is arithmetic operation process information output from the data extension unit 33 in the learning phase, and arithmetic operation process information inferred by the text generation unit 39 in the inference phase. The output sequence information indicates output probability distribution information of the answer information and output probability distribution information of the arithmetic operation process. The encoding unit 35a and decoding unit 36a are described in more detail below.

The encoding unit 35a inputs the question sentence, text information in the document image extracted by the text extraction unit 31a, and object information in the document image extracted by the object extraction unit 32a, and outputs encoded features. Specifically, the encoding unit 35a inputs the question sentence, the document text sequence, the document text rectangular region, the object rectangular region information, the object region semantic label, and the object rectangular region information.

For this reason, the encoding unit 35a encodes the text, objects, and questions in the document image. In this case, it is sufficient to input both image and language features and perform general processing when outputting the combined features. An example of the processing is as follows:

S121: The encoding unit 35a divides the question sentence, the document text sequence, and the object region semantic label into token sequences w ^ocr , w ^q , and w ^obj , which are predetermined processing units. In this embodiment, the Byte-level BPE described in Reference 4 is used as an example of the tokenization process. Note that instead of this step, the encoding unit 35a may obtain the token sequences w ^ocr , w ^q , and w ^obj that have been generated in advance by an external unit, for example, the text extraction unit 31a and the object extraction unit 32a.

Furthermore, the processing unit (token) may be anything obtained by dividing (tokenizing) a text into predetermined units. The token sequence resulting from tokenization is a sequence consisting of one or more tokens. The predetermined unit (token) may be, for example, a word or a subword.
<Reference 4> Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever: Language models are unsupervised multitask learners. Technical report, OpenAI, 2019.
S122: The encoding unit 35a prepares the following input token sequence using the subword token sequence w ^ocr , w ^q , and w ^obj and the object region feature v ^obj .

Here, [CLS] is a special token representing a start symbol, and [SEP] is a special token representing an intermediate symbol. In this embodiment, x is a token sequence with a sequence length U=1024.

S123: The encoding unit 35a defines the kth input embedding sequence in the sequence as follows:

Here, in this embodiment, the normalization method shown in Reference 4 is used for LayerNorm.

is the token sequence

into a corresponding D-dimensional vector. In this embodiment, the embedding vector (D=1024) trained according to Reference 5 is used as the initial value and training parameter. Parameters of other pre-trained language models may also be used.
<Reference 5> Jacob Devlin, Ming-Wei Chang, Kanton Lee, Kristina Toutanova: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL19

are two types of embeddings that distinguish between objects and text, and are D-dimensional vector embeddings.

is a process of converting a token into a D-dimensional vector according to its position in the token sequence. In this embodiment, the method used in Reference 5 is used.

is a D-dimensional embedding representing document text rectangular area information and object rectangular area information of a token sequence. In this embodiment, the method used in Non-Patent Document 2 is used. However, rectangular area information corresponding to a question sentence and an object area label is a D-dimensional 0 vector.

S124: The encoding unit 35a encodes the input embedded sequence.

is input to the L-layer Transformer and encoded as

In this embodiment, the pre-trained Transformer shown in Reference 6 is used. Parameters of other pre-trained language models may also be used. Here, in this embodiment, L=12 is used.
<Reference 6> Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E. Hinton: Layer Normalization. Arxiv, 2016.
The decoding unit 36a receives the encoded features output from the encoding unit 35a, as well as the answer information and arithmetic operation process information, and outputs output sequence information. Here, the answer information is tokenized correct answer information (token sequence after tokenization) output from the parameter learning unit 38a in the learning phase, and tokens of answer information inferred by the text generation unit 39 in the inference phase. The arithmetic operation process information is tokenized arithmetic operation process information output from the parameter learning unit 38a in the learning phase, and arithmetic operation process information inferred by the text generation unit 39 in the inference phase. Furthermore, the output sequence information is output probability distribution information of the answer information and the output probability distribution of the arithmetic operation process.

Therefore, the decoding unit 36a outputs probability distributions for the answer and the arithmetic operation process based on the encoding features obtained by the encoding unit 35a. The process is as follows:

S131: In the learning phase, the decoding unit 36a acquires the correct answer information and the token sequence of the arithmetic operation process output by the parameter learning unit 38a for each output step, and creates an output sequence by combining each token sequence. Furthermore, in the inference phase, the decoding unit 36a combines the tokens recursively output for each output step in the text generation unit 39 to create an output sequence. Furthermore, the decoding unit 36a assigns a start symbol and a terminal symbol to the sequence in the learning phase, and only assigns the start symbol to the sequence in the inference phase. In this embodiment, as an example, the start symbol is assigned the symbol [CLS], and the terminal symbol is assigned the symbol [EOS]. Furthermore, in this embodiment, the Byte-level BPE shown in Reference 3 is used as an example of tokenization processing.

S132: The decoding unit 36a converts the output sequence into the following embedding, similar to the encoding unit 35a.

This embedding and encoding feature are input to the L-layer transformer, and the output token representation is

In this embodiment, a pre-trained Transformer shown in Reference 6 is used. Similarly, parameters of other pre-trained language models may be used. Note that T is the length of the output token sequence, and in this embodiment, T=24.

Here,

is the token sequence

into the corresponding D-dimensional vector. In this embodiment, the embedding vector (D=1024) trained according to Reference 5 is used as the initial value and the training parameter. Parameters of other pre-trained language models may also be used.

are two types of embeddings that distinguish between objects and text, and are D-dimensional vector embeddings.

is a process of converting a token into a D-dimensional vector according to its position in the token sequence. In this embodiment, the method described in Reference 5 is used.

S133: The decoding unit 36a converts the representation of the output token into

Based on this, we apply a linear transformation and softmax function to obtain the probability distribution of the t-th word.

(The output probability distribution information of the answer information and the output probability distribution information of the arithmetic operation process) is calculated. t satisfies 0≦t≦T.

The parameter learning unit 38a inputs output sequence information (output probability distribution information of answer information, output probability distribution information of arithmetic operation process) and correct answer information (correct answer information, arithmetic operation process information), and outputs parameter update information, tokenized correct answer information, and tokenized arithmetic operation process information. However, the correct answer information may also be tokenized information.

For this reason, the parameter learning unit 38a performs tokenization processing on the correct answer information (correct answer information, arithmetic operation process information). However, the correct answer information may be input as information that has already been tokenized. Furthermore, the arithmetic operation process information may be information that has been tokenized by another processing unit (for example, the data expansion unit 33). In this embodiment, the Byte-level BPE shown in Reference 3 is used as an example of tokenization processing.

The parameter learning unit 38a calculates the following losses based on the output sequence information and correct answer information, and updates the model parameters of the machine learning model 41a to minimize them:

is a token of correct answer information in the output text.

Furthermore, the parameter learning unit 38a outputs the correct answer information and arithmetic operation process information that have been subjected to the tokenization process.

This concludes the explanation of the functional configuration of the learning phase.

<Functional configuration of the inference phase>
Next, the functions of the answer generation device 3a in the inference phase will be described. Fig. 7 is a functional configuration diagram of the answer generation device in the inference phase.

As shown in FIG. 7, the answer generation device 3a has an input unit 30, a text extraction unit 31a, an object extraction unit 32a, an encoding unit 35a, a decoding unit 36a, a text generation unit 39b, and an output unit 40a. Each of these units has a function realized by instructions from the CPU 301 in FIG. 2 based on a program. In addition, a trained machine learning model 41b is stored in the RAM 303 or the SSD 304. Note that functional components similar to those in the learning phase are assigned the same reference numerals and will not be described again.

The text generation unit 39 inputs output sequence information from the decoding unit 36a and outputs answer information and arithmetic operation process information. In the first embodiment, the output sequence information is output probability distribution information for the answer information and output probability distribution information for the arithmetic operation process.

For this reason, the text generation unit 39 recursively generates answer information and arithmetic operation process information based on the output probability distribution information of the answer information and the output probability distribution information of the arithmetic operation process included in the output sequence information. For each output process, the text generation unit 39 selects the word with the maximum probability distribution, or generates words by sampling according to the probability distribution, and terminates word generation when the end-of-sentence token [EOS] is generated.

The output unit 40a outputs the inference result from the answer generation device 3a based on the answer information and arithmetic operation process information output by the text generation unit 39. Examples of output include displaying the result on a display connected to the external device connection I/F 305 in Figure 2, or transmitting the result to a communication terminal 5 or the like via the network I/F 306.

[Processing or operation of the answer generation device]
Next, the processing or operation of the answer generation device 3a in the learning phase and inference phase will be described with reference to FIGS.

<Processing or Operation in the Learning Phase>
FIG. 8 is a flowchart showing the machine learning method executed by the answer generation device in the learning phase.

S11: The input unit 30 inputs learning data (correct answer information, document image, question text) from the communication terminal 7 or directly.

S12: The text extraction unit 31a extracts text information from the document image, and the object extraction unit 32a extracts object information from the document information.

S13: The data extension unit 33 generates arithmetic operation process information.

S14: The encoding unit 35a generates encoding features.

S15: The decoding unit 36a generates answer information and arithmetic operation process information.

S16: The parameter learning unit 38a calculates the loss and updates the parameters.

S17: The parameter learning unit 38a determines whether there is unprocessed data. If there is unprocessed data (S17; YES), the process returns to S11. On the other hand, if there is no unprocessed data (S17; NO), the learning phase ends.

This concludes the explanation of the learning phase processing or operation.

<Processing or Operation in the Inference Phase>
FIG. 9 is a flowchart showing an answer generation method executed by the answer generation device in the inference phase.

S21: The input unit 30a inputs input data (document image, question text) from the communication terminal 7 or directly.

S22: The text extraction unit 31a extracts text information from the document image, and the object extraction unit 32a extracts object information from the document information.

S23: The encoding unit 35a generates encoding features.

S24: The text generation unit 39 generates answer information and arithmetic operation process information.

S25: The text generation unit 39 determines whether a sentence-end symbol has been output. If a sentence-end symbol has not been output (S25; NO), the process returns to step S24. On the other hand, if a sentence-end symbol has been output (S25; YES), the process proceeds to step S26.

S26: The output unit 40 outputs information on the inference result, including answer information and arithmetic operation process information.

This concludes the explanation of the processing or operation of the inference phase.

[Experimental conditions and results of the first embodiment]
Next, the experimental conditions and results of this embodiment will be described.

<Experimental conditions>
(Experimental Setup)
The batch size was set to 64 for both pre-training and fine-tuning, and the answer generation device 3a was trained for 30 epochs. Optimization was performed using Adam (Reference 8), with a learning rate of 3e-5. The maximum lengths of v ^obj , w ^obj , w ^ocr , and w ^task were set to 36, 20, 430, and 40, respectively. For evaluation on development and test data, BERT-{base,large} was used as the initial weight values during pre-training. Additionally, for large, the hyperparameters were set based on the base experiment.
<Reference 8> Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
(Evaluation indicators)
We use the ANLS (Reference 9) (average edit distance between predicted sentences and the set of ground truth sentences) adopted in the ICDAR 2021 Competition. ANUM denotes the ANLS when limited to examples requiring arithmetic operations (17.4% of the development data).
<Reference 9> Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomezi Bigorda, Marcal Rusinol, CV Jawahar, Ernest Valveny, and Dimosthenis Karatzas. Scene text visual question answering.In ICCV, pp. 4290-4300, 2019.
<Experimental Results>
19 is a table showing the experimental results according to the first and second embodiments. As shown in the second row "BERT w/o ADA" and the fifth row "IG-BERT w/o SRP + ADA" in the table, by using the concept of arithmetic operation process, the values of ANLS and ANUM were reduced compared to the conventional case where the concept was not used.

[Effects of the First Embodiment]
As described above, according to the first embodiment, in order for the answer generation device 3a to generate arithmetic operation processes required for answer information, the sequence conversion unit 20a uses the machine learning model 41a to perform learning in the learning phase to generate arithmetic operation processes required for answer information. This improves the answer generation device 3a's ability to understand numerical data included in document images, making it possible to generate answer information involving arithmetic operations with higher performance than conventional techniques.

The data extension unit also prepares multiple templates and automatically generates arithmetic operation process information by using a predetermined template. This reduces the cost of creating training data compared to manually creating correct answer information for arithmetic operation processes.

Furthermore, understanding the positional relationships between objects (icons, graphs, etc.) and text (e.g., 60%) contained in a document image is one of the most important factors for a system to understand document images and perform question-answering. The answer generation device 3a of this embodiment is able to understand the many visually represented documents (infographics, PDF documents, HTML documents, etc.) that exist in the real world as knowledge sources and perform question-answering involving arithmetic operations.

Second Embodiment Next, a second embodiment of the present invention will be described using FIGS. 10 to 15 . Compared to conventional techniques that use question text indicating a question sentence and a document image containing the text as input and generate answer information to a question based on the content of the document image, the answer generation device 3 b according to the second embodiment of the present invention provides specific improvements and represents an advancement in the technical field related to generating answer information using a neural network. The overall configuration and hardware configuration of the communication system according to this embodiment are similar to those of the first embodiment, and therefore will not be described here. Furthermore, while FIG. 10 shows the functional configuration of the learning phase and FIG. 15 shows the functional configuration of the inference phase, the answer generation device 3 b may also include the components of FIGS. 10 and 13 collectively.

[Functional configuration of the answer generation device]
The functional configuration of the answer generation device 3b according to this embodiment in the learning phase and inference (prediction) phase will be described.

<Learning phase functional configuration>
The functions of the answer generation device 3b in the learning phase will be described with reference to Fig. 10. Fig. 10 is a functional configuration diagram of the answer generation device in the learning phase according to the second embodiment. Note that the answer generation device 3b is an example of the answer generation device 3.

As shown in FIG. 4 , the answer generation device 3b includes an input unit 30, a text extraction unit 31b, an object extraction unit 32b, a correct layout relation derivation unit 34, an encoding unit 35b, a decoding unit 36b, a layout relation prediction unit 37, and a parameter learning unit 38b. These units are functions implemented by commands from the CPU 301 in FIG. 2 based on a program. A machine learning model 42a and a machine learning-completed machine learning model (W ^srp ) 43 are also built in the RAM 303 or the SSD 304. The input unit 30, the text extraction unit 31b, the object extraction unit 32b, and the encoding unit 35b perform the same processes as the input unit 30, the text extraction unit 31a, the object extraction unit 32a, and the encoding unit 35b in the first embodiment, respectively, and therefore description thereof will be omitted.

The input unit 30 inputs the learning data (data on the question, correct answer information, and document image) to the answer generation device 3b via the communication terminal 7. Alternatively, the input unit 30 inputs the learning data directly to the answer generation device 3a. Figure 11 is a conceptual diagram of the processing of the correct layout relation derivation unit according to the second embodiment. For example, as shown in Figure 11, the question is "What number is written above the woman's icon?" The document image is the same as in Figure 5. The correct answer information is "3.6%."

The correct layout relation derivation unit 34 inputs the rectangular area information of the document text output from the text extraction unit 31b and the rectangular area information of the object output from the object extraction unit 32b, and outputs rectangular selection area information of the document text (an example of first rectangular selection area information), rectangular selection area information of the object (an example of second rectangular selection area information), and correct layout relation information.

Therefore, the correct layout relation derivation unit 34 randomly selects one rectangular area information of a predetermined object and one rectangular area information of a predetermined document text from the rectangular area information of the object and the rectangular area information of the document text, respectively, and defines and outputs a correct layout relation consisting of M classes (any number as long as M=1 or greater). In this embodiment, the layout relation of M=12 defined in Reference 7 (see the lower right of FIG. 11 ) is used. Here, an example is shown in which, regarding the layout relations of dashed-line rectangular areas with respect to solid-line rectangular areas, a layout relation classification task of 11 classes and one class in which the rectangular areas are too far apart are solved, for a total of 12 classes.
<Reference 7> Ting Yao, Yingwei Pan, Yehao Li, Tao Mei: Exploring visual relationship for image captioning. ECCV18
The encoding unit 35b and decoding unit 36b constitute a sequence transformation unit 20b. Fig. 12 is a detailed block diagram of the sequence transformation unit according to the second embodiment. The sequence transformation unit 20b uses neural network model parameters to generate output sequence information based on input information. The model configuration employs a general encoder-decoder model format for generating sequences using sequences as input.

The sequence conversion unit 20b does not have the arithmetic operation process information output unit 358 of the sequence conversion unit 20a in the first embodiment, but does have the other components. Therefore, the same components are assigned the same reference numerals and their descriptions are omitted.

The decoding unit 36b basically performs the same processing as the decoding unit 36a, but since the second embodiment does not have a data extension unit 33, no arithmetic operation process information is input, and no output of output probability distribution information for the arithmetic operation process is performed.

The layout relationship prediction unit 37 inputs the rectangular selection area information of the document text and the rectangular selection area information of the object derived by the correct layout relationship derivation unit 34, as well as the encoded features output from the encoding unit 35b, and outputs predicted layout relationship distribution information.

Therefore, the layout relationship prediction unit 37 predicts the layout relationship between objects and text in the image document based on the encoded features, making it possible to generate answers that take layout relationships into account. The process of the layout relationship prediction unit 37 is as follows:

S211: The layout relationship prediction unit 37 acquires feature quantities e obj and e text corresponding to the rectangular selection area information of the document text and the rectangular selection area information of the object output by the correct layout relationship derivation unit 34, from the encoded feature quantities corresponding to the ^object and the ^text (vectors of a predetermined dimension obtained by converting the information on the document image and question sentence input to the encoding unit 35 a based on the model parameters of the neural network).

S212: The layout relationship prediction unit 37 calculates and outputs predicted layout relationship distribution information based on e ^obj and e ^text as follows:

Since the second embodiment does not include the data extension unit 33, the parameter learning unit 38b does not receive arithmetic operation process information and does not output output probability distribution information of the arithmetic operation process. The parameter learning unit 38b receives the correct answer information output from the input unit 30, the output sequence information (output probability distribution information of answer information) output from the decoding unit 36b, the predicted layout relationship distribution information output from the layout relationship prediction unit 37, and the correct layout relationship information output from the correct layout relationship derivation unit, and outputs parameter update information and tokenized correct answer information. However, the correct answer information may be information that has been tokenized.

For this reason, the parameter learning unit 38b performs tokenization processing on the correct answer information. However, the correct answer information may be input as information that has already been tokenized. In this embodiment, the Byte-level BPE shown in Reference 3 is used as an example of tokenization processing.

The parameter learning unit 38b calculates the following losses based on the output sequence information and correct answer information, and updates the model parameters of the machine learning model 42a to minimize them:

L ^ans is as follows:

is a token of correct answer information in the output text.

L ^srp is as follows: q ^srp is a one-hot vector of the correct configuration relationship.

Furthermore, the parameter learning unit 38b outputs the correct answer information that has been tokenized.

This concludes the explanation of the functional configuration of the learning phase.

<Functional configuration of the inference phase>
Next, the functions of the answer generation device 3b in the inference phase will be described. Fig. 13 is a functional configuration diagram of the answer generation device in the inference phase.

As shown in FIG. 7, the answer generation device 3b has an input unit 30, a text extraction unit 31b, an object extraction unit 32b, an encoding unit 35b, a decoding unit 36b, a text generation unit 39b, and an output unit 40b. Each of these units has a function realized by instructions from the CPU 301 in FIG. 2 based on a program. In addition, the RAM 303 or the SSD 304 stores a trained machine learning model 42b. Note that functional components similar to those in the learning phase are assigned the same reference numerals and will not be described again.

The text generation unit 39b inputs the output sequence information from the decoding unit 36a and outputs answer information. In the second embodiment, the output sequence information is output probability distribution information for the answer information.

For this reason, the text generation unit 39b recursively generates answer information based on the output sequence information. For each output step, the text generation unit 39b selects the word with the maximum probability distribution, or generates words by sampling according to the probability distribution, and ends word generation when the end-of-sentence token [EOS] is generated.

The output unit 40b outputs the inference results from the answer generation device 3b based on the answer information output by the text generation unit 39. Examples of output include displaying the results on a display connected to the external device connection I/F 305 in Figure 2, or transmitting the results to a communication terminal 5 or the like via the network I/F 306.

[Processing or operation of the answer generation device]
Next, the processing or operation of the answer generation device 3b in the learning phase and inference phase will be described with reference to FIGS.

<Processing or Operation in the Learning Phase>
FIG. 14 is a flowchart showing the machine learning method executed by the answer generation device in the learning phase.

S31: The input unit 30 inputs learning data (correct answer information, document image, question text) from the communication terminal 7 or directly.

S32: The text extraction unit 31b extracts text information from the document image, and the object extraction unit 32b extracts object information from the document information.

S33: The correct layout relation derivation unit 34 derives the correct layout relation.

S34: The encoding unit 35b generates encoding features.

S35: The decoding unit 36b predicts the answer information and outputs output probability distribution information for the answer information, and the layout relationship prediction unit 37 predicts the layout relationship information and outputs predicted layout relationship distribution information.

S36: The parameter learning unit 38b calculates the loss and updates the parameters.

S37: The parameter learning unit 38b determines whether there is unprocessed data. If there is unprocessed data (S37; YES), the process returns to S31. On the other hand, if there is no unprocessed data (S37; NO), the learning phase ends.

This concludes the explanation of the learning phase processing or operation.

<Processing or Operation in the Inference Phase>
FIG. 15 is a flowchart showing an answer generation method executed by the answer generation device in the inference phase.

S41: The input unit 30 inputs input data (document image, question text) from the communication terminal 7 or directly.

S42: The text extraction unit 31b extracts text information from the document image, and the object extraction unit 32a extracts object information from the document information.

S43: The encoding unit 35b generates encoding features.

S44: The text generation unit 39 generates answer information.

S45: The text generation unit 39 determines whether a sentence-end symbol has been output. If a sentence-end symbol has not been output (S45; NO), the process returns to step S44. On the other hand, if a sentence-end symbol has been output (S45; YES), the process proceeds to step S46.

S46: The output unit 40b outputs information about the inference result, including the answer information.

This concludes the explanation of the processing or operation of the inference phase.

[Experimental conditions and results]
Next, the experimental conditions and results of the second embodiment will be described. Note that the experimental conditions of the second embodiment are the same as those of the first embodiment, and therefore will not be described here.

<Experimental Results>
As shown in FIG. 19, as shown in the fourth row of the table, "IG-BERT w/o SRP," by using the concept of layout relationship, the values of ANLS and ANUM were reduced compared to the conventional case where the concept was not used.

[Effects of the second embodiment]
As described above, according to the second embodiment, in the answer generation device 3b, in addition to machine learning for generating answer information, the layout relationship prediction unit 37 and the parameter learning unit 38b perform learning on the layout relationship between objects and text in a document image. This makes it possible to generate answer information that takes the layout relationship into consideration. Therefore, the answer generation device 3b's ability to understand the layout relationship between objects and text is improved, making it possible to understand document images and respond to questions with higher performance than conventional techniques.

As a result, the answer generation device 3b can understand the many visually represented documents (infographics, PDF documents, HTML documents, etc.) that exist in the real world as knowledge sources and perform question and answering.

Supplementary Information The present invention is not limited to the above-described embodiment, and may have the following configurations or processes (operations).

(1) The answer generation devices 3a and 3b can be realized using a computer and a program, but this program can also be recorded on a (non-temporary) recording medium or provided via the communication network 100.

(2) In the above embodiment, a notebook computer is shown as an example of the communication terminal 5, but this is not limited to this. For example, the communication terminal 5 may be a desktop computer, a tablet terminal, a smartphone, a smartwatch, a car navigation device, a refrigerator, a microwave oven, etc. (3) Each CPU 301, 501 may be a single CPU or multiple CPUs.

Supplementary Notes The above-described embodiment can also be expressed as the following inventions.

[Additional Note 1]
An answer generation device having a processor that generates answer information to a question based on the content of a document image,
The processor:
an encoding process for outputting encoded features based on text information in the document image, object information indicating objects in the document image, and the question;
a decoding process for outputting output probability distribution information of the answer information based on the encoded feature;
a correct layout relationship derivation process that derives first rectangular selection area information indicating a rectangular area of a predetermined document text selected from rectangular area information indicating rectangular areas of a plurality of document texts in the document image, derives second rectangular selection area information indicating a rectangular area of a predetermined object selected from rectangular area information indicating rectangular areas of one or more objects in the document image, and derives correct layout relationship information indicating a layout relationship of the predetermined document text with respect to the predetermined object;
a layout relationship prediction unit that predicts a layout relationship between the predetermined document text related to the first rectangular selection area information and the predetermined object related to the second rectangular selection area information based on the encoded feature, and outputs predicted layout relationship distribution information;
a parameter learning process for learning model parameters of a neural network based on output probability distribution information of the answer information and correct answer information (so that the output probability distribution information of the answer information approaches the correct answer information), and based on the predicted layout relationship distribution information and the correct layout relationship information (so that the predicted layout relationship distribution information approaches the correct layout relationship information);
An answer generation device that executes the above.

[Additional note 2]
The answer generation device described in Appendix 1, wherein the correct layout relationship derivation process includes a process of deriving the correct layout relationship information by predicting to which of a plurality of classes representing a predetermined layout relationship the layout relationship between the specified document text related to the first rectangular selection area information and the specified object related to the second rectangular selection area information falls.

[Additional note 3]
3. The response generation device according to claim 1 or 2,
the encoding process and the decoding process include a process of outputting output probability distribution information of the answer information based on the document image and the question sentence by using the model parameters previously learned by the parameter learning unit,
the processor executes a text generation process to generate the answer information based on output probability distribution information of the answer information.
Answer generator.

[Additional note 4]
A machine learning method executed by a processor that generates answer information to a question based on the content of a document image,
The processor:
outputting an encoded feature, which is a vector indicating a region in the document image corresponding to the question, based on text information in the document image, object information indicating an object in the document image, and the question;
outputting output probability distribution information of the answer information based on the encoded feature;
deriving first rectangular selection area information indicating a rectangular area of a predetermined document text selected from rectangular area information indicating rectangular areas of a plurality of document texts in the document image; deriving second rectangular selection area information indicating a rectangular area of a predetermined object selected from rectangular area information indicating rectangular areas of one or more objects in the document image; and deriving correct layout relationship information indicating a layout relationship of the predetermined document text with respect to the predetermined object;
predicting a layout relationship between the predetermined document text related to the first rectangular selection area information and the predetermined object related to the second rectangular selection area information based on the encoded feature, and outputting predicted layout relationship distribution information;
performing machine learning of model parameters of a neural network based on output probability distribution information of the answer information, correct answer information, the predicted layout relationship distribution information, and the correct layout relationship information;
Machine learning methods.

[Additional Note 5]
A non-transitory recording medium having recorded thereon a program for causing a computer to implement the answer generation device according to any one of appended items 1 to 3.

1 Communication system 3 Answer generation device 3a Answer generation device 3b Answer generation device 5 Communication terminal 30 Input unit 31a, 31b Text extraction unit 32a, 32b Object extraction unit 33 Data expansion unit 34 Correct layout relationship derivation unit 35a, 35b Encoding unit 36a, 36b Decoding unit 37 Layout relationship prediction unit 38a, 38b Parameter learning unit 39a, 39b Text generation unit 40a, 44b Output unit 20a Sequence conversion unit 20b Sequence conversion unit 41a Machine learning model 41b Trained machine learning model 42a Machine learning model 42b Trained machine learning model

Claims (5)

An answer generation device that generates answer information to a question based on the content of a document image,
an encoding unit that outputs encoded features based on text information in the document image, object information indicating an object in the document image, and the question;
a decoding unit that outputs output probability distribution information of the answer information based on the encoded feature;
a correct layout relation derivation unit that derives first rectangular selection area information indicating a rectangular area of a predetermined document text selected from rectangular area information indicating rectangular areas of a plurality of document texts in the document image, derives second rectangular selection area information indicating a rectangular area of a predetermined object selected from rectangular area information indicating rectangular areas of objects in the document image, and derives correct layout relation information indicating a layout relation of the predetermined document text with respect to the predetermined object;
a layout relationship prediction unit that predicts a layout relationship between the predetermined document text related to the first rectangular selection area information and the predetermined object related to the second rectangular selection area information based on the encoded feature, and outputs predicted layout relationship distribution information;
a parameter learning unit that learns model parameters of a neural network based on output probability distribution information of the answer information, correct answer information, the predicted layout relationship distribution information, and the correct layout relationship information;
An answer generation device having the above. The answer generation device of claim 1, wherein the correct layout relationship derivation unit derives the correct layout relationship information by predicting to which of multiple classes representing predetermined layout relationships the layout relationship between the specified document text related to the first rectangular selection area information and the specified object related to the second rectangular selection area information corresponds. 3. The response generation device according to claim 1,
the encoding unit and the decoding unit include a process of outputting output probability distribution information of the answer information based on the document image and the question sentence, using the model parameters previously learned by the parameter learning unit;
an answer generation device comprising: a text generation unit that generates the answer information based on output probability distribution information of the answer information; A machine learning method executed by an answer generation device that generates answer information to a question based on the content of a document image,
The answer generation device
outputting encoded features based on text information in the document image, object information indicating an object in the document image, and the question;
outputting output probability distribution information of the answer information based on the encoded feature;
deriving first rectangular selection area information indicating a rectangular area of a predetermined document text selected from rectangular area information indicating rectangular areas of a plurality of document texts in the document image; deriving second rectangular selection area information indicating a rectangular area of a predetermined object selected from rectangular area information indicating rectangular areas of objects in the document image; and deriving correct layout relationship information indicating a layout relationship of the predetermined document text with respect to the predetermined object;
predicting a layout relationship between the predetermined document text related to the first rectangular selection area information and the predetermined object related to the second rectangular selection area information based on the encoded feature, and outputting predicted layout relationship distribution information;
learning model parameters of a neural network based on output probability distribution information of the answer information, correct answer information, the predicted layout relationship distribution information, and the correct layout relationship information;
Machine learning methods. A program that causes a computer to implement the answer generation device described in claim 1.