JP7724676B2 - Information processing method, program, and information processing device - Google Patents

Description

The present invention relates to an information processing method, a program, and an information processing device.

An information processing method for recognizing information from documents containing accounting information, etc., has been proposed. (For example, see Patent Document 1.)

Japanese Patent Application Laid-Open No. 2021-34030

In areas of a document where text spans multiple lines, line break symbols are inserted between characters that span lines in order to recognize line breaks. As a result, if an item spans multiple lines, the text on the line above and the text on the line below may be recognized as separate items.

The purpose of this disclosure is to provide an information processing method that can plausibly recognize a string of characters for each item in a document, even if there are line breaks between characters within the item.

The information processing method disclosed herein acquires a group of character strings that includes multiple character strings located across multiple lines within one area of a document and codes that indicate divisions inserted between the character strings, and deletes the codes that should join the preceding and following character strings based on the information contained in the area.

The information processing method disclosed herein determines whether the character strings before and after the code should be combined based on the character strings before and after the code, and if it is determined that the character strings before and after the code should be combined, deletes the code.

The information processing method disclosed herein references a table that records specific character strings and determines whether the character strings before and after the code should be combined.

The information processing method disclosed herein inputs multiple character strings into a learning model that has been trained to output the probability that the character strings before and after the code should be combined when multiple character strings are input, and outputs the probability.

The information processing method disclosed herein compares the probability with a threshold and deletes the code based on the comparison result between the probability and the threshold.

The information processing method disclosed herein inputs the group of character strings into a learning model that has been trained to output a group of character strings from which the code that should join the preceding and following character strings has been deleted when the group of character strings is input, and outputs a group of character strings from which the code that should join the preceding and following character strings has been deleted.

The information processing method disclosed herein removes the code based on information contained in a related region related to the region containing the character string.

The program disclosed herein causes a computer to execute the steps of: acquiring a string group including multiple strings of characters located across multiple lines in one area of a document and a code indicating a division inserted between the strings; and deleting the code if the strings before and after the code should be combined based on the information contained in the area.

The information processing device disclosed herein includes an acquisition unit that acquires a group of character strings including multiple character strings located across multiple lines within one area of a document and a code that indicates a division inserted between the character strings, and a deletion unit that deletes the code when the character strings before and after the code should be combined based on the information contained in the area.

In an information processing method according to an embodiment of the present disclosure, even if there are line breaks between characters within an item in a single area of a document, the character string can be plausibly recognized for each item.

1 is a block diagram of an information processing apparatus according to a first embodiment. FIG. 2 is an explanatory diagram illustrating an example of a document to be read. 10 is a flowchart illustrating an example of a processing procedure performed by a processing unit of the information processing apparatus according to the first embodiment. FIG. 10 is an explanatory diagram showing a character string recognized from within a region of interest of a document. FIG. 2 is an explanatory diagram showing a processing result output by a processing unit of the information processing device. FIG. 10 is a block diagram of an information processing apparatus according to a second embodiment. FIG. 10 is an explanatory diagram of a learning model according to the second embodiment. 10 is a flowchart illustrating an example of a processing procedure performed by a processing unit of an information processing apparatus according to a second embodiment. FIG. 10 is an explanatory diagram of a learning model according to the third embodiment. 11 is a flowchart illustrating an example of a processing procedure performed by a processing unit of an information processing apparatus according to a third embodiment. FIG. 2 is an explanatory diagram showing an example of a region of interest and a related region related to the region of interest.

(Embodiment 1)
Hereinafter, a first embodiment will be described with reference to the drawings. Fig. 1 is a block diagram of an information processing device according to the first embodiment. The information processing device 1 is a computer including a processing unit 11, a storage unit 12, and an input/output I/F 13. The information processing device 1 is, for example, a server computer, a personal computer, a tablet computer, a smartphone, or a cloud server connected via an external network.

The processing unit 11 is a CPU (Central Processing Unit) or an MPU (Micro Processing Unit).
The processing unit 11 reads out and executes a program P and data stored in advance in the storage unit 12, thereby carrying out various control processes, arithmetic processes, and the like.

The storage unit 12 includes a volatile storage area such as a static random access memory (SRAM), a dynamic random access memory (DRAM), or a flash memory, and a non-volatile storage area such as an EEPROM or a hard disk. The storage unit 12 pre-stores a program P and data to be referenced during processing. The program P may be a program stored in a readable recording medium 12a that is read by the processing unit 11 and stored therein. Alternatively, the program P may be a program downloaded from an external computer (not shown) connected to a communication network (not shown) and stored in the storage unit 12.
The storage unit 12 includes an item table 121. The item table 121 stores, for example, item strings, which are predetermined character strings representing items described in the financial statements to be read. The item table 121 may register, as item strings, nouns extracted from a group of books in the same field as the document to be read, or nouns selected by experts in the field to which the document to be read belongs. The item table 121 is referenced by the processing unit 11 during the pattern matching process described below.

The input/output I/F 13 is a group of interfaces for communicating with external devices of the information processing device 1, and is a standardized communication interface such as USB or DSUB. The input/output I/F 13 is connected to an input unit 131 and an output unit 132. The input unit 131 is, for example, a scanner that reads documents on paper, converts them into document files containing layout information such as PDF, and outputs them to the processing unit 11. The input unit 131 may also acquire document files from an external network or external device (not shown), and output the acquired document files to the processing unit 11.

The processing unit 11 performs various processes on the document file acquired from the input unit 131 and outputs the processing results to the output unit 132. The processes performed by the processing unit 11 will be described later. The output unit 132 is, for example, a display, and acquires the processing results from the processing unit 11 and notifies the user by displaying the processing results. The output unit 132 may also output the processing results acquired from the processing unit 11 to an external network or external device (not shown).

Figure 2 is an explanatory diagram showing an example of a document to be read. The document to be read is, for example, a financial statement or accounting table, a document containing areas composed of columns and rows. Based on the position information contained in the layout information of the document file, the processing unit 11 of the information processing device 1 designates the area from which information needs to be acquired as the area of interest and performs the processing described below. The area to be designated as the area of interest may be determined based on information entered by the user into the input unit 131, or each area included in a line containing a predetermined title in the topmost line may be designated as the area of interest. Below, the area surrounded by a dashed line in Figure 2 will be described as the area of interest. For example, within the area of interest, there is an item called "Short-term loans to affiliated companies," which spans multiple lines, with "Short-term loans to affiliated companies" and "Loans" divided into upper and lower lines. In this way, when one item spans multiple lines, the character strings in the upper and lower lines may be recognized as separate items. That is, "Short-term loans to affiliated companies" and "Loans" may each be recognized as a single item. By performing the processing described below, the information processing device 1 recognizes that "short-term loans to affiliated companies" is a single item.

Fig. 3 is a flowchart showing an example of a processing procedure by the processing unit of the information processing apparatus according to embodiment 1. Fig. 4 is an explanatory diagram showing a character string recognized from within a region of interest of a document. Fig. 5 is an explanatory diagram showing a processing result output by the processing unit of the information processing apparatus.
The processing unit 11 of the information processing device 1 acquires a read document file from the input unit 131 (S1). As shown in Fig. 4, the processing unit 11 extracts character strings in a region of interest in the document file acquired from the input unit 131 using OCR (Optical Character Recognition) or the like (S2), and inserts a code indicating a division (e.g., <SEP>) between characters that span lines (S3). Hereinafter, this code will be referred to as a division code, and the character strings in the region of interest will be arranged from the top line, with the division codes inserted between the character strings, forming a group of character strings.

As shown in Figure 5, the processing unit 11 of the information processing device 1 refers to the item table 121 and performs pattern matching on the combined string formed by combining the character strings before and after the division code (S4). If an item string matching the combined character string formed by combining the character strings before and after the division code is registered in the item table 121, it determines that the character strings before and after the division code should be combined and deletes the division code (S5). As mentioned above, the item table 121 stores, for example, item strings to be written on the financial statement to be read. The processing unit 11 performs pattern matching by comparing the combined character string formed by combining the character strings before and after each division code with the item names registered in the item table 121 in a brute-force manner.
For example, as shown in Figure 5, if the item name "short-term loan to affiliated company" which combines "short-term loan to affiliated company" and "loan" is registered in the item table 121, the processing unit 11 determines that the two character strings should be combined and deletes the division code inserted between "short-term loan to affiliated company" and "loan".

Furthermore, the processing unit 11 recognizes each item based on the division codes remaining after the above processing, converts it into a table format with one item per line (S6), and outputs it to the output unit 132 (S7). Note that the format in which the processing unit 11 outputs the group of character strings with division codes inserted to the output unit is not limited to this, and it may also be converted into a format such as HTML, XML, or CSV and output to the output unit 132.

When nouns are extracted from a group of books in the same field as the document to be read and registered in the item table 121, duplication may be allowed. That is, the same number of nouns as the number of nouns listed in the group of books are registered as item strings in the item table 121. At this time, in S5, if the combined string formed by combining the strings before and after the division code is an item string that has been registered a certain number of times in the item table 121, the processing unit 11 deletes the division code.

The above method makes it possible to recognize each item with high plausibility, even when a single item spans multiple lines within the document's focus area.

(Embodiment 2)
The present invention according to embodiment 2 will be described below. Of the configurations according to embodiment 2, the same configurations as those in embodiment 1 are given the same reference numerals, and detailed description thereof will be omitted.
The processing unit 11 according to the first embodiment determines the divided code to be deleted by referring to the item table 121. However, it is also possible to use a learning model to calculate the probability of combining the characters before and after the divided code into one item (combination probability), and determine the divided code to be deleted based on this probability. The combination probability represents a probability, and its value ranges from 0 to 1.

Figure 6 is a block diagram of an information processing device according to embodiment 2. The memory unit 12 of the information processing device 1 according to embodiment 2 stores entity files (neural network (NN) instance files) that constitute the learning model 122 described below. These entity files may be configured as part of a program.

FIG. 7 is an explanatory diagram of a learning model according to the second embodiment. The learning model 122 according to the second embodiment is, for example, a model that extracts features of a string using a neural network. The input layer included in the learning model 122 has a plurality of neurons that receive, as input data, information about a divided code inserted into a group of strings recognized from a document file by the processing unit 11 and the strings before and after the divided code, and passes the input string to the intermediate layer. The intermediate layer has a plurality of neurons that extract features of the string and passes the extracted features to the output layer. The output layer has one or more neurons that output the combination probability of the divided codes included in the string, and outputs the combination probability of each divided code based on the features output from the intermediate layer. Alternatively, the learning model 122 may input the features output from the intermediate layer into an SVM (support vector machine) and output the combination probability. A neural network (learning model 122) trained using training data is expected to be used as a program module that is part of artificial intelligence software. The learning model 122 is used in the information processing device 1 that includes the processing unit 11 and the storage unit 12 as described above, and a neural network system is configured by being executed in the information processing device 1 that has such calculation processing capabilities. That is, the processing unit 11 of the information processing device 1 performs calculations to extract feature amounts of character strings before and after a divided code input to the input layer in accordance with instructions from the learning model 122 stored in the storage unit 12, and outputs the combination probability of each divided code from the output layer.
The learning model 122 may be a recurrent neural network (RNN), a long short-term memory (LSTM), a transformer, etc. Additionally, it may be a pre-trained model such as a bidirectional encoder representations from transformers (BERT).

For example, the training data may be a group of character strings recognized from within a region of interest, into which split codes labeled with a connection probability by the user are inserted. In this case, training data is created by labeling split codes between item names with a connection probability of 0, and split codes between characters within item names with a connection probability of 1.
Furthermore, the storage unit 12 according to the second embodiment may include an item table 121, and training data may be created using the item table 121. Specifically, a plurality of character strings are prepared by randomly inserting split codes labeled with a combination probability of 1 between characters in the item names registered in the item table 121, without allowing overlaps. Next, a split code labeled with a combination probability of 0 is inserted between the plurality of prepared character strings, and the plurality of prepared character strings are combined. Data of the group of character strings created in this manner is used as training data.
The processing unit 11 reads the above-mentioned training data, compares the predicted value of the combination probability of the divided code with the combination probability labeled on the divided code, which is the correct value, and learns the weights in the intermediate layer so that the difference is small, thereby generating a learning model 122.

As shown in Figure 7, the processing unit 11 extracts character strings within the region of interest in the financial statement (see Figure 2) using OCR or the like, inputs information about the split codes inserted between the character strings and the character strings before and after the split codes into the learning model 122, and outputs the probability of combining each split code.

If the combination probability of a division code is equal to or greater than a threshold, the processing unit 11 determines that the character strings before and after the division code should be combined, and deletes the division code. The threshold is, for example, 0.5. Therefore, in Figure 7, the division code inserted between "Affiliate Company Short-Term" and "Loan" is deleted.
The processing unit 11 recognizes the items based on the division codes remaining after the above-mentioned processing, converts the items into a table format in which one item is written per line, and outputs the table to the output unit 132 (see FIG. 5).

Figure 8 is a flowchart showing an example of a processing procedure performed by the processing unit of the information processing device according to the second embodiment. The processing unit 11 acquires a read document file from the input unit 131 (S11). The processing unit 11 extracts character strings in a region of interest within the document file (S12) and inserts split codes at line break positions (S13). The processing unit 11 inputs character strings within the group of character strings into a learning model (S14) and acquires the combination accuracy of the split codes (S15). The processing unit 11 determines whether the combination accuracy is equal to or greater than a threshold (S16). If the combination accuracy is equal to or greater than the threshold (S16: YES), the processing unit 11 deletes the split codes (S17) and converts the group of character strings into a table format (S18). If the combination accuracy is less than the threshold (S16: NO), the processing unit 11 proceeds to S18. The processing unit 11 outputs the group of character strings converted into a table format to the output unit 132 (S19) and ends the processing.

The learning model 122 according to the second embodiment calculates the probability that the combined string formed by combining the character strings before and after the split code is one item (combined probability), but it may also calculate the probability that the combined string formed by combining the character strings before and after the split code is not one item (non-combined probability). In this case, the processing unit 11 deletes split codes whose non-combined probability is less than a threshold.

(Embodiment 3)
The present invention according to embodiment 3 will be described below. Of the configurations according to embodiment 3, the same configurations as those in embodiment 2 are given the same reference numerals, and detailed description thereof will be omitted.
The learning model 122 of embodiment 2 calculates the probability of combining divided codes based on the characteristics of the string, and the processing unit 11 determines the divided code to be deleted based on the probability of combining, while the learning model 122 of embodiment 3 outputs a group of strings from which divided codes that become one item when the preceding and following strings are combined are deleted, based on the characteristics of the string.

FIG. 9 is an explanatory diagram of a learning model according to the third embodiment. The learning model according to the third embodiment is a model capable of outputting a group of character strings or sentences, such as an RNN, an LSTM, or a Transformer. The input layer included in the learning model 122 has a plurality of neurons that accept a group of character strings into which splitting codes have been inserted, and passes the input group of character strings to the intermediate layer. The intermediate layer has a plurality of neurons that extract features of the character strings, and passes the extracted features to the output layer. The output layer has one or more neurons that output a group of character strings from which splitting codes have been deleted, from which the preceding and following character strings become one item when combined, and outputs a group of character strings from which splitting codes have been deleted, from which the preceding and following character strings become one item when combined, based on the features output from the intermediate layer.
The processing unit 11 recognizes the items based on the division codes remaining after the above-mentioned processing, converts them into a table format with one item written per line, and outputs them to the output unit 132 (see FIG. 5).

Figure 10 is a flowchart showing an example of a processing procedure performed by the processing unit of the information processing device according to the third embodiment. In steps S21 to S24, the same processing as in steps S11 to S14 is performed. The processing unit 11 acquires a group of character strings from which dividing codes that form a single item when the preceding and following character strings are combined have been removed (S25), and converts the group of character strings into a table format (S26). The processing unit 11 outputs the group of character strings converted into a table format to the output unit 132 (S27), and ends the processing.

(Modification)
In each of the above-described embodiments, the processing unit 11 processes the information of the character string in the region of interest, but may also refer to information in a related region that includes information related to the information of the character string in the region of interest, and delete the division code.

FIG. 11 is an explanatory diagram showing an example of a region of interest and a related region related to the region of interest. The processing unit 11 of the information processing device, for example, determines the region located to the right of the region of interest as the related region and extracts the character strings contained in the related region using OCR or the like, in the same manner as the region of interest. The processing unit 11 inserts a dividing code between characters that span lines to obtain a group of character strings. The region to be determined as the related region may be determined based on information input by the user to the input unit 131, or may be determined as the related region based on position information contained in the layout information of the document file, and a region that meets a predetermined condition may be determined as the related region. For example, the condition may be set as "a region located to the right of the region of interest that contains only a numeric string." In a financial statement such as that shown in FIG. 11, a numeric string representing an amount rarely spans multiple lines, and it is estimated that one item in the region of interest corresponds to one row of a numeric string in the related region. Therefore, by determining the region that meets the above-mentioned conditions as the related region, and determining the number of divided codes to delete from the group of character strings in the region of interest based on the number of divided codes in the group of character strings extracted from the related region, which is information contained in the related region, and performing the processing according to each embodiment, it is possible to obtain a group of character strings in which divided codes have been inserted in necessary and sufficient positions with higher accuracy. Furthermore, by having the processing unit 11 correspond one-to-one between each item in the region of interest and each line in the related region and outputting this to the output unit 132, it is possible to obtain information that can be used for statistical processing, analysis, etc.
The information contained in the related region is not limited to the number of divided codes, but may also be, for example, character strings within the related region. In this case, in embodiment 1, the storage unit 12 includes multiple item tables, and the processing unit 11 selects an item table to reference based on the degree of match between the character strings in the related region and the item strings registered in each item table, thereby enabling highly accurate pattern matching processing. Furthermore, in embodiment 2 or 3, the character strings in the related region are treated as information contained in the related region, and this information is input to the learning model 122, allowing the combination probability of the divided codes or the group of character strings to be output with higher accuracy.

In each embodiment, whether to combine the character strings before and after the division code is determined based on the character string in the region of interest, but the determination may also be based on other information. For example, the margins and indents between character strings may be recognized based on position information included in the layout information, and the determination may be based on the recognized margins and indents between character strings.
The determination may also be made based on information about the style, such as the font of the character string in the attention area, the color, the frame, the underline, or the ruled line.

In each embodiment, the document to be read is a financial statement, but this is not limited to this. The document to be read may also be, for example, an accounting table, various statistical materials, personnel data, or customer information documents.

The embodiments disclosed herein are illustrative in all respects and should not be considered limiting. The technical features described in each embodiment can be combined with one another, and the scope of the present invention is intended to include all modifications within the scope of the claims and equivalents to the scope of the claims.

REFERENCE SIGNS LIST 1 Information processing device 11 Processing unit 12 Storage unit 121 Item table 122 Learning model 12a Recording medium 13 Input/output I/F
131 Input section 132 Output section P Program

Claims (12)

1. A computer-implemented information processing method, comprising:
A group of character strings is obtained, the group of character strings being located across multiple lines in one area of a document including an area made up of columns and rows , and including multiple character strings made up of multiple items and codes indicating divisions inserted between the character strings;
based on the information included in the area, deleting the code that should be combined with the preceding and following character strings from among the codes in the group of character strings ;
A character string between two of the codes in the character string group from which the code that should join the preceding and following character strings has been deleted is recognized as one item of the plurality of items.
Information processing methods. 1. A computer-implemented information processing method, comprising:
A group of character strings is obtained, the group of character strings including a plurality of character strings located across a plurality of lines in one region of the document and a code indicating a division inserted between the character strings;
determining the number of codes to be removed from the group of character strings in the region based on the number of codes indicating divisions in the group of character strings extracted from the related region related to the region;
Based on the information contained in the area, the code that should be combined with the preceding and following character strings is deleted.
Information processing methods. determining whether the character strings before and after the code should be combined based on the character strings before and after the code;
The information processing method according to claim 1 , further comprising the step of: deleting the code when it is determined that the character strings before and after the code should be combined. The information processing method according to claim 3 , further comprising the step of: referring to a table in which predetermined character strings are recorded, and determining whether the character strings before and after the code should be combined. A learning model is trained to output a probability that the character strings before and after the code should be combined when the character strings are input, and
The information processing method according to claim 3 , further comprising outputting the degree of certainty. comparing the likelihood with a threshold;
The information processing method according to claim 5 , further comprising the step of: deleting the code based on a comparison result between the probability and the threshold value. inputting the group of character strings into a learning model that has been trained so as to output a group of character strings from which the code that should join the preceding and following character strings has been deleted when the group of character strings is input;
The information processing method according to claim 1 , further comprising the step of outputting a group of character strings from which the code that should join the preceding and following character strings has been deleted. The information processing method according to claim 1 , further comprising the step of: deleting the code based on information contained in a related region related to the region containing the character string. A step of acquiring a group of character strings, the group of character strings being located across multiple lines in one area of a document including an area made up of columns and rows , and including a plurality of character strings made up of multiple items and a code indicating a division inserted between the character strings;
a step of deleting a code that should be combined with a preceding or following character string from among the codes in the group of character strings based on information included in the area ;
a step of recognizing a character string between two of the codes in the group of character strings from which the code for combining the preceding and following character strings has been deleted as one item of the plurality of items;
A program that causes a computer to execute the following. obtaining a group of character strings including a plurality of character strings located across a plurality of lines in one region of a document and a code indicating a division inserted between the character strings;
determining the number of codes to be removed from the set of strings in the region based on the number of codes indicating divisions in the set of strings extracted from the related region associated with the region;
a step of deleting the code if the character strings before and after the code should be combined based on the information contained in the area;
A program that causes a computer to execute the following. an acquisition unit that acquires a group of character strings, the group of character strings being located across multiple lines in one area of a document including an area made up of columns and rows , and including multiple character strings made up of multiple items and codes indicating divisions inserted between the character strings;
a deletion unit that deletes, from the codes in the character string group, a code whose preceding and following character strings should be combined based on the information included in the area ;
a recognition unit that recognizes a character string between two of the codes in the character string group from which the code that should combine the preceding and following character strings has been deleted as one item among the plurality of items;
An information processing device comprising: an acquisition unit that acquires a group of character strings including a plurality of character strings located across a plurality of lines in one area of a document and a code indicating a division inserted between the character strings;
a determination unit that determines the number of codes to be deleted from a group of character strings in the region based on the number of codes indicating divisions in the group of character strings extracted from a related region that is related to the region;
a deletion unit that deletes the code when character strings before and after the code should be combined based on information included in the area;
An information processing device comprising: