JP7699773B2 - Document image processing system, document image processing method, and document image processing program - Google Patents

Description

The present invention relates to a document image processing system, a document image processing method, and a document image processing program.

In one document identification system, a document format table is created in advance by the user, and the document format table contains field information indicating the position, size, character type, etc. of the character recognition target area specified by the user. Then, based on this document format (i.e., field information), character information (text data) within the document is obtained from the image data of the document image (see, for example, Patent Document 1).

One image recognition device performs an extraction process that cuts out a partial image from a target image, recognizes letters and numbers in the partial image, and extracts letters and numbers that satisfy a predetermined condition from the letters and numbers (see, for example, Patent Document 2). In the extraction process, the image recognition device, for example, determines whether the recognized characters include a predetermined bank name that has been set in advance, and if the characters include the predetermined bank name, extracts the characters and numbers within a predetermined distance from the characters as a pair of a bank name and an account number.

JP 2016-48444 A JP 2020-170264 A

However, the above-mentioned document identification system uses template data that specifies the layout of documents such as documents (such as information on the position where each attribute is written), so in order to process multiple documents with different layouts, template data must be created in advance for each layout, which requires tedious work in advance. Furthermore, for documents with unknown layouts, the above-mentioned technology has difficulty accurately detecting attribute values for a certain attribute in the document image.

Furthermore, in the above-mentioned image recognition device, template data is not required, but the character string to be extracted (the above-mentioned bank name) must be set in advance, and character strings that are not set will not be extracted. In addition, in the above-mentioned image recognition device, numbers within a specified distance from the above-mentioned bank name are extracted as account numbers, but two character objects may not be related even if the distance between them is short, and may be related even if the distance between them is long, so the desired character string may not be extracted correctly.

Figure 5 is a diagram showing an example of a document image including a table. For example, as shown in Figure 5, document image 101 of a medical examination report includes table 111 showing the results of the medical examination. Table 111 includes combinations of labels indicating test items and values that are the test results of those test items. In table 111, cells are arranged two-dimensionally, with or without lines, and labels and values corresponding to the labels (numeric values or non-numeric character strings) are entered in the cells. However, even if the distance between the two (label and value) is short, the two may not be related, and even if the distance between the two is long, the two may be related.

The present invention has been made in consideration of the above problems, and aims to provide a document image processing system, a document image processing method, and a document image processing program that can accurately identify the attributes of cells in a table without using template data.

The document image processing system according to the present invention includes a table detection unit that detects a table in a document image, detects at least a cell in the table, and generates cell geometric data indicating the position and size of the cell; a text object detection unit that detects a text object in a document image, and generates text object geometric data indicating the position and size of the text object; a character recognition processing unit that performs character recognition processing on the text object to generate text data corresponding to the text object; and a cell attribute identification unit that (a) identifies a text object in a cell based on the cell geometric data and the text object geometric data, (b) generates node data for each cell, including the cell geometric data, the text object geometric data of the text object in the cell, and the text data of the text object in the cell, and generates a node data set including the node data corresponding to the table, and (c) performs a predetermined classification processing on the node data set to identify the attributes of the cell for each cell.

A document image processing method according to the present invention includes the steps of: a computer detecting a table in a document image, the computer detecting at least cells in the table, and the computer generating cell geometry data indicating the positions and sizes of the cells; a computer detecting text objects in the document image, and the computer generating text object geometry data indicating the positions and sizes of the text objects; a computer performing character recognition processing on the text objects to generate text data corresponding to the text objects; and a computer generating a node data set including the node data corresponding to the table by the computer performing a predetermined classification processing on the node data set to identify cell attributes for each cell.

The document image processing program of the present invention causes a computer to function as the above-mentioned table detection unit, the above-mentioned text object detection unit, the above-mentioned character recognition processing unit, and the above-mentioned cell attribute identification unit.

The present invention provides a document image processing system, a document image processing method, and a document image processing program that accurately identify attributes of cells in a table without using template data that specifies the description position of an attribute in a document image.

The above and other objects, features and advantages of the present invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings.

FIG. 1 is a block diagram showing a configuration of a document image processing system according to an embodiment of the present invention. FIG. 2 is a diagram for explaining node data generated in the document image processing system according to the embodiment of the present invention. FIG. 3 is a diagram for explaining the classification process in the document image processing system according to the embodiment of the present invention. FIG. 4 is a flow chart for explaining the operation of the document image processing system shown in FIG. FIG. 5 is a diagram showing an example of a document image including a table.

The following describes an embodiment of the present invention with reference to the drawings.

Figure 1 is a block diagram showing the configuration of a document image processing system according to an embodiment of the present invention. The document image processing system shown in Figure 1 is composed of one information processing device (personal computer, server, etc.), but the processing units described below may be distributed among multiple information processing devices capable of data communication with each other. Furthermore, such multiple information processing devices may include a GPU (Graphics Processing Unit) that processes specific calculations in parallel.

The document image processing system shown in FIG. 1 includes a storage device 1, a communication device 2, an image reading device 3, and a processing device 4.

The storage device 1 is a non-volatile storage device such as a flash memory or a hard disk, and stores various data and programs.

Here, the storage device 1 stores an image processing program 11, and also stores system setting data (such as the coefficient setting values of the neural network used in each processing unit described below) as necessary. Note that the image processing program 11 may be stored on a portable computer-readable recording medium such as a CD (Compact Disk). In that case, for example, the image processing program 11 is installed from the recording medium into the storage device 1. Also, the image processing program 11 may be a single program or a collection of multiple programs.

The communication device 2 is a device capable of data communication, such as a network interface, a peripheral device interface, or a modem, and performs data communication with other devices as necessary. The image reading device 3 optically reads a document image from a document and generates image data (e.g., raster image data) of the document image. The communication device 2 and the image reading device 3 are provided as necessary.

The arithmetic processing device 4 is a computer equipped with a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc., and operates as various processing units by loading programs from the ROM, storage device 1, etc. into the RAM and executing them on the CPU.

Here, by executing the image processing program 11, the calculation processing device 4 operates as a document image acquisition unit 21, a table detection unit 22, a text object detection unit 23, a character recognition processing unit 24, a cell attribute identification unit 25, a data output unit 26, and a machine learning processing unit 27.

The document image acquisition unit 21 acquires a document image as image data such as raster image data. The document image is an image of a document that includes, in a table, attribute labels (text such as headings) and attribute values (text such as numbers and other character strings) for one or more attributes (such as written items), such as receipts (including receipts), invoices, delivery notes, and other forms, flyers such as advertising and announcements, completed questionnaires, and health check reports. For example, the document image acquisition unit 21 reads out document images as image data stored in the storage device 1, acquires document images as image data received by the communication device 2 via a communication path such as a network, and acquires document images as image data generated by the image reading device 3.

The table detection unit 22 detects tables in the acquired document image without using template data, generates table geometry data indicating the position and size of the table, and detects at least cells in the table, and generates cell geometry data indicating the position and size of the cell.

In this embodiment, the table detection unit 22 detects at least one of rows and columns along with cells in a table, and generates at least one of row geometry data indicating the position and size of the row and column geometry data indicating the position and size of the column (here, both).

The table detection unit 22 detects tables in document images, as well as rows, columns, and cells within those tables, and generates their geometric data according to an existing method using a neural network.

The text object detection unit 23 detects text objects in the acquired document image without using template data, and generates text object geometry data indicating the position and size of the text objects.

Specifically, the text object detection unit 23 (a) detects text objects by excluding non-text objects (photo objects, graphic objects, line objects, etc.) in the document image, and (b) extracts text objects by grouping them into "words" based on the position of each text object.

The text object detection unit 23 extracts text objects from within a document image using existing techniques (e.g., region separation processing, machine-learned deep neural networks, etc.).

The character recognition processing unit 24 performs character recognition processing on the detected text object (raster image) to generate text data (character code string) corresponding to the text object. This character recognition processing uses existing technology.

The cell attribute identification unit 25 (a) identifies text objects in the cells based on the above-mentioned cell geometric data and text object geometric data, (b) generates node data for each cell, including the cell geometric data, the text object geometric data of the text objects in the cell, and the text data of the text objects in the cell, and generates a node data set including the node data corresponding to the table, and (c) performs a predetermined classification process on the node data set to identify the cell attributes for each cell.

The positions of cells, rows, columns, and text objects in the node data are
The coordinate values are two-dimensional coordinates, and their sizes are the lengths at the respective two-dimensional coordinates. The positions are indicated by the positions of a predetermined portion (one of the four corners, the center, etc.) of a rectangular area of a cell, row, column, or text object, and may be expressed as a relative position from a predetermined portion (one of the four corners, the center, etc.) of a table. This relative position is derived from the absolute position of the table in the document image (the position in the table geometric data) and the absolute position of the cell, row, column, or text object (the position in the original geometric data).

Specifically, for each detected cell, if the area of a text object (bounding box) identified from the text object geometric data is included within the cell area identified from the cell geometric data of the cell, the cell attribute identification unit 25 determines that the text object is a text object within the cell.

Figure 2 is a diagram explaining node data generated in a document image processing system according to an embodiment of the present invention. Figure 3 is a diagram explaining classification processing in a document image processing system according to an embodiment of the present invention.

In this embodiment, as shown in FIG. 2, the node data of a cell includes not only the cell geometry data of the cell, the text object geometry data of the text object in the cell, and the text data of the text object in the cell, but also at least one of (here, both) the row geometry data of the row to which the cell belongs and the column geometry data of the column to which the cell belongs.

In this embodiment, as shown in FIG. 3, the cell attribute identification unit 25 inputs the above-mentioned node data set as input data to a machine-learned graph neural network (GNN), and identifies the output data of the GNN as the above-mentioned cell attribute. Note that this GNN and its machine learning can be existing ones.

The attributes of a cell include at least the cell type of the cell, and the cell type is either a label or an attribute value (that is, in this case, the cell is classified as either a label cell or an attribute value cell). The output data of the GNN for each node data is the probability (a number in the range of 0 to 1) for each possible value (here, label and attribute value) of the cell type of the cell corresponding to that node data, and based on the value of that probability, for example by classification using a threshold, the cell type is determined to be one of the possible values of the cell type (here, label and attribute value).

The cell attribute identification unit 25 may classify the node data into clusters by clustering the node data based on the features indicated by the node data, and identify the cell attributes based on the clusters. For example, the cell attribute identification unit 25 may identify the cell attributes by the above-mentioned clustering instead of the GNN, or may identify the cell attributes by the above-mentioned clustering instead of the GNN when the reliability of identifying the above-mentioned cell attributes by the GNN is low.

For example, the feature quantity used is a feature vector corresponding to the node data (all or a specific part), which is generated according to an existing method such as Word2vec (Skip-Gram model). In addition, a large number of combinations of node data and the cell type values for that node data are collected, and the median value (average of the feature vector) for each cell type value (here, label or attribute value) is identified (as the center of each cluster), and the cell type value (here, label or attribute value) having the closest median value is selected as the cell type value corresponding to that node data from the position indicated by the feature vector of the node data to be classified.

The data output unit 26 adds the cell attributes corresponding to each node data to the node data, and stores the node data set in a specified data format in the storage device 1 or transmits it via the communication device 2.

This output data (node data set) makes it possible, for example, to identify label cells and attribute value cells in a column, or to identify label cells and attribute value cells in a row.

The machine learning processing unit 27 executes machine learning processing to perform machine learning of the GNN in the cell attribute identification unit 25 described above. Note that the machine learning processing unit 27 described above is not essential and may be provided as necessary. Also, if the machine learning of the cell attribute identification unit 25 (GNN) is completed, the machine learning processing unit 27 does not need to be provided.

Next, the operation of the document image processing system according to this embodiment will be described. Figure 4 is a flowchart explaining the operation of the document image processing system shown in Figure 1.

First, the document image acquisition unit 21 acquires a document image (step S1).

Next, the table detection unit 22 detects a table in the document image without using template data, detects cells, columns, and rows in the table, and generates cell geometry data, column geometry data, and row geometry data (step S2).

The text object detection unit 23 detects text objects in the document image without using template data and generates text object geometric data (step S2). The character recognition processing unit 24 performs character recognition processing on the detected text objects and generates text data for the text objects.

The above-mentioned processing by the table detection unit 22 and the above-mentioned processing by the text object detection unit 23 may be performed in parallel, or when these processing are performed in sequence, either one may be performed first.

Then, for each detected cell, the cell attribute identification unit 25 identifies the text object within the cell based on the above-mentioned cell geometric data and text object geometric data, and generates node data including the cell geometric data, the text object geometric data of the text object within the cell, and the text data of the text object within the cell (step S3).

Next, the cell attribute identification unit 25 generates a node data set for each table using node data corresponding to all cells detected in that table, and performs a predetermined classification process on the node data set to classify the attributes of each cell and identify the attribute of each cell (here, the cell type, which is a label or attribute value) (step S4).

Then, the data output unit 26, for example, adds the attributes of each cell to the node data corresponding to that cell, and stores the node data set as output data for each table in a predetermined data format in the storage device 1 or transmits it via the communication device 2 (step S5).

In this way, cell-level attribute data is generated for each table in the document image.

The data output unit 26 may also receive a search request in a predetermined format, and in accordance with the search request, search for an attribute value corresponding to a label specified by the search request in the node data set generated as described above, and output a combination of the label and the attribute value. For example, first, in the generated node data set, a cell, row, and column including the label to be searched are identified, and a row or column including a cell of an attribute value in the row or column is identified. If there is one cell of the attribute value in the row or column, the attribute value is identified as the attribute value corresponding to the label, and if there are multiple cells of the attribute value, the attribute values are identified as the attribute values corresponding to the label, and the labels of the row or column of the cells of the attribute value (column if a cell of the attribute value is identified in a row including the label to be searched, row if a cell of the attribute value is identified in a column including the label to be searched) may be associated with the attribute values, respectively.

For example, in the node data set of table 111 in Figure 5, when "height" is specified as the search target, three attribute values, "161.0", "161.2", and "161.1", are detected in the row containing "height", and "161.0" is labeled "this time", "161.2" is labeled "last time", and "161.1" is labeled "before last time".

Two labels may also be specified as search target labels in a search request. In this case, an attribute value is detected from one of the two labels in the same manner as described above, and among the detected attribute values, those whose row or column label to which the cell of that attribute value belongs matches the other label are determined to be attribute values corresponding to the two labels and detected.

As described above, according to the above embodiment, the table detection unit 22 detects a table in a document image, detects at least a cell in the table, and generates cell geometric data indicating the position and size of the cell. The text object detection unit 23 detects a text object in a document image, and generates text object geometric data indicating the position and size of the text object. The character recognition processing unit 24 performs character recognition processing on the text object to generate text data corresponding to the text object. The cell attribute identification unit 25 (a) identifies a text object in a cell based on the cell geometric data and the text object geometric data, (b) generates node data for each cell, including the cell geometric data, the text object geometric data of the text object in the cell, and the text data of the text object in the cell, generates a node data set including the node data corresponding to the table, and (c) performs a predetermined classification processing on the node data set to identify the cell attribute for each cell.

This allows the attributes of cells in a table to be accurately identified without using template data. In addition, because the attribute values corresponding to the labels are detected without taking into account the distance (Euclidean distance) between the labels and the attribute values, the combinations of labels and attribute values in a table can be accurately identified regardless of the distance (Euclidean distance) between the labels and the attribute values.

It should be noted that various changes and modifications to the above-described embodiments will be apparent to those skilled in the art. Such changes and modifications may be made without departing from the spirit and scope of the subject matter and without diminishing its intended advantages. In other words, such changes and modifications are intended to be included within the scope of the claims.

For example, in the above embodiment, image data of the document image may be immediately deleted from the system after the above-mentioned processing is completed.

In addition, in the above embodiment, the number of nodes in the input data of the GNN is set to a predetermined value that is equal to or greater than the maximum number of node data in the node data set (i.e., the number of cells in the table), and if the number of node data is less than the number of nodes in the input data of the GNN, a fixed value is used as the missing node data, and the corresponding output data is discarded.

In addition, in the above embodiment, the cell attribute is the cell type, which is either a label or an attribute value, but the cell type may be a header (such as a table title) or other type in addition to the label and attribute value.

In addition, in the above embodiment, rows and columns may not be detected, and row geometry data and column geometry data may not be included in the node data. Even in this case, the attribute value corresponding to the label can be detected by searching and detecting the attribute value cell in the horizontal and vertical directions from the position of the label cell based on the position of the cell in the cell geometry data.

The present invention can be applied to recognition processing of document images such as forms, for example.

4. Processing unit (an example of a computer)
11 Image processing program (an example of a document image processing program)
22 table detection unit 23 text object detection unit 24 character recognition processing unit 25 cell attribute identification unit

Claims (7)

a table detector for detecting tables within a document image, detecting at least cells within said tables, and generating cell geometry data indicative of positions and sizes of said cells;
a text object detector for detecting text objects within the document image and generating text object geometry data indicative of positions and sizes of the text objects;
a character recognition processing unit that performs character recognition processing on the text object to generate text data corresponding to the text object;
(a) identifying the text object in the cell based on the cell geometric data and the text object geometric data; (b) generating, for each of the cells, node data including the cell geometric data, the text object geometric data of the text object in the cell, and the text data of the text object in the cell, and generating a node data set including the node data corresponding to the table; and (c) performing a predetermined classification process on the node data set to identify, for each of the cells, an attribute of the cell.
A document image processing system comprising: The document image processing system according to claim 1, characterized in that the cell attribute identification unit inputs the node data set as input data to a machine-learned graph neural network and identifies the output data of the graph neural network as the attributes of the cell. The document image processing system according to claim 1, characterized in that the cell attribute identification unit performs clustering on the node data based on the feature values indicated by the node data to classify the node data into clusters, and identifies the attributes of the cells based on the clusters. the table detection unit detects at least one of rows and columns along with cells in the table, and generates at least one of row geometry data indicating a position and size of the rows and column geometry data indicating a position and size of the columns;
the node data of a cell further includes at least one of the row geometric data of a row to which the cell belongs and the column geometric data of a column to which the cell belongs;
4. The document image processing system according to claim 1, wherein: the attributes of the cell include at least a cell type of the cell;
the cell type includes a label and an attribute value;
5. The document image processing system according to claim 1, further comprising: a computer detecting a table in a document image, the computer detecting at least cells in the table, and the computer generating cell geometry data indicative of positions and sizes of the cells;
detecting text objects within the document image by the computer and generating text object geometry data indicative of positions and sizes of the text objects by the computer ;
performing character recognition processing on the text object by the computer to generate text data corresponding to the text object by the computer ;
(a) the computer identifies the text object in the cell based on the cell geometry data and the text object geometry data; (b) for each cell, the computer generates node data including the cell geometry data, the text object geometry data of the text object in the cell, and the text data of the text object in the cell , and generates a node data set including the node data corresponding to the table; and (c) the computer executes a predetermined classification process on the node data set, and for each cell, the computer identifies an attribute of the cell.
A document image processing method comprising: Computer,
a table detector for detecting tables within a document image, detecting at least cells within said tables, and generating cell geometry data indicative of positions and sizes of said cells;
a text object detector for detecting text objects within the document image and generating text object geometry data indicative of positions and sizes of the text objects;
a character recognition processing unit that performs character recognition processing on the text object to generate text data corresponding to the text object; and
(a) identifying the text object in the cell based on the cell geometric data and the text object geometric data; (b) generating, for each of the cells, node data including the cell geometric data, the text object geometric data of the text object in the cell, and the text data of the text object in the cell, and generating a node data set including the node data corresponding to the table; and (c) performing a predetermined classification process on the node data set to identify, for each of the cells, an attribute of the cell.
A document image processing program that acts as a

Images