JP7615688B2 - Information processing device and computer program - Google Patents

Description

The present invention relates to an information processing device and a computer program.

BPO (Business Process Outsourcing), in which a company outsources operations that the company does not have the operational know-how to external businesses, is being carried out in a variety of fields. For example, there is an increasing demand for BPO services that comprehensively outsource the receipt, sorting, and review of documents related to various applications.

Document sorting work requires the documents to be scanned. Patent Document 1 discloses a process in which an operator inspects documents scanned by a scanner and then manages them electronically, and document sorting work is often performed manually by classifying documents scanned by a scanner.

JP 2006-94036 A

However, if human error occurs due to worker fatigue, the sorting work must be performed again, which not only reduces work efficiency but also reduces sorting accuracy. Furthermore, if a labor shortage occurs due to a sudden increase in cases, there is a risk that work cannot be outsourced in a timely manner and business opportunities will be missed.

The present invention has been made in consideration of the above circumstances, and aims to provide an information processing device and computer program that can make document sorting more efficient and improve sorting accuracy.

The information processing device includes an acquisition unit that acquires a document image, an image generation unit that performs color reduction processing based on multiple thresholds on the document image acquired by the acquisition unit to generate multiple processed images, a text data generation unit that optically reads each of the multiple processed images generated by the image generation unit to generate text data, and a classifier that classifies the document image based on the multiple text data generated by the text data generation unit.

The computer program causes the computer to execute the following processes: acquire a document image, perform color reduction processing on the acquired document image based on multiple thresholds to generate multiple processed images, optically read each of the multiple processed images to generate text data, and classify the document image based on the multiple text data generated.

The present invention makes document sorting more efficient and improves sorting accuracy.

FIG. 1 is a schematic diagram illustrating an example of a configuration of an information processing system. FIG. 2 is a schematic diagram showing an example of a document image. 11 is a schematic diagram showing an example of a document image after color reduction processing; FIG. FIG. 2 is a schematic diagram showing an example of text data. FIG. 2 is a schematic diagram showing an example of text data obtained by concatenating text-converted data; FIG. 13 is a schematic diagram showing an example of character segmentation using character 2-grams. FIG. 13 is a schematic diagram showing an example of the frequency of appearance of divided character strings. FIG. 13 is a schematic diagram showing an example of a method for calculating weights. FIG. 13 is a schematic diagram showing an example of character feature amounts to which weighting is applied. FIG. 1 is a schematic diagram showing a first example of a document classification method using a trained model. FIG. 13 is a schematic diagram showing a second example of a document classification method using a trained model. FIG. 1 is a schematic diagram showing an example of a rule-based document classification method. FIG. 13 is a schematic diagram showing an example of a method for identifying a contributing portion. FIG. 13 is a schematic diagram showing an example of a highlighted document image. 10 is a flowchart showing an example of a procedure for a document image classification process performed by an information processing apparatus.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a schematic diagram showing an example of the configuration of an information processing system. The information processing system includes an information processing device 50 and a terminal device 10. The information processing device 50 and the terminal device 10 are connected via a communication network 1. The terminal device 10 is configured as a personal computer, a tablet terminal, or the like, and is used by a worker.

The information processing device 50 includes a control unit 51 that controls the entire device, a communication unit 52, a memory unit 53, an image processing unit 54, an OCR processing unit 55, a feature extraction unit 56, a classifier 57, a weighting unit 58, an identification unit 59, and an output unit 60. The control unit 51 can be composed of a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), etc.

The communication unit 52 has a function of communicating with the terminal device 10 via the communication network 1, and can transmit and receive required information. More specifically, the communication unit 52 has a function as an acquisition unit, and acquires document images from the terminal device 10.

Figure 2 is a schematic diagram showing an example of a document image. Figure 2 illustrates an image of a driver's license as an example of a document image. Documents include documents for identity verification, and in addition to driver's licenses, include various documents such as passports, My Number cards, insurance cards (health insurance cards, nursing care insurance cards, etc.), national pension booklets, disability certificates, resident registration cards, certified copies of family registers and certified seal certificates.

The storage unit 53 is composed of a semiconductor memory or a hard disk, and can store document images acquired via the communication unit 52. The storage unit 53 can also store required data such as the processing results in the information processing device 50.

The image processing unit 54 has a function of performing image pre-processing on the document image acquired via the communication unit 52. The image pre-processing includes sharpening, color reduction, resizing, etc. In this specification, the image processing unit 54 is configured to include sharpening, color reduction, and resizing, but the sharpening, color reduction, and resizing processes may each be configured as separate processing units. Each type of image pre-processing is described below.

The image processing unit 54 performs a sharpening process on the document image acquired via the communication unit 52 (or the image after a predetermined image processing has been performed on the document image). The sharpening process emphasizes contours using a filter that emphasizes edge contrast. The sharpening process can be performed on the image before or after the color reduction process.

Figure 3 is a schematic diagram showing an example of a document image after color reduction processing. The image processing unit 54 has a function as an image generating unit, and performs color reduction processing based on multiple thresholds on the document image acquired via the communication unit 52 to generate multiple processed images. In the color reduction processing, pixel values equal to or greater than the threshold are erased. In the example of Figure 3, two document images after color reduction processing are generated based on two different thresholds Th1 and Th2. The pixel values of R (red), G (green), and B (blue) are expressed as 0 (black) to 255 (white). For example, the threshold Th1 can be (R: 100, G: 100, B: 100), and the threshold Th2 can be (R: 200, G: 200, B: 200), but are not limited to this. In addition, the thresholds are not limited to the same value for each of R, G, and B, and different sets of thresholds may be used for R, G, and B.

In the area where the background and the characters overlap, the visibility of the characters decreases depending on the combination of the background color and the character color. In order to improve the visibility of the character part, the background color is removed by the color reduction process, but depending on the threshold, the character part may also be removed. Therefore, by performing the color reduction process using two different thresholds, even if the character is removed by the color reduction process based on one threshold, it is possible to pick up the character that was missed by the color reduction process based on the other threshold. In FIG. 3A, the threshold value Th1 = (R: 100, G: 100, B: 100), so that it is possible to erase pixels with R, G, and B pixel values of 100 or more. In the example of FIG. 3A, an example is shown in which the characters "driver's license" have been removed. On the other hand, in FIG. 3B, the threshold value Th1 = (R: 200, G: 200, B: 200), so that it is possible to erase pixels with R, G, and B pixel values of 200 or more. In the example of Figure 3B, the characters "Driver's License" that were removed in the example of Figure 3A remain untouched.

As described above, by performing color reduction processing on a document image based on multiple thresholds, the character recognition accuracy of the OCR (Optical Character Recognition) processing described below can be improved.

The image processing unit 54 performs a resizing process on the document image acquired via the communication unit 52 (or the image after a specified image processing has been performed on the document image). The size of the document image may vary depending on the type of document. By performing a resizing process, the image processing unit 54 can convert the image into an image of an optimal size for OCR processing. The resizing process can be performed on the document image after the sharpening process, but it may also be performed on each of the two post-color reduction images generated by the color reduction process. The resizing process can be performed on the image before or after the color reduction process.

The OCR processing unit 55 functions as a text data generating unit, and generates text data by optically reading an image that has been pre-processed by the image processing unit 54. More specifically, the OCR processing unit 55 generates text data by concatenating multiple pieces of text data obtained by optically reading each of the multiple processed images generated by the image processing unit 54.

Figure 4 is a schematic diagram showing an example of text data. Figure 4A is a text version of the post-color reduction processing image shown in Figure 3A, in which the characters "Valid until O month O day" and "123456789000" have been converted to text. Figure 4B is a text version of the post-color reduction processing image shown in Figure 3B, in which the characters "Valid until O month O day", "Driver's license", and "123456789000" have been converted to text.

Figure 5 is a schematic diagram showing an example of text data in which the converted text data is linked. In the example of Figure 5, the converted text data of Figure 4A and Figure 4B are linked. In the example of Figure 5, the converted text data of Figure 4B is placed below Figure 4A to link the text, but this is not limited to the above example, and Figure 4B may be placed to the right, left, or above Figure 4A to link the text. By linking the two text files, they are combined into one.

The feature extraction unit 56 extracts character features from the text data generated by the OCR processing unit 55. Specifically, the feature extraction unit 56 extracts character features based on the frequency of occurrence in the text data of each character string into which the text data generated by the OCR processing unit 55 is divided. Character n-grams can be used to divide the text data. Character n-grams are a method of dividing a document into n consecutive characters. Below, character 2-grams are explained as an example of character n-grams.

Figure 6 is a schematic diagram showing an example of character segmentation using character 2-grams. The text data shown in Figure 5 is used as the document before segmentation. Character 2-grams segment the text data in Figure 5 into two consecutive characters. In the example in Figure 6, the data is segmented into two characters, such as "O month", "O day", {until}, ....

Figure 7 is a schematic diagram showing an example of the frequency of occurrence of divided character strings. The character string "O month" appears twice in the text data, so its frequency is 2. The character string "O day" appears twice in the text data, so its frequency is 2. The same applies below. Furthermore, the character string "drive" appears once in the text data, so its frequency is 1. The character string "license" appears once in the text data, so its frequency is 1. The character string "certificate" appears once in the text data, so its frequency is 1.

The feature extraction unit 56 extracts a feature vector (also called a character feature) in which each divided string is an element and the frequency of the string is the element value. For example, if the number of divided strings is 100, the feature vector will be a 100-dimensional vector.

The divided strings can be weighted in advance according to their importance. The weighting method is explained below.

The weighting unit 58 assigns weights to the frequency of character strings. For example, the TF-IDF method can be used as a weighting method. TF-IDF stands for Term Frequency (TF) and Inverse Document Frequency (IDF). TF represents the frequency of occurrence of a certain character string in a document. Here, "in document" refers to all the text data collected to determine the weighting. The more frequently a character string appears in the document data, the more important the character string is considered to be. IDF represents the inverse of the document frequency at which a certain character string appears. A character string that appears in many documents is considered to have low importance within a document. The weighting unit 58 can calculate the weighting by multiplying TF and IDF.

Figure 8 is a schematic diagram showing an example of a method for calculating weights. The weighting unit 58 uses document data as learning data to calculate a weight for each character string. The document data is text data generated from multiple documents such as driver's licenses, passports, health insurance cards, and resident's certificates. In the example of Figure 8, the weighting unit 58 generates weighting information 581 using the TF-IDF method. The weighting information 581 is information that associates character strings with the weighting of the character strings. In the example of Figure 8, the weighting is set to 5.0 for the character strings "insurance" and "driving", and 4.5 for the character string "resident". The same applies to other character strings.

As described above, if a feature appears frequently in the document data, it is more likely to be an important feature and is weighted more heavily; on the other hand, if a feature appears in many sentences, it is less likely to be an important feature and is weighted less heavily, making it easier for the classifier 57 to classify document images.

The feature extraction unit 56 can extract character features by weighting the frequency of each divided character string.

Figure 9 is a schematic diagram showing an example of a character feature to which weighting has been applied. In Figure 9, the relationship between the character string and the corresponding frequency is the same as the example in Figure 7. As described above, the weighting unit 58 generates weighting information in advance. As shown in Figure 9, the weighting of each of the character strings "O month" to "valid" is 0.5, the weighting of each of the character strings "12" to "00" is 0.2, the weighting of each of the character strings "driver" and "license" is 5.0, and the weighting of the character string "certificate" is 1.0. A feature vector as a character feature is obtained by multiplying the frequency and weighting of each character string, and the values of each element x1 to x13 of the feature vector are calculated. For example, element x1 corresponds to the character string "O month", and the value of element x1 is 2 x 0.5 = 1.0. Also, element x11 corresponds to the character string "driver", and the value of element x11 is 1 x 5.0 = 5.0. The same applies to the other elements.

The classifier 57 classifies the document image based on the text data generated by the OCR processing unit 55. More specifically, the classifier 57 can classify the document image based on the character features extracted by the feature extraction unit 56. The document classification by the classifier 57 may use a trained model generated by machine learning, or may use rules that associate character features with document types. First, the case where a trained model is used will be described.

Figure 10 is a schematic diagram showing a first example of a document classification method using a trained model. Classifier 57 has neural network 571 as a trained model. Classifier 57 can be configured by combining hardware such as a CPU (e.g., a multi-processor having multiple processor cores), GPUs (Graphics Processing Units), DSPs (Digital Signal Processors), and FPGAs (Field-Programmable Gate Arrays).

When the character features (feature vector) extracted by the feature extraction unit 56 are input to the neural network 571, the neural network 571 outputs a probability (a value between 0 and 1) for each document type. Examples of document types include driver's license, health insurance card, passport, resident's card, and others. Note that the document types are not limited to the example in FIG. 10. The classifier 57 classifies the document image based on the type with the largest value among the probabilities for each document type output by the neural network 571. For example, if the probabilities for driver's license, health insurance card, passport, resident's card, and others are 0.8, 0.4, 0.3, 0.1, and 0.1, respectively, the document image is classified as a driver's license. Note that other machine learning models may be used instead of the neural network 571.

Figure 11 is a schematic diagram showing a second example of a document classification method using a trained model. The classifier 57 has a first classifier 572, a second classifier 573, and an averaging unit 574. When the character features (feature vectors) extracted by the feature extraction unit 56 are input, the classifiers 572 and 573 each output a probability (a value between 0 and 1) for each document type. The classifier 572 outputs probabilities of driver's license, health insurance card, passport, resident's card, and others from output terminals A1 to A5. The classifier 573 outputs probabilities of driver's license, health insurance card, passport, resident's card, and others from output terminals B1 to B5. The averaging unit 574 averages the probabilities output from output terminals A1 and B1 to output the probability of driver's license. The same applies to health insurance card, passport, resident's card, and others. The classifier 57 classifies the document image based on the type with the largest probability for each document type output by the averaging unit 574.

For example, a linear SVM (support vector machine) can be used as the first classifier 572, and for example, a gradient boosting tree can be used as the second classifier 573. By performing ensemble learning and combining multiple machine learning models, the classification accuracy of document images can be improved. Note that the machine learning models to be combined are not limited to linear SVMs and gradient boosting trees.

Next, we'll explain the rule-based case.

Figure 12 is a schematic diagram showing an example of a rule-based document classification method. Classifier 57 has rule-based classifier 575 and rule DB 577. Classifier 575 has similarity calculation section 576. Rule DB 577 stores rules that associate feature vectors with document types in advance. For example, feature vector V1 is associated with a driver's license, and feature vector V2 is associated with an insurance card. The same applies to other document types.

When the feature vector Vx extracted by the feature extraction unit 56 is input, the classifier 575 uses the function of the similarity calculation unit 576 to calculate the similarity between the input feature vector Vx and each feature vector recorded in the rule DB 577, and outputs the type associated with the feature vector with the greatest similarity as the document type. The similarity can be calculated using, for example, cosine similarity.

In general document classification using image processing, the type of document is determined based on the document's design and format, such as by pattern matching, making it difficult to classify non-standard documents (such as insurance cards and resident registration cards). As described above, the information processing device 50 performs optical character recognition using the OCR processing unit 55, so that even non-standard documents can be classified with high accuracy, regardless of the document's design or format.

In addition, the accuracy of character recognition by OCR processing is not 100%, and misrecognition does occur. As described above, the information processing device 50 uses the image processing unit 54 to perform multiple pre-processing steps (color reduction processing) with different thresholds, which can reduce the loss of characters and, as a result, can improve the character recognition accuracy of the subsequent OCR processing.

Conventionally, document sorting work has often been performed manually by classifying documents scanned by a scanner, which is prone to human error. As described above, the information processing device 50 uses a classifier generated by machine learning or a rule-based classifier, making it possible to automate document sorting work, improving the efficiency of document sorting work and reducing sorting errors.

Since document images are classified by the information processing device 50, it is possible to automate the document sorting process. However, in the review process of documents related to various applications after the sorting process, it is necessary to accurately determine review items such as whether all documents necessary for the application are present and whether the documents are valid, and manual review is essential. The following describes a method for providing information that is useful for document review.

The identification unit 59 identifies the contributing portion of the text data that contributed to the classification of the document image by the classifier 57. The output unit 60 outputs the document image in which the area corresponding to the contributing portion is highlighted. For example, the output unit 60 can output the highlighted document image to the terminal device 10 via the communication unit 52. Note that the output unit 60 may output only the document type to the terminal device 10.

Figure 13 is a schematic diagram showing an example of a method for identifying a contributing portion. The elements of a feature vector are character strings in text data, and the frequency of occurrence of the character strings (weighted frequency) becomes the value of the element. That is, since each element of the feature vector corresponds to a character string in the text data, the position or coordinates of the character string in the text data can be recorded. Since the element that contributes to the classification of a document image is considered to be the part with a large weighted frequency value, if the position or coordinates of the character string in the text data are known, the contributing portion that contributed to the classification of the document image can be identified based on the position or coordinates. And if the contributing portion in the text data is known, the area corresponding to the contributing portion of the document image can also be known.

Figure 14 is a schematic diagram showing an example of a highlighted document image. In the example of Figure 14, the document is a health insurance card, and the area corresponding to the contributing portion that contributed to the classification is surrounded by a thick line. The words "health insurance" and "insured person certificate" are written in the area corresponding to the contributing portion. The highlighting may be in the form of a thick line surrounding the document as in Figure 14, or may be in any display form that allows the person conducting the review to easily determine the type of document, such as highlighting using a color or pattern.

As described above, the explanatory reasons that contributed to the classification of a document can be shown by highlighting text on the document image, allowing document reviewers to accurately and quickly determine review items.

Figure 15 is a flowchart showing an example of the procedure for classification processing of document images by the information processing device 50. For convenience, the following description will be given with the control unit 51 as the subject of the processing. The control unit 51 acquires a document image (S11), and performs a sharpening process on the acquired document image (S12). The control unit 51 also performs a resizing process (S13) as necessary.

The control unit 51 performs color reduction processing based on multiple thresholds (S14), performs OCR processing on the multiple processed images generated by the color reduction processing to convert them into text (S15), and concatenates the converted text data to generate text data (S16).

The control unit 51 extracts character features (feature vectors) from the text data (S17), and inputs the extracted character features to the classifier 57 to classify the document image (S18). The control unit 51 outputs to the terminal device 10 a document image in which the areas corresponding to the contributing parts that contributed to the classification are highlighted (S19).

The control unit 51 determines whether there are other document images (S20), and if there are other document images (YES in S20), repeats the processing from step S11 onwards, and if there are no other document images (NO in S20), ends the processing.

The information processing device 50 can be realized, for example, by using a computer equipped with a CPU (e.g., a multi-processor having multiple processor cores), a GPU (Graphics Processing Units), a RAM, etc. The information processing device 50 can be realized on a computer by loading a computer program (which can be recorded on a recording medium) that defines the processing procedure as shown in FIG. 15 into the RAM of the computer and executing the computer program with the CPU (processor).

The information processing device includes an acquisition unit that acquires a document image, an image generation unit that performs color reduction processing based on multiple thresholds on the document image acquired by the acquisition unit to generate multiple processed images, a text data generation unit that optically reads each of the multiple processed images generated by the image generation unit to generate text data, and a classifier that classifies the document image based on the multiple text data generated by the text data generation unit.

The information processing device is equipped with a sharpening processing unit that performs a sharpening process on the image before or after the color reduction process.

The information processing device includes a resizing processing unit that performs resizing processing on the image before or after the color reduction processing.

In the information processing device, the text data generation unit generates text data by concatenating multiple pieces of text data obtained by optically reading each of the multiple processed images generated by the image generation unit.

In the information processing device, the classifier classifies the document image based on the character features extracted from the text data generated by the text data generation unit, using rules that associate character features with document types, or using a trained model that outputs a document type when character features are input.

The information processing device includes a feature extraction unit that extracts the character features based on the frequency of occurrence in the text data of each character string into which the text data generated by the text data generation unit is divided.

In the information processing device, the feature extraction unit includes a character n-gram method.

The information processing device includes a weighting unit that pre-weights the frequency of character strings, and the feature extraction unit extracts the character features by weighting the frequency of each of the divided character strings.

In the information processing device, the weighting unit includes a TF-IDF method.

The information processing device includes an identification unit that identifies a contributing portion of the text data that contributed to the classification of the document image, and an output unit that outputs the document image with the area corresponding to the contributing portion highlighted.

The computer program causes the computer to execute the following processes: acquire a document image, perform color reduction processing on the acquired document image based on multiple thresholds to generate multiple processed images, optically read each of the multiple processed images to generate text data, and classify the document image based on the multiple text data generated.

REFERENCE SIGNS LIST 1 Communication network 10 Terminal device 50 Information processing device 51 Control unit 52 Communication unit 53 Storage unit 54 Image processing unit 55 OCR processing unit 56 Feature extraction unit 57, 572, 573, 575 Classifier 574 Additive average unit 576 Similarity calculation unit 577 Rule DB
571 Neural network 58 Weighting unit 581 Weighting information 59 Identification unit 60 Output unit

Claims (10)

An acquisition unit for acquiring a document image;
an image generating unit that performs color reduction processing based on a plurality of threshold values on the document image acquired by the acquiring unit to generate a plurality of processed images;
a text data generating unit that optically reads each of the plurality of processed images generated by the image generating unit and generates text data;
a classifier that classifies the document image based on the plurality of text data generated by the text data generating unit ,
The text data generation unit
generating text data by concatenating a plurality of text data obtained by optically reading each of the plurality of processed images generated by the image generating unit ;
Information processing device. A sharpening processing unit that performs a sharpening process on the image before or after the color reduction process is provided.
The information processing device according to claim 1 . a resizing processing unit that performs a resizing process on the image before or after the color reduction process,
3. The information processing device according to claim 1 or 2. The classifier may be
classifying the document image based on the character features extracted from the text data generated by the text data generation unit, using a rule associating character features with document types, or using a trained model that outputs a document type when character features are input;
The information processing device according to claim 1 . a feature extraction unit that extracts the character feature based on a frequency of appearance in the text data of each character string obtained by dividing the text data generated by the text data generation unit,
The information processing device according to claim 4 . The feature extraction unit includes a character n-gram method.
The information processing device according to claim 5 . A weighting unit that pre-weights the frequency of character strings,
The feature extraction unit is
weighting the frequency of each of the divided character strings to extract the character feature amount;
7. The information processing device according to claim 5 or 6 . The weighting unit includes a TF-IDF method.
The information processing device according to claim 7 . an identification unit that identifies a contribution portion of the text data that contributed to the classification of the document image;
and an output unit that outputs the document image in which an area corresponding to the contributing portion is highlighted.
The information processing device according to claim 1 . On the computer,
Get document images,
performing color reduction processing based on a plurality of threshold values on the acquired document image to generate a plurality of processed images;
Each of the generated multiple processed images is optically read to generate text data;
classifying the document image based on the generated plurality of text data;
generating text data by concatenating a plurality of text data obtained by optically reading each of the generated plurality of processed images ;
A computer program that executes a process.

Images