JP7526692B2 - Recognition method and device - Google Patents

Description

The present invention relates to a recognition method and a recognition device.

Conventionally, technology is known that performs character recognition for specific items from an optically read image of a form.

For example, a technology is known that preregisters candidate strings of characters to be printed in multiple-choice items and then recognizes the registered strings in areas marked with a circle or check mark.

In addition, there is known a technique in which a shape such as a double line used to select a character string is registered in advance, and the selected character string is identified by reading the registered shape.

Japanese Patent Application Publication No. 11-345281 JP 2005-173673 A

However, conventional techniques require the advance registration of character strings and figures, which means that it is sometimes not possible to easily and universally recognize selected character strings on a form.

For example, consider a situation in which an account subject is selected on a form at a bank when applying to open an account or making a deposit. In this case, the characters to be recognized are assumed to be "regular" and "current."

In this case, in addition to "regular" and "current," the character string "tax preparation" is provided as an option for the subject, and if "tax preparation" is circled by hand, conventional technology may not be able to recognize the selected character string.

In one aspect, the aim is to make it easy and versatile to recognize selected character strings on a form.

In one aspect, the recognition method involves a computer executing a process to extract a first area from a form that is filled in by hand, extract a second area from the form that is printed with type, and select from the second area an area whose degree of overlap with the first area satisfies a predetermined condition.

In one aspect, it allows for easy and versatile recognition of selected character strings on a form.

FIG. 1 is a diagram illustrating an example of the configuration of a recognition system. FIG. 2 is a diagram illustrating an example of the configuration of a recognition device. FIG. 3 is a diagram showing an example of selected character string information. FIG. 4 is a diagram for explaining extraction of a handwritten region. FIG. 5 is a diagram for explaining extraction of a print region. FIG. 6 is a diagram showing examples of items on a form. FIG. 7 is a diagram showing an example of an overlapping region. FIG. 8 is a flowchart showing the flow of the recognition process. FIG. 9 is a diagram illustrating an example of a hardware configuration.

The following describes in detail the recognition method and recognition device according to the present invention with reference to the drawings. Note that the present invention is not limited to these examples. Furthermore, the examples can be combined as appropriate within a range that does not cause inconsistencies.

The configuration of a recognition system according to an embodiment will be described with reference to FIG. 1. FIG. 1 is a diagram showing an example of the configuration of a recognition system. As shown in FIG. 1, the recognition system 1 includes a recognition device 10 and a scanner 20.

The recognition device 10 accepts input of an image of a form generated by the scanner 20 and outputs a recognition result. In the example of FIG. 1, the recognition device 10 outputs the recognition result "Gender: Male". The recognition result in FIG. 1 means that the option "Male" was selected from among the options for the "Gender" item on the form.

In this embodiment, the form is a medium such as paper on which items and options are printed, and the person filling out the form handwrites the option for each item. For example, the form is a bank account opening application form, a deposit request form, an answer sheet for a multiple-choice test such as a mark sheet, a questionnaire, etc.

The recognition device 10 may be realized by a personal computer, an automatic teller machine (ATM), a smartphone, etc.

The recognition device 10 may also be realized by combining a terminal and a server. In that case, the terminal transmits the image received from the scanner 20 to the server. The server then returns the recognition result based on the image to the terminal.

The scanner 20 is a device that optically reads media such as paper and generates an image. For example, the scanner 20 may be part of the functions of a multifunction device or an ATM that can accept handwritten forms. The scanner 20 may also be a mobile terminal with a camera, such as a smartphone.

The scanner 20 reads the form 30. On the form 30, the item name "Gender" is printed, and the options "(1) Male" and "(2) Female" are printed below it. In addition, a handwritten circle is written near the option "(1) Male."

In the following explanation, handwriting is not limited to writing characters, but also refers to writing and drawing any shape by human hand.

The shapes referred to here are not limited to geometric shapes such as circles and rectangles, but include any type of shape such as letters, check marks, and fills.

On the other hand, type is the characters that are generated by a word processor or the like and printed onto a form by a printer or the like.

In the example of FIG. 1, the circles on form 30 are handwritten. On the other hand, "Gender," "(1) Male," and "(2) Female" are printed on form 30 in type.

Figure 2 is a diagram showing an example of the configuration of a recognition device. As shown in Figure 2, the recognition device 10 has an IF (interface) unit 11, a storage unit 12, and a control unit 13.

The IF unit 11 is an interface for inputting and outputting data. For example, the IF unit 11 is a NIC (Network Interface Card). The IF unit 11 can send and receive data to and from other devices including the scanner 20.

The IF unit 11 may also be connected to input devices such as a mouse and a keyboard. The IF unit 11 may also be connected to output devices such as a display and a speaker.

The memory unit 12 is an example of a storage device that stores data, programs executed by the control unit 13, etc., and is, for example, a hard disk or memory. The memory unit 12 stores handwritten area extraction model information 121, printed character area extraction model information 122, dictionary information 123, and selected character string information 124.

The handwritten area extraction model information 121 is parameters for constructing a handwritten area extraction model. For example, the handwritten area extraction model is an image recognition model that uses a neural network or the like, and is used by the extraction unit 133 described below. For example, the handwritten area extraction model information 121 is a weight matrix and bias value of a neural network.

The typed character area extraction model information 122 is parameters for constructing a typed character area extraction model. For example, the typed character area extraction model is an image recognition model that uses a neural network or the like, and is used by the extraction unit 133 described below. For example, the typed character area extraction model information 122 is a weight matrix and bias value of a neural network.

Dictionary information 123 is a set of characters for character recognition. Dictionary information 123 is a dictionary used in existing OCR (Optical Character Recognition) software, etc., and may include the characteristics of characters such as alphabets, kanji, hiragana, Arabic numerals, and symbols.

The selection string information 124 is a combination of item names and strings used as options. FIG. 3 is a diagram showing an example of selection string information. As shown in FIG. 3, for example, the selection string information 124 includes a combination of the item "Gender" and the options "Male, Female, Male, Female, ...".

The character strings included in the item names and options of the selected character string information 124 are designated in advance as character strings that are frequently used as item names and options, respectively. The selected character string information 124 is used to assist in the character recognition process described below.

Returning to FIG. 2, the control unit 13 is realized, for example, by a CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphics Processing Unit), etc., which executes a program stored in an internal storage device using RAM (Random Access Memory) as a working area. The control unit 13 may also be realized, for example, by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

The control unit 13 includes a scanner control unit 131, an analysis unit 132, an extraction unit 133, a selection unit 134, a recognition unit 135, a correction unit 136, and an output control unit 137.

The scanner control unit 131 controls the scanner 20. The scanner control unit 131 instructs the scanner 20 to read the document, generate an image, and transfer the image.

The analysis unit 132 analyzes the image received from the scanner 20 and identifies the position of the selected item on the form. For example, the analysis unit 132 can extract the items using the method described in Reference 1 (JP Patent Publication 2010-3155A).

The extraction unit 133 extracts handwritten areas and typed areas at the positions identified by the analysis unit 132. The extraction unit 133 extracts handwritten areas from the form where entries have been made by hand. The handwritten areas are an example of a first area. The extraction unit 133 also extracts typed areas from the form where type is printed. The typed areas are an example of a second area.

The extraction unit 133 extracts the handwritten area using an image recognition model that has learned the characteristics of handwritten figures. The extraction unit 133 uses a handwritten area extraction model, which is an image recognition model constructed based on the handwritten area extraction model information 121.

The handwritten region extraction model may be trained using specific shapes such as circles and check marks as training data, or it may be trained using any handwritten shape as training data, not limited to specific shapes.

Figure 4 is a diagram illustrating the extraction of a handwritten region. In the example of Figure 4, the extraction unit 133 extracts an area 51 in which a handwritten circle is written as a handwritten region.

The extraction unit 133 extracts the type region using an image recognition model that has learned the characteristics of type. The extraction unit 133 uses a type region extraction model, which is an image recognition model constructed based on the type region extraction model information 122.

Figure 5 is a diagram illustrating the extraction of a type region. In the example of Figure 5, the extraction unit 133 extracts the region 52 in which "(1) Male" is printed in type, and the region 53 in which "(2) Female" is printed in type, as type regions.

The selection unit 134 selects an area from the type region where the degree of overlap with the handwritten region satisfies a predetermined condition. For example, the selection unit 134 selects an area from the type region where at least a portion of the area overlaps with the handwritten region.

For example, in the example of FIG. 5, region 51, which is a handwritten region, and region 52, which is a type region, partially overlap. On the other hand, region 51, which is a handwritten region, and region 53, which is a type region, do not overlap. Therefore, the selection unit 134 selects region 52 as the type region corresponding to region 51.

In addition, for example, multiple type regions may overlap with a handwritten region. In such a case, the selection unit 134 can select the type region that has the largest area of overlap with the handwritten region.

Figure 6 shows an example of the items on the form. In the example in Figure 6, the options "1. Regular," "2. Current," and "3. Tax Preparation" are printed.

Consider the case where a circle is handwritten on the form shown in Figure 6. In this case, the circle is written in the position shown in Figure 7. Figure 7 shows an example of an overlapping area.

The extraction unit 133 extracts area 54 as a handwritten area, and areas 55, 56, and 57 as typed areas.

Area 58 is an area where area 54 and area 56 overlap. Area 59 is an area where area 54 and area 57 overlap. Here, since the area of area 59 is larger than the area of area 58, the selection unit 134 selects area 57, which is a handwritten area that corresponds to area 59.

The recognition unit 135 recognizes the character string printed in the area selected by the selection unit 134. The recognition unit 135 recognizes the character string in the typed area. In the example of FIG. 5, the recognition unit 135 recognizes the character string "(1) Man." In the example of FIG. 7, the recognition unit 135 recognizes the character string "3 Preparation for paying taxes."

The recognition unit 135 performs character recognition by referring to the dictionary information 123. Furthermore, the recognition unit 135 may preferentially recognize character strings included as options in the selection character string information 124.

For example, assume that the recognition unit 135 calculates the same recognition accuracy (probability) for the string of options for an item named "occupation" for "company employee" and "company employee."

On the other hand, when referring to the selected character string information 124 shown in FIG. 3, "company employee" is included in the options for the item name "occupation", but "company employee" is not included in the options for the item name "occupation". In this case, the recognition unit 135 recognizes the character string as "company employee".

In this way, the recognition unit 135 recognizes combinations of item names and options that are associated in advance.

The correction unit 136 removes pre-specified character strings from the character strings recognized by the recognition unit 135. For example, the correction unit 136 removes "(1)" from "(1) male." For example, the correction unit 136 deletes pre-specified information such as item numbers.

The correction function of the correction unit 136 can be enabled or disabled at the administrator's discretion.

The output control unit 137 outputs the recognition result by the recognition unit 135 in a predetermined format. For example, the output control unit 137 may output a combination of the item name and the recognized option, such as "Gender: Male."

Figure 8 is a flowchart showing the flow of the recognition process. As shown in Figure 8, first, the recognition device 10 reads the form (step S101). Next, the recognition device 10 identifies the position of the item on the form (step S102).

Then, the recognition device 10 uses the handwritten area extraction model to extract the handwritten area around the item (step S103). The recognition device 10 also uses the typed area extraction model to extract the typed area around the item (step S104).

Here, the recognition device 10 selects a typed region that corresponds to the handwritten region (step S105). For example, the recognition device 10 selects the typed region that has the largest area of overlap with the handwritten region.

Then, the recognition device 10 recognizes the character string written in the selected type area (step S106). Furthermore, the recognition device 10 corrects the character string (step S107). Note that step S107 may be omitted depending on the settings. Then, the recognition device 10 outputs the recognized character string (step S108).

As described above, the extraction unit 133 extracts handwritten areas from the form where text has been written by hand. The extraction unit 133 extracts typed areas from the form where type characters are printed. The selection unit 134 selects from the typed areas an area whose degree of overlap with the handwritten area satisfies a predetermined condition. In this way, the recognition device 10 can select handwritten areas and typed areas in association with each other without specifying the character strings and figures to be recognized in advance. As a result, according to this embodiment, it is possible to easily and generally recognize selected character strings on a form.

The extraction unit 133 extracts handwritten regions using an image recognition model that has learned the characteristics of handwritten figures. In this way, by having the image recognition model learn the characteristics of handwriting, it becomes possible to extract regions that include handwritten figures with incomplete shapes.

The selection unit 134 selects the region of the type that has the largest area of overlap with the handwritten region. This allows the recognition device 10 to quantitatively identify the selected option.

The recognition unit 135 recognizes the character string printed in the area selected by the selection unit 134. The correction unit 136 removes pre-specified character strings from the character string recognized by the recognition unit 135. This allows the recognition device 10 to remove unnecessary information from the recognition result and leave only important information.

The recognition unit 135 recognizes combinations of item names and options that are associated in advance. This allows the recognition device 10 to preferentially recognize character strings intended as options for an item.

The information, including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings, may be changed as desired unless otherwise specified. Furthermore, the specific examples, distributions, numerical values, etc. described in the embodiments are merely examples and may be changed as desired.

In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure. In other words, all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. Furthermore, each processing function performed by each device can be realized in whole or in any part by a CPU and a program analyzed and executed by the CPU, or can be realized as hardware using wired logic.

Figure 9 is a diagram illustrating an example of a hardware configuration. As shown in Figure 9, the recognition device 10 has a communication interface 10a, a HDD (Hard Disk Drive) 10b, a memory 10c, and a processor 10d. In addition, each part shown in Figure 9 is connected to each other via a bus or the like.

The communication interface 10a is a network interface card or the like, and communicates with other servers. The HDD 10b stores the programs and DBs that operate the functions shown in FIG. 2.

The processor 10d is a hardware circuit that operates a process that executes each function described in FIG. 2 and the like by reading a program that executes the same processes as the respective processing units shown in FIG. 2 from the HDD 10b and the like and expanding it in the memory 10c. In other words, this process executes the same functions as the respective processing units of the recognition device 10.

Specifically, processor 10d reads out from HDD 10b or the like a program having the same functions as scanner control unit 131, analysis unit 132, extraction unit 133, selection unit 134, recognition unit 135, correction unit 136, and output control unit 137. Then, processor 10d executes a process that executes the same processing as scanner control unit 131, analysis unit 132, extraction unit 133, selection unit 134, recognition unit 135, correction unit 136, and output control unit 137, etc.

In this way, the recognition device 10 operates as an information processing device that executes a learning method by reading and executing a program. The recognition device 10 can also realize functions similar to those of the above-mentioned embodiment by reading the program from a recording medium using a media reading device and executing the read program. Note that the program in these other embodiments is not limited to being executed by the recognition device 10. For example, the present invention can be similarly applied to cases where another computer or server executes a program, or where these cooperate to execute a program.

This program can be distributed via a network such as the Internet. In addition, this program can be recorded on a computer-readable recording medium such as a hard disk, a flexible disk (FD), a CD-ROM, an MO (Magneto-Optical disk), or a DVD (Digital Versatile Disc), and can be executed by being read from the recording medium by a computer.

REFERENCE SIGNS LIST 10 Recognition device 11 IF section 12 Storage section 13 Control section 51, 52, 53, 54, 55, 56, 57, 58, 59 Area 121 Handwritten area extraction model information 122 Printed area extraction model information 123 Dictionary information 124 Selected character string information 131 Scanner control section 132 Analysis section 133 Extraction section 134 Selection section 135 Recognition section 136 Correction section 137 Output control section

Claims (5)

Extracting a first area where a handwritten entry is made from the form;
Extracting a second area in which type is printed from the document;
selecting an area from the second area whose degree of overlap with the first area satisfies a predetermined condition ;
Recognizing a character string printed in the area selected by the selection process and included in a choice that is previously associated with an item name
A recognition method characterized in that the processing is executed by a computer. The recognition method according to claim 1, characterized in that the process of extracting the first region extracts the first region using an image recognition model that has learned the characteristics of handwritten figures. The recognition method according to claim 1 or 2, characterized in that the selection process selects the second region that has the largest area of overlap with the first region. 4. The method according to claim 1, further comprising the step of: excluding a character string designated in advance from the character string recognized by the recognition process. a handwritten area extraction unit that extracts a first area that is handwritten from a form;
a typed character area extraction unit that extracts a second area in which typed characters are printed from the document;
a selection unit that selects, from the second region, a region whose degree of overlap with the first region satisfies a predetermined condition;
a recognition unit that recognizes a character string printed in the area selected by the selection unit and that is included in an option that is previously associated with an item name;
A recognition device comprising:

Images