JP7779130B2 - Image recognition device and image recognition method - Google Patents

Description

The present invention relates to an image recognition device and an image recognition method.

Image recognition devices are known that use OCR (Optical Character Reader) technology to recognize images of characters and figures formed on recording media.

One such image recognition device has been disclosed that, when defining a form image to be recognized, accepts the specification of an area containing items and options for the items included in the form image, and displays a definition that associates the items with the options extracted from the area (see, for example, Patent Document 1).

Image recognition devices are required to have excellent recognition accuracy for selection acceptance figures that are included in input images such as form images and that accept selections for items.

The present invention aims to provide an image recognition device with excellent recognition accuracy for selection acceptance figures contained in input images.

An image recognition device according to one aspect of the present invention is an image recognition device capable of recognizing a selection acceptance figure that is included in an input image into which at least a portion of a manuscript is input, and that accepts selections for items.The image recognition device has: a vertex candidate output unit that outputs a detection result of a vertex candidate of the selection acceptance figure based on characteristic points of the external shape of the figure included in the input image; a vertex candidate evaluation unit that determines whether the vertex candidate is a vertex candidate of the selection acceptance figure based on the detection result of the vertex candidate received from the vertex candidate output unit; and a recognition result output unit that outputs a recognition result of the selection acceptance figure based on the determination result by the vertex candidate evaluation unit.The vertex candidate evaluation unit determines whether the vertex candidate is a vertex candidate of the selection acceptance figure by determining whether a straight line connecting multiple vertex candidates is colored, or by comparing the distance between the vertex candidates with the size of the characters included in the input image .

This invention provides an image recognition device with excellent recognition accuracy for selection acceptance figures contained in input images.

1 is a diagram illustrating an example of an input image of an image recognition device according to an embodiment; 1 is a diagram illustrating an example of an output image of an image recognition device according to an embodiment; 1 is a diagram illustrating an example of the configuration of an image forming apparatus including an image recognition device according to an embodiment. 1 is a block diagram of an example of the configuration of an image forming apparatus according to an embodiment; FIG. 2 is a block diagram of an example of the hardware configuration of a controller according to the embodiment. FIG. 2 is a block diagram of an example of the hardware configuration of an IPU according to an embodiment. FIG. 2 is a block diagram illustrating an example of a functional configuration of an IPU according to an embodiment. FIG. 4 is a flowchart of an example of an operation of reading a document by the image forming apparatus according to the embodiment. FIG. 10 is a flow diagram of an example of recognition processing by an IPU according to an embodiment. FIG. 10 is a diagram illustrating an example of an input image according to the embodiment. FIG. 10 is a diagram illustrating a histogram of an input image according to the embodiment. FIG. 10 is a diagram illustrating an example of a binarized image according to the embodiment. FIG. 10 is a diagram showing an example of an outer shape image according to the embodiment; FIG. 1 is a first diagram illustrating an example of a vertex candidate detection process according to the embodiment. FIG. 10 is a second diagram illustrating an example of a vertex candidate detection process according to the embodiment. FIG. 3 is a third diagram illustrating an example of a vertex candidate detection process according to the embodiment. 10A to 10C are diagrams illustrating an example of a vertex candidate evaluation process according to the embodiment. FIG. 10 is a diagram illustrating a first example of an output image according to the embodiment. FIG. 10 is a diagram showing a second example of an output image according to the embodiment. 10A to 10C are diagrams illustrating an example of tilt detection processing according to an embodiment. 10A to 10C are diagrams illustrating an example of color inversion processing according to an embodiment. 10A to 10C are diagrams illustrating an example of a process for enlarging a black region according to an embodiment. FIG. 10 is a diagram showing a first example of an output image according to another embodiment. FIG. 10 is a block diagram illustrating an example of the functional configuration of an IPU according to another embodiment. FIG. 10 is a diagram showing a second example of an output image according to another embodiment.

The following describes in detail the embodiments of the present invention with reference to the drawings. In each drawing, the same components are designated by the same reference numerals, and duplicate explanations will be omitted where appropriate.

The image recognition device according to the embodiment is capable of recognizing a selection acceptance graphic that accepts selections for items and that is included in an input image that contains at least a portion of a manuscript.

A selection acceptance figure is a figure that is formed around an item to be selected and into which a mark indicating that it has been selected is entered. For example, a selection acceptance figure is a rectangular figure formed around an item to be selected that is included in a manuscript, such as a checkbox, into which a check mark indicating that it has been selected is added when the item is selected. A target item is an item that is selected using a selection acceptance figure.

In this embodiment, we will explain the case where the selection acceptance shape is a check box as an example.

Figures 1 and 2 are diagrams illustrating input and output images for an image recognition device according to an embodiment, with Figure 1 illustrating input image Im0 and Figure 2 illustrating output image Q.

As shown in Figure 1, input image Im0 includes an image of a target item 510 and a check box 511 formed adjacent to the target item 510.

The image recognition device according to the embodiment recognizes check boxes 511 from the input image Im0 and outputs an output image Q to which a recognition mark 521 indicating the recognized check box 511 has been added, as shown in FIG. 2.

By recognizing the check box 511 using the image recognition device according to the embodiment, a downstream processing device can accurately recognize whether or not a check mark 522 has been added to the check box 511 in the document Pi.

The image recognition device according to the embodiment includes a vertex candidate output unit that outputs the detection results of vertex candidates for the check box 511 based on feature points of the external shape of a figure included in the input image Im0, and a recognition result output unit that outputs the recognition results of the check box 511 based on the vertex candidate detection results output from the vertex candidate output unit. This configuration provides an image recognition device with excellent recognition accuracy for the check box 511 included in the input image Im0. Here, a vertex refers to a corner of the figure in the check box 511.

The following describes an image forming device having an image recognition device according to an embodiment as an example. This image forming device is a multi-function peripheral (MFP) that has the function of forming an image on a recording medium such as paper, and the function of reading an original document such as paper on which an image has been formed. The image recognition device according to an embodiment recognizes a check box 511 from an input image Im0, which contains at least a portion of an image of an original document read by the image forming device.

<Configuration example of image forming apparatus 1>
(Overall configuration example)
3 is a diagram showing an example of the overall configuration of the image forming apparatus 1. As shown in FIG.

The paper feed unit 400 has paper feed cassettes 421 and 422 that store paper of different sizes, and a paper feed means 423 consisting of various rollers that transport the paper stored in the paper feed cassettes 421 and 422 to the image formation position by the plotter 300.

The plotter 300 has an exposure device 331, a photosensitive drum 332, a developing device 333, a transfer belt 334, and a fixing device 335. The plotter 300 exposes the photosensitive drum 332 using the exposure device 331 to form a latent image on the photosensitive drum 332 based on image data of an original scanned by the scanner 100 or image data input from an external PC or the like via a network I/F. The plotter 300 develops the image by supplying different color toners to the photosensitive drum 332 using the developing device 333. The plotter 300 transfers the toner image developed on the photosensitive drum 332 using the transfer belt 334 to paper supplied from the paper feed unit 400, and then the fixing device 335 melts the toner that makes up the toner image transferred to the paper, fixing the color image to the paper.

Scanner 100 has an ADF (Auto Document Feeder) 41, a scanner unit 42, and a paper output tray 43. Scanner 100 drives ADF 41 to transport a document placed on ADF 41 to scanner unit 42. Scanner 100 drives scanner unit 42 to capture an image of the document transported from ADF 41. If a document is not placed on ADF 41 and is placed directly on scanner unit 42, scanner unit 42 captures an image of the placed document. In other words, scanner unit 42 operates as a document imaging unit.

The scanner unit 42 shown in Figure 3 is, for example, a differential mirror drive system with an integrated optical sensor drive system. The scanner unit 42 has a first mirror unit 204, a second mirror unit 210, a lens 216, and a first sensor board 215.

The first mirror unit 204 is an optical unit that includes a light source, such as an LED (Light Emitting Diode), and a mirror. The first mirror unit 204 illuminates the placed document with light from the light source and reflects the light reflected by the document toward the second mirror unit 210.

The second mirror unit 210 reflects the light from the first mirror unit 204 toward the lens 216. The lens 216 causes the light from the second mirror unit 210 to form an approximate image on the photoelectric conversion element 214 provided on the first sensor board 215. The photoelectric conversion element 214 photoelectrically converts the approximate image of the original document into an analog image signal, thereby reading the original document.

Figure 4 is a block diagram illustrating the configuration of the image forming device 1. The image forming device 1 includes a scanner 100, a controller 200, a plotter 300, an image memory 401, and an IPU (Image Processing Unit) 500.

The image forming device 1 uses the IPU 500 to perform image recognition processing on the input image Im0, which is at least a portion of the original document Pi scanned by the scanner 100. Note that the IPU 500 may also perform image processing other than image recognition processing. The image forming device 1 forms an image on paper using the plotter 300 based on the image data after image processing by the IPU 500 and image data of a test chart (test pattern), and outputs the paper with the image formed on it as a printout Po.

The controller 200 controls the entire image forming device 1.

The image memory 401 includes volatile memory, a hard disk, etc. The image memory 401 provides temporary storage for sending and receiving image data between the scanner 100, controller 200, plotter 300, and IPU 500, as well as permanent storage for later use.

IPU500 is an example of an image recognition device that can recognize check boxes 511 included in input image Im0 and that accept selections for items.

Image forming device 1 can connect to PC 2, server 3, etc. via network NW to send and receive image data, etc.

<Example of Hardware Configuration of Controller 200>
Fig. 5 is a block diagram showing an example of the hardware configuration of the controller 200. As shown in Fig. 5, the controller 200 has a CPU (Central Processing Unit) 10, a RAM (Random Access Memory) 11, a ROM (Read Only Memory) 12, a HDD (Hard Disk Drive) 13, and a communication I/F (Interface) 14. These are electrically connected to each other via a bus 15. The communication I/F 14 is also connected to a display device 16 and an input operation unit 17.

The CPU 10 is a computing unit that controls the overall operation of the image forming apparatus 1. The RAM 11 is a volatile storage medium that allows high-speed reading and writing of information. The CPU 10 uses the RAM 11 as a working area when processing information. The ROM 12 is a read-only non-volatile storage medium that stores programs such as firmware.

The HDD 13 is a non-volatile storage medium that can read and write information, and stores the OS (Operating System), various control programs, application programs, etc. The communication I/F 14 is an interface that connects and controls the bus 15 with various hardware and networks, etc.

The display device 16 is a visual user interface that allows the user to check the status of the image forming device 1. The input operation unit 17 is a user interface, such as a keyboard or mouse, that allows the user to input information into the image forming device 1.

In the image forming device 1, a software control unit is configured by loading a program stored on a recording medium such as ROM 12, HDD 13, or optical disk into RAM 11 and running it under the control of CPU 10 in the hardware configuration described above. The image forming device 1 realizes its functions by combining the software control unit configured in this way with hardware.

<Example of hardware configuration of IPU500>
Fig. 6 is a block diagram showing an example of the hardware configuration of the IPU 500. As shown in Fig. 6, the IPU 500 has a CPU 20, a RAM 21, a ROM 22, a HDD 23, and a communication I/F 24. These are electrically connected to each other via a bus 25.

The CPU 20 is a computing unit that controls the overall operation of the IPU 500. The RAM 21 is a volatile storage medium that allows high-speed reading and writing of information. The CPU 20 uses the RAM 21 as a working area when processing information. The ROM 22 is a read-only non-volatile storage medium that stores programs such as firmware.

The HDD 23 is a non-volatile storage medium that can read and write information, and stores the OS, various control programs, application programs, etc. The communication I/F 24 is an interface that connects and controls the bus 25 with various hardware and networks, etc.

In the above hardware configuration, the IPU 500 configures a software control unit by loading a program stored on a recording medium such as the ROM 22, HDD 23, or optical disk into the RAM 21 and running it under the control of the CPU 20. The software control unit thus configured configures the IPU 500 into functional blocks that realize the functions of the IPU 500.

<Example of functional configuration of IPU 500>
7 is a block diagram showing an example of the functional configuration of the IPU 500. As shown in FIG. 7, the IPU 500 has an input unit 31, a binarization processing unit 32, a contour shape extraction unit 33, a feature point extraction unit 34, a vertex candidate output unit 35, a vertex candidate evaluation unit 36, a registration unit 37, and a recognition result output unit 38. The IPU 500 realizes these functions by having the CPU 20 read out a program stored in the ROM 22 and execute the program using the RAM 21 as a work area.

The input unit 31 acquires an input image Im0 from the scanner 100, which is at least a portion of the image of the original document Pi scanned by the scanner 100. The input unit 31 also acquires information about the size, resolution, color, background, etc. of the input image Im0. The input unit 31 outputs the acquired input image Im0 and information about the size, resolution, color, background, etc. of the input image Im0 to the binarization processing unit 32.

The binarization processing unit 32 performs binarization processing on the input image Im0 using a binarization threshold based on a histogram generated from the pixel values of each of the multiple pixels contained in the input image Im0, and outputs the resulting binarized image Im1 to the outer shape extraction unit 33.

The binarization threshold does not have to be based on a histogram, but may be determined based on color information of the input image Im0, etc. However, determining the binarization threshold based on a histogram is more preferable, as it reduces the influence of images other than figures in the input image Im0. Furthermore, in order to perform the binarization process quickly, preprocessing of the binarization process may involve monochrome image conversion if the input image Im0 is a color image, or conversion that reduces the image resolution.

The outer shape extraction unit 33 extracts the outer shape of a figure contained in a binarized image Im1 based on the input image Im0, and outputs the resulting outer shape image Im2 to the feature point extraction unit 34. For example, the outer shape extraction unit 33 detects colored pixels, such as pixels colored black, from among the multiple pixels contained in the input image Im0, and extracts the outer shape of a figure composed of these colored pixels.

The feature point extraction unit 34 extracts feature points in the outer shape of the figure extracted by the outer shape extraction unit 33, and outputs the extraction results to the vertex candidate output unit 35. Feature points refer to characteristic parts of the figure extracted by the outer shape extraction unit 33. Characteristic parts can be defined in advance depending on the figure to be recognized. In this embodiment, for example, corners in the outer shape are extracted as feature points. A corner refers to a part where two straight lines intersect. In this embodiment, the extraction result is position information of the feature points within the input image Im0. For example, performing feature point extraction processing by linear approximation of the outer shape is preferable, as it simplifies the processing and shortens the processing time.

The vertex candidate output unit 35 outputs the detection result of the vertex candidate of the check box 511 based on the feature points extracted by the feature point extraction unit 34. A vertex candidate means the position of a part that may be the vertex of the check box 511. The detection result of the vertex candidate is, for example, the position of the vertex candidate within the input image Im0. In this embodiment, the vertex candidate output unit 35 outputs a vertex candidate image Im3 containing vertex candidate information to the vertex candidate evaluation unit 36 as the detection result of the vertex candidate.

In this embodiment, the vertex candidate output unit 35 outputs vertex candidates for the check box 511 based on feature points extracted from the binarized image Im1 obtained by binarizing the input image Im0 using a binarization threshold based on a histogram generated from the pixel values of each of the multiple pixels included in the input image Im0.

In addition, in this embodiment, the vertex candidate output unit 35 outputs the detection result of the vertex candidates of the check box 511 based on the number of pixels included in the image area connecting the multiple feature points extracted by the feature point extraction unit 34, whose pixel values match the pixel values of the pixels included in the feature points.

The vertex candidate output unit 35 may be configured to output vertex candidates for the check box 511 based on feature points of the input image Im0, which are distinguished into handwritten and non-handwritten areas, based on at least one of the color component information and histogram information of the input image Im0. Here, a handwritten area refers to an area in the input image Im0 that corresponds to the handwritten area when a document Pi containing an area handwritten by a user of the image forming device 1 is read by the scanner 100.

Since handwritten regions are based on patterns or marks handwritten by a user on the document Pi, the image density is not constant compared to images formed on the document Pi by an image forming device or the like. Therefore, the vertex candidate output unit 35 can distinguish, for example, regions in the input image Im0 where the density fluctuation range of the image is greater than a predetermined fluctuation range threshold as handwritten regions, and regions below the fluctuation range threshold as non-handwritten regions.

The vertex candidate output unit 35 may also be configured to output vertex candidates for the check box 511 based on feature points of the input image Im0, which have been corrected for tilt and are detected from the ruled line area included in the input image Im0. Here, the ruled line area refers to the area that corresponds to the ruled line included in the input image Im0 when the original Pi containing the ruled line is read by the scanner 100.

The vertex candidate output unit 35 may also be configured to output vertex candidates for the check box 511 based on feature points of the input image Im0, in which the pixel values of each of the multiple pixels contained in the input image Im0 are color-inverted. Color inversion refers to converting a specific color into a color that has a complementary relationship with the input image Im0. For example, color inversion is black-and-white inversion, in which white image areas in a black-and-white image are converted into black, and black image areas are converted into white.

The vertex candidate output unit 35 may also be configured to output vertex candidates for the check box 511 based on feature points in a color component image obtained by decomposing the pixel values of each of the multiple pixels included in the input image Im0 into color components. Here, a color component image refers to an image obtained by color-separating the color input image Im0, which uses multiple primary colors, into each primary color. For example, if the input image Im0 is a color image with red (R), green (G), and blue (B) as the three primary colors, the color component images correspond to an image consisting of only red, an image consisting of only green, and an image consisting of only blue.

The vertex candidate output unit 35 may also be configured to output vertex candidates for the check box 511 based on feature points of the input image Im0, which are separated into form areas and non-form areas in the input image Im0, based on at least one of the color component information and histogram information of the input image Im0. Here, the form area refers to the area corresponding to the form included in the input image Im0 when a document Pi including the form is read by the scanner 100. A form is a document with blank spaces for filling in information, and is a general term for ledgers, slips, etc.

The vertex candidate output unit 35 may also be configured to output vertex candidates for the check box 511 based on feature points of the input image Im0, which is an enlarged version of a black area contained in the input image Im0. A black area refers to an image area where pixel values indicate black.

The vertex candidate evaluation unit 36 evaluates the vertex candidate information contained in the vertex candidate image Im3 received from the vertex candidate output unit 35 to determine whether the vertex candidate is the vertex of the check box 511.

Because the check box 511 is a rectangular shape, the vertex candidate evaluation unit 36 can determine whether a vertex candidate is a vertex of the check box 511 by evaluating whether the vertex candidate corresponds to one of the four corner vertices of the rectangular shape. In other words, the vertex candidate evaluation unit 36 can determine whether a vertex candidate corresponds to one of the four corner vertices of the rectangular shape based on at least one of the size information and position information of the check box 511.

The evaluation criteria used by the vertex candidate evaluation unit 36 include, for example, the following:
(1) The lines connecting multiple vertex candidates are colored.
(2) The vertex candidates form part of a rectangle or square.
(3) The distance between the vertex candidates is an appropriate size compared to the size of the characters contained in the input image Im0.

When two or three vertex candidates are detected from the input image Im0, the vertex candidate evaluation unit 36 performs calculations to estimate the other vertex candidates out of the four vertex candidates based on the detected vertex candidates, and evaluates the vertex candidates using these estimation results as the vertex candidates.

The registration unit 37 receives the evaluation result from the vertex candidate evaluation unit 36, and if the vertex candidate is the vertex of the check box 511, it registers information corresponding to the presence of a vertex and the position information of this vertex. If there is no vertex of the check box 511, the registration unit 37 registers information corresponding to the absence of a vertex. The registration unit 37 then outputs the position information of the vertex to the recognition result output unit 38.

The recognition result output unit 38 outputs the recognition result of the check box 511 received from the registration unit 37. In other words, the recognition result output unit 38 outputs the recognition result of the check box 511 based on the vertex candidate detection result output from the vertex candidate output unit 35. In this embodiment, the recognition result output unit 38 outputs information on the presence or absence of a vertex as the recognition result of the check box 511, and, if a vertex is present, the position information of the check box 511. Because the check box 511 has a rectangular outer shape, the recognition result of the check box 511 includes the position of at least one vertex of the rectangle in the check box 511.

The recognition result output unit 38 outputs, for example, the vertex position information of the check box 511 to the controller 200, which is an external device of the IPU 500. However, the external device is not limited to the controller 200, and may be other external devices such as the plotter 300 or the display device 16.

<Example of document reading operation by image forming apparatus 1>
8 is a flowchart showing an example of an original reading operation by the image forming apparatus 1. The image forming apparatus 1 starts the operation of FIG. 8 when it receives an instruction to start original reading from a user using the input operation unit 17.

First, in step S81, the image forming device 1 uses the scanner 100 to read the original document Pi placed on the ADF 41 or scanner unit 42.

Next, in step S82, the image forming device 1, via the controller 200, counts the number of original documents Pi read by the scanner 100.

Next, in step S83, the image forming apparatus 1 determines, via the controller 200, whether the scanner 100 has read all of the original documents Pi. For example, the controller 200 can determine whether all of the original documents Pi have been read by determining whether original documents Pi are placed on the ADF 41 or the scanner unit 42, or whether the number of original documents Pi that have been read has reached the specified number of original documents Pi to be read that was input using the input operation unit 17.

If it is determined in step S83 that not all of the original documents Pi have been read (step S83, No), the image forming device 1 performs the operations from step S81 onwards again. On the other hand, if it is determined that all of the original documents Pi have been read (step S83, Yes), in step S84, the image forming device 1 displays information about the original documents Pi on the display device 16. This information about the original documents Pi is, for example, information such as the number of pages of the original documents Pi that have been read or the size of the original documents Pi.

Next, in step S85, the image forming device 1 determines, via the controller 200, whether or not to recognize the check box 511 on the scanned document Pi. This determination can be made based on, for example, a user operation using the input operation unit 17.

If it is determined in step S85 that the check box 511 is not recognized (step S85, No), the image forming device 1 terminates operation. On the other hand, if it is determined that the check box 511 is recognized (step S85, Yes), in step S86, the image forming device 1 executes recognition processing for the check box 511. Details of this recognition processing will be described below with reference to Figure 9. The image forming device 1 terminates operation after completing recognition processing for the check box 511.

In this way, the image forming device 1 can read the original document Pi.

<Example of recognition processing by IPU 500>
9 is a flowchart showing an example of a process for recognizing the check box 511 by the IPU 500. The IPU 500 starts the process of FIG. 9 at the timing of step 86 in the operation of FIG.

First, in step S91, the IPU 500 acquires the input image Im0 by inputting it from the scanner 100 via the input unit 31. The input unit 31 also acquires information about the size, resolution, color, or background of the input image Im0. The input unit 31 outputs the acquired input image Im0, as well as information about the size, resolution, color, or background of the input image Im0, to the binarization processing unit 32.

Next, in step S92, the IPU 500 causes the binarization processing unit 32 to perform binarization processing of the input image Im0 using a binarization threshold based on a histogram generated from the pixel values of each of the multiple pixels contained in the input image Im0, and outputs the processed result, the binarized image Im1, to the outer shape extraction unit 33.

Next, in step S93, the IPU 500 causes the outer shape extraction unit 33 to extract the outer shapes of the figures contained in the binarized image Im1 based on the input image Im0, and outputs the extracted outer shape image Im2 to the feature point extraction unit 34. If multiple figures are contained in the binarized image Im1, the outer shape extraction unit 33 outputs the outer shape image Im2 containing the extracted outer shapes of all of the figures to the feature point extraction unit 34.

Next, in step S94, the IPU 500 causes the feature point extraction unit 34 to extract feature points in the outer shape of the figure extracted by the outer shape extraction unit 33, and outputs the extraction results to the vertex candidate output unit 35. If the outer shape of the figure includes multiple feature points, the feature point extraction unit 34 outputs the extraction results of all feature points to the vertex candidate output unit 35. Furthermore, if the outer shapes of multiple figures are included in the outer shape image Im2, the feature point extraction unit 34 outputs the extraction results of all feature points in the outer shapes of all figures to the vertex candidate output unit 35.

Next, in step S95, the IPU 500 causes the vertex candidate output unit 35 to output the detection results of the vertex candidates for the check box 511 based on the feature points extracted by the feature point extraction unit 34. For example, the vertex candidate output unit 35 outputs a vertex candidate image Im3 containing vertex candidate information to the vertex candidate evaluation unit 36 as the vertex candidate detection result. If multiple vertex candidates are included, the vertex candidate output unit 35 outputs a vertex candidate image Im3 containing the detection results of all vertex candidates to the vertex candidate evaluation unit 36.

Next, in step S96, the IPU 500 determines whether the vertex candidate is the vertex of the check box 511 by using the vertex candidate evaluation unit 36 to evaluate the vertex candidate information contained in the vertex candidate image Im3 received from the vertex candidate output unit 35. If there are multiple vertex candidates, the vertex candidate evaluation unit 36 determines whether all of the vertex candidates are the vertices of the check box 511.

Next, in step S97, the IPU 500 receives the evaluation result from the vertex candidate evaluation unit 36 via the registration unit 37, and if the vertex candidate is the vertex of the check box 511, registers the position information of this vertex. The registration unit 37 outputs the position information of all vertices to the recognition result output unit 38.

Next, in step S98, the IPU 500 outputs the recognition result of the check box 511 received from the registration unit 37 via the recognition result output unit 38.

In this way, the IPU 500 can perform recognition processing of the check box 511 contained in the input image Im0.

<Example of recognition processing result by IPU 500>
Next, an example of the result of the recognition process by the IPU 500 will be described.

(Example of input image Im0)
10 is a diagram showing an example of input image Im0. Input image Im0 shown in Fig. 10 shows an extracted area surrounding check boxes 511a and 511b, which is an even smaller area in input image Im0 shown in Fig. 1.

(Example of binarized image Im1)
FIG. 11 is a diagram illustrating a histogram of input image Im0. In FIG. 11, the horizontal axis represents the pixel values of pixels included in input image Im0, and the vertical axis represents frequency. Based on the histogram shown in FIG. 11, the binarization processing unit 32 can determine an appropriate binarization threshold value for extracting shapes included in input image Im0. Through the binarization processing, the binarization processing unit 32 can obtain a binarized image Im1 as shown in FIG. 12.

(Example of outer shape image Im2)
13 is a diagram showing an example of the outer shape image Im2. As shown in Fig. 13, the outer shape extractor 33 extracts the outer shape 512 of the figure included in the input image Im0.

(An example of a vertex candidate 514)
14 to 16 are diagrams for explaining an example of the process of detecting the vertex candidates 514, with FIG. 14 being FIG. 1, FIG. 15 being FIG. 2, and FIG. 16 being FIG. 3.

As shown in Figure 14, the feature point extraction unit 34 extracts feature points 513 based on the outer shape 512 contained in the outer shape image Im2.

Figure 15 shows the area around the check box 511a in the outer shape image Im2. The vertex candidate output unit 35 detects vertex candidates 514a and 514b, which are two of the four corners of the check box 511a.

Figure 16 is a diagram showing check box 511a in Figure 15 at an even larger scale than Figure 15. As shown in Figure 16, the positions of vertex candidates 514a and 514b correspond to the outer frame portion of the black image area in check box 511a. Therefore, the vertex candidate output unit 35 outputs position information of vertex candidate 514a corrected by referring to outer position 514a' and inner position 514a'' of the black image area near vertex candidate 514a. In addition, the vertex candidate output unit 35 outputs position information of vertex candidate 514b corrected by referring to outer position 514b' and inner position 514b'' of the black image area near vertex candidate 514b.

Figure 17 is a diagram illustrating an example of the evaluation process for vertex candidates 514.

The vertex candidate evaluation unit 36 evaluates the vertex candidates 514a and 514b based on the number of pixels included in the image region connecting the detected vertex candidates 514a and 514b whose pixel values match the pixel values corresponding to the color black included in the feature point 513. If the number of pixels included in the image region connecting the vertex candidates 514a and 514b whose pixel values correspond to black is greater than a predetermined pixel number threshold, the vertex candidate evaluation unit 36 designates the vertex candidates 514a and 514b as vertex candidates, respectively.

In Figure 17, for example, if a falsely detected vertex candidate 515 is detected, many of the pixels included in the image area connecting the falsely detected vertex candidate 515 and vertex candidate 514a will have pixel values corresponding to white, and the number of pixels corresponding to black will be below a predetermined pixel number threshold. Therefore, the vertex candidate evaluation unit 36 evaluates the falsely detected vertex candidate 515 as being inappropriate as a vertex candidate, and can remove the falsely detected vertex candidate 515 from the vertex candidates.

(Example of output image)
18 and 19 are diagrams showing output images: Fig. 18 shows an output image Q according to the first example, and Fig. 19 shows an output image Q' according to the second example.

In Figure 18, output image Q includes a recognition mark 521 indicating the check box 511. On the other hand, in Figure 19, output image Q' includes, in addition to the recognition mark 521, an item mark 523 indicating the target item 510.

(Tilt detection processing example)
Fig. 20 is a diagram illustrating an example of the inclination detection process performed by the vertex candidate output unit 35. Fig. 20 shows a binarized image Im1 obtained by binarizing the input image Im0.

Here, if the scanner 100 reads the original document Pi while it is tilted, the input image Im0 and the binarized image Im1 will be tilted, and the IPU 500 may not be able to detect the vertex candidate 514 of the check box 511.

As shown in Figure 20, the ruled line area 516 corresponding to the ruled lines included in the document Pi is arranged along the outline of the document Pi. Therefore, the vertex candidate output unit 35 corrects the inclination of the input image Im0 and the binary image Im1 based on the inclination of the ruled line area 516, and detects the vertex candidates 514 from the corrected binary image Im1. This allows the IPU 500 to ensure high accuracy in detecting the vertex candidates 514 of the check boxes 511.

(Example of color inversion processing)
Fig. 21 is a diagram illustrating an example of color inversion processing, showing a color-inverted image Im1' obtained by inverting black and white from a binarized image Im1 obtained by binarizing an input image Im0.

The vertex candidate output unit 35 can detect the vertex candidates 514 using the color-inverted image Im1' even if the document Pi contains a color-inverted image or the background of the image is colored. This allows the IPU 500 to ensure high accuracy in detecting the vertex candidates 514 of the check box 511.

(Example of enlarging the black area)
22 is a diagram illustrating an example of the enlargement process of a black region. The binarized image Im1 is an image obtained by binarizing the input image Im0, and shows the state before the enlargement process of the black region is performed. The enlarged black region image Im1'' is an image in which the area of the black region is increased by thickening the frame of the frame figure of the check box 511 in the binarized image Im1.

For example, if the document Pi has a check box 511 that is missing a portion due to degradation of the image formed on the document Pi, the accuracy of recognizing the check box 511 may decrease. By performing a black area enlargement process, the missing portion of the check box 511 can be filled in, allowing the IPU 500 to ensure high accuracy in detecting the vertex candidates 514 of the check box 511.

<Actions and Effects of IPU500>
Next, the effects of the IPU 500 will be described.

With conventional software using OCR technology, check boxes could not be correctly recognized unless they were individually defined. Furthermore, when recognizing forms using OCR technology, simply defining the frame that contains the check boxes could result in check boxes not being recognized correctly or reduced recognition accuracy. Therefore, in order to improve check box recognition accuracy, users had to manually define each of the multiple check boxes formed within the ruled lines included in the form, which was time-consuming.

The IPU 500 according to the embodiment is an image recognition device capable of recognizing a check box 511 (selection acceptance figure) that accepts selection for an item and is included in an input image Im0 into which at least a portion of a document Pi has been input. The IPU 500 also has a vertex candidate output unit 35 that outputs a detection result of vertex candidates 514 of the check box based on feature points 513 of the outer shape of the figure included in the input image Im0, and a recognition result output unit 38 that outputs a recognition result of the check box 511 based on the detection result of the vertex candidates 514 output from the vertex candidate output unit 35. For example, the check box 511 has a rectangular outer shape, and the recognition result of the check box 511 includes the position of at least one vertex of the rectangle in the check box 511.

In this embodiment, the check box 511 is recognized based on vertex candidates 514 based on feature points 513 of the outer shape, thereby improving the recognition accuracy of the check box 511. As a result, in this embodiment, an IPU 500 can be provided that has excellent recognition accuracy for the check box 511 included in the input image Im0.

In this embodiment, a rectangular frame-shaped checkbox 511 is used as an example of the selection acceptance shape, but this is not limited to this. As long as the shape has vertices, it may be a polygon other than a rectangle, such as a triangle or hexagon, and it does not have to be a frame-shaped shape.

In addition, in this embodiment, the vertex candidate output unit 35 outputs the detection result of the vertex candidates 514 of the check box 511 based on the number of pixels included in the image area connecting the multiple feature points 513, whose pixel values match the pixel values of the pixels included in the feature points 513. As a result, in this embodiment, it is possible to remove erroneously detected vertex candidates 514 and detect the vertex candidates 514 of the check box 511, thereby providing an IPU 500 with excellent recognition accuracy for the check box 511.

In addition, in this embodiment, the vertex candidate output unit 35 outputs the vertex candidates 514 of the check box 511 based on feature points 513 extracted from the binarized image Im1 obtained by binarizing the input image Im0 using a binarization threshold based on a histogram generated from the pixel values of each of the multiple pixels included in the input image Im0. This makes it possible to detect the vertex candidates 514 of the check box 511 even in a document Pi on which a handwritten pattern has been written or a document Pi on which a noise pattern has been overwritten on the check box 511, thereby providing an IPU 500 with excellent recognition accuracy for the check box 511.

The vertex candidate output unit 35 may be configured to output vertex candidates 514 of the check box 511 based on feature points 513 of the input image Im0, which are distinguished into handwritten and non-handwritten areas in the input image Im0, based on at least one of the color component information and histogram information of the input image Im0. This makes it possible to detect vertex candidates 514 of the check box 511 even in a document Pi on which a handwritten pattern has been entered, thereby providing an IPU 500 with excellent recognition accuracy for the check box 511.

The recognition result output unit 38 may also be configured to output the recognition result of the check box 511 based on at least one of the size information and position information of the check box 511 figure. This makes it possible to provide an output image Q including target items 510 that accept selection using the check box 511 to a device that performs processing in a process downstream of the IPU 500.

The vertex candidate output unit 35 may also be configured to output vertex candidates 514 of the check box 511 based on feature points 513 of the input image Im0, which has been corrected for tilt and is detected from the ruled line area 516 included in the input image Im0. This makes it possible to detect vertex candidates 514 of the check box 511 even for an input image Im0 obtained by scanning a tilted document Pi, thereby providing an IPU 500 with excellent recognition accuracy for the check box 511.

The vertex candidate output unit 35 may also be configured to output vertex candidates 514 of the check box 511 based on feature points 513 of the input image Im0, in which the pixel values of each of the multiple pixels contained in the input image Im0 are inverted. This makes it possible to detect the vertex candidates 514 of the check box 511 even in documents Pi on which a color-inverted image is formed or documents Pi with a colored background, thereby providing an IPU 500 with excellent recognition accuracy for the check box 511.

The vertex candidate output unit 35 may also be configured to output vertex candidates 514 of the check box 511 based on feature points 513 in a color component image obtained by decomposing the pixel values of each of the multiple pixels included in the input image Im0 into color components. This makes it possible to detect vertex candidates 514 of the check box 511 even in a color document Pi or a document Pi with a handwritten pattern, thereby providing an IPU 500 with excellent recognition accuracy for the check box 511.

The vertex candidate output unit 35 may also be configured to output vertex candidates 514 of check boxes 511 based on feature points 513 of input image Im0, which distinguish between form regions and non-form regions in the input image Im0, based on at least one of the color component information and histogram information of the input image Im0. This makes it possible to detect vertex candidates 514 of check boxes 511 even in documents Pi on which handwritten patterns have been written or documents Pi on which check boxes 511 have been overwritten with noise patterns, thereby providing an IPU 500 with excellent recognition accuracy for check boxes 511.

The vertex candidate output unit 35 may also be configured to output vertex candidates 514 of the check box 511 based on feature points 513 of the input image Im0, which is an enlarged version of the black area contained in the input image Im0. This makes it possible to detect the vertex candidates 514 of the check box 511 even in documents Pi in which part of the check box 511 is missing due to deterioration of the image formed on the document Pi, thereby providing an IPU 500 with excellent recognition accuracy for the check box 511.

[Other Preferred Embodiments]
Other embodiments will be described below. Note that the same components as those in the above-described embodiment are denoted by the same reference numerals, and redundant description will be omitted as appropriate.

Figure 23 is a diagram showing an output image Qa according to another embodiment. The output image Qa includes a peripheral area 524 of a recognition mark 521 indicating a check box 511, which is determined based on the detection results of the vertex candidates 514 output from the vertex candidate output unit 35. By outputting such an output image Qa, the recognition result output unit 38 can recognize the check box 511 even if the handwritten mark does not fit within the check box 511, thereby providing an IPU 500 with excellent recognition accuracy for the check box 511.

Figure 24 is a block diagram showing an example of the functional configuration of an IPU 500b according to another embodiment. As shown in Figure 24, the IPU 500b has an error recognition evaluation unit 39.

The misrecognition evaluation unit 39 outputs the detection results of at least one of the characters and figures other than the check box 511 based on the recognition result of the check box 511 output from the recognition result output unit 38.

Figure 25 shows an output image Qb according to another embodiment. The output image Qb' in Figure 25 includes an incorrectly recognized check box 525 in which the character "団" (group), which is not a check box 511, is incorrectly recognized as a check box.

On the other hand, output image Qb in Figure 25 includes an error recognition mark 526 indicating that the error recognition check box 525 has been erroneously recognized. This error recognition mark 526 is a mark that is detected by the error recognition evaluation unit 39 based on the recognition result of the check box 511 output from the recognition result output unit 38, and is added by the error recognition evaluation unit 39. In this way, IPU 500b can prevent erroneous recognition due to characters and figures other than the check box 511, and improve the recognition accuracy of the check box 511.

The above describes a preferred embodiment in detail, but the present invention is not limited to the above embodiment, and various modifications and substitutions can be made to the above embodiment without departing from the scope of the claims.

In the above-described embodiment, an electrophotographic image forming apparatus using toner was exemplified, but the present invention is not limited to this, and the embodiment can also be applied to a liquid ejection image forming apparatus using a liquid such as ink. Similar to toner, ink can also have uneven adhesion to the recording medium, scattering, bleeding, etc., and therefore can achieve the same effects as when toner is used.

Embodiments also include an image recognition method. For example, the image recognition method is an image recognition method using an image recognition device that is included in an input image in which at least a portion of a document is input and that is capable of recognizing a selection acceptance figure that accepts selections for items, in which the image recognition device outputs, via a vertex candidate output unit, a detection result of vertex candidates for the selection acceptance figure based on feature points of the outer shape of the figure included in the input image, and outputs, via a recognition result output unit, a recognition result of the selection acceptance figure based on the detection result of the vertex candidates output from the vertex candidate output unit. Such an image recognition method can achieve the same effects as the IPU 500 described above.

All ordinal numbers, quantities, and other figures used in the description of the embodiments are provided as examples to specifically explain the technology of the present invention, and the present invention is not limited to these figures. Furthermore, the interconnections between components are provided as examples to specifically explain the technology of the present invention, and do not limit the interconnections that realize the functions of the present invention.

Each function of the embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" as used herein includes processors programmed to perform each function by software, such as processors implemented as electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), FPGAs (Field Programmable Gate Arrays), and conventional circuit modules designed to perform each of the functions described above.

1 Image forming device 2 PC
3 Servers 10 and 20 CPU
11, 21 RAM
12, 22 ROM
13, 23 HDD
14, 24 Communication I/F
15, 25 Bus 16 Display device 17 Input operation unit 41 ADF
42 Scanner unit 43 Paper output tray 31 Input unit 32 Binarization processing unit 33 Outer shape extraction unit 34 Feature point extraction unit 35 Vertex candidate output unit 36 Vertex candidate evaluation unit 37 Registration unit 38 Recognition result output unit 39 Error recognition evaluation unit 100 Scanner 200 Controller 204 First mirror unit 210 Second mirror unit 214 Photoelectric conversion element 215 First sensor board 216 Lens 300 Plotter 331 Exposure device 332 Photosensitive drum 333 Developing device 334 Transfer belt 335 Fixing device 401 Image memories 421, 422 Paper feed cassette 423 Paper feed means 500 IPU (an example of an image recognition device)
510: Target items 511, 511a, 511b: Check boxes (examples of selection acceptance shapes)
512 Outer shape 513 Feature points 514, 514a, 514b Vertex candidate 515 Falsely detected vertex candidate 516 Ruled line area 521 Recognition mark 522 Check mark 523 Item mark 524 Peripheral area 525 Falsely recognized check box 526 Falsely recognized mark Im0 Input image Im1 Binarized image Im2 Outer shape image NW Network Pi Original Po Printed matter Q Output image

Japanese Patent Application Laid-Open No. 2021-039429

Claims (14)

An image recognition device capable of recognizing a selection acceptance figure that accepts a selection for an item, the selection acceptance figure being included in an input image that includes at least a part of a document,
a vertex candidate output unit that outputs a detection result of vertex candidates of the selection receiving figure based on feature points of the outer shape of the figure included in the input image;
a vertex candidate evaluation unit that determines whether or not the vertex candidate is a vertex candidate of the selection receiving figure based on the vertex candidate detection result received from the vertex candidate output unit;
a recognition result output unit that outputs a recognition result of the selection receiving figure based on a determination result by the vertex candidate evaluation unit ,
The vertex candidate evaluation unit determines whether the vertex candidate is a vertex candidate of the selection receiving figure by determining whether the straight lines connecting the multiple vertex candidates are colored, or by comparing the distance between the vertex candidates with the size of the characters contained in the input image . the selection receiving figure has a rectangular outer shape,
The image recognition device according to claim 1 , wherein the recognition result of the selection receiving figure includes the position of at least one vertex of the rectangle in the selection receiving figure. The image recognition device of claim 1 or 2, wherein the vertex candidate output unit outputs the detection result of the vertex candidate based on the number of pixels included in an image region connecting the feature points, the pixel values of which match the pixel values of pixels included in the feature points. The image recognition device of any one of claims 1 to 3, wherein the vertex candidate output unit outputs the vertex candidates based on the feature points extracted from a binarized image obtained by binarizing the input image using a binarization threshold based on a histogram generated from the pixel values of each of a plurality of pixels included in the input image. The image recognition device described in any one of claims 1 to 4, wherein the vertex candidate output unit outputs the vertex candidates based on the feature points of the input image that are distinguished into handwritten areas and non-handwritten areas based on at least one of color component information and histogram information of the input image. The image recognition device of any one of claims 1 to 5, wherein the recognition result output unit outputs the recognition result of the selection acceptance figure based on at least one of size information and position information of the selection acceptance figure. The image recognition device described in any one of claims 1 to 6, wherein the vertex candidate output unit outputs the vertex candidates based on the feature points of the input image after the tilt of the input image has been corrected and detected from a ruled line region included in the input image. The image recognition device described in any one of claims 1 to 7, wherein the vertex candidate output unit outputs the vertex candidates based on the feature points of the input image in which the pixel values of each of the multiple pixels included in the input image have been color-inverted. The image recognition device described in any one of claims 1 to 8, wherein the vertex candidate output unit outputs the vertex candidates based on the feature points in a color component image obtained by decomposing the pixel values of each of a plurality of pixels included in the input image into color components. The image recognition device described in any one of claims 1 to 9, wherein the vertex candidate output unit outputs the vertex candidates based on the feature points of the input image, which are distinguished into form regions and regions other than the form regions based on at least one of color component information and histogram information of the input image. The image recognition device described in any one of claims 1 to 10, wherein the vertex candidate output unit outputs the vertex candidates based on the feature points in the input image in which a black area included in the input image has been enlarged. The image recognition device of any one of claims 1 to 11, wherein the recognition result output unit outputs the recognition result of the selection acceptance figure, including the peripheral area of a recognition mark indicating the selection acceptance figure, determined based on the vertex candidate detection result output from the vertex candidate output unit. The image recognition device of any one of claims 1 to 12, further comprising an error recognition evaluation unit that outputs a detection result of at least one of characters and figures other than the selection acceptance figure based on the recognition result of the selection acceptance figure output from the recognition result output unit. An image recognition method using an image recognition device capable of recognizing a selection acceptance graphic that accepts a selection for an item, the selection acceptance graphic being included in an input image that includes at least a part of a document, the image recognition device comprising:
a vertex candidate output unit outputs a detection result of vertex candidates of the selection receiving figure based on feature points of the outer shape of the figure included in the input image;
a vertex candidate evaluation unit determines whether or not the vertex candidate is a vertex candidate of the selection receiving figure based on the detection result of the vertex candidate received from the vertex candidate output unit;
a recognition result output unit outputs a recognition result of the selection receiving figure based on the vertex candidate detection result output from the vertex candidate output unit ;
The vertex candidate evaluation unit determines whether the vertex candidate is a vertex candidate of the selection receiving figure by determining whether the straight lines connecting the multiple vertex candidates are colored, or by comparing the distance between the vertex candidates with the size of the characters contained in the input image .