JP7761130B2 - Character recognition system, character recognition method, and character recognition program - Google Patents

Description

The present invention relates to a character recognition system, etc.

Optical Character Recognition (OCR) is widely used. OCR scans handwritten characters written on forms as images and converts them into text data by recognizing the characters in the image. Character recognition using OCR is performed, for example, by recognizing characters written on preprinted forms using a learning model generated by machine learning. However, even when the same characters are written, the shape of the characters written on preprinted forms and the position of the characters on the preprint vary depending on the person who wrote the characters. Furthermore, images obtained by scanning characters written on preprinted forms contain a mixture of preprinted and unprinted characters. Therefore, a learning model for recognizing handwritten characters on forms may be required to accurately recognize characters written on preprinted forms, which come in a variety of shapes and positions, from images that contain a mixture of preprinted and unprinted characters. Therefore, a technology capable of accurately recognizing characters on preprinted forms is desirable.

The image processing system in Patent Document 1 uses a learning model to extract handwritten characters written within a preprint frame. The image processing system in Patent Document 1 extracts handwritten characters by erasing the preprint frame from an image of the handwritten characters written within the preprint frame using image processing.

Japanese Patent Application Laid-Open No. 2021-39424

The information processing device in Patent Document 1 may have difficulty accurately recognizing characters written on a preprint.

In order to solve the above problems, the main objective of the present invention is to provide a character recognition system, etc. that can improve the recognition accuracy of characters written on preprints.

To solve the above problem, the character recognition system of the present invention comprises an acquisition means for acquiring an image of characters written on a preprint of a document including the preprint, a recognition means for recognizing characters written on the preprint of the acquired image from the acquired image and the preprint image using a recognition model that recognizes characters written on the preprint from the image of the characters written on the preprint and the preprint image of the preprint, and an output means for outputting the recognition results.

The character recognition method of the present invention acquires an image of characters written on a preprint of a document including the preprint, and uses a recognition model that recognizes characters written on the preprint from the image of the characters written on the preprint and a preprint image of the preprint, recognizes the characters written on the preprint in the acquired image from the acquired image and the preprint image, and outputs the recognition results.

The recording medium of the present invention non-temporarily records a character recognition program that causes a computer to execute the following processes: acquiring an image of characters written on a preprint of a document including the preprint; recognizing characters written on the preprint of the acquired image from the acquired image and the preprint image using a recognition model that recognizes characters written on the preprint from the image of the characters written on the preprint and the preprint image of the preprint; and outputting the recognition results.

This invention makes it possible to improve the recognition accuracy of characters written on preprints.

1 is a diagram illustrating an example of a configuration of a first exemplary embodiment of the present invention. FIG. 2 is a diagram illustrating an example of a form according to the first embodiment of the present invention. FIG. 2 is a diagram showing an example of an image on which characters are written according to the first embodiment of the present invention. FIG. 2 is a diagram illustrating an example of a preprint image according to the first embodiment of the present invention. FIG. 2 is a diagram showing an example of an image on which characters are written according to the first embodiment of the present invention. FIG. 2 is a diagram illustrating an example of a preprint image according to the first embodiment of the present invention. 1 is a diagram illustrating an example of the configuration of a character recognition system according to a first embodiment of the present invention. FIG. 2 is a diagram showing an example of an image on which characters are written according to the first embodiment of the present invention. FIG. 2 is a diagram illustrating an example of a preprint image according to the first embodiment of the present invention. FIG. 2 is a diagram showing an example of an image on which characters are written according to the first embodiment of the present invention. FIG. 2 is a diagram illustrating an example of a preprint image according to the first embodiment of the present invention. FIG. 2 is a diagram showing an example of an image on which characters are written according to the first embodiment of the present invention. FIG. 2 is a diagram illustrating an example of a preprint image according to the first embodiment of the present invention. FIG. 2 is a diagram illustrating an example of an operation flow of the character recognition system according to the first exemplary embodiment of the present invention. FIG. 2 is a diagram illustrating an example of an operation flow of the character recognition system according to the first exemplary embodiment of the present invention. FIG. 10 is a diagram illustrating an example of a configuration of a second exemplary embodiment of the present invention. FIG. 10 is a diagram illustrating an example of the configuration of a character recognition system according to a second embodiment of the present invention. FIG. 10 is a diagram schematically illustrating a flow of data processing in a second embodiment of the present invention. FIG. 10 is a diagram illustrating an example of an operation flow of a character recognition system according to a second exemplary embodiment of the present invention. FIG. 10 is a diagram illustrating an example of an operation flow of a character recognition system according to a second exemplary embodiment of the present invention. FIG. 10 is a diagram illustrating an example of an operation flow of a character recognition system according to a second exemplary embodiment of the present invention. FIG. 10 is a diagram illustrating an example of the configuration of another embodiment of the present invention.

(First embodiment)
A first embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 is a diagram showing an example of the configuration of a form processing system according to this embodiment. The form processing system includes, as an example, a character recognition system 10, a scanner 20, and an information processing server 30. The character recognition system 10 is connected to the scanner 20 via a network, for example. The character recognition system 10 is also connected to the information processing server 30 via the network. There may be a plurality of scanners 20 and information processing servers 30. The number of scanners 20 and information processing servers 30 is not particularly limited.

The character recognition system 10 acquires, for example, an image of a form scanned by the scanner 20. A preprint on which characters are written is printed on the form paper. The preprint is, for example, a frame or line on the form that indicates the position where characters are to be written. The character recognition system 10 acquires, for example, an image of handwritten characters written on the preprint. The characters written on the preprint may be printed. The characters written on the preprint are not limited to the above examples.

The character recognition system 10 uses a recognition model to recognize characters written on a preprint from an image of the characters written on the preprint acquired from the scanner 20 and a preprint image of the preprint. The recognition model is a learning model that recognizes characters written on the preprint from an image of the characters written on the preprint and the preprint image. The character recognition system 10 outputs the recognition results of the characters written on the preprint to, for example, the information processing server 30. The information processing server 30 is a server that performs processing according to the purpose of the recognition results of the characters written on the preprint.

The character recognition system 10 uses the preprint image in addition to the image of the characters written on the preprint to be recognized to recognize the characters written on the preprint, thereby reducing the impact of the preprint on character recognition.

Figure 2 is a diagram showing an example of a form. In the example of the form in Figure 2, the name of the form is written at the top as "Payment Slip." The example of the form in Figure 2 is, for example, a document that is submitted to a financial institution when depositing money into an account at the financial institution. In the example of the form in Figure 2, entry fields for "Account Number" and "Amount" are set. In the example of the form in Figure 2, the boxes for entering numbers in the "Account Number" and "Amount" entry fields are preprinted.

The characters written on the preprint are, for example, the characters written within the frame of the preprint. The characters written on the preprint may be written so that they overlap the frame of the preprint. An image with characters written on the preprint is an image that includes both the preprint and the characters written on the preprint. A preprint image is an image that shows only the preprint with no characters written on it. In the example form in Figure 2, numbers are written on the preprint, but the characters written on the preprint are not limited to numbers. The characters written on the preprint may also include symbols.

Figure 3 is a diagram showing an example of an image in which characters written on a preprint have been copied. Figure 3 is an image in which the "Account Number" entry field has been extracted from the example form in Figure 2. Figure 4 is a preprint image of the "Account Number" entry field in the example form in Figure 2. In the example of an image in which characters written on a preprint in Figure 3 have been copied, the characters "01778543" have been handwritten on the preprint shown in Figure 4.

Images that are preprinted only may contain preprinted characters. Preprinted characters may be, for example, characters indicating monetary digits, characters indicating items, or characters indicating units. Preprinted characters are not limited to the above, as long as they are printed on paper as preprints.

Figure 5 is a diagram showing an example of an image in which characters written on a preprint have been copied. Figure 5 is an image in which the "Amount" entry field has been extracted from the example form in Figure 2. Figure 6 is a preprint image of the "Amount" entry field in the example form in Figure 2. In the example preprint image in Figure 6, "yen" indicating the unit of amount is printed as part of the preprint at the bottom of the right-hand frame. In the example image in which characters written on a preprint in Figure 5 have been copied, the characters "40000" have been handwritten on the preprint shown in Figure 6.

Forms are documents used for procedures at, for example, financial institutions, government agencies, educational institutions, hospitals, transportation facilities, or companies. Forms may also be documents attached to managed items. Examples of forms are not limited to the above. A preprint indicates, for example, the location on a form where the date, name, affiliation, address, telephone number, email address, age, gender, occupation, or amount should be entered. A preprint is composed of, for example, fields to be filled in and boxes into which characters should be entered. When multiple characters are entered in one field, the preprint may be a series of multiple boxes. Also, preprints for multiple fields may be printed on a single form. For example, when a preprint is printed on paper as a fill-in field consisting of multiple boxes, the character recognition system 10 outputs the recognized characters as string data in the order of the boxes.

The configuration of the character recognition system 10 will be described. Figure 7 is a diagram showing an example of the configuration of the character recognition system 10. The character recognition system 10 basically comprises an acquisition unit 11, a recognition unit 13, and an output unit 14. The character recognition system 10 further comprises an image extraction unit 12, a generation unit 15, and a memory unit 16. The acquisition unit 11, image extraction unit 12, recognition unit 13, output unit 14, and memory unit 16 recognize characters written on a preprint, for example, from an image of the characters written on the preprint. The acquisition unit 11, generation unit 15, and memory unit 16 also generate, for example, a recognition model.

The acquisition unit 11 acquires an image of the characters written on the preprint. The acquisition unit 11 acquires, for example, from the scanner 20, an image of a form with characters written on the preprint. The acquisition unit 11 may also acquire an image in which the portion of the form with characters written on the preprint has been extracted. The image of the portion of the preprint with characters written is, for example, the image shown in the examples of Figures 3 and 5. When extracting a preprint image from a form, the acquisition unit 11 may also acquire an image of the form in which no characters are written on the preprint. The acquisition unit 11 acquires, for example, from the scanner 20, an image of the form in which no characters are written on the preprint.

When the character recognition system 10 generates a recognition model, the acquisition unit 11 may acquire training data to be used in generating the recognition model. The generation unit 15 acquires, for example, an image of characters written on a preprint and data associating the preprint image with the characters written on the preprint as training data. The training data is input, for example, by an operator's operation, into the character recognition system 10 or another terminal device connected to the character recognition system 10.

The image extraction unit 12 extracts a preprint image corresponding to an image of the characters written on the preprint acquired by the acquisition unit 11. The image extraction unit 12 extracts the preprint image from, for example, form data stored in the memory unit 16. The form data includes, for example, an image of the form and definition data. The definition data includes, for example, information about the items to be written on the form and the position on the form of the preprint corresponding to the items to be written. Information about the position of the preprint is, for example, information indicating the area on the form where the preprint is printed. The items to be written are, for example, one or more of name, postal code, address, telephone number, age, personal identification number, account number, amount, and date. The items to be written are not limited to the above examples.

The image extraction unit 12 identifies the position of the preprint on the form, for example, based on information about the preprint position included in the definition data. The image extraction unit 12 then extracts the preprint image by cutting out the image at the identified preprint position from the image stored in the memory unit 16. The image extraction unit 12 may also extract the preprint image from an image of the form acquired by the acquisition unit 11 in which no text is written on the preprint.

The recognition unit 13 uses a recognition model to recognize characters in an image of characters written on a preprint acquired by the acquisition unit 11 from the preprint image. The recognition model is a learning model that recognizes characters written on a preprint from an image of characters written on a preprint and the preprint image. For example, the recognition unit 13 inputs the image of characters written on a preprint acquired by the acquisition unit 11 and the preprint image into the recognition model. The recognition unit 13 then recognizes characters written on the preprint using the recognition model. The recognition unit 13 may recognize characters written on a preprint using a preprint image that has been extracted in advance. The recognition unit 13 may also recognize characters written on a preprint using a preprint image that has been generated in advance as an image of the preprint portion. The recognition unit 13 recognizes characters written on a preprint using, for example, a preprint image stored in the memory unit 16.

The recognition unit 13 extracts an image showing the characters written on the preprint by identifying the position of the preprint, for example, based on information about the position of the preprint included in the definition data. Then, using a recognition model, the recognition unit 13 recognizes the characters written on the preprint from the extracted image showing the characters written on the preprint and the preprint image extracted by the image extraction unit 12.

The recognition unit 13, for example, combines an image of characters written on a preprint with the preprint image to create a single piece of data and inputs the data into the recognition model. Combining an image of characters written on a preprint with the preprint image means generating image data by overlaying the two images. If the image of characters written on a preprint and the preprint image each have three channels (RGB) per pixel, the recognition unit 13, for example, combines the data of the two images to create image data with six channels per pixel. The recognition unit 13 then inputs the combined six-channel image data into the recognition model.

The recognition unit 13, for example, combines an image of characters written on a preprint with a preprint image based on preset conditions. For example, the recognition unit 13 combines two images by overlaying an image of characters written on a preprint extracted at the same size with an image of characters written on a preprint and a preprint image based on the outer periphery of the preprint image. For example, when the two images are overlaid, the recognition unit 13 combines image data of corresponding pixels. Then, the recognition unit 13 inputs the combined data into a recognition model to recognize the characters written on the preprint.

The recognition unit 13 may recognize characters other than those written on the preprint in an image of the form. The recognition unit 13 may, for example, identify the type of form from the image of the form acquired by the acquisition unit 11. The recognition unit 13 then identifies the position of the preprint based on definition data included in the form data corresponding to the identified type of form, thereby recognizing the characters written on the preprint. The recognition unit 13 may, for example, identify the type of form by recognizing the form name or form number printed on the form in the image of the form. The relationship between the form name or form number printed on the form and the form type is preset. The recognition model used by the recognition unit 13 may also be a learning model generated outside the character recognition system 10.

Figure 8 is a diagram showing an example image of characters written on a preprint. The preprint in Figure 8 differs from the example image in Figure 3 in terms of the preprint configuration. The preprint in the example image in Figure 8 differs, for example, from the example image in Figure 3 in terms of line thickness and type. Figure 9 is a preprint image for the example image in Figure 8. In the example image of characters written on a preprint in Figure 8, the characters "13758047" are handwritten on the preprint shown in Figure 9. When the example image in Figure 8 and the example image in Figure 9 are input, a recognition model outputs "13758047" as the recognition result. Even a recognition model generated using, for example, the preprint in the example image in Figure 4 as training data can recognize characters written on the preprint in the example image in Figure 9. In other words, by inputting an image of characters written on a preprint and a preprint image, the recognition model can recognize characters written on an untrained preprint.

Figure 10 is a diagram showing an example of an image in which characters indicating a year in the Gregorian calendar are copied onto a preprint. In the example image of Figure 10, the characters "Gregorian calendar" and "year" are pre-printed within the preprint frame. In the example image of Figure 10, the characters "2022" are handwritten on the preprint image. Figure 11 is a preprint image of the example image of Figure 10. When the example image of Figure 10 and the example image of Figure 11 are input, the recognition model outputs "2022" as the recognition result.

Figure 12 shows an example of an image similar to the example image of Figure 10, in which the first two digits, "20," indicating the year in the Gregorian calendar are preprinted. That is, in the example image of Figure 12, the "year," "20," and "year" are preprinted as preprints. In the example image of Figure 12, the "22" of "2022" is handwritten on the preprint. Figure 13 shows a preprint image of the example image of Figure 12. When the example image of Figure 12 and the example image of Figure 13 are input, the recognition model outputs "22" as the recognition result. Even a learning model that does not use the preprints of the example images of Figures 11 and 13 as training data can recognize the image written on the preprint from the input image. In this way, the recognition model can recognize characters written on various types of preprints by inputting an image containing characters written on the preprint and the preprint image. Furthermore, in the above example, examples of preprints with different line thicknesses, line types, and preprinted characters were shown, but the recognition model can also perform recognition in the case of preprints with different frame shapes and colors, even if it is not trained using all types of preprints as training data.

The output unit 14 outputs the recognition results obtained by the recognition unit 13. The output unit 14 outputs the characters recognized by the recognition unit 13 to, for example, the information processing server 30. The output unit 14 outputs, for example, the items corresponding to the preprint and the recognized characters in association with each other. When the target of recognition is an account number as shown in the example image of Figure 3, the output unit 14 outputs, for example, information indicating that it is an account number in association with the recognized character string. The output unit 14 may output the recognition results to a display device (not shown) connected to the character recognition system 10.

When generating a recognition model in the character recognition system 10, the generation unit 15 performs processing related to the generation of the recognition model. The generation unit 15 learns images of characters written on a preprint, and the relationship between the preprint image and the characters written on the preprint. The generation unit 15 then generates a recognition model that recognizes characters in an image from the image of the characters written on the preprint and the preprint image.

The generation unit 15 generates a recognition model by learning the relationship between, for example, an image showing characters written on a preprint, data obtained by combining the preprint image, and the characters written on the preprint. If the image showing characters written on a preprint and the preprint image each have three channels (RGB) per pixel, the generation unit 15 combines the data of the two images to create image data with six channels per pixel. The generation unit 15 then generates a recognition model by learning the relationship between the combined six-channel image data and the characters written on the preprint.

The image of the characters written on the preprint and the preprint included in the preprint image that the generation unit 15 uses as training data do not have to be the actual preprint image used. When generating a recognition model, the generation unit 15 may perform training using a randomly shaped figure as the preprint. When a randomly shaped figure is used as the preprint, the generation unit 15 generates a recognition model using, for example, an image of the characters written on the randomly shaped figure and an image of the same figure as the figure with the characters written on top as training data.

The generation unit 15 generates a recognition model by deep learning using, for example, a DNN (Deep Neural Network). The machine learning algorithm for generating the recognition model is not limited to deep learning using a DNN.

The memory unit 16 stores, for example, a recognition model used by the recognition unit 13 to recognize characters in an image. The memory unit 16 stores, for example, a preprint image. The memory unit 16 stores, for example, form data. The form data includes, for example, image data of the form and definition data. The form data may include a preprint image extracted in advance. The memory unit 16 stores, for example, an image of characters written on a preprint, the preprint image, and the characters written on the preprint as learning data. Note that the recognition model used by the recognition unit 13 may be stored in a storage means other than the memory unit 16.

The scanner 20, for example, optically reads the form and generates an image of the form. The scanner 20 then outputs the image of the form to the character recognition system 10. The scanner 20 may extract an image of a preprinted portion from the image of the form. When extracting an image of a preprinted portion, the scanner 20 outputs the extracted preprinted image to the character recognition system 10. Furthermore, if the form is a document attached to an item to be managed, the scanner 20 may generate an image of the form by photographing the form.

The information processing server 30 obtains, for example, from the character recognition system 10, the recognition results of characters written on a form. The information processing server 30 uses the recognition results to perform processing according to the application. For example, the information processing server 30 uses the recognition results to process applications related to account management and deposits and withdrawals at financial institutions. For example, the information processing server 30 may use the recognition results to process application documents at government agencies, educational institutions, hospitals, or transportation facilities. The information processing server 30 may use the recognition results to process invoices at companies. The information processing server 30 may also use the identification results to manage goods in distribution. Examples of uses for the identification results are not limited to those mentioned above.

The following describes the operation of the character recognition system 10 when recognizing characters written on a preprint. Figure 14 is a diagram showing an example of the operation flow when the character recognition system 10 recognizes characters written on a preprint.

The acquisition unit 11 acquires an image showing the characters written on the preprint (step S11). The acquisition unit 11 acquires, for example, from the scanner 20, an image of the form showing the characters written on the preprint.

The image extraction unit 12 also extracts a preprint image corresponding to the image acquired by the acquisition unit 11 (step S12). The image extraction unit 12 extracts a preprint image corresponding to the image acquired by the acquisition unit 11, for example, from data stored in the memory unit 16.

Once the preprint image is extracted, the recognition unit 13 uses a recognition model to recognize characters in the image from the image acquired by the acquisition unit 11 and the preprint image (step S13). The recognition model recognizes characters written on the preprint from an image showing the characters written on the preprint and the preprint image.

Once the characters in the image are recognized, the output unit 14 outputs the recognition result (step S14). The output unit 14 outputs the recognition result to, for example, the information processing server 30.

The following describes the operation of the character recognition system 10 when generating a recognition model. Figure 15 is a diagram showing an example of the operation flow when the character recognition system 10 generates a recognition model.

The acquisition unit 11 acquires, as learning data, an image of characters written on a preprint, a preprint image, and the characters written on the preprint (step S21).

After acquiring the training data, the generation unit 15 learns the image of the characters written on the preprint and the relationship between the preprint image and the characters written on the preprint, and generates a recognition model (step S22). The generation unit 15, for example, combines the image of the characters written on the preprint with the preprint image. The generation unit 15 then learns the relationship between the combined data and the characters written on the preprint that are included as correct answer data in the training data, and generates a recognition model.

Once the recognition model is generated, the generation unit 15 stores the generated recognition model (step S23). The generation unit 15 stores the generated recognition model, for example, in the storage unit 16.

The character recognition system 10 of the form processing system of this embodiment uses a recognition model to recognize characters written on a preprint from an image of the characters written on the preprint and the preprint image. The character recognition system 10 recognizes characters written on a preprint using the preprint image in addition to the image of the characters written on the preprint to be recognized, thereby reducing the impact of the preprint on character recognition. As a result, the character recognition system 10 can improve the accuracy of recognizing characters written on a preprint.

Furthermore, the recognition model used by character recognition system 10 can recognize characters written on preprints in untrained forms by taking an image of characters written on a preprint and the preprint image as input and recognizing the characters written on the preprint. Therefore, by taking an image of characters written on a preprint and the preprint image as input and recognizing the characters written on the preprint, character recognition system 10 can recognize characters written on various forms of preprints. Furthermore, when generating a recognition model, character recognition system 10 does not need to prepare training data for each form of preprint actually used for recognition. Furthermore, when generating a recognition model, character recognition system 10 does not need to train training data for each form of preprint actually used for recognition, which reduces the amount of training required when generating a recognition model. Therefore, character recognition system 10 can reduce the computer resources required for generating a recognition model. Therefore, character recognition system 10 can efficiently generate recognition models.

Furthermore, by using randomly shaped figures as preprints when generating a recognition model, the character recognition system 10 can generate a recognition model that can recognize characters written on various preprint images. In other words, by using a recognition model generated using randomly shaped figures as preprints, the character recognition system 10 can accurately recognize characters written on preprints even if the shape of the preprint images differs for each form.

Furthermore, if a character recognition method different from that of the present embodiment is used, for example, a method in which characters written on a preprint are erased from an image containing the characters before character recognition is performed, erasing the preprint may require significant computer resources. Furthermore, there is a risk that some of the characters may be lost when erasing the preprint. On the other hand, the character recognition system 10 of the present embodiment recognizes characters by inputting an image containing characters written on a preprint and data combining the preprint image into a recognition model, thereby eliminating the need to erase the preprint as a preprocessing step for character recognition. Furthermore, because the preprint erasure process is not performed, the impact of the process related to erasing the preprint on character recognition can be reduced. Therefore, the character recognition system 10 of the present embodiment can improve recognition accuracy while reducing the resources required for recognizing characters written on a preprint.

Second Embodiment
A second embodiment of the present invention will be described in detail with reference to the drawings. FIG. 16 is a diagram showing an example of the configuration of a form processing system of this embodiment. The form processing system includes, as an example, a character recognition system 40, a scanner 20, and an information processing server 30. The character recognition system 40 is connected to the scanner 20, for example, via a network. The character recognition system 40 is also connected to the information processing server 30 via the network. There may be multiple scanners 20 and multiple information processing servers 30. The number of scanners 20 and multiple information processing servers 30 is not particularly limited. The functions of the scanner 20 and the information processing server 30 of this embodiment are similar to those of the scanner 20 and the information processing server 30 of the first embodiment.

The character recognition system 10 of the first embodiment uses, for example, a recognition model to input data that combines an image of characters written on a preprint with the preprint image, and recognizes the characters on the preprint. The character recognition system 10 then outputs the recognition results. In addition to this configuration, the character recognition system 40 of the present embodiment, for example, when combining an image of characters written on a preprint with the preprint image, combines the two images after performing a conversion process on the preprint image using a conversion model in order to improve the accuracy of overlaying the two images. The conversion model is a learning model that estimates conversion parameters used when performing a conversion process on the preprint image.

The configuration of the character recognition system 40 will be described. Figure 17 is a diagram showing an example of the configuration of the character recognition system 40. The character recognition system 40 includes an acquisition unit 11, an image extraction unit 12, a recognition unit 41, an output unit 14, a generation unit 42, and a memory unit 16. The recognition unit 41 also includes a conversion unit 51 and an image recognition unit 52. The configurations and functions of the acquisition unit 11, image extraction unit 12, output unit 14, and memory unit 16 of the character recognition system 40 are similar to those of the acquisition unit 11, image extraction unit 12, output unit 14, and memory unit 16 of the character recognition system 10 of the first embodiment, respectively.

The transformation unit 51 of the recognition unit 41 transforms the preprint image using, for example, a transformation model. The transformation model, for example, performs an affine transformation on the preprint image. The recognition unit 41 transforms the preprint image so that it overlaps with the image to be combined, for example, by rotating, adjusting the size, and translating the preprint image. The transformation model, for example, estimates transformation parameters to be used when rotating, adjusting the size, and translating the preprint image.

The transformation unit 51, for example, uses a transformation model to estimate affine transformation parameters from data obtained by combining an image containing text written on the preprint with the preprint image under preset conditions. The transformation unit 51 then uses the estimated parameters to perform affine transformation on the preprint image. For example, the transformation unit 51 combines the image containing text written on the preprint with the preprint image by aligning their respective peripheries under preset conditions, thereby overlapping the two images. The transformation unit 51 then uses the transformation model to estimate transformation parameters from the data combined under preset conditions. The transformation parameters are parameters for transforming the preprint image to improve the accuracy of the overlay compared to when the images are combined under preset conditions. After estimating the transformation parameters, the transformation unit 51 performs an affine transformation on the preprint image using the transformation parameters, thereby improving the accuracy of the overlay.

The transformation model is, for example, a learning model that uses a DNN known as an STN (Spatial Transformer Network). An image transformation method using an STN is described, for example, in Max Jaderberg et al. "Spatial Transformer Networks," NIPS'15: Proceedings of the 28th International Conference on Neural Information Processing Systems, Volume 2, December 2015, pp. 2017-2025.

The image recognition unit 52 of the recognition unit 41 uses a recognition model to recognize characters written on the preprint from an image showing the characters written on the preprint and the preprint image. The image recognition unit 52 combines the image showing the characters written on the preprint with the preprint image on which the transformation unit 51 has performed an affine transformation. The image recognition unit 52 then uses a discriminative model to recognize the characters written on the preprint from the combined data. The transformation model and the recognition model may be learning models generated outside the character recognition system 40.

Figure 18 is a diagram schematically illustrating the processing flow when the recognition unit 41 recognizes characters written on a preprint. In the example of Figure 18, assume that an image showing characters written on a preprint and a preprint image are input to the recognition unit 41. The conversion unit 51 combines the image showing characters written on the preprint and the preprint image, for example, according to preset conditions. The preset conditions are, for example, set so that the peripheries of the two images are aligned. After combining the images, the conversion unit 51 estimates affine transformation parameters using a transformation model. Then, the conversion unit 51 performs an affine transformation on the preprint image using the estimated affine transformation parameters. The conversion unit 51 outputs the affine-transformed preprint image to the image recognition unit 52. When the affine-transformed preprint image is input, the image recognition unit 52 combines the image showing characters written on the preprint with the affine-transformed image. Once the images are combined, the image recognition unit 52 uses a recognition model to recognize characters written on the preprint from the combined data.

The character recognition system 40 generates, for example, only the recognition model out of the conversion model and the recognition model. When generating only the recognition model, for example, a training model generated outside the character recognition system 40 is used as the conversion model. When generating only the recognition model out of the conversion model and the recognition model, the generation unit 42 combines, for example, an image of characters written on a preprint included in the training data with the preprint image using the conversion model. The generation unit 42 then learns the relationship between the combined data and the characters written on the preprint included as correct data in the training data, and generates a recognition model. The generation unit 42 saves the generated conversion model and recognition model in the memory unit 16.

The character recognition system 40 may generate both a transformation model and a recognition model. When generating both a transformation model and a recognition model, the generation unit 42 uses the transformation model to estimate transformation parameters from data obtained by combining an image of characters written on a preprint with the preprint image under preset conditions. The generation unit 42 also uses the recognition model to recognize the characters written on the preprint from the combined data. The generation unit 42 updates the parameters of the transformation model so that the difference between the affine transformation parameters estimated by the transformation model and the affine transformation parameters included in the training data becomes smaller. The generation unit 42 also updates the parameters of the recognition model so that the difference between the identification result and the correct data becomes smaller.

After updating the conversion parameters of the conversion model and the parameters of the recognition model, the generation unit 42 repeats the above process using the updated models. The generation unit 42 generates a conversion model and a recognition model by repeating the above process, for example, until the estimated results of the conversion parameters of the conversion model and the accuracy of the recognition results of the recognition model satisfy a preset standard. The generation unit 42 also generates an identification model by, for example, updating the parameters of the recognition model so that the difference between the identification result and the correct data is reduced. The generation unit 42 stores the generated conversion model and recognition model in, for example, the storage unit 16.

The following describes the operation of the character recognition system 40 when recognizing characters written on a preprint. Figure 19 is a diagram showing an example of the operation flow when the character recognition system 40 recognizes characters written on a preprint.

The acquisition unit 11 acquires an image showing the characters written on the preprint (step S31). The acquisition unit 11 acquires, for example, from the scanner 20, an image of the form showing the characters written on the preprint.

The image extraction unit 12 also extracts a preprint image corresponding to the image acquired by the acquisition unit 11 (step S32). The image extraction unit 12 extracts a preprint image corresponding to the image acquired by the acquisition unit 11, for example, from data stored in the memory unit 16.

When the preprint image is acquired, the conversion unit 51 of the recognition unit 41 uses a conversion model to estimate conversion parameters to be used when converting the preprint image. The conversion unit 51 then converts the preprint image using the estimated conversion parameters (step S33). Once the preprint image has been converted, the image recognition unit 52 combines the image of the preprint with text written on it and the converted preprint image. The image recognition unit 52 then uses the recognition model to recognize the text in the image from the combined data (step S34).

Once the characters in the image are recognized, the output unit 14 outputs the recognition results (step S35). The output unit 14 outputs the recognition results to, for example, the information processing server 30.

This section explains the operation of the character recognition system 40 when it generates only the recognition model out of the conversion model and the recognition model. Figure 20 is a diagram showing an example of the operation flow when the character recognition system 40 generates only the recognition model.

The acquisition unit 11 acquires, as learning data, an image of characters written on a preprint, a preprint image, and the characters written on the preprint (step S41).

Once the learning data is acquired, the generation unit 42 uses the conversion model to estimate the conversion parameters to be used when converting the preprint image. Then, the generation unit 42 uses the estimated conversion parameters and the conversion model to convert the preprint image (step S42).

After converting the preprint image, the generation unit 42 combines the image containing the characters written on the preprint with the converted preprint image. The generation unit 42 then learns the relationship between the combined data and the characters written on the preprint, and generates a recognition model (step S43).

Once the recognition model is generated, the generation unit 42 stores the generated recognition model (step S44). The generation unit 42 stores the generated recognition model, for example, in the storage unit 16.

This section explains the operation of the character recognition system 40 when it generates a conversion model and a recognition model. Figure 21 is a diagram showing an example of the operation flow when the character recognition system 40 generates a conversion model and a recognition model.

The acquisition unit 11 acquires, as learning data, data combining an image of characters written on the preprint with the preprint image, conversion parameters, and the characters written on the preprint (step S51).

Once the training data is acquired, the generation unit 42 generates a conversion model by learning the relationship between the data, which is a combination of an image of characters written on a preprint and the preprint image, included in the training model, and the parameters included in the training model. The generation unit 42 also generates a recognition model by learning the relationship between the data, which is a combination of an image of characters written on a preprint and the preprint image, and the characters written on the preprint (step S52).

After generating the conversion model and the recognition model, the generation unit 42 stores the generated conversion model and the recognition model (step S53). The generation unit 42 stores the generated conversion model and the recognition model, for example, in the storage unit 16.

The character recognition system 40 of this embodiment uses a transformation model to combine an image showing characters written on a preprint with the preprint image. Then, the character recognition system 40 uses the recognition model to recognize the characters written on the preprint from the combined data. By using the preprint image converted using the transformation model, the character recognition system 40 can improve the accuracy of overlay when combining the image showing characters written on the preprint with the preprint image. By using the combined data in this manner, the character recognition system 40 can recognize the characters on the preprint using the recognition model while suppressing variations in the misalignment between the image showing characters written on the preprint and the preprint image. By recognizing the characters written on the preprint using the recognition model while suppressing variations in the misalignment between the two images, the character recognition system 40 can improve the accuracy of recognizing characters written on the preprint.

Furthermore, when generating a conversion model using training data, the character recognition system 40 can generate a conversion model that suppresses misalignment between an image of characters written on a preprint and the preprint image, which may occur in actual usage situations. Therefore, the character recognition system 40 can suppress fluctuations in misalignment between an image of characters written on a preprint and the preprint image, depending on actual usage situations. Therefore, when generating a conversion model using training data, the character recognition system 40 can further improve the recognition accuracy of characters written on a preprint.

The processes in the character recognition system 10 of the first embodiment and the character recognition system 40 of the second embodiment can be realized by executing a computer program on a computer. Figure 22 shows an example of the configuration of a computer 200 that executes a computer program that performs the processes in the character recognition system 10 of the first embodiment and the character recognition system 40 of the second embodiment. The computer 200 includes a CPU (Central Processing Unit) 201, memory 202, a storage device 203, an input/output I/F (Interface) 204, and a communication I/F 205.

The CPU 201 reads and executes computer programs for each process from the storage device 203. The CPU 201 may be configured as a combination of multiple CPUs. The CPU 201 may also be configured as a combination of a CPU and another type of processor. For example, the CPU 201 may be configured as a combination of a CPU and a GPU (Graphics Processing Unit). The memory 202 is configured with a DRAM (Dynamic Random Access Memory) or the like, and temporarily stores computer programs executed by the CPU 201 and data being processed. The storage device 203 stores computer programs executed by the CPU 201. The storage device 203 is configured with, for example, a non-volatile semiconductor storage device. Other storage devices such as a hard disk drive may also be used for the storage device 203. The input/output I/F 204 is an interface that accepts input from an operator and outputs display data, etc. The communication I/F 205 is an interface that transmits and receives data between the scanner 20 and the information processing server 30. The information processing server 30 may also have a similar configuration.

The computer programs used to execute each process can also be distributed by storing them on a computer-readable recording medium that non-temporarily stores data. Examples of recording media that can be used include magnetic tape for recording data and magnetic disks such as hard disks. Optical disks such as CD-ROMs (Compact Disc Read Only Memory) can also be used as recording media. Non-volatile semiconductor storage devices can also be used as recording media.

The present invention has been described above using the above-mentioned embodiment as an example. However, the present invention is not limited to the above-mentioned embodiment. In other words, the present invention can be applied in various aspects that are understandable to those skilled in the art within the scope of the present invention.

REFERENCE SIGNS LIST 10 Character recognition system 11 Acquisition unit 12 Image extraction unit 13 Recognition unit 14 Output unit 15 Generation unit 16 Storage unit 20 Scanner 30 Information processing server 40 Character recognition system 41 Recognition unit 42 Generation unit 51 Conversion unit 52 Image recognition unit 100 Computer 101 CPU
102 Memory 103 Storage device 104 Input/output I/F
105 Communication I/F

Claims (9)

An acquisition means for acquiring an image of characters written on a preprint of a form including the preprint;
a recognition means for recognizing characters written on a preprint of an acquired image from pixel data of the acquired image and data obtained by combining pixel data of the preprint image for each pixel , using a recognition model for recognizing characters written on the preprint from data obtained by combining pixel data of an image of characters written on the preprint and pixel data of a preprint image of the preprint for each pixel;
and an output means for outputting the result of the recognition. further comprising a transforming means for transforming the preprint image using a transformation parameter;
the recognition means recognizes characters written on the preprint of the acquired image from data obtained by combining pixel data of the acquired image and pixel data of the converted preprint image for each pixel;
The character recognition system of claim 1 . the conversion means converts the preprint image using a conversion model that estimates conversion parameters from data obtained by combining, for each pixel, pixel data of the image and pixel data of the converted preprint image;
The character recognition system of claim 2 . the recognition means identifies a type of form for which characters written on the preprint are to be recognized from the image, and recognizes the characters written on the preprint based on definition data corresponding to the identified type of form.
4. A character recognition system according to claim 1. the recognition means recognizes characters written on the preprint based on definition data that defines the position of the preprint on the form;
5. A character recognition system according to claim 1. The apparatus further comprises a generating means for learning the relationship between pixel data of an image of characters written on a preprint and data obtained by combining pixel data of the preprint image on a pixel- by-pixel basis, and the characters written on the preprint, and for generating a recognition model for recognizing characters written on the preprint in the image from the pixel data of the image of characters written on the preprint and data obtained by combining pixel data of the preprint image on a pixel-by-pixel basis.
6. A character recognition system according to claim 1. the generating means learns the relationship between pixel data of an image of characters written on a preprint, data obtained by combining pixel-by-pixel pixel data of the preprint image, and transformation parameters, and generates a transformation model for estimating transformation parameters to be used in transforming the preprint image.
7. The character recognition system of claim 6 . An image of the characters written on the preprint of the document containing the preprint is acquired,
using a recognition model that recognizes characters written on the preprint from pixel data of an image in which characters written on the preprint are copied and data obtained by combining pixel data of a preprint image in which the preprint is copied, the characters written on the preprint of the acquired image are recognized from pixel data of the acquired image and data obtained by combining pixel data of the preprint image,
outputting the results of the recognition;
Character recognition method. A process of acquiring an image of characters written on a preprint of a form including the preprint;
a process of recognizing characters written on a preprint of the acquired image from pixel data of the acquired image and data obtained by combining pixel data of the preprint image, using a recognition model that recognizes characters written on the preprint from data obtained by combining pixel data of an image of characters written on the preprint and pixel data of a preprint image of the preprint;
and a process of outputting the results of the recognition.