JP7619566B2 - Information processing device, program, and information processing method - Google Patents

Description

The present invention relates to an information processing device and a program.

Technology has been proposed for character recognition processing that recognizes characters contained in an image (see, for example, Patent Document 1).

JP 2015-069256 A

However, in conventional technology, no matter how much one tries to improve the accuracy of character recognition in character recognition processing, it is difficult to completely prevent the occurrence of misrecognition in character recognition processing. Furthermore, when a misrecognition occurs in character recognition processing, there is a possibility that an operator checking the results of the character recognition processing may overlook the occurrence of the misrecognition in the character recognition processing.

The present invention has been made in consideration of the above-mentioned circumstances, and one of the problems to be solved is to provide a technology that can reduce the risk of overlooking misrecognition during character recognition processing compared to conventional technologies.

In order to solve the above problems, the information processing device according to the present invention includes a recognition unit that recognizes a character represented in a character image by a first character recognition process that recognizes a character included in an image and a second character recognition process that recognizes a character included in an image, and a determination unit that, if a first character that is the recognition result of the one character by the first character recognition process and a second character that is the recognition result of the one character by the second character recognition process are the same character, determines the first character and the second character as a confirmed character corresponding to the recognition result of the one character, and, if the first character and the second character are different characters, determines one of the first character and the second character as a candidate character corresponding to the recognition result of the one character; and a display control unit that causes the confirmed character or the candidate character determined by the determination unit to be displayed on a display unit, and the display control unit causes the candidate character to be displayed on the display unit in a manner different from the confirmed character.

The program according to the present invention causes a processor to function as a recognition unit that recognizes a character represented in a character image by a first character recognition process that recognizes a character included in an image and a second character recognition process that recognizes a character included in an image, a determination unit that determines the first character and the second character as confirmed characters corresponding to the recognition result of the one character when a first character that is the recognition result of the one character by the first character recognition process and a second character that is the recognition result of the one character by the second character recognition process are the same character, and determines one of the first character and the second character as a candidate character corresponding to the recognition result of the one character when the first character and the second character are different characters, and a display control unit that displays the confirmed character or the candidate character determined by the determination unit on a display unit, and the display control unit displays the candidate character on the display unit in a manner different from the confirmed character.

Compared to conventional techniques, the present invention makes it possible to reduce the risk of overlooking misrecognition during character recognition processing.

FIG. 1 is an explanatory diagram showing an example of an overview of a character recognition system Sys according to an embodiment of the present invention. 1 is an explanatory diagram showing an example of an outline of an application form image GM. FIG. FIG. 13 is an explanatory diagram showing an example of an outline of an application content display image GH. FIG. 13 is an explanatory diagram showing an example of an outline of an application content display image GH. 1 is a block diagram showing an example of a configuration of a character recognition device 1. FIG. 1 is a block diagram showing an example of a hardware configuration of a character recognition device 1. FIG. FIG. 2 is a diagram showing an example of the data structure of application form information JM. FIG. 13 is a diagram showing an example of the data configuration of algorithm characteristic information JR. 4 is a flowchart showing an example of an operation of the character recognition device 1. 4 is a flowchart showing an example of an operation of the character recognition device 1. 11A and 11B are explanatory diagrams for explaining an example of a character recognition process and a display character determination process; 4 is a flowchart showing an example of an operation of the character recognition device 1. FIG. 13 is an explanatory diagram showing an example of an overview of a character recognition system Sys-B according to a modified example 4. FIG. 2 is a block diagram showing an example of the configuration of a character recognition device 1B.

Below, the embodiments for carrying out the present invention will be described with reference to the drawings. Note that in each drawing, the dimensions and scale of each part are appropriately different from the actual ones. In addition, the embodiments described below are preferred specific examples of the present invention, and therefore various technically preferable limitations are applied, but the scope of the present invention is not limited to these embodiments unless otherwise specified in the following description to the effect that the present invention is limited.

A. Embodiments
Hereinafter, an embodiment of the present invention will be described.

1. Overview of the character recognition system
The character recognition system Sys will be outlined below with reference to FIGS.

Figure 1 is an explanatory diagram for explaining an example of the configuration of a character recognition system Sys according to this embodiment.

As illustrated in FIG. 1, the character recognition system Sys includes a character recognition device 1, a terminal device 3, and an imaging device 5.

The character recognition device 1 (an example of an "information processing device") executes character recognition processing to recognize characters contained in an image, for example, using optical character recognition (OCR) technology. In this embodiment, it is assumed, as an example, that the image to be subjected to character recognition processing by the character recognition device 1 is an application form image GM obtained by capturing an application form for joining a specified service. The application form image GM will be described later with reference to FIG. 2.

The imaging device 5 is, for example, an optical device such as a camera or an image scanner, which captures an image of the membership application form and supplies image information JG representing the application form image GM obtained as a result of the capture to the terminal device 3.

The terminal device 3 includes a display unit 31 capable of displaying images, and an operation unit 32 that accepts operations by an operator of the terminal device 3. The terminal device 3 may be a stationary information device such as a personal computer, or may be a portable information terminal such as a tablet terminal or a smartphone.

In this embodiment, the terminal device 3 can communicate with the character recognition device 1 via the network NW, and transmits the image information JG supplied from the imaging device 5 to the character recognition device 1 via the network NW. When the image information JG is supplied from the terminal device 3, the character recognition device 1 performs character recognition processing on the application form image GM indicated by the image information JG supplied from the terminal device 3. The character recognition device 1 then generates display information JH for displaying the application content display image GH on the display unit 31 based on the recognition result of the character recognition processing on the application form image GM, and transmits the generated display information JH to the terminal device 3. When the display information JH is supplied from the character recognition device 1, the terminal device 3 displays the application content display image GH on the display unit 31 based on the display information JH supplied from the character recognition device 1. The application content display image GH will be described later in FIG. 3 and FIG. 4.

Figure 2 is an explanatory diagram for explaining an example of an application form image GM.

As shown in FIG. 2, the application form image GM includes multiple fields. Each of the multiple fields includes an item name writing area AM in which an item name, which is the name of an item of the application content to be written in that field, is written, and an application content writing area AK in which the application content related to that field is written. In the following, an image of the application form image GM that shows a character string related to the application content written in the application content writing area AK is referred to as a character string image GCL. In the following, an image of the character string image GCL that shows each of the multiple characters included in the character string shown by the character string image GCL is referred to as a character image GC. In other words, the character string image GCL is an image that includes multiple character images GC.

In this embodiment, as an example, it is assumed that the membership application form has a field for the "application number," a field for the "kana name," a field for the "name," a field for the "postal code," a field for the "address," a field for the "email," and a field for the "telephone number," as shown in Figure 2. In the following, in the application form image GM, the character string image GCL showing the application contents written in the field related to "application number" may be referred to as character string image GCL1, the character string image GCL showing the application contents written in the field related to "kana name" may be referred to as character string image GCL2, the character string image GCL showing the application contents written in the field related to "name" may be referred to as character string image GCL3, the character string image GCL showing the application contents written in the field related to "zip code" may be referred to as character string image GCL4, the character string image GCL showing the application contents written in the field related to "address" may be referred to as character string image GCL5, the character string image GCL showing the application contents written in the field related to "email" may be referred to as character string image GCL6, and the character string image GCL showing the application contents written in the field related to "telephone number" may be referred to as character string image GCL7.

In the following, it is assumed that the character recognition device 1 performs character recognition processing on an application form image GM having the format exemplified in FIG. 2, but the present invention is not limited to this aspect. In this embodiment, the character recognition device 1 can perform character recognition processing on documents having any format. In the following, it is assumed that the application contents written in the entry field are written in handwritten characters, which are characters written by hand, but the present invention is not limited to this aspect. In this embodiment, some or all of the multiple application contents written in the entry field may be written in type.

Figure 3 is an explanatory diagram illustrating an example of an application content display image GH.

As shown in FIG. 3, the application content display image GH includes a plurality of item names that correspond one-to-one to a plurality of entry fields in the application form image GM, a plurality of character string images GCL (character string images GCL1 to GCL7) that correspond one-to-one to a plurality of entry fields in the application form image GM, and a plurality of recognition character strings LMK (recognition character strings LMK1 to LMK7) that correspond one-to-one to a plurality of entry fields in the application form image GM.

Here, the recognized character string LMK is a character string obtained based on the result of character recognition processing performed by the character recognition device 1 on the character string image GCL included in the application form image GM. Hereinafter, each of the multiple characters included in the recognized character string LMK is referred to as a display character RH. In other words, the display character RH is a character obtained based on the result of character recognition processing performed by the character recognition device 1 on the character image GC.

As shown in FIG. 3, the application content display image GH may include a warning mark MK. Here, the warning mark MK is an image that is displayed in the application content display image GH when the character recognition device 1 executes a character recognition process on the character image GC and determines the display character RH based on the result of the character recognition process, and the character recognition device 1 determines that the character indicated by the character image GC and the display character RH may be different characters. For example, the example shown in FIG. 3 illustrates a case in which the character recognition device 1 determines that the character indicated by the character image GC1 included in the character string image GCL5 and the display character RH1 "丈" included in the recognized character string LMK5 may be different characters, and therefore displays the warning mark MK1 at a position corresponding to the character image GC1 and the display character RH1. In addition, the example shown in FIG. 3 illustrates a case in which the character recognition device 1 determines that the character indicated by the character image GC2 included in the character string image GCL7 and the display character RH2 "7" included in the recognized character string LMK7 are likely to be different characters, and therefore displays a warning mark MK2 at a position corresponding to the character image GC2 and the display character RH2.

As shown in FIG. 3, the application content display image GH includes a correction button SSB.
Here, the correction button SSB is a button for making the displayed character RH modifiable when the operator operating the terminal device 3 compares the character image GC with the displayed character RH corresponding to the character image GC in the application content display image GH and determines that the displayed character RH is a character different from the character indicated by the character image GC.

Figure 4 is an explanatory diagram showing an example of the application content display image GH when the edit button SSB is pressed.

As shown in FIG. 4, when the edit button SSB is pressed in the application content display image GH, the operator operating the terminal device 3 becomes able to edit each recognition character string LMK. Specifically, when the edit button SSB is pressed in the application content display image GH, the operator operating the terminal device 3 places the cursor CSR at a position corresponding to the display character RH selected by the operator operating the terminal device 3, thereby making the display character RH editable.

Also, as shown in FIG. 4, when the correction button SSB is pressed in the application content display image GH, and the operator who operates the terminal device 3 places the cursor CSR at the position corresponding to the display character RH and makes the display character RH editable, a pop-up image PRZ in which a correction candidate character RZ corresponding to the display character RH is displayed is displayed at the position corresponding to the display character RH. Here, the correction candidate character RZ is a character that is a correction candidate for the display character RH. More specifically, the correction candidate character RZ is a recognition candidate character R different from the display character RH among a plurality of recognition candidate characters R obtained by performing a character recognition process on the character image GC corresponding to the display character RH. Here, the recognition candidate character R is a character that is a candidate for the character indicated by the character image GC obtained by performing a character recognition process on the character image GC corresponding to the display character RH.
Then, the operator operating the terminal device 3 can replace the displayed character RH with the correction candidate character RZ by selecting the correction candidate character RZ displayed in the popup image PRZ.

In addition, FIG. 4 illustrates a case where an operator operating the terminal device 3 places the cursor CSR on the display character RH1 "丈" included in the recognition character string LMK5, and the display character RH1 is in an editable state. In addition, FIG. 4 illustrates a case where the correction candidate character RZ-1 "太" (thick), the correction candidate character RZ-2 "大" (large), and the correction candidate character RZ-3 "犬" (dog) are present corresponding to the display character RH1 "丈", and a pop-up image PRZ displaying these correction candidate characters RZ1 to RZ3 is displayed near the display character RH1. In addition, in FIG. 4, the operator operating the terminal device 3 can correct the display character RH1 to one correction candidate character RZ by selecting one correction candidate character RZ from the correction candidate characters RZ1 to RZ3 displayed in the pop-up image PRZ. For example, in FIG. 4, when an operator operating the terminal device 3 selects the correction candidate character RZ-2 "大" from among the correction candidate characters RZ-1 (bold), RZ-2 (large), and RZ-3 (dog) displayed in the pop-up image PRZ, the displayed character RH1 is corrected (changed) from "丈" to "大".

As shown in Figs. 3 and 4, the application content display image GH includes a confirmation button KTB.
Here, the Confirm button KTB is a button for confirming the multiple recognized character strings LMK included in the application content display image GH. When the Confirm button KTB is pressed, each display character RH included in each recognized character string LMK is confirmed, and the operator operating the terminal device 3 cannot change each display character RH. As a result, the character recognition system Sys confirms the recognition result of the characters included in the application form image GM as the multiple display characters RH displayed in the application content display image GH.

In this embodiment, an example is given of an aspect in which the operator operating the terminal device 3 can edit each recognition character string LMK only when the edit button SSB is pressed while the application content display image GH is displayed on the display unit 31, but the present invention is not limited to such an aspect. For example, the operator operating the terminal device 3 may be able to edit the recognition character string LMK at the point when the application content display image GH is displayed on the display unit 31. In this case, the application content display image GH does not need to include the edit button SSB.

2. Functions of the character recognition device
The functions of the character recognition device 1 will be described below with reference to FIGS.

Figure 5 is a functional block diagram showing an example of the configuration of the character recognition device 1.

As illustrated in FIG. 5, the character recognition device 1 includes a control unit 11 that controls each part of the character recognition device 1, a storage unit 12 that stores various information, and a communication unit 13 that performs communication with an external device that exists outside the character recognition device 1.

As illustrated in FIG. 5, the control unit 11 includes an image information acquisition unit 111, a character area identification unit 112, a written content recognition unit 113, a recognition result determination unit 114, a display information generation unit 115, an operation reception unit 116, and a learning execution unit 117.

The image information acquisition unit 111 acquires image information JG from the terminal device 3.
The character area identification unit 112 identifies a plurality of application content description areas AK in the application form image GM indicated by the image information JG acquired by the image information acquisition unit 111. Specifically, the character area identification unit 112 identifies a plurality of application content description areas AK included in the application form image GM based on, for example, information (not shown) relating to the layout of the application form image GM stored in the storage unit 12.

The description content recognition unit 113 (an example of a "recognition unit") performs character recognition processing on each character image GC contained in the character string image GCL displayed in the application content description area AK, thereby identifying one or more recognition candidate characters R as the recognition result of the character indicated by the character image GC.

Specifically, the written content recognition unit 113 includes M character recognition units RG[1] to RG[M]. Here, the value M is a natural number that satisfies M≧2. In the following, the m-th character recognition unit RG among the M character recognition units RG[1] to RG[M] may be referred to as the character recognition unit RG[m]. Here, the variable m is a natural number that satisfies 1≦m≦M. The character recognition unit RG[m] performs character recognition processing on the character image GC, and can identify one or more recognition candidate characters R as the recognition result of the character indicated by the character image GC.
Note that various methods can be applied to the character recognition process by the character recognition unit RG[m]. In this embodiment, however, as an example, it is assumed that a method based on optical character recognition (OCR) is applied.

In this embodiment, the written content recognition unit 113 selects W character recognition units RG from among the M character recognition units RG[1] to RG[M], and causes each of the selected W character recognition units RG to execute character recognition processing. Here, W is a natural number that satisfies 2≦W≦M. Note that in this embodiment, as an example, the case where "W=2" is assumed. In the following, the selected W (=2) character recognition units RG are referred to as character recognition unit RG[m1] and character recognition unit RG[m2]. Here, variable m1 is a natural number that satisfies 1≦m1≦M, and variable m2 is a natural number that satisfies 1≦m2≦M.

The recognition result determination unit 114 (an example of a "determination unit") determines one of the one or more recognition candidate characters R identified by the content recognition unit 113 as the recognition result of the character indicated by the character image GC to be the display character RH.
The display information generation unit 115 (an example of a “display control unit”) generates display information JH for displaying, on the display unit 31, the application content display image GH representing the display character RH determined by the recognition result determination unit 114.

The operation reception unit 116 (an example of a “reception unit”) receives operation information J S indicating the operation content of the operation unit 32 by the operator who operates the terminal device 3 .
The learning execution unit 117 (an example of a “learning unit”) causes a learning model N N [m], which will be described later, to learn the relationship between the character image G C and the character indicated by the character image G C .

As shown in FIG. 5, the memory unit 12 stores application form information JM, algorithm characteristic information JR, M learning models NN[1] to NN[M] in one-to-one correspondence with the M character recognition units RG[1] to RG[M], and a control program PG1 for controlling each unit of the character recognition device 1. If there are multiple types of documents, such as membership applications, that can be subject to character recognition processing by the character recognition device 1, the memory unit 12 stores multiple pieces of application form information JM in one-to-one correspondence with the multiple types of documents. The application form information JM will be described later in FIG. 7. The algorithm characteristic information JR will be described later in FIG. 8.

The learning model N N [m] is, for example, a multi-layer neural network, and is a learning model that learns the relationship between images containing one or more types of characters and the characters contained in the images, for that type or types of characters.
For ease of explanation, in this embodiment, it is assumed that the types of characters shown in the image are six types: kanji, hiragana, katakana, Arabic numerals (hereinafter simply referred to as "numerals"), alphabets, and symbols. Here, symbols are a concept including, for example, dots, commas, periods, commas, hyphens, and arithmetic symbols.

In this embodiment, the character recognition unit RG[m] uses the learning model N N[m] to recognize the character indicated by the character image G C. That is, in this embodiment, the character recognition unit RG[m] uses the learning model N N[m] stored in the storage unit 12 to perform character recognition processing by AI (Artificial Intelligence)-OCR.
In addition, in this embodiment, the M character recognition units RG use different algorithms to perform the character recognition process. That is, the algorithm used by the character recognition unit RG[m1] in the character recognition process is different from the algorithm used by the character recognition unit RG[m2] in the character recognition process. Here, "different algorithms" may mean, for example, that the types of learning models N N used in the character recognition process (e.g., types of neural networks) are different, or that the data sets used when training the learning models N N used in the character recognition process are different, or may include both of these concepts.
Specifically, in this embodiment, as an example, it is assumed that the characters included in the image used when the character recognition unit RG[m1] trains the learning model NN[m1] used in the character recognition process include characters of a different type from the characters included in the image used when the character recognition unit RG[m2] trains the learning model NN[m2] used in the character recognition process. In this case, the characters included in the image used when the character recognition unit RG[m1] trains the learning model NN[m1] used in the character recognition process may include the same type of characters as the characters included in the image used when the character recognition unit RG[m2] trains the learning model NN[m2] used in the character recognition process. In addition, in this embodiment, as an example, it is assumed that the neural network constituting the learning model NN[m1] used in the character recognition process by the character recognition unit RG[m1] and the neural network constituting the learning model NN[m2] used in the character recognition process by the character recognition unit RG[m2] are different types of neural networks. However, the present invention is not limited to this embodiment. The neural network constituting the learning model N[m1] used by the character recognition unit RG[m1] in the character recognition process and the neural network constituting the learning model N[m2] used by the character recognition unit RG[m2] in the character recognition process may be the same type of neural network.

In this embodiment, as an example, it is assumed that the storage unit 12 stores M learning models NN[1] to NN[M] that correspond one-to-one to the M character recognition units RG[1] to RG[M], but the present invention is not limited to this form. For example, some or all of the M character recognition units RG[1] to RG[M] may be components that perform character recognition processing without using a neural network.

Figure 6 is a configuration diagram showing an example of the hardware configuration of the character recognition device 1.

As shown in FIG. 6, the character recognition device 1 includes a processor 1001 that controls each part of the character recognition device 1, a storage device 1002 that stores various information, and a communication device 1003 for communicating with an external device that is outside the character recognition device 1.

Of these, the storage device 1002 includes one or both of a volatile memory such as a RAM (Random Access Memory) that functions as a working area for the processor 1001, and a non-volatile memory such as an EEPROM (Electrically Erasable Programmable Read-Only Memory) that stores various information such as the control program PG1, and provides the function of a storage unit 12.
The processor 1001 is configured to include, for example, one or more central processing units (CPUs). The processor 1001 executes a control program PG1 stored in the storage device 1002 and operates according to the control program PG1 to function as the control unit 11. The processor 1001 may include hardware such as a graphics processing unit (GPU), a digital signal processor (DSP), or a field programmable gate array (FPGA) in addition to one or more CPUs, or in place of a part or all of the one or more CPUs. In this case, a part or all of the control unit 11 realized by the processor 1001 may be realized by hardware such as a DSP.
The communication device 1003 is hardware for communicating with an external device located outside the character recognition device 1 via one or both of a wired network and a wireless network, and provides the function of the communication unit 13 .

Figure 7 is an explanatory diagram showing an example of the data structure of application form information JM. Here, application form information JM is information regarding multiple fields included in the application form image GM.

As shown in FIG. 7, the application form information JM has multiple records that correspond one-to-one to the multiple fields in the application form image GM. Each record in the application form information JM has an item ID, an item name, and character type information JMM.

Of these, the item ID is information for identifying each entry field among the multiple entry fields contained in the application form image GM.
Further, the item name is the name written in the item name writing area AM of each entry field of the application form image GM.

Furthermore, the character type information JMM is information regarding the type of characters that may be entered in the application content entry area AK of each entry field of the application form image GM. Specifically, in this embodiment, the character type information JMM includes kanji use or not information JMM1 indicating whether kanji may be entered in the application content entry area AK, hiragana use or not information JMM2 indicating whether hiragana may be entered in the application content entry area AK, katakana use or not information JMM3 indicating whether katakana may be entered in the application content entry area AK, number use or not information JMM4 indicating whether numbers may be entered in the application content entry area AK, alphabet use or not information JMM5 indicating whether alphabets may be entered in the application content entry area AK, and symbol use or not information JMM6 indicating whether symbols may be entered in the application content entry area AK.
For example, in Fig. 7, when the kanji usage information JMM1 indicates "1", it indicates that there is a possibility that a kanji will be written in the entry field corresponding to the kanji usage information JMM1, and when the kanji usage information JMM1 indicates "0", it indicates that there is no possibility that a kanji will be written in the entry field corresponding to the kanji usage information JMM1. However, the kanji usage information JMM1 is not limited to binary information indicating either "1" or "0", but may be information indicating the possibility that a kanji will be written in the entry field corresponding to the kanji usage information JMM1, for example, with a real number between 0 and 1. Note that the hiragana usage information JMM2, the katakana usage information JMM3, the number usage information JMM4, the alphabet usage information JMM5, and the symbol usage information JMM6 are similar to the kanji usage information JMM1, except that the types of characters are different.

Figure 8 is an explanatory diagram showing an example of the data structure of algorithm property information JR. Here, algorithm property information JR is information that indicates the properties of the algorithm used by the character recognition unit RG in the character recognition process.

As shown in FIG. 8, the algorithm property information JR has M records that correspond one-to-one to the M character recognition units RG[1] to RG[M]. Each record in the algorithm property information JR has an algorithm ID, learning presence/absence information, learning model information, and valid character type information JRM.

Of these, the algorithm ID is information for identifying the algorithm that the character recognition unit RG[m] uses when performing character recognition processing from among multiple algorithms that may be used in the written content recognition unit 113.
The learning information is information indicating whether or not the algorithm used by the character recognition unit RG[m] when performing character recognition processing is an algorithm that uses a neural network.
Furthermore, the learning model information is information indicating the learning model N[m] configured by a neural network when the algorithm used by the character recognition unit R[m] when performing character recognition processing is an algorithm using a neural network. In other words, the learning model information is information for identifying the learning model N[m] to be used by the character recognition unit R[m] from among the multiple learning models N stored in the memory unit 12.

The valid character type information JRM is information that indicates the type of characters that can be recognized by the algorithm used when the character recognition unit RG[m] performs character recognition processing. Specifically, in this embodiment, the valid character type information JRM includes kanji correspondence presence/absence information JRM1 indicating whether the algorithm used by the character recognition unit RG[m] when performing character recognition processing is an algorithm suitable for recognizing kanji, hiragana correspondence presence/absence information JRM2 indicating whether the algorithm used by the character recognition unit RG[m] when performing character recognition processing is an algorithm suitable for recognizing hiragana, katakana correspondence presence/absence information JRM3 indicating whether the algorithm used by the character recognition unit RG[m] when performing character recognition processing is an algorithm suitable for recognizing katakana, number correspondence presence/absence information JRM4 indicating whether the algorithm used by the character recognition unit RG[m] when performing character recognition processing is an algorithm suitable for recognizing numbers, alphabet correspondence presence/absence information JRM5 indicating whether the algorithm used by the character recognition unit RG[m] when performing character recognition processing is an algorithm suitable for recognizing the alphabet, and symbol correspondence presence/absence information JRM6 indicating whether the algorithm used by the character recognition unit RG[m] when performing character recognition processing is an algorithm suitable for recognizing symbols.
8, when the kanji support presence/absence information JRM1 indicates "1", it indicates that the algorithm used in the character recognition process by the character recognition unit RG corresponding to the kanji support presence/absence information JRM1 is an algorithm suitable for recognizing kanji, and when the kanji support presence/absence information JRM1 indicates "0", it indicates that the algorithm used in the character recognition process by the character recognition unit RG corresponding to the kanji support presence/absence information JRM1 is an algorithm not suitable for recognizing kanji. However, the kanji support presence/absence information JRM1 is not binary information indicating either "1" or "0", but may be information indicating the degree of suitability of the algorithm used in the character recognition process by the character recognition unit RG corresponding to the kanji support presence/absence information JRM1 for use in recognizing kanji, using a real number between 0 and 1. Hiragana support information JRM2, katakana support information JRM3, number support information JRM4, alphabet support information JRM5, and symbol support information JRM6 are similar to kanji support information JRM1, except for the types of characters.

3. Operation of the character recognition device
The operation of the character recognition device 1 will be described below with reference to FIGS.

<3.1. Application content display process>
Figures 9 and 10 are flowcharts showing an example of the operation of the character recognition device 1 when the character recognition device 1 executes a process (hereinafter referred to as the "application content display process") of generating display information JH indicating an application content display image GH based on image information JG indicating an application form image GM.

As shown in FIG. 9, when the application content display process is started, the image information acquisition unit 111 acquires the image information JG supplied from the terminal device 3 (S101).
Next, the character area identification unit 112 identifies a plurality of application content writing areas AK in the application form image GM indicated by the image information JG acquired by the image information acquisition unit 111 in step S101 (S103).
Then, the character area specification unit 112 selects one application content description area AK from the multiple application content description areas AK specified in step S103 (S105). Note that in step S105, the character area specification unit 112 selects an application content description area AK that has not been selected since the application content display process was started from among the multiple application content description areas AK.

As illustrated in FIG. 9, the description content recognition unit 113 selects W character recognition units RG for performing character recognition processing on the character string image GCL displayed in the application content description area AK selected in step S105 (S107).
In this embodiment, in step S107, the description content recognition unit 113 selects the character recognition unit RG[m1] and the character recognition unit RG[m2] from among the M character recognition units RG[1] to RG[M] based on the character type information JMM corresponding to the application content description area AK and the algorithm characteristic information JR. Specifically, in step S107, the description content recognition unit 113 selects the character recognition unit RG[m1] and the character recognition unit RG[m2] from among the M character recognition units RG[1] to RG[M] based on the similarity between the character type information JMM corresponding to the application content description area AK and the valid character type information JRM corresponding to each character recognition unit RG. More specifically, in step S107, the content recognition unit 113 selects two character recognition units RG from the character recognition units RG[1] to RG[M] that have a higher similarity between the character type information JMM corresponding to the application content entry area AK and the valid character type information JRM corresponding to each character recognition unit RG than the others, as character recognition unit RG[m1] and character recognition unit RG[m2].
Here, as the similarity between the character type information JMM and the valid character type information JRM, for example, the inner product of a vector VM having each of the pieces of information (JMM1 to JMM6) included in the character type information JMM as its elements and a vector VR having each of the pieces of information (JRM1 to JRM6) included in the valid character type information JRM as its elements may be used. More specifically, when the vector VM is set as VM=(JMM1, JMM2, JMM3, JMM4, JMM5, JMM6) and the vector VR is set as VR=(JRM1, JRM2, JRM3, JRM4, JRM5, JRM6), the inner product of the vector VM and the vector VR (JMM1*JRM1+JMM2*JRM2+JMM3*JRM3+JMM4*JRM4+JMM5*JRM5+JMM6*JRM6) may be used as the similarity between the character type information JMM and the valid character type information JRM.

As illustrated in FIG. 9, each of the character recognition units RG[m1] and RG[m2] divides the character string image GCL displayed in the application content entry area AK selected in step S105 into a plurality of character images GC (S109).
Then, each of the character recognition units RG[m1] and RG[m2] selects one character image GC from the multiple character images GC classified in step S109 (S111).
Next, each of the character recognition units RG[m1] and RG[m2] executes character recognition processing on the character image GC selected in step S111 (S113).
10, the recognition result determination unit 114 determines the display character RH based on the result of the character recognition process in step S113 (S115). Note that, hereinafter, the process of determining the display character RH by the recognition result determination unit 114 is referred to as a display character determination process.

Figure 11 is an explanatory diagram for explaining the character recognition process performed by the character recognition unit RG[m1] and the character recognition unit RG[m2] in step S113, and the display character determination process performed by the recognition result determination unit 114 in step S115.

As shown in Fig. 11, the character recognition unit RG[m1] performs character recognition processing on the character image GC to identify Q1 recognition candidate characters R[m1][1] to R[m1][Q1]. Here, the value Q1 is a natural number that satisfies Q1 ≥ 1. Hereinafter, the q1th recognition candidate character R among the Q1 recognition candidate characters R[m1][1] to R[m1][Q1] is referred to as the recognition candidate character R[m1][q1]. Here, the variable q1 is a natural number that satisfies 1 ≤ q1 ≤ Q1.
The character recognition unit RG[m1] also calculates a recognition accuracy D[m1][q1] corresponding to the recognition candidate character R[m1][q1]. Here, the recognition accuracy D[m1][q1] is an index value indicating the likelihood of the recognition result of the character represented in the character image GC when the character recognition unit RG[m1] performs character recognition processing on the character image GC. In this embodiment, the higher the recognition accuracy, the more likely the recognition result is. In this embodiment, the Q1 recognition candidate characters R[m1][1] to R[m1][Q1] are determined such that the Q1 recognition accuracy D[m1][1] to D[m1][Q1] that correspond one-to-one to the Q1 recognition candidate characters R[m1][1] to R[m1][Q1] satisfy the relationship "D[m1][1] ≧ D[m1][2] ≧ ... ≧ D[m1][Q1]". That is, the recognition candidate character R[m1][1] is the recognition candidate character R with the highest recognition accuracy among the Q1 recognition candidate characters R[m1][1] to R[m1][Q1] identified by the character recognition unit RG[m1].
In this embodiment, the character recognition process performed by the character recognition unit RG[m1] is an example of a "first character recognition process", the recognition candidate character R[m1][1] with the highest recognition accuracy identified by the character recognition unit RG[m1] is an example of a "first character", and the learning model N N[m1] used in the character recognition process by the character recognition unit RG[m1] is an example of a "first learning model".

As shown in Fig. 11, the character recognition unit RG[m2] performs character recognition processing on the character image GC to identify Q2 recognition candidate characters R[m2][1] to R[m2][Q2]. Here, the value Q2 is a natural number that satisfies Q2 ≥ 1. Hereinafter, the q2th recognition candidate character R among the Q2 recognition candidate characters R[m2][1] to R[m2][Q2] is referred to as the recognition candidate character R[m2][q2]. Here, the variable q2 is a natural number that satisfies 1 ≤ q2 ≤ Q2.
The character recognition unit RG[m2] also calculates a recognition accuracy D[m2][q2] corresponding to the recognition candidate character R[m2][q2]. Here, the recognition accuracy D[m2][q2] is an index value indicating the likelihood of the recognition result of the character represented in the character image GC when the character recognition unit RG[m2] performs character recognition processing on the character image GC. In this embodiment, the Q2 recognition candidate characters R[m2][1] to R[m2][Q2] are determined such that the Q2 recognition accuracy D[m2][1] to D[m2][Q2] that correspond one-to-one to the Q2 recognition candidate characters R[m2][1] to R[m2][Q2] satisfy the relationship "D[m2][1] ≧ D[m2][2] ≧ ... ≧ D[m2][Q2]". That is, the recognition candidate character R[m2][1] is the recognition candidate character R with the highest recognition accuracy among the Q2 recognition candidate characters R[m2][1] to R[m2][Q2] identified by the character recognition unit RG[m2].
In this embodiment, the character recognition process performed by the character recognition unit RG[m2] is an example of a "second character recognition process", and the recognition candidate character R[m2][1] with the highest recognition accuracy identified by the character recognition unit RG[m2] is an example of a "second character".

As illustrated in FIG. 11 , the recognition result determination unit 114 executes a display character determination process that determines a display character RH based on the recognition candidate character R[m1][1] with the highest recognition accuracy among the Q1 recognition candidate characters R[m1][1] to R[m1][Q1] identified as a result of the character recognition process by the character recognition unit RG[m1], and the recognition candidate character R[m2][1] with the highest recognition accuracy among the Q2 recognition candidate characters R[m2][1] to R[m2][Q2] identified as a result of the character recognition process by the character recognition unit RG[m2].
Specifically, in the display character determination process, if the recognition candidate character R[m1][1] and the recognition candidate character R[m2][1] are the same character, the recognition result determination unit 114 determines the recognition candidate character R[m1][1] and the recognition candidate character R[m2][1] as the confirmed recognition character RX (an example of a "confirmed character"). In addition, if the recognition candidate character R[m1][1] and the recognition candidate character R[m2][1] are different characters and the recognition accuracy D[m1][1] and the recognition accuracy D[m2][1] satisfy D[m1][1]>D[m2][1], the recognition result determination unit 114 determines the recognition candidate character R[m1][1] as the likely recognition character RY (an example of a "candidate character"). Furthermore, when the recognition candidate character R[m1][1] and the recognition candidate character R[m2][1] are different characters and the recognition accuracy D[m1][1] and the recognition accuracy D[m2][1] satisfy D[m2][1]≧D[m1][1], the recognition result determination unit 114 determines the recognition candidate character R[m2][1] as a likely recognized character RY (another example of a "candidate character").Then, the recognition result determination unit 114 determines the confirmed recognition character RX or the likely recognized character RY as the display character RH.

In addition, FIG. 11 illustrates an example in which, of the Q1 recognition candidate characters R[m1][1] to R[m1][Q1] identified as a result of the character recognition process by the character recognition unit RG[m1], the recognition candidate character R[m1][1] is "丈" and has a recognition accuracy D[m1][1] of "98%", the recognition candidate character R[m1][2] is "大" and has a recognition accuracy D[m1][2] of "85%", and the recognition candidate character R[m1][Q1] is "燃" and has a recognition accuracy D[m1][1] of "25%". 11 illustrates an example in which, among the Q2 recognition candidate characters R[m2][1] to R[m2][Q2] identified as a result of the character recognition process by the character recognition unit RG[m2], the recognition candidate character R[m2][1] is “bold” and the recognition accuracy D[m2][1] is “92%”, the recognition candidate character R[m2][2] is “dog” and the recognition accuracy D[m2][2] is “80%”, and the recognition candidate character R[m2][Q2] is “person” and the recognition accuracy D[m2][Q2] is “45%”. As a result, in the example illustrated in FIG. 11, the recognition candidate character R[m1][1] “jo” and the recognition candidate character R[m2][1] “bold” are different characters, so the recognition result determination unit 114 determines in the display character determination process that there is no confirmed recognition character RX. In the example shown in FIG. 11, the recognition accuracy D[m1][1] corresponding to the recognition candidate character R[m1][1] "丈" is "98%", which is greater than the recognition accuracy D[m2][1] corresponding to the recognition candidate character R[m2][1] "太" which is "92%". Therefore, in the display character determination process, the recognition result determination unit 114 determines the recognition candidate character R[m1][1] "丈" as the likely recognized character RY. In the example shown in FIG. 11, since there is no confirmed recognition character RX and the recognition candidate character R[m1][1] "丈" is determined as the likely recognized character RY, the recognition result determination unit 114 determines the recognition candidate character R[m1][1] "丈" determined as the likely recognized character RY as the display character RH in the display character determination process.

Returning to the explanation of FIG.
As illustrated in FIG. 10, the content recognition unit 113 determines whether or not the character recognition process in step S113 and the display character determination process in step S115 are completed for all of the multiple character images GC separated from the character string image GCL in step S109 (S117).
If the result of the determination in step S117 is negative, the description content recognition unit 113 advances the process to step S111.

As illustrated in FIG. 10, if the result of the judgment in step S117 is positive, the character area identification unit 112 determines whether or not the character recognition process in step S113 and the display character determination process in step S115 have been completed for all of the multiple application content description areas AK identified in step S103 (S119).
If the result of the determination in step S119 is negative, the description content recognition unit 113 advances the process to step S105.

As illustrated in FIG. 10, if the result of the determination in step S119 is positive, the display information generating unit 115 generates display information JH indicating an application content display image GH including the display characters RH (S121).
Specifically, in step S121, the display information generating unit 115 generates the display information JH so that the display character RH corresponding to the recognition candidate character R determined to be the likely recognition character RY in the display character determination process and the display character RH corresponding to the recognition candidate character R determined to be the confirmed recognition character RX in the display character determination process are displayed in different modes in the application content display image GH. More specifically, in step S121, when the display character RH is the recognition candidate character R determined to be the likely recognition character RY in the display character determination process, the display information generating unit 115 generates the display information JH so that an attention drawing mark MK is placed at a position corresponding to the display character RH in the application content display image GH.
In the application content display image GH illustrated in Fig. 3, among the multiple display characters RH included in the recognition character strings LMK1 to LMK7, the display character RH1 "丈" included in the recognition character string LMK5 and the display character RH2 "７" included in the recognition character string LMK7 correspond to the recognition candidate character R determined as the likely recognition character RY, and the display characters RH other than the display characters RH1 and RH2 correspond to the recognition candidate character R determined as the confirmed recognition character RX. For this reason, in the application content display image GH illustrated in Fig. 3, the attention calling mark MK1 is displayed at a position corresponding to the display character RH1, and the attention calling mark MK2 is displayed at a position corresponding to the display character RH2.

When the processing of step S121 is completed, the control unit 11 ends the application content display processing shown in Figures 9 and 10.

As described above, according to this embodiment, the display information generating unit 115 generates the display information JH showing the application content display image GH in which the display character RH corresponding to the likely recognized character RY and the display character RH corresponding to the confirmed recognized character RX are displayed in different manners. Specifically, according to this embodiment, the display information generating unit 115 generates the display information JH showing the application content display image GH in which the display character RH corresponding to the likely recognized character RY is highlighted by the attention mark MK compared to the display character RH corresponding to the confirmed recognized character RX. Therefore, according to this embodiment, when the character indicated by the character image GC and the display character RH are likely to be different characters, it is possible to highlight the display character RH in the application content display image GH compared to when the possibility is low. Therefore, according to this embodiment, it is possible to reduce the possibility that the operator operating the terminal device 3 will overlook that the character indicated by the character image GC and the display character RH are different characters, compared to when the display character RH corresponding to the likely recognized character RY and the display character RH corresponding to the confirmed recognized character RX are displayed in the same manner.

Furthermore, according to this embodiment, the written content recognition unit 113 selects the character recognition unit RG[m1] and the character recognition unit RG[m2] suitable for the character recognition process for the character string image GCL from among the M character recognition units RG[1] to RG[M], and executes the character recognition process using the two character recognition units RG[m1] and RG[m2]. As described above, the learning model N N[m1] used by the character recognition unit RG[m1] in the character recognition process is different from the learning model N N[m2] used by the character recognition unit RG[m2] in the character recognition process. Therefore, in this embodiment, when the result of the character recognition process by the character recognition unit RG[m1] and the result of the character recognition process by the character recognition unit RG[m2] match, the accuracy of character recognition of the character image GC by the character recognition device 1 is higher than the accuracy of character recognition of the character image GC by the character recognition device 1 when the result of the character recognition process by the character recognition unit RG[m1] and the result of the character recognition process by the character recognition unit RG[m2] differ. That is, in this embodiment, when the result of the character recognition process by the character recognition unit RG[m1] and the result of the character recognition process by the character recognition unit RG[m2] match, it is estimated that the accuracy of character recognition of the character image GC by the character recognition device 1 is relatively high, whereas when the result of the character recognition process by the character recognition unit RG[m1] and the result of the character recognition process by the character recognition unit RG[m2] differ, it is estimated that the accuracy of character recognition of the character image GC by the character recognition device 1 is relatively low. In this embodiment, when the result of the character recognition process by the character recognition unit RG[m1] differs from the result of the character recognition process by the character recognition unit RG[m2], the display character RH related to the result of the character recognition process is highlighted with an attention mark MK. Therefore, according to this embodiment, the operator who operates the terminal device 3 can easily visually recognize the display character RH for which the accuracy of character recognition by the character recognition device 1 is relatively low.
Furthermore, in this embodiment, when character recognition processing is performed using two character recognition units RG, character recognition unit RG[m1] and character recognition unit RG[m2], and the result of the character recognition processing by character recognition unit RG[m1] matches the result of the character recognition processing by character recognition unit RG[m2], the accuracy of character recognition of the character image GC is at least equal to or greater than the accuracy of character recognition of the character image GC when character recognition processing is performed using one of character recognition units RG[m1] and RG[m2]. That is, in this embodiment, when character recognition processing is performed using two character recognition units RG, RG[m1] and RG[m2], the accuracy of character recognition of the character image GC recognized as the confirmed recognized character RX is equal to or higher than the accuracy of character recognition of the character image GC when character recognition processing is performed using one of the character recognition units RG[m1] and RG[m2]. Therefore, according to this embodiment, it is possible to reduce the possibility that the character indicated by the character image GC and the display character RH determined as the confirmed recognized character RX without the attention mark MK are different characters, compared to the case where the character recognition unit RG is executed using one character recognition unit RG.

<3.2. Character Correction Processing>
12 is a flowchart showing an example of the operation of the character recognition device 1 when the character recognition device 1 executes a process of correcting the display characters RH displayed in the application content display image GH based on an operation by an operator of the terminal device 3 (hereinafter referred to as the "character correction process"). In this embodiment, the character correction process is started when the operator operating the terminal device 3 presses the correction button SSB in the application content display image GH. As described above, the character correction process may be started regardless of whether the correction button SSB is pressed. Specifically, the character correction process may be started, for example, when the application content display image GH is displayed on the display unit 31.

As illustrated in FIG. 12, when the character correction process is started, the operation reception unit 116 identifies the display character RH at the position corresponding to the cursor CSR in the application content display image GH based on the operation information JS supplied from the terminal device 3 (S201).

Next, the written content recognition unit 113 identifies a plurality of recognition candidate characters R corresponding to the display character RH identified in step S201 (S203).
Specifically, in step S203, when the content recognition unit 113 performs character recognition processing on the character image GC corresponding to the display character RH identified in step S201, it identifies Q1 recognition candidate characters R[m1][1] to R[m1][Q1] and Q2 recognition candidate characters R[m2][1] to R[m2][Q2] that have been identified as the recognition results of the character image GC.

Then, the written content recognition unit 113 identifies a correction candidate character Rz from among the plurality of recognition candidate characters R identified in step S203 (S205).
Specifically, the written content recognition unit 113 identifies, as the correction candidate character RZ, the recognition candidate character R whose recognition accuracy D is equal to or greater than a predetermined threshold Dth and is other than the display character RH, among the multiple recognition candidate characters R identified in step S203. In this embodiment, as an example, it is assumed that the threshold Dth is "80%". Therefore, for example, in the example shown in FIG. 11, "bold" which is the recognition candidate character R[m2][1] with a recognition accuracy D[m2][1] of "92%", "large" which is the recognition candidate character R[m1][2] with a recognition accuracy D[m1][2] of "85%", and "dog" which is the recognition candidate character R[m2][2] with a recognition accuracy D[m2][2] of "80%" are identified as the correction candidate character RZ. In the example shown in FIG. 11, the recognition candidate character R[m1][1] “JO”, which has a recognition accuracy D[m1][1] of 98%, is determined as the display character RH, and therefore the recognition candidate character R[m1][1] is not identified as the correction candidate character RZ.

12, the display information generating unit 115 updates the display information JH indicating the application content display image GH so that a pop-up image PRZ is displayed in the application content display image GH and the correction candidate character RZ identified in step S205 is displayed in the pop-up image PRZ, and controls each part of the character recognition device 1 so that the updated display information JH is transmitted to the terminal device 3 (S207). Therefore, for example, as shown in the example of FIG. 4, in the application content display image GH displayed on the display unit 31, a pop-up image PRZ displaying the correction candidate character RZ-1 "bold", the correction candidate character RZ-2 "large", and the correction candidate character RZ-3 "dog" is displayed near the display character RH1 "length".

As illustrated in FIG. 12, the operation reception unit 116 determines whether or not operation information J S indicating an instruction to modify the display character R H has been supplied from the terminal device 3 (S209).
If the result of the determination in step S209 is negative, the operation reception unit 116 advances the process to step S213.
If the result of the determination in step S209 is positive, the operation receiving unit 116 receives operation information JS indicating the contents of an instruction to correct the display character RH.

If the result of the determination in step S209 is positive, the learning execution unit 117 generates or updates the correction history information J SH based on the operation information JS accepted by the operation acceptance unit 116 in step S209 (S211), and the process proceeds to step S213.
Here, the correction history information JSH is information including the correction content in step S209 (the display character RH after correction), the display character RH corrected in step S209 (the display character RH before correction), the character image GC corresponding to the display character RH before correction, and the character recognition unit RG[m1] and the character recognition unit RG[m2] which have identified the recognition candidate character R corresponding to the display character RH before correction.

Thereafter, the operation reception unit 116 determines whether or not operation information JS indicating that the confirmation button KTB has been pressed has been supplied from the terminal device 3 (S213).
If the result of the determination in step S213 is negative, the operation reception unit 116 advances the process to step S201.

If the result of the judgment in step S213 is positive, the learning execution unit 117, based on the correction history information JSH, teaches the learning model N[m1] corresponding to the character recognition unit RG[m1] and the learning model N[m2] corresponding to the character recognition unit RG[m2] that the character image G represents the corrected display character R (S215).
In addition, in step S215, the learning execution unit 117 causes the learning model NN[m1] corresponding to the character recognition unit RG[m1] to learn that the character image GC represents the corrected display character RH only when the corrected display character RH and the recognition candidate character R[m1][1] are different characters. However, in step S215, even when the corrected display character RH and the recognition candidate character R[m1][1] are the same character, the learning execution unit 117 may cause the learning model NN[m1] corresponding to the character recognition unit RG[m1] to learn that the character image GC represents the corrected display character RH.
Also, in step S215, the learning execution unit 117 causes the learning model NN[m2] corresponding to the character recognition unit RG[m2] to learn that the character image GC represents the corrected display character RH only when the corrected display character RH and the recognition candidate character R[m2][1] are different characters. However, in step S215, even when the corrected display character RH and the recognition candidate character R[m2][1] are the same character, the learning execution unit 117 may cause the learning model NN[m2] corresponding to the character recognition unit RG[m2] to learn that the character image GC represents the corrected display character RH.

As described above, according to this embodiment, the learning model NN[m1] corresponding to the character recognition unit RG[m1] and the learning model NN[m2] corresponding to the character recognition unit RG[m2] are trained to know that the character image GC represents the corrected display character RH based on the correction history information JSH. Therefore, according to this embodiment, the learning model NN[m] can improve the accuracy of recognition of the character represented by the character image GC by the character recognition unit RG[m] compared to a case where learning based on the results of the character correction process is not performed.

4. Conclusion of the embodiment
As described above, in this embodiment, the character recognition device 1 (an example of an "information processing device") recognizes a character (an example of a "first character") represented in the character image G by each of the character recognition process performed by the character recognition unit RG[m1] (an example of a "first character") and the character recognition process performed by the character recognition unit RG[m2] (an example of a "second character"), when the recognized candidate character R[m1][1] (an example of a "first character"), which is the recognition result of the character represented in the character image G by the character recognition process performed by the character recognition unit RG[m1], and the recognized candidate character R[m2][1] (an example of a "second character"), which is the recognition result of the character represented in the character image G by the character recognition process performed by the character recognition unit RG[m2], are the same character, The system includes a recognition result determination unit 114 (an example of a "determining unit") that determines R[m1][1] and the recognition candidate character R[m2][1] as a confirmed recognition character RX (an example of a "confirmed character") corresponding to the recognition result of the character image GC, and, if the recognition candidate character R[m1][1] and the recognition candidate character R[m2][1] are different characters, determines one of the recognition candidate character R[m1][1] and the recognition candidate character R[m2][1] as a likely recognition character RY (an example of a "candidate character") corresponding to the recognition result of the character image GC, and a display information generation unit 115 (an example of a "display control unit") that generates display information JH for displaying the confirmed recognition character RX or the likely recognition character RY on the display unit 31, and the display information generation unit 115 causes the display unit 31 to display the likely recognition character RY in a manner different from the confirmed recognition character RX based on the display information JH.

That is, according to this embodiment, the display information generating unit 115 differentiates the display mode of the likely recognized character RY when the recognition result determining unit 114 determines the recognition result of the character image GC as the likely recognized character RY and the character represented in the character image GC is likely to differ from the display mode of the confirmed recognized character RX when the recognition result determining unit 114 determines the recognition result of the character image GC as the confirmed recognized character RX and the character represented in the character image GC is unlikely to differ from the confirmed recognized character RX. Thus, according to this embodiment, it is possible to reduce the possibility of overlooking that the recognition result of the character represented in the character image GC is not accurate, compared to the case where the likely recognized character RY and the confirmed recognized character RX are displayed in the same manner.
Furthermore, according to this embodiment, the character recognition process is performed by the character recognition unit RG[m1] and the character recognition unit RG[m2]. Therefore, according to this embodiment, the recognition accuracy in the character recognition process can be improved compared to the case where the character recognition process is performed by a single character recognition unit RG.

In addition, in this embodiment, the character recognition process executed by the character recognition unit RG[m1] is characterized in that it is a process of recognizing characters shown in the character image GC using a learning model NN[m1] (an example of a "first learning model") that has learned the relationship between an image containing characters and the characters shown in the image.

Therefore, according to this embodiment, the recognition accuracy in the character recognition process can be improved compared to when character recognition process is performed without using a learning model.

In addition, in this embodiment, the character recognition device 1 is characterized by having an operation reception unit 116 (an example of a "reception unit") that receives corrections to the confirmed recognition character RX or likely recognition character RY displayed on the display unit 31, and a learning execution unit 117 (an example of a "learning unit") that, when the operation reception unit 116 receives a correction to change the confirmed recognition character RX or likely recognition character RY to a character represented in the character image GC, causes the learning model NN[m1] to learn that the character image GC indicates the character represented in the character image GC.

Therefore, according to this embodiment, the recognition accuracy in the character recognition process can be improved compared to when character recognition process is performed without using a learning model.

In addition, in this embodiment, the character recognition device 1 is characterized by including an operation acceptance unit 116 that accepts corrections to the confirmed recognition character RX or likely recognition character RY displayed on the display unit 31, and a learning execution unit 117 that, when the operation acceptance unit 116 accepts a correction to change the confirmed recognition character RX or likely recognition character RY to a character represented in the character image GC and the recognition candidate character R[m1][1] differs from the character represented in the character image GC, causes the learning model NN[m1] to learn that the character image GC indicates the character represented in the character image GC.

Therefore, according to this embodiment, the recognition accuracy in the character recognition process can be improved compared to when character recognition process is performed without using a learning model.

In addition, in this embodiment, the character recognition device 1 includes an operation receiving unit 116 that receives a correction to the confirmed recognition character RX or the likely recognition character RY displayed on the display unit 31, and when the recognition result determination unit 114 determines one of the recognition candidate character R[m1][1] and the recognition candidate character R[m2][1] as the likely recognition character RY and the operation receiving unit 116 receives a correction to the likely recognition character RY, the display information generation unit 115 displays the other of the recognition candidate character R[m1][1] and the recognition candidate character R[m2][1] as the correction candidate character RZ on the display unit 31.

Therefore, according to this embodiment, the burden on the operator who corrects the confirmed recognized character RX or the likely recognized character RY can be reduced.

Furthermore, in this embodiment, the display information generating unit 115 displays the character image GC and the confirmed recognition character RX or the likely recognition character RY on the display unit 31 based on the display information JH.

Therefore, according to this embodiment, it is possible to reduce the possibility of overlooking the fact that the recognition result of the characters represented in the character image GC is inaccurate.

Furthermore, in this embodiment, the content recognition unit 113 selects character recognition unit RG[m1] and character recognition unit RG[m2] from among M character recognition units RG[1] to RG[M] that perform character recognition processing using mutually different algorithms according to the type of character represented in the character image GC.

Therefore, according to this embodiment, the recognition accuracy in the character recognition process can be improved compared to when the character recognition process is performed using a single character recognition unit RG.

B. Modifications
Each of the above embodiments may be modified in various ways. Specific modified embodiments are exemplified below. Two or more embodiments arbitrarily selected from the following examples may be appropriately combined within a range that does not contradict each other. In the modified examples exemplified below, the elements whose actions and functions are equivalent to those of the embodiment will be appropriately omitted by using the symbols referenced in the above description.

<Modification 1>
In the above embodiment, the display information generating unit 115 displays the attention mark MK in the application content display image GH at a position corresponding to the display character RH determined as the likely recognition character RY, but the present invention is not limited to such an embodiment. The display information generating unit 115 may display the display character RH determined as the likely recognition character RY and the display character RH determined as the confirmed recognition character RX in the application content display image GH in a distinguishable manner.
For example, the display information generation unit 115 may display the display character RH determined as the likely recognition character RY and the display character RH determined as the confirmed recognition character RX in the application content display image GH as characters of different sizes, different colors, or different fonts.
Also, for example, the display information generation unit 115 may display the display character RH determined as the likely recognition character RY in the application content display image GH in an emphasized manner compared to the display character RH determined as the confirmed recognition character RX.

<Modification 2>
In the above embodiment, the recognition result determination unit 114 determines whether the recognition candidate character R[m1][1] and the recognition candidate character R[m2][1] are the same character in the display character determination process to determine the confirmed recognition character RX or the likely recognition character RY, and determines the confirmed recognition character RX or the likely recognition character RY as the display character RH, but the present invention is not limited to such an embodiment. For example, the recognition result determination unit 114 may determine the recognition candidate character R[m1][1] as the display character RH when the recognition accuracy D[m1][1] and the recognition accuracy D[m2][1] satisfy D[m1][1]>D[m2][1], and may determine the recognition candidate character R[m2][1] as the display character RH when the recognition accuracy D[m1][1] and the recognition accuracy D[m2][1] satisfy D[m2][1]≧D[m1][1]. In this case, the display information generation unit 115 may determine whether or not to display a warning mark MK at a position based on the display character RH based on the difference value between the recognition accuracy D[m1][1] and the recognition accuracy D[m2][1].

<Modification 3>
In the above-described embodiment and variants 1 and 2, the character recognition unit RG[m] divides the character string image GCL into a plurality of character images GC and performs character recognition processing on each of the divided character images GC, but the present invention is not limited to such an embodiment.
For example, the M character recognition units RG[1] to RG[M] included in the written content recognition unit 113 may include a word recognition unit RGT that can recognize an image representing a character string included in the character string image GCL as a character string (word). Note that, hereinafter, the character recognition unit RG that performs character recognition processing on each character image GC, like the character recognition unit RG described in the above embodiment, will be referred to as an individual character recognition unit RGk to be distinguished from the word recognition unit RGT.

For example, in this modification, the written content recognition unit 113 may include M word recognition units RGT as the M character recognition units RG[1] to RG[M]. That is, in this modification, all of the M character recognition units RG[1] to RG[M] included in the written content recognition unit 113 may be word recognition units RGT.
In this case, when a character string (word) recognized by the character recognition unit RG[m1] matches a character string (word) recognized by the character recognition unit RG[m2], the recognition result determination unit 114 may determine each of the characters constituting the character string (word) as a confirmed recognition character RX. In addition, in this case, when a character string recognized by the character recognition unit RG[m1] does not match a character string recognized by the character recognition unit RG[m2], the recognition result determination unit 114 may determine each of the characters constituting one of the character strings recognized by the character recognition unit RG[m1] and the character string recognized by the character recognition unit RG[m2] as a likely recognition character RY. In this case, the other character string may be selected based on the recognition accuracy D of each of the characters constituting the character string.

Also, for example, in this modified example, the written content recognition unit 113 may include one or more individual character recognition units RGk and one or more word recognition units RGT as the M character recognition units RG[1] to RG[M]. In this case, the written content recognition unit 113 may select the word recognition unit RGT as the character recognition unit RG[m1], and select the individual character recognition unit RGk as the character recognition unit RG[m2]. In this case, when a character constituting a character string (word) recognized by the character recognition unit RG[m1], which is the word recognition unit RGT, matches a character recognized by the character recognition unit RG[m2], which is the individual character recognition unit RGk, the recognition result determination unit 114 may determine the character as the confirmed recognized character RX. In this case, if the characters constituting the character string (word) recognized by the character recognition unit RG[m1], which is the word recognition unit RGT, do not match the characters recognized by the character recognition unit RG[m2], which is the individual character recognition unit RGk, the recognition result determination unit 114 may determine one of the characters constituting the character string recognized by the character recognition unit RG[m1] and the characters recognized by the character recognition unit RG[m2] as the likely recognized character RY. In this case, the other character may be selected based on the recognition accuracy D.

<Modification 4>
In the above-described embodiment and modified examples 1 to 3, the character recognition device 1 and the terminal device 3 are separate entities. However, the character recognition device 1 and the terminal device 3 may be integrated into one device.

Figure 13 is an explanatory diagram for explaining an example of the configuration of a character recognition system Sys-B according to this modified example.

As illustrated in FIG. 13, the character recognition system Sys-B is configured in the same manner as the character recognition system Sys according to the embodiment, except that it includes a character recognition device 1B instead of a character recognition device 1 and does not include a terminal device 3.

Figure 14 is a functional block diagram showing an example of the configuration of character recognition device 1B.

As illustrated in FIG. 14, the character recognition device 1B is configured in the same manner as the character recognition device 1 according to the embodiment, except that it includes a display unit 31 and an operation unit 32.

1...character recognition device, 3...terminal device, 5...imaging device, 11...control unit, 12...storage unit, 13...communication unit, 31...display unit, 32...operation unit, 111...image information acquisition unit, 112...character area identification unit, 113...written content recognition unit, 114...recognition result determination unit, 115...display information generation unit, 116...operation acceptance unit, 117...learning execution unit, RG...character recognition unit, NN...learning model.

Claims (10)

A first character recognition process for recognizing characters included in an image; and
Each of the second character recognition processes for recognizing characters included in the image
a recognition unit that recognizes one character of a character string represented for each item in a character image;
a first character that is a recognition result of the one character by the first character recognition process; and
When a second character that is a recognition result of the first character by the second character recognition process is the same character,
determining the first character and the second character as confirmed characters corresponding to a recognition result of the one character;
When the first character and the second character are different characters,
a determining unit that determines one of the first character and the second character as a candidate character corresponding to a recognition result of the one character;
a display control unit that causes the determined character or the candidate character determined by the determination unit to be displayed on a display unit;
Equipped with
In the recognition unit, the first character recognition process and the second character recognition process are selected based on a similarity between effective character type information in each of a plurality of character recognition processes prepared in advance and character type information in the item,
The display control unit is
displaying the candidate characters on the display unit in a manner different from that of the finalized characters;
23. An information processing apparatus comprising: 2 . The information processing apparatus according to claim 1 , wherein the similarity is calculated by an inner product of a vector having elements each of the effective character type information and a vector having elements each of the character type information in the item. The first character recognition process includes:
Using a first learning model that has learned a relationship between an image including characters and the characters shown in the image,
A process of recognizing the one character represented in the character image.
3. The information processing device according to claim 1, wherein the information processing device further comprises: a reception unit that receives a correction to the candidate character or the final character displayed on the display unit;
When the reception unit receives a correction to change the candidate character or the confirmed character to the one character,
a learning unit that causes the first learning model to learn that the character image represents the one character;
Equipped with
4. The information processing device according to claim 3 . a reception unit that receives a correction to the candidate character or the final character displayed on the display unit;
When the reception unit receives a correction to change the candidate character or the confirmed character to the one character,
When the first character and the one character are different,
a learning unit that causes the first learning model to learn that the character image represents the one character;
Equipped with
4. The information processing device according to claim 3 . a reception unit that receives a correction to the candidate character or the final character displayed on the display unit,
The display control unit is
When the determination unit determines one of the first character and the second character as the candidate character,
When the reception unit receives a correction to the candidate character,
displaying the other of the first character and the second character on the display unit as a correction candidate for the candidate character;
6. The information processing device according to claim 1, wherein the information processing device further comprises: The display control unit is
displaying the character image and the confirmed character or the candidate character on the display unit;
7. The information processing device according to claim 1, wherein the information processing device is a data processing device. The recognition unit is
According to the type of the one character represented in the character image,
Among the multiple character recognition processes that recognize characters contained in images,
selecting the first character recognition process and the second character recognition process;
8. The information processing device according to claim 1, wherein the information processing device is a data processing device. The processor,
A first character recognition process for recognizing characters included in an image; and
Each of the second character recognition processes for recognizing characters included in the image
a recognition unit that recognizes one character of a character string represented for each item in a character image;
a first character that is a recognition result of the one character by the first character recognition process; and
When a second character that is a recognition result of the first character by the second character recognition process is the same character,
determining the first character and the second character as confirmed characters corresponding to a recognition result of the one character;
When the first character and the second character are different characters,
a determining unit that determines one of the first character and the second character as a candidate character corresponding to a recognition result of the one character;
a display control unit that causes the determined character or the candidate character determined by the determination unit to be displayed on a display unit;
and make it work.
In the recognition unit, the first character recognition process and the second character recognition process are selected based on a similarity between effective character type information in each of a plurality of character recognition processes prepared in advance and character type information in the item,
The display control unit is
displaying the candidate characters on the display unit in a manner different from that of the finalized characters;
A program characterized by: Recognizing a character of a character string represented for each item in a character image by a first character recognition process and a second character recognition process;
when a first character that is a recognition result of the one character by the first character recognition process and a second character that is a recognition result of the one character by the second character recognition process are the same character, determining the first character and the second character as confirmed characters corresponding to the recognition result of the one character;
When the first character and the second character are different characters, determining one of the first character and the second character as a candidate character corresponding to a recognition result of the one character;
An information processing method for displaying the determined confirmed character or the candidate character on a display unit,
the first character recognition process and the second character recognition process are selected based on a similarity between effective character type information in each of a plurality of character recognition processes prepared in advance and character type information in the item;
The information processing method according to claim 1, wherein the candidate characters are displayed on the display unit in a manner different from that of the confirmed characters.

Images