JP7618458B2 - Character recognition device, character recognition method, and program - Google Patents

Description

Embodiments of the present invention relate to a character recognition device, a character recognition method, and a program.

Conventionally, character recognition technology has been known that recognizes characters contained in an input image. It is known that such conventional technology can improve recognition accuracy by recognizing characters without explicitly separating the boundaries of each character. However, when recognizing characters without separating their boundaries, there are cases where a character is recognized as being duplicated. Also, depending on the input image, there are cases where a character is recognized without being read through. In other words, conventional technology has a problem in that it is not possible to correctly recognize characters contained in an input image.

JP 2013-097590 A

The problem that the present invention aims to solve is to provide a character recognition device, a character recognition method, and a program that can correctly recognize characters contained in an input image.

A character recognition device according to an embodiment includes a first score calculation unit, a character region estimation unit, a second score calculation unit, and a selection unit. The first score calculation unit is a first score indicating the likelihood of a character string, and calculates the first score for each of a plurality of candidate character strings that are candidates for character strings included in an input image. The character region estimation unit estimates an area of the input image corresponding to each character included in the candidate character string. The second score calculation unit calculates a second score indicating consistency of characters included in the candidate character string based on the estimated area. The selection unit selects one or more character strings from the plurality of candidate character strings based on the calculated first score and second score. The input image includes a plurality of character input areas, and the second score calculation unit calculates the second score based on an area corresponding to each character included in the candidate character string and the character input area.

FIG. 1 is a schematic diagram showing an example of the functional configuration of a character recognition system according to a first embodiment. FIG. 4 is a view showing an example of an input image according to the first embodiment. FIG. 4 is a diagram for explaining a first score according to the first embodiment. FIG. 6 is a diagram for explaining a second score according to the first embodiment. 4 is a flowchart showing a series of operations of the character recognition system according to the first embodiment. FIG. 13 is a diagram showing a modified example of the functional configuration of a second score calculation unit according to the first embodiment. FIG. 4 is a diagram for explaining overlapping reading according to the first embodiment; FIG. 4 is a diagram showing an example of the functional configuration of a skip score calculation unit according to the first embodiment. 5 is a diagram for explaining a character-likeness map according to the first embodiment; FIG. FIG. 13 is a diagram showing a modified example of the functional configuration of a first score calculation unit according to the first embodiment. FIG. 11 is a diagram for explaining a beam search according to the second embodiment. 13A and 13B are diagrams for explaining a match score for each partial input image in the beam search according to the second embodiment. 10 is a flowchart showing a series of operations of a character recognition system according to a second embodiment. FIG. 13 is a view showing an example of an input image according to the third embodiment. FIG. 13 is a diagram showing an example of a second score according to the third embodiment. 13A to 13C are diagrams for explaining an example of the operation of a character region estimation unit according to the fourth embodiment. 13A to 13C are diagrams for explaining a modified example of the operation of the character region estimation unit according to the fourth embodiment. 13A and 13B are diagrams for explaining modified examples of the character-likeness map according to the fifth embodiment; FIG. 13 is a diagram showing an example of the functional configuration of a modified example of a character-likeliness map generation unit according to the fifth embodiment. 13A and 13B are diagrams showing examples of input data and teacher data according to the fifth embodiment. FIG. 1 is a diagram for explaining overlapping reading and skipping according to the conventional technology.

The character recognition device, character recognition method, and program of the embodiment will be described below with reference to the drawings.

[Prior Art]
Problems that arise when using a character recognition method according to the prior art will be described with reference to Fig. 21. Fig. 11 is a diagram for explaining overlapping reading and skipping reading according to the prior art. Column 90 shows an example of overlapping reading, and column 95 shows an example of skipping.

In an example of overlapping reading, a case where an input image 91 is input will be described. When the character string written in the input image 91 is correctly recognized as a character, regions 92 and 93 are estimated as the positions where the characters are written. Since "川" is written in region 92 and "崎" is written in region 93, it can be recognized that "川崎" is written in the input image 91 according to the character recognition method of the conventional technology. On the other hand, when the character string written in the input image 91 is erroneously recognized as a character, regions 92, 93, and 94 are estimated as the positions where the characters are written. Since "川" is written in region 92, "崎" is written in region 93, and "山" is written in region 94, it is erroneously recognized that "川山崎" is written in the input image 91 according to the character recognition method of the conventional technology. Thus, in this example of overlapping reading, after the radical of "崎" is recognized as "山", "崎" is recognized again, there is a problem in that the character "山" is recognized overlappingly.

Next, an example of skipping will be described in which input image 96 is input. When the character string written in input image 96 is correctly recognized as a character, regions 97, 98, and 99 are estimated as the positions where the characters are written. Since "長" is written in region 97, "谷" is written in region 98, and "川" is written in region 99, it can be recognized that "Hasegawa" is written in input image 96 according to the character recognition method of the conventional technology. On the other hand, when the character string written in input image 96 is erroneously recognized as a character, regions 97 and 98 are estimated as the positions where the characters are written. Since "長" is written in region 97 and "川" is written in region 99, it is erroneously recognized that "長川" is written in input image 96 according to the character recognition method of the conventional technology. Thus, in this example of skipping, there was a problem in that the character "谷" written in region 98 was skipped over and recognized.

[First embodiment]
The character recognition device according to this embodiment suppresses the problems of the conventional technology as described above. The character recognition device according to this embodiment performs character string recognition on a character string written in an input image. Character string recognition is a task of recognizing a character string in an image by inputting an image including the character string. In this embodiment, a horizontally written character string read from left to right will be described. Note that this embodiment is not limited to a horizontally written character string read from left to right, and the same applies to a vertically written character string. Note that an image including a character string broadly includes an image of handwritten characters, a photographed signboard, a road sign, and the like. Note that in this embodiment, a case will be described in which the number of characters included in the character string is 0 or more.

FIG. 1 is a schematic diagram showing an example of the functional configuration of a character recognition system according to a first embodiment. The functional configuration of the character recognition system 1 will be described with reference to the diagram. The character recognition system 1 includes an input image acquisition unit 21, a candidate character string calculation unit 22, a character recognition device 10, and an output unit 23.

The input image acquisition unit 21 acquires an input image IM. FIG. 2 is a diagram showing an example of an input image IM according to the first embodiment. A character string S is written in the input image IM. Specifically, the input image IM contains the character string S, which is the handwritten character string "Kawasaki." In this embodiment, an example of the case where the handwritten character string S is written in the input image IM will be described.

Returning to FIG. 1, the candidate character string calculation unit 22 calculates candidate character strings CS that are candidates for the character string S written in the input image IM acquired by the input image acquisition unit 21. The candidate character string calculation unit 22 calculates the candidate character strings CS using a known character recognition technique (e.g., pattern matching, feature detection, etc.). The candidate character string calculation unit 22 outputs multiple character strings S as candidate character strings CS.

Based on the input image IM acquired by the input image acquisition unit 21 and the multiple candidate strings CS calculated by the candidate string calculation unit 22, the character recognition device 10 selects a likely string as a selected string SS from the multiple candidate strings CS calculated by the candidate string calculation unit 22. The character recognition device 10 includes a first score calculation unit (character recognition unit) 110, a character region estimation unit 120, a second score calculation unit (region consistency score calculation unit) 130, and a selection unit 140.

The output unit 23 outputs the selected character string SS selected by the character recognition device 10. For example, the output unit 23 outputs information for displaying the selected character string SS on a display unit (not shown), outputs information for audio output from an audio output unit (not shown), or outputs the selected character string SS wirelessly to an information processing device (not shown).

The first score calculation unit 110 calculates a first score S1 for each of the multiple candidate strings CS calculated by the candidate string calculation unit 22. The candidate strings CS are candidates for the string S included in the input image IM. The first score S1 indicates the likelihood of the string. In other words, the first score calculation unit 110 calculates the first score S1 indicating the likelihood of the string, for each of the multiple candidate strings CS that are candidates for the string S included in the input image IM.

FIG. 3 is a diagram for explaining the first score according to the first embodiment. A specific example of the first score S1 calculated by the first score calculation unit 110 will be described with reference to the figure. The figure shows an example in which the input image IM includes the character string "Kawasaki". In this example, the candidate character string calculation unit 22 calculates "Kawayamazaki" as the candidate character string CS-1, "Kawasaki" as the candidate character string CS-2, and "Kawayamaki" as the candidate character string CS-3. The first score calculation unit 110 calculates the first score S1 for each of the candidate character strings CS-1, CS-2, and CS-3. In this example, the first score S1-1 of the candidate character string CS-1 is "0.5", the first score S1-2 of the candidate character string CS-2 is "0.5", and the first score S1-3 of the candidate character string CS-3 is "0.1".

Returning to FIG. 1, the character area estimation unit 120 estimates a character area CA based on the candidate character string CS and the input image IM. The character area CA is an area corresponding to each character C included in the candidate character string CS. In other words, the character area estimation unit 120 estimates a character area CA, which is an area of the input image IM that corresponds to each character C included in the candidate character string CS.

The second score calculation unit 130 calculates a second score S2 indicating the consistency of the characters included in the candidate character string CS based on the character area CA, which is the area estimated by the character area estimation unit 120. Here, the consistency of the characters included in the candidate character string CS refers to spatial consistency. If there is no spatial consistency, characters may overlap or characters may be skipped.

FIG. 4 is a diagram for explaining the second score according to the first embodiment. With reference to the figure, the estimation of the character area CA performed by the character area estimation unit 120 and a specific example of the second score S2 calculated by the second score calculation unit 130 will be explained. The figure shows an example in which the input image IM includes the character string "Kawasaki". In this example, the character area estimation unit 120 estimates character areas CA, which are areas corresponding to each of the characters C included in the candidate character string CS. For example, the character area estimation unit 120 estimates the areas corresponding to the characters C included in the candidate character string CS-1, "Kawasaki", as character areas CA1-1, CA2-1, and CA3-1. In addition, the character area estimation unit 120 estimates the areas corresponding to the characters C included in the candidate character string CS-2, "Kawasaki", as character areas CA1-2 and CA2-2. Additionally, the character area estimation unit 120 estimates the areas corresponding to the character C in the candidate character string CS-3, "川山奇," as character areas CA1-3, CA2-3, and CA3-3.

The second score calculation unit 130 calculates a second score S2 for each of the multiple candidate strings CS. In this example, the second score S2-1 of candidate string CS-1 is "0.1", the second score S2-2 of candidate string CS-2 is "1.0", and the second score S2-3 of candidate string CS-3 is "1.0".

Returning to FIG. 1, the selection unit 140 selects one or more character strings S from among the multiple candidate character strings CS as the selected character string SS based on the calculated first score S1 and second score S2. For example, the selection unit 140 selects the candidate character string CS that has the largest value as the selected character string SS when the first score S1 and the second score S2 are multiplied together.

Figure 5 is a flowchart showing a series of operations of the character recognition system according to the first embodiment. Below, the series of operations of the character recognition device 10 will be explained according to the flowchart shown in the figure.

(Step S110) The input image acquisition unit 21 acquires an input image IM. The candidate character string calculation unit 22 calculates candidate character strings CS that are candidates for the character string S written in the input image IM. In this flowchart, a case will be described in which the candidate character string calculation unit 22 calculates n candidate character strings CS (n is an integer equal to or greater than 1).

(Step S120) The first score calculation unit 110 calculates a first score S1 for each of the calculated candidate character strings CS. That is, when the candidate character string CS is y _n and the first score S1 is α _n , the first score calculation unit 110 calculates (y _1, α ₁ )...(y _n, α _n ).

(Step S130) The second score calculation unit 130 sets counter i to 1.

(Step S140) Character region estimation unit 120 estimates regions of input image IM corresponding to each of multiple characters C included in candidate character string CS. In this flowchart, a case where candidate character string CS includes m characters (m is an integer equal to or greater than 1) will be described. That is, candidate character string CS _yi includes characters C yi _,1 , ..., _yi,m . In this case, character region estimation unit 120 estimates character regions CA _s1 , ..., _sm corresponding to each character C.

(Step S150) The second score calculation unit 130 calculates a second score S2 for y _i , which is a candidate character string CS. The second score S2 is also referred to as β _n . β _n , which is the second score S2, is calculated based on s ₁ , ..., s _m .

(Step S160) The selection unit 140 calculates γ _i based on α _i which is the first score S1 and β _i which is the second score S2.

(Step S170) If i<n, the second score calculation unit 130 advances the process to step S190. That is, the second score calculation unit 130 repeats the process from step S140 to step S160 until the counter i reaches n, which is the number of candidate character strings CS calculated by the candidate character string calculation unit 22. If i<n is not true, that is, if the counter i reaches n, which is the number of candidate character strings CS calculated by the candidate character string calculation unit 22, the second score calculation unit 130 advances the process to step S180.

(Step S190) The second score calculation unit 130 increments the counter i and proceeds to step S140.

(Step S180) The selection unit 140 selects the candidate character string CS with the maximum γ _n as the selected character string SS. In this flowchart, the selection unit 140 selects the selected character string SS using the maximum point set. Note that the selection unit 140 may select the selected character string SS using the minimum point set depending on the calculation method of α _n and β _n .

[Modification of the first embodiment]
6 is a diagram showing a modified example of the functional configuration of the second score calculation unit according to the first embodiment. With reference to the figure, a second score calculation unit 130A, which is a modified example of the second score calculation unit 130, will be described. The second score calculation unit 130A includes a duplicate reading score calculation unit 131, a skipping score calculation unit 132, and a second score integration unit 133.

The overlapping reading score calculation unit 131 calculates an overlapping reading score S21, which is a score indicating the amount of overlapping of the candidate string CS. The amount of overlapping of the candidate string CS is, specifically, the amount of overlapping of the areas corresponding to the characters C included in the candidate string CS. The second score calculation unit 130A calculates a second score S2 based on the calculated overlapping reading score S21. That is, in this embodiment, the second score calculation unit 130A calculates the second score S2 based on the amount of overlapping of the areas corresponding to the characters C included in the candidate string CS.

FIG. 7 is a diagram for explaining overlapping reading according to the first embodiment. With reference to the same figure, the overlapping amount calculated by the overlapping reading score calculation unit 131 will be explained. In this example, the input image IM contains the character "Kawasaki", and the character area estimation unit 120 estimates character area CA1, character area CA2, and character area CA3 as the character area CA. Here, the overlapping amount is the overlapping area CA-DP, which is an area where the character area CA2 and the character area CA3 overlap each other. Specifically, when the overlapping amount is m(y), the overlapping reading score calculation unit 131 calculates the following formula (1) as the overlapping consistency score P _ovlp .

Here, C _OP is a constant between 0 and 1, and the smaller the C OP is, the smaller the overlap consistency score P _ovlp becomes. The value of C _OP may be determined experimentally.

Returning to FIG. 6, the skipping score calculation unit 132 calculates a skipping score S22, which is a score indicating whether skipping has occurred, based on the character C included in the candidate character string CS and the area estimated by the character area estimation unit 120. The second score calculation unit 130 calculates a second score S2 based on the calculated skipping score S22. That is, the second score calculation unit 130 calculates the second score S2 based on the character C included in the candidate character string CS and the area estimated by the character area estimation unit 120.

FIG. 8 is a diagram showing an example of the functional configuration of the skip score calculation unit according to the first embodiment. An example of the functional configuration of the skip score calculation unit 132 will be described with reference to the diagram. The skip score calculation unit 132 includes a character likelihood map generation unit 1321 and a skip score integration unit 1322. In this embodiment, the skip score calculation unit 132 calculates the second score S2 based on the likelihood that some character C exists in the area of the input image IM.

The character-likelihood map generator 1321 generates a character-likelihood map CLM. The character-likelihood map CLM indicates the likelihood that some character C exists in the image area of the input image IM.

The skip score integration unit 1322 calculates the skip score S22 based on the character area CA estimated by the character area estimation unit 120 and the character likeness map CLM generated by the character likeness map generation unit 1321.

FIG. 9 is a diagram for explaining the character-likeness map according to the first embodiment. With reference to the figure, an overview of the character-likeness map CLM and the processing performed by the skip score integration unit 1322 will be explained.

Figure 9 (A) is a diagram showing character area CA when the character area estimation unit 120 estimates character area CA for a candidate character string CS that has been erroneously recognized as skipping. In the figure, the character "長" in the character string "Hasegawa" contained in the input image IM is estimated as character area CA1, and the character "川" as character area CA2. The character "谷" is not estimated as character area CA. In other words, skipping has occurred.

Figure 9 (B) is a diagram showing an example of a character likelihood map CLM. In the example shown in the figure, there is a high probability that characters exist in areas AR1, AR2, and AR3. In other words, the character likelihood map generating unit 1321 generates the likelihood that some character C exists in an image area of the input image IM as the character likelihood map CLM, and therefore estimates the likelihood that a character C exists for the character string S, including characters that have been skipped.

Figure 9 (C) is a diagram showing an example of the mask MSK. The skip score integrating unit 1322 generates a mask MSK based on the character area CA estimated by the character area estimation unit 120. The mask MSK indicates an area where a character included in the candidate string CS exists or does not exist. The skip score integrating unit 1322 filters the character likelihood map CLM using the generated mask MSK. By filtering the character likelihood map CLM, the skip score integrating unit 1322 estimates areas where a character is highly likely to exist even though it is not included in the candidate string CS.

Figure 9 (D) shows the character-likeness map CLM after filtering by the skip score integrating unit 1322. Area AR2 is an area where there is a high probability that a character exists, but is not included in the candidate character string CS. In other words, the larger area AR2 is, the more likely skipping has occurred.

Here, an image region of the input image IM in which there is a high probability that some character C exists and which is included in the character-likeliness map CLM after filtering is defined as U _j (y). When the image region of the input image IM is divided into a region with a width W and a height H, the skipping score integrating unit 1322 calculates the following formula (2) as the skipping consistency score P _SKIP (y). Note that the image region of the input image IM may be divided into pixel units of the input image IM, or may be divided into units of a predetermined range consisting of a plurality of pixels.

Here, _CSP is a constant equal to or greater than 0, and the larger _CSP is, the smaller the skipping consistency score _PSKIP is. The value of _CSP may be obtained experimentally. If no skipping penalty is imposed, _CSP may be set to 0.

Returning to FIG. 6, the second score integrating unit 133 calculates the second score S2 based on the overlapping reading score S21 calculated by the overlapping reading score calculating unit 131 and the skipping score S22 calculated by the skipping score calculating unit 132. For example, the second score integrating unit 133 calculates the second score S2 as a value obtained by multiplying the overlapping reading score S21 and the skipping score S22.

Fig. 10 is a diagram showing a modified functional configuration of the first score calculation unit according to the first embodiment. With reference to the figure, a first score calculation unit 110A, which is a modified example of the first score calculation unit 110, will be described. The first score calculation unit 110A includes a character recognition score calculation unit 111, a knowledge processing score calculation unit 112, and a first score integration unit 113.

The character recognition score calculation unit 111 calculates a character recognition score S11 for each candidate character string CS. The character recognition score S11 indicates the likelihood of the character string.

The knowledge processing score calculation unit 112 calculates the knowledge processing score S12 for each candidate character string CS. The knowledge processing score calculation unit 112 is used when the candidate character strings CS to be written in the input image IM are limited. A case in which the candidate character strings CS to be written in the input image IM are limited is, for example, when information such as the postal code, address, name, etc. is obtained in advance for the input image IM. If it is known that the input image IM is a postal code, and if the candidate character string CS is not a number, the knowledge processing score S12 is calculated to be lower. Also, if it is known that the input image IM is an address, the knowledge processing score S12 is calculated to be lower for "Kawayamaki" than for "Kawasaki".

The first score integration unit 113 calculates a first score S1 based on the character recognition score S11 calculated by the character recognition score calculation unit 111 and the knowledge processing score S12 calculated by the knowledge processing score calculation unit 112. The selection unit 140 selects a selected character string SS based on the calculated first score S1 and second score S2.

Here, an example will be described in which the selection unit 140 selects the selected character string SS based on the character recognition score S11, the knowledge processing score S12, the overlapping reading score S21, and the skipping score S22. In this case, the selection unit 140 selects the selected character string SS based on the following formula (3).

Specifically, the selection unit 140 selects, as the selected string SS, the candidate string CS that maximizes the product of P _OCR , which is the character recognition score S11, P _LM , which is the knowledge processing score S12, P _ovlp , which is the overlapping reading score S21, and P _skip , which is the skipping score S22.

[Summary of the first embodiment]
According to the above-described embodiment, the character recognition device 10 includes the first score calculation unit 110 to calculate a first score S1 indicating the likelihood of the character string S for each candidate character string CS, the character region estimation unit 120 to estimate a region for each character C included in the character string S, the second score calculation unit 130 to calculate a second score S2 indicating the consistency of the character C, and the selection unit 140 to select a selected character string SS based on the first score S1 and the second score S2. That is, according to the above-described embodiment, the most likely character string is selected in consideration of the consistency of the region in which the character C exists. Therefore, the character recognition device 10 can correctly recognize the character C included in the input image IM.

Furthermore, according to the above-described embodiment, the second score calculation unit 130 calculates the second score S2 based on the overlapping reading score S21. The overlapping reading score S21 is a score according to the amount of overlap between the areas corresponding to the characters C included in the complementary string CS. Therefore, according to this embodiment, the character recognition device 10 can suppress overlapping reading, and therefore can correctly recognize the character C included in the input image IM.

Furthermore, according to the embodiment described above, the second score calculation unit 130 calculates the second score S2 based on the skipping score S22. The skipping score S22 is a score based on the character C included in the candidate string CS and the character area CA estimated by the character area estimation unit 120, and if skipping occurs, the penalty given is large. Therefore, according to the embodiment, the character recognition device 10 can prevent skipping, and can therefore correctly recognize the character C included in the input image IM.

Here, according to the conventional technology, the improvement of duplicate reading and the improvement of skipping are in a trade-off relationship, and improving one makes the other problem more likely to occur. According to the above-mentioned embodiment, the duplicate reading score S21 and the skipping score S22 are calculated separately, and the selected character string SS is selected comprehensively, so that both the duplicate reading and skipping problems can be improved.

In addition, according to the embodiment described above, the second score calculation unit 130 calculates the skipping score S22 by using the character-likelihood map CLM. The character-likelihood map CLM indicates the likelihood that some character C exists in an area of the input image IM. According to this embodiment, skipping can be easily prevented.

Second Embodiment
An example of a character recognition device 10A according to the second embodiment will be described with reference to Figs. 11 to 13. The character recognition device 10A according to the second embodiment performs character recognition of a character string S included in an input image IM using a beam search algorithm. Here, when candidate characters C are calculated for each of a plurality of characters C included in the character string S and a combination of the candidates for each character C is set as a candidate character string CS, there is a problem that the more characters C included in the character string S, the more candidate character strings CS there are. When the number of candidate character strings CS increases, it takes time and resources to select a selection character string SS. Therefore, in this embodiment, the beam search algorithm is used to perform character recognition with less time and resources.

FIG. 11 is a diagram for explaining beam search according to the second embodiment. Beam search according to the second embodiment will be explained with reference to the same figure. In this embodiment, the character recognition device 10A divides the input image IM into a plurality of partial input images IMP and performs character recognition. In the example shown in the same figure, the input image IM is divided into partial input images IMP-1, IMP-2, and IMP-3. The partial input images IMP may be divided according to, for example, a predetermined number of pixels. The predetermined number of pixels may be determined according to the width in which the letter C is likely to be written.

Specifically, first, character recognition device 10A selects one or more selected character strings SS for partial input image IMP-1 of input image IM. Next, character recognition device 10A selects one or more selected character strings SS for partial input image IMP-1 and partial input image IMP-2. At this time, one or more selected character strings SS have already been selected for partial input image IMP-1, so the number of candidate character strings CS for partial input image IMP-1 and partial input image IMP-2 is reduced. Furthermore, character recognition device 10A selects final selected character strings SS for partial input image IMP-1, partial input image IMP-2, and partial input image IMP-3. At this time, one or more selected character strings SS have already been selected for partial input image IMP-1 and partial input image IMP-2, so the number of candidate character strings CS for partial input image IMP-1, partial input image IMP-2, and partial input image IMP-3 is reduced. In this way, in this embodiment, the overall processing time is shortened by narrowing down the candidate character strings S for each partial input image IMP.

Figure 12 is a diagram for explaining the consistency score for each partial input image in the beam search according to the second embodiment. With reference to the figure, the consistency score for each partial input image IMP in the case where the input image IM contains "Kawasaki City" will be explained. Figure 12 (A) shows the correspondence between the candidate character strings CS and the consistency scores when the character recognition device 10A performs character recognition on the partial input image IMP-1, and Figure 12 (B) shows the correspondence between the candidate character strings CS and the consistency scores when the character recognition device 10A performs character recognition on the partial input images IMP-1 and IMP-2. Here, the consistency score is a score used by the selection unit 140 when selecting the selection character string SS, and is, for example, a score obtained by multiplying the first score S1 and the second score S2.

In FIG. 12(A), character recognition device 10A calculates "川" as candidate string CS-11, "川1" as candidate string CS-12, and "ノリ" as candidate string CS-13. The consistency scores of each candidate string CS are "1.0", "0.3", and "1.0", respectively. Character recognition device 10A selects candidate string CS-11, which is the most likely string, and candidate string CS-13 as selected strings SS. In other words, character recognition device 10A excludes candidate string CS-12 from the candidates.

12B, character recognition device 10A calculates "Kawayamazaki" as candidate character string CS-21, "Kawasaki" as candidate character string CS-22, "Noriyamazaki" as candidate character string CS-23, "Norizaki" as candidate character string CS-24, "Kawayamaki" as candidate character string CS-25, and "Noriyamaki" as candidate character string CS-26. The consistency scores of each candidate character string CS are "0.1", "1.0", "0.1", "1.0", "1.0", and "1.0", respectively. Character recognition device 10A selects candidate character strings CS-22, CS-24, CS-25, and CS-26, which are likely character strings, as the selected character string SS. In other words, character recognition device 10A excludes candidate character strings C-21 and CS-23 from the candidates. Here, because candidate character string CS-12, "川1", has been excluded from the candidates when examining partial input image IMP-1, the number of candidate character strings S can be reduced when examining partial input image IMP-1 and partial input image IMP-2.

FIG. 13 is a flowchart showing a series of operations of the character recognition system according to the second embodiment. With reference to the same figure, a series of operations of the character recognition system 1A according to the second embodiment will be described. Step S100 is similar to the operation of the character recognition system according to the first embodiment described in FIG. 5, and therefore description thereof will be omitted.

(Step S210) Character recognition device 10A sets x to δ. δ is a predetermined integer indicating the range of the partial input image IMP. x indicates the range of characters recognized by character recognition device 10A. In this flowchart, character recognition device 10A first calculates candidate character string CS for the range from 0 to x. Here, x, which is the range of characters recognized by character recognition device 10A, corresponds to the partial input image IMP in the example described with reference to FIG. 11.

(Step S220) The character recognition device 10A sets the candidate set Φ to an empty set (empty set).

(Step S230) First score calculation unit 110 included in character recognition device 10A calculates a first score S1 for each of multiple candidate character strings CS included in partial input image IMP. That is, when candidate character string CS is y _n and first score S1 is α _n , first score calculation unit 110 calculates (y _1, α ₁ )...(y _n, α _n ).

(Step S240) The character recognition device 10A selects a selection character string SS in the partial input image IMP. Specifically, the character recognition device 10A selects R pairs of y _i and γ _i with large γ _i , and sets them as a candidate set Φ. R is the number of character strings to be candidates when performing character recognition on the next partial input image IMP. By making R smaller, the processing time can be shortened, but if R is too small, the possibility of erroneous recognition may increase.

(Step S250) Character recognition device 10A determines whether character recognition has been performed on the entire input image IM. Specifically, if x is smaller than W, character recognition device 10A proceeds to step S270. If x is not smaller than W, character recognition device 10A proceeds to step S260.

(Step S270) Character recognition device 10A expands the range in which character recognition is performed. Specifically, character recognition device 10A sets the value obtained by adding δ to x as x, and proceeds to step S230.

(Step S260) The character recognition device 10A outputs the character string y _k with the maximum γ _k as a selected character string SS.

[Summary of the second embodiment]
According to the above-described embodiment, the first score calculation unit 110 provided in the character recognition device 10A calculates a first score S1 for a partial input image IMP that is a part of the input image IM. In other words, the first score calculation unit 110 calculates a first score S1 for a candidate character string CS that is a candidate for a character string S that includes some characters among a plurality of characters C that constitute the character string S included in the input image IM. In addition, the second score calculation unit 130 provided in the character recognition device 10A calculates a second score S2 for a partial input image IMP that is a part of the input image IM. In other words, the second score calculation unit 130 calculates a second score S2 for a candidate character string CS that is a candidate for a character string S that includes some characters among a plurality of characters C that constitute the character string S included in the input image IM. Since the character recognition device 10A calculates a candidate character string CS for each part of the input image IM, the number of candidates for the entire character string included in the input image IM can be reduced. Therefore, according to this embodiment, character recognition can be performed with less time and resources by using a beam search algorithm.

[Third embodiment]
An example of a character recognition device 10B according to the third embodiment will be described with reference to Figures 14 and 15. The third embodiment differs from the other embodiments in that a reference character spacing or a character area to be written is defined in the input image IM. The present embodiment aims to perform character recognition based on the reference character spacing or the character area to be written, thereby more accurately recognizing the character C included in the input image IM.

Figure 14 is a diagram showing an example of an input image IM according to the third embodiment. With reference to the figure, the reference character spacing or the area of the character to be written, which is determined in the input image IM, will be described. Figure 14 (A) is an example of an input image IM in this embodiment. Figure 14 (B) is an example of an input image IM in this embodiment in which characters are written.

The input image IM shown in FIG. 14(A) includes multiple character input areas IAR. Specifically, the input image IM includes character input area IAR1, character input area IAR2, and character input area IAR3. The character input area IAR is provided, for example, to a user who writes a character string S in the input image IM as a reference for writing characters. In other words, the character input area IAR determines the reference character spacing or the area of the characters to be written. In the following explanation, the character input area IAR may be referred to as a "frame."

The character C is written in the input image IM shown in FIG. 14(B). Specifically, the character C-1 is written in the character input area IAR1, the character C-2 is written in the character input area IAR2, and the character C-3 is written in the character input area IAR3. The character C-1 is "川", the character C-2 is "崎", and the character C-3 is "市".

FIG. 15 is a diagram showing an example of the second score according to the third embodiment. The second score S2 calculated by the second score calculation unit 130 will be described with reference to the figure. In the example shown in the figure, the candidate character string calculation unit 22 calculates the candidate character strings CS of "Kawayamaki City" and "Kawasaki City".

When the candidate character string CS is "川山奇市", character input area IAR1 includes character area CA1-1, character input area IAR2 includes character area CA2-1 and character area CA3-1, and character input area IAR3 includes character area CA4-1. In this case, character input area IAR2 includes character area CA2-1 and character area CA3-1, so two character areas CA exist in one frame (character input area IAR). In this case, when multiple character areas CA exist in one frame, second score calculation unit 130 sets the overlap amount between the smaller character area CA and the frame area as m(y) and calculates the match score P _BOX based on the following formula (4).

Here, _CBP is a constant between 0 and 1, and the smaller CBP is, the smaller the match score _PBOX becomes. The value of _CBP may be determined experimentally.

Here, an example will be described in which the selection unit 140 further selects the selected character string SS based on the match score P _BOX . In this case, the selection unit 140 selects the selected character string SS based on the following formula (5).

That is, in this embodiment, the second score calculation unit 130 calculates the second score S2 based on the character areas CA, which are areas corresponding to each of the characters C included in the candidate string CS, and the character input area IAR.

[Summary of the third embodiment]
According to the above-described embodiment, the character recognition device 10B includes the second score calculation unit 130, and calculates the second score S2 based on the character area CA and the character input area IAR. For example, the second score calculation unit 130 calculates a low value for the second score S2 when a single box contains multiple characters. By calculating a low value for the second score S2 when a single box contains multiple characters, the second score calculation unit 130 can prevent erroneous recognition, such as character recognition by dividing a kanji character into its radicals, side characters, etc. Therefore, according to this embodiment, the character C included in the input image IM can be more accurately recognized.

[Fourth embodiment]
A fourth embodiment will be described with reference to Fig. 16 and Fig. 17. In the fourth embodiment, a specific example of character region estimation unit 120 will be described. An example of character region estimation unit 120 will be described with reference to Fig. 16, and a modified example of character region estimation unit 120 will be described with reference to Fig. 17.

Figure 16 is a diagram for explaining an example of the operation of the character region estimation unit according to the fourth embodiment. An example of the character region estimation unit 120 will be described with reference to the figure. First, input data DI is input to the neural network NN1. In this example, an example in which the input data DI is an image of a character string will be described. Specifically, an example in which the input data DI is an image of a character string "Kawasaki City" handwritten horizontally from left to right will be described.

The neural network NN1 calculates a series of feature values F for the input character string. When the input data DI is a character string image handwritten horizontally from left to right, the neural network NN1 recognizes the series of feature values F from left to right for the width of the judgment range. In this example, the neural network NN1 calculates feature values F1 to F6. Here, the neural network NN1 calculates a number of feature values F according to the length of the line of the input data DI.

The neural network NN2 calculates a probability distribution P for each feature F calculated by the neural network NN1. In this example, the neural network NN1 calculates features F1 to F6, so the neural network NN2 calculates probability distributions P1 corresponding to feature F1 to P6 corresponding to feature F6.

CTC (Connectionist Temporal Classification) 80 integrates each of the calculated probability distributions, calculates a probability distribution P of the character string corresponding to the input data DI, and outputs the character string recognized from the calculated probability distribution P as output data DO.

The estimation unit 85 acquires the feature F calculated by the neural network NN1. The estimation unit 85 estimates the range in which elements to which a specified label should be assigned may exist from the acquired feature F using the neural network NN3.

The estimation unit 85 associates each label of the output data DO recognized by the CTC 80 with each feature F. When one label in the label string of the output data DO is associated with multiple feature F, the estimation unit 85 integrates and outputs the range estimated from the multiple feature F associated with the one label. The output result output by the estimation unit 85 specifies the range of each label in the input data DI. In the example shown in the same figure, range A1 specifies the range of "river", range A2 specifies the range of "cape", and range A3 specifies the range of "city".

Figure 17 is a diagram for explaining a modified example of the operation of the character region estimation unit according to the fourth embodiment. With reference to the figure, a modified example of the operation of the character region estimation unit 120 will be explained. In the modified example of the operation of the character region estimation unit 120, object detection is applied to estimate the character region.

First, input data DI is input to neural network NN4. In this example, an example will be described in which the input data DI is an image of a character string. Specifically, an example will be described in which the input data DI is an image of a character string "Kawasaki" handwritten horizontally from left to right. The neural network NN4 is a detection DNN (Deep Neural Network). The neural network NN4 receives an image as input, and outputs multiple candidate rectangles R and the scores of the characters corresponding to each candidate rectangle R.

Specifically, the neural network NN4 outputs candidate rectangle R1, candidate rectangle R2, candidate rectangle R3, candidate rectangle R4, candidate rectangle R5, candidate rectangle R6, and character scores corresponding to each candidate rectangle R. More specifically, the character "川" corresponding to candidate rectangle R1 has a score of "0.8" and the character "周" has a score of "0.1", the character "り" corresponding to candidate rectangle R2 has a score of "0.5" and the character "い" has a score of "0.2", the character "１" corresponding to candidate rectangle R3 has a score of "0.3" and the character "ノ" has a score of "0.1", the character "崎" corresponding to candidate rectangle R4 has a score of "0.8" and the character "崎" has a score of "0.1", the character "山" corresponding to candidate rectangle R5 has a score of "0.5" and the character "凸" has a score of "0.1", and the character "奇" corresponding to candidate rectangle R6 has a score of "0.7" and the character "嵜" has a score of "0.1".

[Summary of the Fourth Embodiment]
According to the above-described embodiment, the character region estimation unit 120 includes the estimation unit 85, and estimates a region where the character C may exist based on the feature F acquired from the input data DI, associates the character C with at least one of the feature F, and identifies a region corresponding to each character C by integrating multiple ranges associated with one label. By using this embodiment, an efficient search can be performed using a beam search algorithm. Furthermore, the estimation of the character region according to this embodiment can be easily implemented.

Furthermore, according to the above-described embodiment, the character region estimation unit 120 receives an image as input and outputs multiple candidate rectangles R and character scores corresponding to each candidate rectangle R. By using this embodiment, it is possible to estimate a character region with fewer resources.

[Fifth embodiment]
The fifth embodiment will be described with reference to Fig. 18 to Fig. 20. In the fifth embodiment, a modified example of the character-likeliness map CLM generated by the character-likeliness map generating unit 1321 will be described. Fig. 18 is a diagram for explaining a modified example of the character-likeliness map according to the fifth embodiment. Fig. 18(A) shows the character-likeliness map CLM described in the first embodiment.

Fig. 18(B) shows a character-likeness map CLM1, which is a first modified example of the character-likeness map CLM according to the fifth embodiment. The character-likeness map CLM1 differs from the character-likeness map CLM in that the character-likeness is expressed in gradations for each region made up of multiple pixels. In this way, the character-likeness map CLM1 may be calculated for each predetermined range of the input image IM.

Fig. 18C shows a character-likeness map CLM2, which is a second modified example of the character-likeness map CLM according to the fifth embodiment. The character-likeness map CLM2 includes a correspondence between the x-coordinate of the input image IM and the number of black pixels at each x-coordinate. That is, the character-likeness map CLM2 may be a luminance histogram. In this embodiment, the input image IM is a character written horizontally, so the X-coordinate, which is the direction in which the character is written, is used. If the input image IM is a character written vertically, the Y-coordinate may be used. The character-likeness map CLM2 uses information on the number of black pixels at each X-coordinate, so the character-likeness map CLM2 can be easily created.

Fig. 18(D) shows a character-likeliness map CLM3, which is a third modified example of the character-likeliness map CLM according to the fifth embodiment. The character-likeliness map CLM3 is obtained by normalizing the character-likeliness map CLM2 so that it takes values between 0 and 1.

Alternatively, the character-likeness map CLM may be obtained by dividing the input image IM into small grid-like regions, and may be based on the total number of black pixels in each small region.

Fig. 19 is a diagram showing an example of the functional configuration of a modified example of the character-likeliness map generating unit according to the fifth embodiment. With reference to the figure, the character-likeliness map generating unit 1321A, which is a modified example of the character-likeliness map generating unit 1321, will be described. The character-likeliness map generating unit 1321A differs from the character-likeliness map generating unit 1321 in that it includes a character-likeliness calculation neural network DNN.

The character-likeness calculation neural network DNN is a neural network that has been trained in advance to predict character-likeness. FIG. 20 is a diagram showing an example of input data DI and teacher data DT according to the fifth embodiment. An example of input data DI and an example of teacher data DT will be described with reference to the diagram.

In the example shown in FIG. 20, input data DI1 corresponds to teacher data DT1, and input data DI2 corresponds to teacher data DT2. Input data DI1 includes characters C-11 and C-12, and teacher data DT1 includes area AR11 corresponding to character C-11 and area AR12 corresponding to character C-12. Input data DI2 includes characters C-21, C-22, and C-23, and teacher data DT2 includes area AR21 corresponding to character C-21, area AR22 corresponding to character C-22, and area AR23 corresponding to character C-23.

[Summary of the Fifth Embodiment]
According to the embodiment described above, the character-likeliness map generating unit 1321 can easily generate the character-likeliness map CLM by using the character-likeliness map CLM1, the character-likeliness map CLM2, or the character-likeliness map CLM3.

Furthermore, according to the above-described embodiment, the character-likeness map generating unit 1321 is equipped with a character-likeness calculation neural network DNN, and is therefore able to generate a character-likeness map CLM by machine learning. According to the above-described embodiment, the use of machine learning makes it possible to be resistant to noise and to prevent erroneous recognition. Furthermore, according to the above-described embodiment, the use of machine learning makes it possible to correctly recognize input images IM with different backgrounds.

As explained above, multiple modified examples are described in the embodiment. Here, multiple embodiments and multiple modified examples may be combined to the extent that they are possible to combine.

The functions of the information processing device in the above-mentioned embodiment may be realized by a computer. In that case, a program for realizing the functions may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed to realize the functions. The term "computer system" here includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, CD-ROMs, DVD-ROMs, and USB memories, and storage devices such as hard disks built into computer systems. The term "computer-readable recording medium" may also include devices that dynamically hold a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, and devices that hold a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in such a case. The above-mentioned program may be for realizing part of the above-mentioned functions, or may be capable of realizing the above-mentioned functions in combination with a program already recorded in the computer system.

According to at least one of the embodiments described above, by having a first score calculation unit, a character area estimation unit, a second score calculation unit, and a selection unit, it is possible to correctly recognize characters contained in an input image.

Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. These embodiments and their modifications are within the scope of the invention and its equivalents as set forth in the claims, as well as the scope and gist of the invention.

1...character recognition system, 10...character recognition device, 21...input image acquisition unit, 22...candidate character string calculation unit, 23...output unit, 24...input rule information storage unit, 110...first score calculation unit, 120...character area estimation unit, 130...second score calculation unit, 140...selection unit, 111...character recognition score calculation unit, 112...knowledge processing score calculation unit, 113...first score integration unit, 131...overlapping reading score calculation unit, 132...skipping score calculation unit, 133...second score integration unit , 1321...character-likeness map generation unit, 1322...skipping score integration unit, IM...input image, S...character string, C...character, CS...candidate character string, CA...character area, S1...first score, S2...second score, S11...character recognition score, S12...knowledge processing score, S21...overlapping reading score, S22...skipping score, SS...selected character string, IR...input rule, IAR...character input area, CLM...character-likeness map, MSK...mask, IMP...partial input image

Claims (7)

a first score calculation unit that calculates a first score indicating a likelihood of a character string, the first score being a candidate for a character string included in an input image;
a character region estimation unit that estimates regions of the input image corresponding to each of the characters included in the candidate character string;
a second score calculation unit that calculates a second score indicating consistency of characters included in the candidate character string based on the region estimated by the character region estimation unit;
a selection unit that selects one or more character strings from among the plurality of candidate character strings based on the calculated first score and the calculated second score ,
the input image includes a plurality of character input areas;
The second score calculation unit calculates the second score based on an area corresponding to each character included in the candidate character string and the character input area.
Character recognition device. The character recognition device according to claim 1 , wherein the first score calculation unit calculates the first score of the candidate character string included in a partial input image that is a part of the input image. The second score calculation unit,
The character recognition device according to claim 1 or 2, wherein the second score is calculated based on an amount of overlap between regions corresponding to the characters included in the candidate character string. The second score calculation unit,
The character recognition device according to claim 1 , wherein the second score is calculated based on characters included in the candidate character string and a region estimated by the character region estimation unit. The second score calculation unit,
The character recognition device according to claim 4 , wherein the second score is calculated based on the likelihood that a character exists in the region of the input image. a first score calculation step in which the computer calculates a first score indicating a likelihood of a character string, the first score being a candidate for a character string included in the input image for each of a plurality of candidate character strings;
a character region estimation step of estimating , by a computer, regions of the input image corresponding to each of the characters included in the candidate character string;
a second score calculation step of calculating a second score indicating consistency of characters included in the candidate character string based on the estimated region by the computer ;
a selection step of selecting one or more character strings from among the plurality of candidate character strings based on the calculated first score and the calculated second score ,
the input image includes a plurality of character input areas;
The second score calculation step calculates the second score based on an area corresponding to each character included in the candidate character string and the character input area.
Character recognition methods. On the computer,
a first score calculation step of calculating a first score indicating a likelihood of a character string, the first score being a candidate for a character string included in the input image for each of a plurality of candidate character strings;
a character region estimation step of estimating regions of the input image corresponding to each of the characters included in the candidate character string;
a second score calculation step of calculating a second score indicating consistency of characters included in the candidate character string based on the estimated region;
a selection step of selecting one or more character strings from among the plurality of candidate character strings based on the calculated first score and the calculated second score;
the input image includes a plurality of character input areas;
The second score calculation step calculates the second score based on an area corresponding to each character included in the candidate character string and the character input area.
program.

Images