JP7599861B2 - Information processing device, information processing method, and program - Google Patents

Description

The present invention relates to a technology for recognizing characters contained in document images.

Conventionally, there is a technology for reading and recognizing characters in a document image obtained by scanning a document. This technology is generally called OCR (Optical Character Recognition). OCR processing is usually composed of image preprocessing for modifying a document image into an image suitable for OCR, character string detection processing for identifying an area in which characters are written (character string area) from the document image, and character recognition processing for identifying each character included in the detected character string area. Regarding such OCR processing, a technology is known in which rules and classification items for classifying acquired character string areas are determined in advance, and optimal character recognition is performed for each character string area. For example, Patent Document 1 discloses a technology for classifying character string areas obtained from a document image based on the positional relationship between predetermined character string areas (layout information) and characteristics of classification items, and for improving recognition accuracy by matching the character string areas with a word dictionary corresponding to each classification item.

Japanese Patent Application Publication No. 5-20505

The technology of Patent Document 1 requires that rules for classification, such as layout information of character string areas and characteristics of classification items, be determined in advance, but for documents that do not fit the rules, character string areas cannot be properly classified, and as a result, characters cannot be recognized correctly. For example, in the name section of a business card, which may be vertical or horizontal, or may have a combination of vertical and horizontal writing, the number of characters is small and the spacing between characters is wide, making it difficult to obtain an appropriate character string area and perform character recognition with high accuracy.

The technology disclosed herein has been developed in consideration of the above problems, and aims to improve character recognition accuracy in areas of a document image where character string regions are likely to fail to be acquired.

Means for solving the invention

The information processing device according to the present disclosure is characterized by comprising: a first processing means for performing a first character recognition process on a character string area obtained by performing a character string detection process on a document image obtained by reading a document; a specifying means for specifying name characters constituting a person's name in the document based on the result of the first character recognition process; an acquiring means for acquiring a character string area corresponding to a character line representing the name by integrating an area corresponding to the name characters specified by the specifying means and an area corresponding to one or more characters belonging to the same line as the name characters; a second processing means for performing a second character recognition process on the character string area corresponding to the character line representing the name; and a determining means for determining a character code for the characters constituting the character line representing the name based on the result of the first character recognition process and the result of the second character recognition process.

The technology disclosed herein can improve character recognition accuracy even in cases where it is difficult to properly obtain a character string area, such as the characters of a name on a business card.

FIG. 1 is a diagram showing a hardware configuration of an information processing system. 4 is a flowchart showing the flow of processing in the entire system. 11 is a flowchart showing details of a recognition result improvement process. 11 is a flowchart showing details of a name line acquisition process. 11 is a flowchart showing details of a name character identification process. 11 is a flowchart showing details of a character code replacement process. 11 is a flowchart showing details of a name row reevaluation process. 11 is a flowchart showing details of a blob integration process. 11 is a flowchart showing details of a blob attribute determination process. 10 is a flowchart showing details of a recognition result comparison process. 5A is a diagram showing an example of a business card image, and FIG. 5B is a diagram showing an example of a character string region before a recognition result improvement process is performed. FIG. 11 is a diagram showing an example of a recognition result before a recognition result improvement process is performed. 11 is a diagram showing an example of a character string area corresponding to a name line acquired from a business card image. FIG. 14 is a diagram showing an example of a recognition result for the character string region in FIG. 13 . FIG. 11 is a diagram showing an example of a result of a name character identification process. FIG. 1A is a diagram showing a state in which the inter-character distance is positive in the case of horizontal writing, and FIG. 1B is a diagram showing a state in which the inter-character distance is positive in the case of vertical writing. 11A and 11B are diagrams for explaining overlap determination of rectangular areas. 11A and 11B are diagrams for explaining overlap determination between original blobs. FIG. 11 is a diagram showing a character string region after a recognition result improvement process is performed. FIG. 11 is a diagram showing an example of a recognition result after a recognition result improvement process is performed.

The following describes an embodiment of the present invention with reference to the drawings. Note that the embodiment does not limit the present invention, and all of the configurations described in the embodiment are not necessarily essential means for solving the problems of the present invention.

[Embodiment 1]
[Hardware configuration]
1 is a diagram showing the hardware configuration of an information processing system according to this embodiment. The information processing system includes a copying machine 100 and an information processing device 110.

The copier 100 has a scanner 101 and a copier-side communication unit 102. The scanner 101 scans a document and generates a document image. The copier-side communication unit 102 communicates with external devices, including an information processing device 110, via a network.

The information processing device 110 has a system control unit 111, a ROM 112, a RAM 113, a HDD 114, a display unit 115, an input unit 116, and an information processing device side communication unit 117. The system control unit 111 is composed of a calculation device such as a CPU, and reads out a control program stored in the ROM 112 to execute various processes. The RAM 113 is used as a temporary storage area such as the main memory and work area of the system control unit 111. The HDD 114 stores various data and various programs. The functions and processes of the information processing device 110 described later are realized by the system control unit 111 reading out a program stored in the ROM 112 or the HDD 114 and executing this program. The information processing device side communication unit 117 performs communication processing with external devices including the copier 100 via a network. The display unit 115 displays various information. The input unit 116 has a keyboard and a mouse, and accepts various operations by the user. The display unit 115 and the input unit 116 may be integrated into one unit, such as a touch panel. The display unit 115 may project an image using a projector, and the input unit 116 may use a camera to recognize the position of a fingertip relative to the projected image.

In this embodiment, the scanner 101 of the copier 100 scans a paper document such as a business card to generate a document image. The document image is transmitted to the information processing device 110 by the copier side communication unit 102. In the information processing device 110, the information processing device side communication unit 117 receives the document image and stores it in a storage device such as the HDD 114.

The hardware configuration in FIG. 1 is an example of a configuration for realizing this embodiment, and for example, some functions of the display unit 115 and the input unit 116 may be included in the copier 100. Also, the copier 100 and the information processing device 110 may be integrated into one configuration.

[Overall processing flow]
Next, the operation flow of software for implementing OCR processing on document images in the information processing system according to this embodiment will be described with reference to Fig. 2. The series of processes shown in the flowchart in Fig. 2 are implemented by the system control unit 111 reading out a predetermined program from the ROM 112 or the like and executing it. In the following description, the symbol "S" means a step.

First, in S201, document image data is acquired from HDD 114. Next, in S202, a process is executed to determine the type of document for the acquired document image. In this embodiment, it is determined whether the document type is a business card or non-business card. This determination is made based on the scan resolution and document image size. For example, when the scan resolution is 300 DPI, if the long side of the document image obtained by scanning is within the range of 1040 pixels ± 5% and the short side is within the range of 615 pixels ± 5%, it is determined to be a business card, and if it is outside the range, it is determined to be a non-business card. Note that the method of determining the document type is not limited to the method based on the image size. For example, information specifying the document type may be received from input unit 116, or a feature amount may be calculated from the document image and the determination may be made using a discrimination model that has been learned in advance.

Based on the result of the determination in S202, the process flow branches in S203. If it is determined to be a non-business card, the process proceeds to S204, and if it is determined to be a business card, the process proceeds to S205.

In S204, a general-purpose character recognition process is performed that is not dependent on the document type. When the character recognition process is complete, this process ends.

In S205, a process for correcting the skew of the document image is executed. The document image generated by the scanner 101 is skewed depending on how the document is placed on the platen. Therefore, a skew correction process is performed to obtain a document image without skew. The skew correction process may be performed using a known method disclosed in, for example, Japanese Patent No. 4114959.

Next, in S206, a process is performed to determine the character direction in the document image. Here, the character direction is defined as the direction of the characters in the document image when the direction in which the characters are upright is set to 0 degrees. In the document image that has undergone the tilt correction process in S205 above, the original may be rotated in 90 degree increments. Therefore, the character direction of the document image is obtained by applying a known method disclosed in Patent No. 3727971, etc.

Next, in S207, a process is performed to rotate the document image based on the character direction determined in S206. This results in a document image with upright characters.

Next, in S208, a process is performed to extract character pixels from the document image. In this embodiment, character pixels are extracted by binarization, which involves threshold processing of the luminance value of each pixel. For binarization, a known method such as Otsu's binarization may be applied. Since characters in a document are usually printed in a darker color than the background, pixels with a luminance value smaller than the threshold value are considered to be character pixels. Note that the method of extracting character pixels is not limited to the method using threshold processing. For example, it may be realized by a method in which image features are extracted from the surrounding area centered on an arbitrary pixel of interest, and whether or not the pixel of interest is a character pixel is inferred based on the features and a discrimination model that has been previously trained.

Next, in S209, a process is performed to determine the line direction based on the character pixels extracted in S208. The line direction here refers to the direction in which characters are lined up in the document image, and in this embodiment, is either the horizontal or vertical direction. One method of determination is, for example, to generate a projection histogram for the entire document image in both the vertical and horizontal directions, and determine the direction with the smallest variance as the line direction. With this line direction determination process, for example, the line direction of the business card image shown in Figure 11 (a) is determined to be vertical.

In the next step S210, the process flow branches based on the result of the determination in S209. If the result of the determination is that the row direction is horizontal, the process transitions to S211, and if the row direction is vertical, the process transitions to S212.

In S211, when the line direction is horizontal, a character string detection process suitable for identifying horizontally written character lines is executed. When the characters in a document image are arranged horizontally, the gap between characters (the distance between black pixel clusters = the number of white pixels) is usually narrower in the horizontal direction (left-right direction) than in the vertical direction (up-down direction). Therefore, character string regions are detected from the document image by performing region analysis with processing parameters adjusted to be suitable for identifying a group of horizontally written characters. A specific method for detecting character strings is a publicly known method disclosed in JP-A-7-200733, etc.

In S212, when the line direction is vertical, a rotation process is performed to rotate the document image by 90 degrees prior to the character string detection process. For example, the business card image in FIG. 11(a), which has a vertical line direction, is rotated by 90 degrees to generate a business card image in which the top and bottom of the business card are aligned horizontally. As a result, characters that are aligned vertically on the original (business card) are aligned horizontally in the rotated image.

In the next step S213, similar to S211 described above, a character string detection process suitable for identifying horizontally written lines is executed. When the processing target is a vertically long business card with vertical writing, the image is rotated 90 degrees as in this embodiment before the character string detection process is executed, so that character string areas in the character portion where the characters are widely spaced and discretely arranged, such as a name, can be more appropriately detected.

Next, in S214, the character string area detected in S213 is rotated by -90 degrees, that is, the same angle as the rotation in S212 is rotated in the opposite direction. This causes the characters in the character string area to return to an upright state. At this time, the partial image corresponding to the detected character string area may be rotated by -90 degrees, or the coordinate information of the detected character string area may be rotated by -90 degrees and the coordinate information after the rotation may be applied to the document image obtained in S207 in the orientation in which the characters are formed.

In S215, character recognition processing is performed on each character string area detected in S211 or acquired in S214. As an example of character recognition processing, there is a method of inferring the character code of a character included in a partial image corresponding to a character string area using a trained model. The trained model is an identification model trained to input a character image and output its character code. Furthermore, in the character recognition processing of this embodiment, a process of acquiring the reliability of the character as the recognition result (hereinafter referred to as "recognized character") is also performed. The reliability is a numerical value that indicates the likelihood of the recognized character. The higher the numerical value, the higher the accuracy of the recognized character. When the image part corresponding to the recognized character is in good condition (low noise, etc.) and there is no overlap with other characters, a high numerical reliability tends to be obtained. Conversely, when the image part corresponding to the recognized character is in poor condition (high noise, etc.) and there is overlap with other characters, the numerical reliability tends to be low. The numerical value is also affected by the font size. In this way, in the character recognition process of this embodiment, the character code of the recognized character, area information (size information) indicating the circumscribing rectangle of the recognized character unit, and the reliability of the recognized character are obtained.

Next, in S216, a process (recognition result improvement process) is performed to improve the character recognition result obtained in S215. As described above, the accuracy of character recognition decreases if the results of the character string detection process in S211 or S213 are inappropriate. FIG. 11(b) is a diagram showing an example in which the character string area of the name portion could not be properly extracted from the business card image in FIG. 11(a) described above, and FIG. 12 shows the result of character recognition process for a partial image (character string rectangular image) corresponding to that character string area. As a result of the detected character string area being inappropriate, it can be seen that the kanji character "新", which is part of the family name, was not correctly recognized. Therefore, in this embodiment, the character string area is reacquired for the part where it is highly likely that an appropriate character string area was not detected in the character string detection process, and character recognition process is performed again, thereby improving the recognition result. Details of the recognition result improvement process will be described later. When the recognition result improvement process is completed, this process ends.

The above is the operational flow of the software that realizes OCR processing of document images.

<Outline of recognition result improvement process>
Next, the recognition result improvement process of S216 will be outlined with reference to the flowchart of Fig. 3. This process is executed for each character string region extracted from the document image.

In S301, it is determined whether the character recognition process executed in S216 was character recognition process in Japanese. The target language for the character recognition process is set in advance, and the determination is made by referring to the setting information. If it was character recognition process in Japanese, the process transitions to S302, and if it was character recognition process in another language, the process transitions to S303.

In S302, a process is executed to obtain, based on the recognition result, a character string area corresponding to a specific character line to be subjected to character recognition processing again from the document image. Here, the specific character line is a character line constituting the name of a person on a business card (hereinafter, referred to as a "name line"). The name line obtaining process will be described in detail later.

Next, in S303, the process flow branches based on the result of the name line acquisition process. If the character string area corresponding to the name line can be acquired, the process proceeds to S304, and if not, the process exits. In S304, the same character recognition process as that performed in S215 is executed again on the character string area of the name line acquired in S302. However, it does not necessarily have to be exactly the same character recognition process, and for example, a different character recognition process may be performed using a trained model that limits the character types to be recognized to those that can be used in personal names, such as kanji, katakana, and hiragana.

Next, in S305, a process (character code replacement process) is executed to replace the character code of the recognized character obtained in the second character recognition process in S304 with the character code obtained in the first character recognition process under certain conditions. Details of the character code replacement process will be described later. When the character code replacement process is completed, this process ends.

This concludes the overview of the recognition result improvement process.

<Name line acquisition process>
Next, the name line acquisition process in S302 will be described with reference to the flowchart in FIG.

In S401, a process (name character identification process) is executed to identify characters that appear to be a person's name (hereinafter referred to as "name characters") from the recognition results obtained in S215. Each recognized character identified as a name character by this name character identification process is treated as a unit called a "blob" hereafter. This "blob" includes the coordinate information (rectangle information) of the circumscribing rectangle that surrounds the corresponding character or character string area and its character code. The name character identification process will be described in detail later. Table 1 below summarizes the various types of blobs that will appear in the following explanation.

In S402, candidates for name lines (character strings consisting of multiple name characters) are generated by combining name characters obtained in the name character identification process that are aligned in lines. Note that whether or not line alignment is possible is determined by the blob of each name character, and when they are combined, a blob corresponding to the name line candidate is generated.

In S403, the characters existing in the range before and after the name line candidate obtained in S402 are combined as part of the name line candidate and incorporated into the name line candidate.

In S404, pixel blocks existing in the range before and after the name line candidate obtained in S403 are extracted from the document image and further added to the name line candidates.

In S405, unnecessary name line candidates are deleted from all the name line candidates obtained up to this point, and the remaining name line candidates are determined to be name lines. Whether or not a name line candidate is unnecessary is determined using rectangular information within the blob and information on the reliability of the recognition result. If a judgment condition that combines this information is not met, it is determined to be an unnecessary name line candidate. One example of the condition is when there are two or more blobs of name characters, or when there are no name character blobs but the name line consists of multiple lines, etc.

The above is the content of the name line acquisition process. FIG. 13 is a diagram showing an example of a character string area corresponding to the name line acquired from the business card image of FIG. 11(a) by this process. The area including the title and name in the business card image of FIG. 11(a) (solid line frame in the figure) is acquired as the character string area corresponding to the name line. FIG. 14 shows the recognition result when character recognition processing is performed on a partial image (character string rectangular image) corresponding to the character string area shown in FIG. 13. In this case, it can be seen that the name portion (the four kanji characters "Arata", "No", "Yasu", and "Suke") is correctly recognized, but the title portion spanning two lines is not correctly recognized.

<Name character identification processing>
Next, the name character specification process of S401 will be described with reference to the flowchart of Fig. 5. The series of processes shown below are executed for each recognized character obtained in the character recognition process of S215. The results of the character recognition process referred to in this flow include information on the number of first candidate characters for each recognized character, in addition to information on the character code, size, and reliability of the recognized characters.

In S501, it is determined whether the size in the line height direction of the recognized character of interest is less than a predetermined threshold. In this case, the threshold may be determined by tallying up the frequency of occurrence of each character size that can be used as a valid recognition result from the character recognition results for the entire document image, and setting the most frequent size as the threshold. In this case, the conditions for a valid recognition result are as follows:
- The character width and aspect ratio are equal to or greater than the specified value. (Barcode-sized characters are not considered characters.)
The reliability of the recognition result is equal to or higher than a predetermined value.
- The character size in the line height direction (width for vertical writing, height for horizontal writing) is equal to or greater than a specified value.

Note that the predetermined values for each of the above conditions are values determined empirically. If the result of the determination is that the size of the recognized character in the line height direction of the target character is less than the threshold, the process transitions to S508, and if it is equal to or greater than the threshold, the process transitions to S502.

In S502, it is determined whether the number of characters obtained as the first candidate in the recognition result for the recognized character of interest is multiple (i.e., other than one). Here, "first candidate" means the one with the highest accuracy among the multiple recognition candidates for that character. "Multiple characters obtained" means that two or more recognition candidates are output for one character, such as when the kanji character "誠" is recognized as being separated into "言" and "成". If the number of characters obtained as the first candidate in the recognition result is other than one, the process transitions to S508, and if the number of characters is one, the process transitions to S503.

In S503, it is determined whether the first candidate character code for the recognized character of interest is a character code other than kanji or kana (specifically, a character code for alphanumeric characters, symbols, etc.). If the first candidate character code is a character code other than kanji or kana, the process transitions to S508, and if it is a kanji or kana character code, the process transitions to S504.

In S504, it is determined whether the character code of the recognized character of interest is a character code to be excluded. Character codes to be excluded include, for example, character codes for lowercase letters such as "ぁ" and "ｱ" and are specified in advance. If the character code is to be excluded, the process proceeds to S508, and if the character code is not to be excluded, the process proceeds to S505.

In S505, it is determined whether the reliability of the recognized character in question is lower than a threshold value. The threshold value here may be set based on experience, such that a certain value or higher indicates that the character is generally reliable. If the reliability is lower than the threshold value, the process transitions to S508, and if it is equal to or higher than the threshold, the process transitions to S506.

In S506, it is determined whether the aspect ratio of the circumscribing rectangle of the recognized character of interest is approximately 1. Since the recognition result may contain errors, if the aspect ratio is within 1 ± a threshold value (e.g., approximately 0 to 0.6), it is determined to be "approximately 1." If the aspect ratio is other than approximately 1, the process transitions to S508, and if it is approximately 1, the process transitions to S507.

If none of the conditions in S501 to S506 above are met, then in S507 the recognized characters of interest are determined to be name characters. If any of the conditions in S501 to S506 above are met, then in S508 the recognized characters of interest are determined to not be name characters. Once the determination is complete, this process ends. Figure 15 is a diagram showing an example of the results of this process performed on the character recognition results for the business card image in Figure 11(a), and shows that the characters "株", "式", "会", "社", "野", "幸", and "輔" have been determined to be name characters.

<Character code replacement process>
Next, the character code replacement process in S305 will be described with reference to the flowchart in FIG.

In S601, it is determined whether or not there is a valid recognized character in the recognition result of the second character recognition process (S304). Specifically, if the recognized character obtained by the second character recognition process is a character other than a space, the recognized character is determined to be a valid recognized character. If the recognized character is a character other than a space, it may be a character that can be used to replace the recognition result of the first character recognition process. Therefore, if it is determined to be a valid recognized character, the blob is collected as a valid recognized blob (see Table 1 above) and the process proceeds to S602. If there is no valid recognized character, the process is terminated. Note that even if the character is not a space, if its character height is less than the threshold and its character length is less than the threshold, it is collected as a valid recognized blob and also counted as a noise character. In this case, the threshold value is appropriately about one-fourth of the line height (width if written vertically, height if written horizontally).

In S602, a process of reevaluating the name line (name line reevaluation process) is executed based on the results of the second character recognition process (S304) and the results of the initial character recognition process (S215). The details of the name line reevaluation process will be described later.

In S603, the process flow branches based on the result of the name line reevaluation process. If the reevaluation result indicates that a name line containing name characters for which replacement is valid exists, the process proceeds to S604; if not, the process exits.

In S604, the character codes of the name characters in the name line for which replacement is valid are replaced with the character codes obtained by the second character recognition process. When the character code replacement is complete, this process ends.

The above is the content of the character code replacement process.

<Name row reevaluation process>
Next, the name line reevaluation process of S602 will be described with reference to the flowchart of FIG.

In S701, a process (blob integration process) is executed for integrating blobs that overlap in the row direction for the valid recognition blobs collected in S601. At this time, an attribute flag is set for the valid recognition blobs after integration. The blob integration process will be described in detail later.

In S702, the attributes of the valid recognized blobs after the blob integration process are determined, and a process (blob attribute determination process) is executed to set an attribute flag. The details of the blob attribute determination process will be described later.

In S703, a process is performed to determine whether or not a recognized character corresponding to a valid recognition blob for which no attribute flag was set in S701 or S702 is a name character. In this case, there are four types of attribute flags: noise flag, integration flag, high-reliability original multi-line flag, and high-reliability original single-line flag. For a recognized character of a valid recognition blob for which any of these four types of flags is set, a determination is not made as to whether or not it is a name character. Note that this determination can be made by performing the same process as the name character identification process described above.

In S704, the recognized characters determined to be name characters in S703 are integrated to generate a name row. This means that the rectangular information of both recognized characters determined to be name characters is combined based on the blobs of the recognized characters and treated as a single character string area.

In S705, among the recognized characters located before and after the recognized characters determined to be name characters in S703, characters that satisfy a predetermined condition are added to the name row generated in S704. This means that the rectangular information of the blob of the recognized characters determined to be name characters and the rectangular information of the blog of the characters before and after it are integrated into one character string area. For example, in the case of a sequence of characters "ai", if the recognized character "ai" is determined to be a name character, if the recognized character "a" located before "ai" satisfies a predetermined condition, it is taken in as a name row and a character string area including the two characters "ai" is obtained. Here, the predetermined condition is a combination of information such as the character code obtained by the character recognition process, the reliability of the recognition result, and the distance between the character determined to be a name character. For example, conditions such as the size of the character being equal to or larger than a threshold and the reliability of the recognition result being equal to or larger than a threshold are set, and characters that satisfy all the conditions are taken in to the name row. This results in a character string area of a character row consisting of recognized characters that are approximately like a person's name.

The above is the content of the name line reevaluation process.

<Blob integration processing>
Next, the blob integration process in S701 will be described with reference to the flowchart in Fig. 8. Note that the series of processes shown below are executed in order for all valid recognition blobs.

In S801, it is determined whether the value indicating the distance between the end of the recognized character corresponding to the valid recognition blob of interest (hereinafter referred to as the "blob of interest") and the beginning of the recognized character corresponding to the next valid recognition blob (hereinafter referred to as the "next blob") is positive or negative. Here, at the start of processing, each valid recognition blob is sorted by distance from the beginning of the line. Therefore, the "next blob" means the valid recognition blob that is closest among the unprocessed valid recognition blobs in the same line. FIG. 16(a) is a diagram showing a state in which the distance (inter-character distance) between the end of the character of the blob of interest and the beginning of the character of the next blob is positive in the case of horizontal writing, and FIG. 16(b) is a diagram showing a state in which the inter-character distance is positive in the case of vertical writing. In other words, when the inter-character distance is negative, it means that the recognized character of the blob of interest and the recognized character of the next blob overlap. Hereinafter, the absolute value of the inter-character distance when it is negative will be called the "inter-character overlap size". If the inter-character distance value is positive, the process transitions to S805; if it is negative, the process transitions to S802.

In S802, it is determined whether the overlap size between characters is equal to or smaller than a threshold value. The threshold value is determined based on the line height (character width for vertical writing, character height for horizontal writing). For example, a value less than one time the line height is set as the threshold value. If it is equal to or smaller than the threshold value, the process proceeds to S804, and if it is greater than the threshold value, the process proceeds to S803.

In S803, a duplicate character merge flag is set for the blob of interest. After setting, the process proceeds to S809.

In S804, it is determined whether the size of the recognized characters corresponding to the focus blob and the next blob is greater than a threshold. The threshold is determined based on the line height (character width for vertical writing, character height for horizontal writing), similar to the threshold in S802. For example, a value less than one time the line height is set as the threshold. If the size of both recognized characters is greater than the threshold, the recognized characters of the focus blob and the next blob are both determined to be the size of name characters, and processing for the focus blob is completed. On the other hand, if the size of both recognized characters is equal to or less than the threshold, the process proceeds to S803 above, and an overlapping character integration flag is set for the focus blob.

In S805, if the inter-character distance is positive, it is determined whether the inter-character distance is greater than a threshold. The threshold in this case is also determined based on the line height (character width for vertical writing, character height for horizontal writing) as with the threshold in S802 and S804. For example, a value less than one time the line height is set as the threshold. If the inter-character distance is greater than the threshold, it is determined that the name characters are separated, and processing of the target blob is completed without merging with the recognized characters of the next blob (S809). On the other hand, if the inter-character distance is equal to or less than the threshold, the process transitions to S806.

In S806, it is determined whether the recognized characters corresponding to the valid recognized blob immediately preceding the target blob (hereinafter referred to as the "previous blob") are closer than twice the distance between the recognized characters of the target blob and the previous blob. If they are close, processing for the target blob ends without merging with the recognized characters of the previous blob (S809). On the other hand, if they are not close, the process transitions to S807. Note that if there is no previous blob for the target blob, the line height of the target blob (character width if written vertically, character height if written horizontally) can be treated as the distance between the two characters and a determination can be made.

In S807, it is determined whether the length of the characters when the recognized characters of the target blob and the recognized characters of the next blob are combined is greater than a threshold. Here, the "character length when combined" is the character height in the case of vertical writing, and the character width in the case of horizontal writing. The threshold is determined based on the line height (character width in the case of vertical writing, character height in the case of horizontal writing). For example, a value that is more than one time the line height is set as the threshold. If the length of the characters when combined is greater than the threshold, processing of the target blob is completed without combining with the recognized characters of the next blob. On the other hand, if the length of the characters when combined is equal to or less than the threshold, the process transitions to S808.

In S808, a separated character integration flag is set for the blob of interest. After setting, the process proceeds to S809.

In S809, the focus blob and the next blob are integrated to generate a new integrated valid recognition blob. After the integration, processing for the focus blob is completed. Then, when the above processing is completed for all valid recognition blobs, the process transitions to S810.

In S810, an attribute flag is set for the valid recognition blob after integration generated in S809. Specifically, either a noise flag or an integration flag is set as the attribute flag according to the following determination conditions:

<Judgment conditions>
・If the integrated blob contains a certain amount of noise characters or more, set the noise flag. ・If the integrated blob contains the overlapping character integration flag or the separated character integration flag, set the integration flag.

When attribute flags have been set for all valid recognition blobs after integration according to the above criteria, this process ends.

The above is the content of the blob integration process.

<Blob attribute determination process>
Next, the blob attribute determination process of S702 will be described with reference to the flowchart of Fig. 9. The series of processes described below are executed for all valid recognition blobs in order. The valid recognition blobs that are the subject of this process include the valid recognition blobs that have been integrated in the above-mentioned blob integration process and the valid recognition blobs that remain unintegrated.

In S901, a process (recognition result comparison process) is executed to compare the recognition result of the first character recognition process (S215) with the recognition result of the second character recognition process (S304) for the valid recognition blob of interest (hereinafter referred to as the "blob of interest"). The details of the recognition result comparison process will be described later.

In S902, based on the comparison result in S901, it is determined whether the blob of interest is a blob that constitutes a multiline. Here, a multiline refers to a character string consisting of multiple character lines, such as the character string enclosed in the dashed frame 1301 in FIG. 13 described above. If it is determined that the blob of interest is a blob that constitutes a multiline, the process transitions to S903, and if it is determined that the blob of interest is a blob that does not constitute a multiline, the process transitions to S904.

In S903, a high-reliability original multi-line flag is set for the blob of interest. After setting, processing for the blob of interest is terminated.

In S904, based on the comparison result in S901, it is determined whether the number of highly reliable original characters is equal to or greater than a threshold value (whether the character string region is strip-shaped). Here, the "number of highly reliable original characters" represents the number of characters determined to have a higher reliability for the recognized characters in the first character recognition process than in the second character recognition process in the recognition result comparison process described next. If the number of highly reliable original characters is equal to or greater than the threshold value, the process proceeds to S905, and if it is less than the threshold, the process for the blob of interest is terminated.

In S905, a high-reliability original single-line flag is set for the blob of interest. After this is set, processing for the blob of interest is terminated.

When the above process has been performed on all valid recognition blobs, this process ends.

The above is the content of the blob attribute determination process.

<Recognition result comparison process>
Next, the recognition result comparison process of S703 will be described with reference to the flowchart of Fig. 10. Note that the series of processes shown below are executed for all original blobs in order.

In S1001, it is determined whether the area indicated by the rectangular information of the valid recognition blob of interest overlaps with the area indicated by the rectangular information of the original blob to be processed (see Table 1 above). Figure 17 is a diagram for explaining overlap determination of rectangular areas. In Figure 17, two vertically long frames indicate rectangular areas of the original blob to be processed, and a substantially square frame indicates a rectangular area of the valid recognition blob of interest. In the example shown in Figure 17, both of the vertically long rectangular areas are determined to overlap with the substantially square rectangular area. In this way, if both rectangular areas overlap, the process proceeds to S1002. On the other hand, if both rectangular areas do not overlap, the process for the original blob is terminated.

In S1002, it is determined whether there are recognized characters that correspond to the original blob being processed. In cases where no character recognition results are returned for a character string region, information indicating this is stored in the blob, so this information is used in the process to determine whether any recognized characters correspond. If there are recognized characters, the process proceeds to S1003. If there are no recognized characters, the process for the original blob ends.

In S1003, the reliability of the valid recognition blob of interest is compared with the reliability of the original blob to be processed. If the reliability of the original blob is higher than that of the valid recognition blob, i.e., if the original blob to be processed is a "high reliability original blob (see Table 1 above)", the process proceeds to S1004. On the other hand, if the reliability of the original blob to be processed is lower than that of the valid recognition blob, the process for that original blob is terminated.

In S1004, it is determined whether the original blob to be processed and another high-confidence original blob are separated in the height direction and overlap in the row direction. If they are separated in the height direction and overlap in the row direction, the process transitions to S905. Figure 18 is a diagram explaining this overlap determination, and in the example shown in the figure, they are separated in the height direction and overlap in the row direction, so the process transitions to S1005. On the other hand, if they are not separated in the height direction or do not overlap in the row direction, the process for that original blob is terminated.

In S1005, it is determined that the recognized character corresponding to the original blob being processed is a character that constitutes a multiline consisting of multiple lines. In the following S1006, the blob being processed is counted as a high-confidence original blob, and processing for the original blob is terminated.

When the above process has been performed for all original blobs, this process ends.

The above is the content of the recognition result comparison process.

<Effects of the recognition result improvement process>
Finally, the effect of the recognition result improvement process explained using the flowcharts of Figures 3 to 10 will be described. Figure 19 is a diagram showing character string areas when the recognition result improvement process is performed on the business card image shown in Figure 11(a) described above. As is clear from a comparison with Figure 11(b), it can be seen that the character string areas of the name portion have been properly extracted. Also, Figure 20 shows the results of character recognition processing for each character string area shown in Figure 19. As is clear from a comparison with Figure 12, it can be seen that the character string areas have been properly extracted, resulting in correct character recognition.

As described above, according to this embodiment, the lines of characters (name lines) that make up the name of a person on a business card are identified from the business card image, and character recognition processing is performed again on that character string area. The result of the initial character recognition processing is then compared with the result of the second character recognition processing, and the more likely result is determined to be the final recognition result. This makes it possible to make the recognition result of the name area on the business card more likely, improving the accuracy of the character recognition processing.

<Other embodiments>
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

Claims (16)

a first processing means for performing a first character recognition process on a character string area obtained by performing a character string detection process on a document image obtained by reading a document;
an identifying means for identifying name characters constituting a person's name in the document based on a result of the first character recognition process;
an acquiring means for acquiring a character string area corresponding to a character line representing the name by integrating an area corresponding to the name characters identified by the identifying means and an area corresponding to one or more characters belonging to the same line as the name characters;
a second processing means for performing a second character recognition process on a character string area corresponding to the character line representing the name;
a determination means for determining character codes for characters constituting the character line representing the name based on a result of the first character recognition process and a result of the second character recognition process;
An information processing device comprising: 2 . The information processing apparatus according to claim 1 , wherein the second character recognition process is a character recognition process in which a character type to be recognized is limited to a character type that can be used in a person's name. 3. The information processing apparatus according to claim 2 , wherein the types of characters that can be used in the person's name include at least kanji and kana. The determining means is
generating a line of characters representing the name using recognized characters other than spaces from the result of the second character recognition process;
if a recognized character included in the generated character line representing the name is a valid name character, the character code obtained by the first character recognition process is replaced with the character code of the recognized character that is the valid name character;
4. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer . The information processing apparatus according to claim 4, characterized in that the determining means generates the character line representing the name by integrating recognized characters other than the space, the recognized characters having a negative inter-character distance, and treating them as a single recognized character. The information processing device according to any one of claims 1 to 5, characterized in that the identification means identifies a recognized character as the name character when the size in the line height direction of the recognized character included in the result of the first character recognition process is equal to or larger than a threshold value . The information processing device according to any one of claims 1 to 5, characterized in that, when the number of first candidate characters of the recognized character included in the result of the first character recognition process is one, the identification means identifies the recognized character as the name character. The information processing device according to any one of claims 1 to 5, characterized in that when a character code of a first candidate for a recognized character included in the result of the first character recognition process is kanji or kana, the identification means identifies the recognized character as the name character. The information processing device according to any one of claims 1 to 5, characterized in that the identification means identifies the recognized character as the name character when the character code of the recognized character included in the result of the first character recognition process is a character code other than a character code specified as an exclusion target. 6. The information processing apparatus according to claim 1, wherein the identifying means identifies a recognized character included in the result of the first character recognition process as the name character when the reliability of the recognized character is equal to or higher than a threshold value. The information processing device according to claim 1, characterized in that the acquiring means acquires a character string area corresponding to the character line representing the name by combining name characters that are aligned with each other for the plurality of name characters identified by the identifying means. 12. The information processing apparatus according to claim 11 , wherein the acquiring means acquires a character string area corresponding to the character line representing the name by further combining characters before and after the character string area obtained by the combining. The information processing device according to claim 12, characterized in that the acquiring means acquires a character string area corresponding to the line of characters representing the name by adding pixel chunks before and after the character string area obtained by combining the characters before and after the character string area. 14. The information processing apparatus according to claim 11, wherein the acquiring means comprises a means for deleting a character string area that satisfies a certain condition from among the acquired character string areas corresponding to the character line representing the name. a processing step of performing a first character recognition process on a character string area obtained by performing a character string detection process on a document image obtained by reading the document;
identifying name characters constituting a person's name in the document based on a result of the first character recognition process;
an acquiring step of acquiring a character string area corresponding to a character line representing the name by integrating an area corresponding to the name characters identified in the identifying step and an area corresponding to one or more characters belonging to the same line as the name characters;
a processing step of performing a second character recognition process on a character string area corresponding to the character line representing the name;
a determining step of determining character codes for characters constituting the character line representing the name based on a result of the first character recognition process and a result of the second character recognition process;
13. An information processing method comprising: A program for causing a computer to function as the information processing device according to any one of claims 1 to 14 .

Images