JP7516170B2 - Image processing device, image processing method, and program - Google Patents

Description

This disclosure relates to a technique for extracting indexes contained in an image.

There is a method for extracting character strings (hereafter referred to as indexes) of desired items contained in a scanned image obtained by scanning a paper document such as a form with an image reading device. OCR processing is required to extract an index from the contents of the document. However, performing OCR processing on the entire scanned image increases the processing load, and this can lead to increased waiting times for users.

Patent document 1 discloses a method of registering in advance information about areas that contain indexes for each type of document, and then performing OCR processing partially on the registered index areas to extract the index from the scanned image.

JP 2019-128715 A

However, even for documents of the same type, the position of the character string area (hereafter referred to as the text block) containing the index may differ depending on the content written. For this reason, even if OCR processing is performed partially on the registered index area, extraction of the index may fail.

The technology disclosed herein aims to extract the index of the extraction target even if the position of the text block in the scanned image is different from the registered position.

The image processing device disclosed herein is characterized by having a detection means for detecting text blocks in an input image, an identification means for identifying a registered document corresponding to the input image from among a plurality of registered documents, a determination means for determining a text block corresponding to the item to be processed in the input image based on a partial layout defined in the identified registered document, the partial layout including a first text block corresponding to the item to be processed and at least one second text block existing in the vicinity of the first text block, and an acquisition means for acquiring a character string corresponding to the item to be processed in the input image by performing character recognition processing on the determined text block.

The technology disclosed herein makes it possible to extract the index of the target text block even if the position of the text block in the scanned image differs from that of the registered document.

FIG. 1 illustrates an example of a system configuration. FIG. 2 illustrates an example of a hardware configuration of an image forming apparatus. FIG. 2 is a diagram illustrating a functional configuration of the image forming apparatus. 13 is a flowchart of a file generation process of a scanned image. 13 is a flowchart of an index extraction process. FIG. 11 is a diagram illustrating an example of a block selection process. FIG. 13 is a diagram illustrating an example of an index extraction rule. 13 is a flowchart of an index block estimation process. FIG. 11 is a diagram for explaining a method for determining pair blocks. FIG. 13 is a diagram showing an example of a partial pattern. 13A and 13B are diagrams illustrating a process of determining a Y candidate position. FIG. 13 is a diagram illustrating an example of a histogram of shift amounts in the Y direction. FIG. 13 is a diagram illustrating calculation of a degree of match of a partial pattern. FIG. 13 is a diagram illustrating calculation of a degree of match of a partial pattern. FIG. 13 is a diagram for explaining a method for determining a partial pattern range. 13 is a flowchart of an index block estimation process. FIG. 13 is a diagram showing an example of a partial pattern. FIG. 13 is a diagram showing an example of a group of XY candidate positions. FIG. 13 is a diagram showing an example of a group of similar positions. 13 is a diagram for explaining correspondence between a group of similar positions and a group of XY candidate positions. FIG.

The following embodiments are described in detail with reference to the attached drawings. Note that the following embodiments do not limit the disclosed technology according to the claims, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the disclosed technology.

<Embodiment 1>
The image forming apparatus of this embodiment scans a document, combines character strings of predetermined items contained in the image of the first page of the obtained scanned image, and generates a file name. The generated file name is then recommended to the user as the file name of the scanned image. However, extracting character strings of predetermined items from the scanned image may increase the processing load.

For this reason, position information for text blocks of specified items is registered for each document type. It is then possible to identify the document type of the scanned image and extract character strings of specified items from the scanned image based on the positions of the registered text blocks in the identified document. However, even in this case, even for the same document type, the positions of text blocks in the scanned image may differ from the registered positions due to changes in the written content, etc.

For example, assume that the document in FIG. 11(a) is registered, and information indicating the position of text block 1003 is registered as information on an area containing a character string indicating the issuing company name. Meanwhile, FIG. 11(b) is a scanned image obtained by scanning the same type of document as FIG. 11(a), but the number of item rows in the table structure has increased, and the text block 1101 of the issuing company name to be extracted has shifted downward compared to FIG. 11(a). For this reason, even if the document scanned to obtain the scanned image in FIG. 11(b) can be identified as being of the same type as FIG. 11(a), extraction of the character string indicating the issuing company name from the image in FIG. 11(b) may fail. Note that FIG. 11(c) will be explained later.

Therefore, in this embodiment, in order to extract text blocks of items included in a scanned image, a layout of text blocks indicating items in a document of the same type as the scanned document manuscript and at least one other text block is used. This embodiment describes a method of searching the scanned image for an area that has a high degree of match with the layout, and estimating text blocks of items included in the scanned image based on the search results.

In this embodiment, the coordinate system used for the image has an origin at the top left, a vertical direction in the Y direction, and a horizontal direction in which a string of characters extends in the X direction. The position of each text block is stored as the top left coordinate value, for example.

[System configuration]
Fig. 1 is a diagram showing the overall configuration of a system to which this embodiment can be applied. A system 105 of this embodiment has an image forming apparatus 100 and a terminal 101. As shown in Fig. 1, the image forming apparatus 100 is connected to a LAN 102 and is capable of communicating with a terminal 101 such as a PC via the Internet 103 or the like. Note that in this embodiment, the terminal 101 does not have to be included, and the system may be configured with only the image forming apparatus 100.

The image forming device 100 is a multifunction peripheral (MFP) having a display and operation unit 123 (see FIG. 2), a scanner unit 122 (see FIG. 2), a printer unit 121 (see FIG. 2), and the like. The image forming device 100 can be used as a scanning terminal that scans document manuscripts using the scanner unit 122. The image forming device 100 also has a display and operation unit 123 such as a touch panel or hard buttons, and displays a user interface for displaying recommended file names and storage destinations, and for receiving instructions from the user.

[Hardware Configuration of Image Forming Apparatus]
2 is a block diagram showing a hardware configuration of the image forming apparatus 100. The image forming apparatus 100 of this embodiment has a display and operation unit 123, a scanner unit 122, a printer unit 121, and a control unit 110.

The control unit 110 has a CPU 111, a storage device 112 (ROM 118, RAM 119, HDD 120), a printer I/F unit 113, a network I/F unit 114, a scanner I/F unit 115, and a display and operation I/F unit 116. Furthermore, in the control unit 110, each of these units is connected to each other so that they can communicate with each other via a system bus 117. The control unit 110 controls the operation of the entire image forming apparatus 100.

The CPU 111 reads and executes the control programs stored in the storage device 112, thereby functioning as a means for executing various processes such as reading control, image processing, and display control in the flowcharts described below.

The storage device 112 stores and holds control programs, image data, metadata, setting data, processing result data, etc. The storage device 112 includes a ROM 118 which is a non-volatile memory, a RAM 119 which is a volatile memory, and a HDD 120 which is a large-capacity storage area. The ROM 118 is a non-volatile memory which holds the control programs, etc., and the CPU 111 reads and controls the control programs. The RAM 119 is a volatile memory which is used as a temporary storage area such as the main memory and work area of the CPU 111.

The network I/F unit 114 connects the control unit 110 (image forming device 100) to the LAN 102 via the system bus 117. The network I/F unit 114 transmits image data to external devices on the LAN 102 and receives various information from external devices on the LAN 102.

The scanner I/F unit 115 connects the scanner unit 122 and the control unit 110 via the system bus 117. The scanner unit 122 reads a document original to generate scanned image data, and inputs the scanned image data to the control unit 110 via the scanner I/F unit 115. The scanner unit 122 is equipped with a document feeder, and is capable of feeding multiple documents placed on a tray one by one, enabling them to be read continuously.

The display and operation I/F unit 116 connects the display and operation unit 123 and the control unit 110 via the system bus 117. The display and operation unit 123 is equipped with a liquid crystal display unit with a touch panel function, hard buttons, etc.

The printer I/F unit 113 connects the printer unit 121 and the control unit 110 via the system bus 117. The printer unit 121 receives image data generated by the CPU 111 via the printer I/F unit 113, and performs printing processing on recording paper using the received image data. As described above, the image forming device 100 according to this embodiment is capable of providing an image processing function with the above hardware configuration.

[Functional Configuration of Image Forming Apparatus]
Fig. 3 is a block diagram showing the functional configuration of the image forming apparatus 100. Note that Fig. 3 shows only functions related to the process of scanning a document original, digitizing it (making it into a file), and storing it, among the various functions of the image forming apparatus 100.

The display control unit 301 displays a user interface screen (UI screen) for receiving various user operations on the touch panel of the display/operation unit 123. The various user operations include, for example, scan settings, instructions to start scanning, file name settings, and instructions to save a file.

In response to a user operation performed on the UI screen (e.g., pressing the "Start Scan" button), the scan control unit 302 instructs the scan execution unit 303 to execute a scan process together with scan setting information. In accordance with the instruction to execute a scan process from the scan control unit 302, the scan execution unit 303 causes the scanner unit 122 to execute a read operation of the document original via the scanner I/F unit 115, and generates scan image data. The generated scan image data is stored in the HDD 120 by the scan image management unit 304.

The image processing unit 305 performs image analysis processes, such as text block detection, OCR (character recognition), and similar document determination, on the scanned image data, as well as image processing processes such as rotation and tilt correction. The image processing unit 305 also allows the image forming device 100 to function as an image processing device. Character string areas detected from the scanned image are also called "text blocks." Details of the image processing will be described later.

The functions of each part in FIG. 3 are realized by the CPU of the image forming device 100 expanding program code stored in the ROM into the RAM and executing it. Alternatively, some or all of the functions of each part in FIG. 3 may be realized by hardware such as an ASIC or electronic circuit.

[Flowchart of scanned image file generation process]
The entire process in which the image forming device 100 reads a document original, performs image processing on the scanned image of the first page of the document original, generates a file name using a character string contained in the scanned image, and recommends the file name to the user via the display/operation unit 123 will be described below.

The series of processes shown in the flowchart of FIG. 4 are performed by the CPU of the image forming device 100 by expanding the program code stored in the ROM into the RAM and executing it. In addition, some or all of the functions of the steps in FIG. 4 may be realized by hardware such as an ASIC or electronic circuit. Note that the symbol "S" in the explanation of each process indicates a step in the flowchart, and the same applies to subsequent flowcharts.

In S400, when the scan control unit 302 receives a scan instruction from the user via the display and operation unit 123, it causes the scan execution unit 303 to read (scan) multiple document originals one by one from the document feeder tray of the scanner unit 122. Then, the scan control unit 302 obtains image data of the image obtained as a result of the scan (called a scanned image).

In S401, the image processing unit 305 analyzes the image data acquired in S400 and performs processing to extract indexes contained in the scanned image (index extraction processing). An "index" is a character string of a specific item such as a document title, management number, or company name. In this embodiment, the index is used as a file name or metadata when saving the scanned image. Details of the index extraction processing in this step will be described later with reference to FIG. 5.

The use of the index is not limited to generating file names or extracting metadata. It may also be used to set other property information such as folder paths. In other words, file names and metadata are types of information that are set as properties (attributes) related to scanned image data.

In S402, the display control unit 301 generates a file name using the index extracted in S401, and displays the generated file name and metadata on the display and operation unit 123 to present (recommend) to the user. The display control unit 301 also accepts confirmation or correction of the presented file name by the user. When the display control unit 301 accepts confirmation or correction from the user via the display and operation unit 123, the presented file name or the corrected file name if corrected, is determined as the file name of the scanned image. If the user makes corrections via the display and operation unit 123, the index extraction rule is updated. The index extraction rule will be described later.

In S403, the image processing unit 305 creates a file from the image data acquired in S400, and sets the file name determined in S402. In this embodiment, as an example, the scanned image is saved in a file format of PDF (Portable Document Format). In the case of PDF, it is possible to save the image data divided into pages, and when multiple document manuscripts are scanned in S400, the image data corresponding to each document manuscript is saved as separate pages in a single file.

In S404, the scan image management unit 304 transmits the file created in S403 to a specified destination via the LAN 102.

[Regarding index extraction process (S401)]
Fig. 5 is a flowchart showing the details of the index extraction process in S401. The details of the index extraction process will be described with reference to Fig. 5. In the index extraction process, the orientation of one page of image data is corrected, the document type is identified, and an index corresponding to the document type is extracted.

In S500, the image processing unit 305 detects the angle of inclination of the scanned image from the image data, and corrects the inclination of the scanned image by rotating the image in the opposite direction by the detected inclination. The inclination that is the subject of inclination correction occurs, for example, when a document is scanned and not read straight due to wear of the rollers in the document feeder of the scanner unit 122. Or, it occurs when the scanned document is not printed straight when it is printed.

The method of detecting the angle of inclination is to first detect objects contained in the image data and connect adjacent objects in the horizontal or vertical direction. Then, the angle connecting the center positions of the connected objects is calculated to determine the degree of inclination from the horizontal or vertical direction. Note that the method of detecting inclination is not limited to this method. For example, a method may be used in which the center coordinates of objects contained in the image data are obtained, the center coordinates are rotated in 0.1 degree increments, and the angle at which the center coordinates are most frequently aligned horizontally or vertically is determined as the inclination of the scanned image. Correcting the inclination of the scanned image can improve the processing accuracy of the rotation correction, block selection processing, and OCR processing that are performed subsequently.

In S501, the image processing unit 305 rotates the scanned image after tilt correction obtained as a result of the processing in S500 in 90 degree increments so that the characters in the image are oriented upright. For example, the method of rotation correction is to use the scanned image after tilt correction as a reference image and prepare four images: the reference image, an image rotated by 90 degrees from the reference image, an image rotated by 180 degrees from the reference image, and an image rotated by 270 degrees from the reference image. Then, a simple OCR process that can be processed at high speed is performed on each image, and the image with the largest number of characters recognized with a certain level of confidence or higher is used as the image after rotation correction. However, the method of rotation correction is not limited to this method. Note that hereinafter, the term "scanned image" refers to the scanned image corrected in S500 and S501 unless otherwise specified.

In S502, the image processing unit 305 performs block selection processing on the scanned image. Block selection processing is a process in which the image is classified into foreground and background regions, the foreground region is divided into text blocks and other blocks, and the text blocks are detected.

Specifically, the contour of the scanned image that has been binarized to black and white is traced to extract a cluster of pixels surrounded by a black pixel contour. For clusters of black pixels whose area is larger than a certain size, the contour of the white pixels inside is also traced to extract a cluster of white pixels, and then a cluster of black pixels is recursively extracted from inside the cluster of white pixels whose area is equal to or larger than a certain size. The cluster of black pixels thus obtained is determined as the foreground region. The determined foreground region is classified by size and shape into regions with different attributes. For example, a foreground region with an aspect ratio close to 1 and a certain range of size is determined as a pixel cluster corresponding to a character, and a region where adjacent characters can be grouped in good alignment is determined as a character string region (TEXT). A flat pixel cluster is determined as a line region (LINE). A region occupied by a black pixel cluster that is equal to or larger than a certain size and contains a rectangular white pixel cluster in good alignment is determined as a table region (TABLE). Areas where irregular pixel clusters are scattered are determined to be photograph areas (PHOTO). Pixel clusters of any other shape are determined to be picture areas (PICTURE). From among the areas divided according to object attributes in this way, foreground areas (TEXT) determined to have character attributes are detected as text blocks.

Figure 6 shows an example of the results of block selection processing. Figure 6(a) shows a scanned image after rotation correction. Figure 6(b) shows the results of block selection processing on the scanned image of Figure 6(a), where the rectangles indicated by dotted lines represent foreground regions. Note that in Figure 6(b), the attributes of all foreground regions have been determined, but the attributes are only displayed for some of the foreground regions. Information about each text block detected in this step (attributes and information indicating the position and size of each block) is used in subsequent processes such as OCR processing and similarity calculation.

In the block selection process in this step, only text blocks are detected. This is because the position of character strings well represents the structure of a scanned image and is closely related to index information. Therefore, this does not exclude the use of information about blocks determined to have other attributes, such as photo areas or table areas, in subsequent processing.

In S503, the image processing unit 305 retrieves the index extraction rules from the HDD 120 and loads them into the RAM 119.

Figure 7 is a diagram showing a portion of an index extraction rule (hereinafter simply referred to as an extraction rule). Figure 7 shows an extraction rule record in which "0001" has been assigned as the document ID included in the extraction rule and registered. In the extraction rule, for each registered document, the following data are associated on a record-by-record basis: "document ID", "thumbnail", "document identification information", and "index information". The extraction rule holds as many combinations (records) of these as there are registered documents. The document ID is a unique ID that indicates the type of document.

Document identification information is information about the position and size of text blocks obtained by performing block selection processing on a scanned image of a registered document. Document identification information is information for identifying the type of document and is used in document matching, which will be described later.

Index information is information for extracting indexes contained in a scanned image. The index is used to determine the file name or metadata to be assigned to a file. Specifically, index information includes information on the coordinates and size of text blocks that contain the character strings (index) of each item in a registered document. The "index information" image 701 in Figure 7 illustrates the position and size of the text blocks that contain each item, arranged at coordinates on the image. Index information also includes information indicating the indexes used to generate a file name and their order, and information to be assigned as metadata.

The "File name rule" in the index information indicates that the file name is generated by connecting the indexes of the fields title (title), issuing company name (sender), and document number (number) with an underscore separator. Additionally, "Metadata" indicates that the index of the field total amount (total_price) is used as metadata. In other words, by extracting the indexes of specified fields, it is possible to generate file names to be recommended to users and extract metadata.

In this embodiment, an example is shown in which the extracted index is used as a file name or metadata, but rules for determining the destination folder information of the file, which is other property information, may also be held. In this case, too, the property information generated using the index is recommended to the user in S402, and the property information is set in the scanned image file in S403.

In addition, as an extraction rule for registered documents, thumbnails of scanned images corresponding to the registered documents may also be stored together, as shown in "Thumbnail" in Figure 7.

In S504, the image processing unit 305 performs document matching on the scanned image. In document matching, it is determined whether a document of the same type as the document (input document) scanned to obtain the scanned image is present in the document group registered in the extraction rule. If it is determined that a document of the same type as the input document is registered, the process identifies the type.

In this embodiment, first, the scanned image is compared one-to-one with each document registered in the extraction rule, and a similarity is calculated to indicate how similar the shapes and arrangements of the included text blocks are. As a method of calculating the similarity, for example, the entire text blocks of the scanned image are aligned with the entire text blocks of the registered document. Then, the square of the sum of the overlapping areas of each text block of the scanned image and each text block of the registered document (value A) is calculated. Furthermore, the product of the sum of the areas of the text blocks of the scanned image and the sum of the areas of the text blocks of the registered document (value B) is calculated. Then, the value A divided by value B is used as the similarity. This calculation of similarity is performed between the scanned image and all documents registered in the extraction rule.

The document registered in the extraction rule that has the highest similarity and is equal to or greater than a predetermined value is identified as the same type of document as the scanned input document. Furthermore, if there is no document in the extraction rule with a similarity equal to or greater than a predetermined value, it is determined that no document of the same type as the input document is registered in the extraction rule.

In S505, the image processing unit 305 determines whether a document of the same type as the input document has been registered in the extraction rules as a result of the document matching performed in S504. If the input document is not a registered document (NO in S505), the processing of this flowchart ends. If it is not a registered document, a new ID is assigned as described above, and the layout information of the text block detected in S502, etc. is registered in the extraction rule. In this case, in S402, the file name and metadata are not recommended to the user, and the display control unit 301 accepts input of a file name by the user. When the display control unit 301 accepts input from the user via the display and operation unit 123, the input file name is determined as the file name of the scanned image.

If a document of the same type as the input document is registered (YES in S505), in S506 the image processing unit 305 assigns to the scanned image the same document ID as the document of the extraction rule identified in S504 as being of the same type as the input document.

In S507, the image processing unit 305 executes an index block estimation process to estimate a text block of an index of an item to be extracted (processed) in the scanned image based on the extraction rule linked to the document ID assigned in S506. A text block that includes character strings (indexes) indicating items such as a title, issuing company name, and form number is sometimes called an index block. Details of the index block estimation process will be described later.

In S508, the image processing unit 305 performs partial OCR on the index blocks of each item estimated in S507, and extracts character strings corresponding to each index block as an index.

[Regarding the index block estimation process (S507)]
Fig. 8 is a flowchart of the index block estimation process in S507. Details of the index block estimation process will be described with reference to Fig. 8. Note that, hereinafter, a registered document refers to a document that corresponds to the document ID assigned to the scanned image in S506, among the documents registered in the extraction rule acquired in S503. In the description of this flowchart, the registered document will be described as the document with document ID "0001" in Fig. 7.

In S800, the image processing unit 305 obtains document identification information linked to the document ID assigned in S506 from the extraction rule. Then, the image processing unit 305 performs overall alignment between the entire text block in the scanned image and the entire text block of the registered document.

The input document of the scanned image acquired in S400 is the same type of document as the registered document, and each item is printed at the same coordinates as the item in the registered document. However, due to differences in printing and scanning timing or the printing device, etc., there may be a misalignment between the position of the text block on the scanned image and the position of the text block in the registered document. Therefore, in this step, overall alignment is performed to reduce the effect of the misalignment and improve the accuracy of subsequent processing. Note that in this embodiment, since tilt correction is performed in S500 of FIG. 5, an example will be described in which the overall alignment in this step only involves correction to shift (translate) the entire text block on the scanned image.

For overall alignment, the shift amount is calculated - how much the text blocks in the scanned image are shifted relative to the text blocks in the registered document - and the coordinates are corrected so that each text block in the scanned image is shifted by the shift amount.

Figure 9 is a diagram of a portion of an image in which a text block of the scanned image and a text block of a registered document are drawn in the same coordinate system. A specific procedure for calculating the shift amount for overall alignment will be explained using Figure 9. In Figure 9, the solid-line rectangle indicates one text block 900 selected from the group of text blocks in the scanned image, and the dashed-line rectangle indicates text blocks 901-903 of the registered document that surround text block 900. Also in Figure 9, the dash-dotted circle 904 indicates an area with a certain radius centered on the upper left vertex of text block 900 of the scanned image.

To calculate the shift amount, we determine the text blocks (called pair blocks) in the registered document that are candidates for each text block in the scanned image. Here we explain how to determine pair blocks for text blocks in the scanned image.

First, among the text blocks 901-903 of the registered document, a text block whose upper left vertex falls within a circle 904 whose center is the upper left vertex of one text block 900 selected from the text blocks in the scanned image is searched for. In FIG. 9, text blocks 901 and 902 fit this criteria. Next, the overlap rate is calculated between the text block 900 of the scanned image and each of the text blocks 901 and 902 of the registered document. The overlap rate is calculated by aligning the upper left vertices of the text block of the scanned image and the text block of the registered image to calculate the area of the common part of both text blocks. The overlap rate is then calculated by dividing the area of the common part by the larger area of both text blocks.

A text block in a registered document whose overlap rate meets a specified condition is considered to be a paired block. The specified condition is, for example, that the overlap rate with the text block in the scanned image is equal to or greater than the maximum overlap rate multiplied by coefficient α, and equal to or greater than a specified threshold. In this case, coefficient α is used to select a combination with an overlap rate close to the maximum overlap rate, and is set to a value less than 1.0, such as 0.5 to 0.8. The specified threshold defines a minimum line, and is set to a value less than 1.0, such as 0.3 to 0.7.

In FIG. 9, of the text blocks 901 and 902 in the registered document, only text block 901, which has a shape similar to that of text block 900 in the scanned image, is selected as a pair block. If there are other text blocks that satisfy certain conditions, multiple pair blocks may be selected. In this way, for each of the pair blocks corresponding to one text block selected from the scanned image, the difference (shift amount) in the X and Y directions of the upper left vertex between the text block selected from the scanned image is calculated. The difference amount is then voted for in a shift amount histogram. In this case, the range of the bins of the histogram can be any range.

In the case of FIG. 9, for text block 900, the difference (shift amount) in the X and Y directions of the upper left vertex between the text block 900 and the text block 901 of the registered document is calculated, and the shift amount is voted for in the shift amount histogram.

The process of determining pair blocks corresponding to text blocks in the scanned image and voting for the shift amount histogram is performed for each text block in the scanned image. Then, the position that is the maximum peak point in the final shift amount histogram is determined. The shift amount indicated by the determined position is used as the shift amount for the overall alignment.

If the influence of noise is a concern, the generated shift amount histogram may be smoothed. Local peak points other than the maximum peak point may also be selected as candidates for the shift amount, and the shift amount to be used for overall alignment may be selected from the candidates. For example, for each candidate shift amount, the coordinates of the text block in the scanned image may be shifted, and a similarity calculation may be performed in the same way as in document matching in S504 of FIG. 5, and the candidate with the highest similarity may be determined as the final shift amount.

By shifting the coordinates of each text block in the scanned image by the shift amount determined by the above procedure, a group of aligned text blocks in the scanned image can be obtained. Note that the method of aligning text blocks is not limited to the above method. An example has been described in which only correction is made regarding the shift (parallel movement) of the entire scanned image, but if a deviation in magnification is expected as a deviation between printing and scanning, alignment can be performed taking into account not only the shift amount but also the deviation in magnification.

Note that in the following steps, the scanned image or text blocks of the scanned image refer to this entire aligned scanned image or text blocks.

Next, index information of the registered document linked to the document ID assigned in S506 is obtained. Then, in S801, one of the index items included in the index information is selected as the processing target, and S801 to S810 are repeated. Then, a process is performed to estimate the text block of the processing target item from the text block group of the scanned image. When the processing of the processing target item is completed, another processing target item is selected from the unprocessed items.

In S801, the image processing unit 305 selects one unprocessed index item from among the items registered in the index information of the registered document, and sets it as the item to be processed. In this embodiment, one of the items held in the index information in FIG. 7, the title (title), issuing company name (sender), document number (number), and total amount (total_price), is selected as the item to be processed.

In S802, the image processing unit 305 acquires a "partial pattern" of the item to be processed. The partial pattern includes information on the layout of a portion of a text block included in the registered document (partial layout) and information on the range that includes the partial layout (partial pattern range).

Figure 10(a) illustrates the position and size of the index blocks for each item in the registered document registered in Figure 7 with document ID "0001." The dashed rectangles in Figure 10(a) represent index blocks 1000 to 1003 for each of the items: title, document number, total amount, and issuing company name.

Figure 10(b) is a diagram showing a partial pattern for the "issuing company name (sender)" item. The range indicated by the dashed-dotted rectangle in Figure 10(b) indicates the partial pattern range 1006 for the "issuing company name (sender)" item. The partial pattern range 1006 is determined using a preset value based on index block 1003, which is the text block for the "issuing company name (sender)" item.

Text blocks 1004 and 1005 represent text blocks in the registered document that are at least partially included in partial pattern range 1006. These text blocks 1004 and 1005 and the partial layout in the registered document represented by index block 1003 are the partial layout of the issuing company name item. The partial layout is represented by the text block of the item to be processed and at least one text block other than the text block of the item to be processed. The layout is information that represents the position information of each text block and the size of each text block.

As information contained in the partial pattern of the issuing company name item, partial pattern range 1006 and a partial layout consisting of index block 1003 and text blocks 1004 and 1005 are determined. In this way, partial patterns corresponding to each item of the registered document are determined and stored.

As will be described in more detail below, in this embodiment, a position in the scanned image that is similar or matches the arrangement of the partial layout is searched for, and the text block of the item to be processed in the scanned image is estimated.

Figure 10(c) shows the partial pattern for the "title" item. Similarly, for the title, a partial pattern range 1007 and a partial layout consisting of the title index block 1000 and text blocks 1001, 1008 to 1013 included in the partial pattern range 1007 have been determined as the partial pattern.

Note that the size of partial pattern range 1007 is different from the size of partial pattern range 1006 in FIG. 10(b). In this way, the partial pattern size may be made different depending on the properties of the item. Alternatively, the size of the partial pattern range may be a common size for all items. A method for determining the size of the partial pattern range will be described in embodiment 2.

Note that the partial pattern does not need to be determined each time the index extraction process is performed after scanning a document manuscript. For example, when registering a document, a partial pattern may be determined for each item and stored in advance as part of the extraction rules shown in FIG. 7. In other words, in S802, it is sufficient to obtain the partial pattern of the stored item to be processed.

In the next steps S803 and S804, positions (XY candidate positions) in the scanned image that have an area that matches highly with the partial layout of the item to be processed are determined. As a method for determining the XY candidate positions, for example, the candidate positions may be estimated by scanning a partial pattern in a search range in the scanned image and calculating the degree of match, as in template matching. In this embodiment, in order to reduce the amount of calculations, candidate positions in the Y direction in the search range are determined and the Y direction positions (Y positions) are narrowed down. Then, for each group of Y position candidates (Y candidate positions), a partial pattern is scanned in the X direction to determine the XY candidate positions, thereby reducing the amount of calculations.

In S803, the image processing unit 305 determines a group of Y candidate positions from the group of text blocks in the scanned image that are similar to the text blocks of the partial pattern of the item to be processed in the registered document.

Figure 11 is a diagram for explaining the process of determining the Y candidate position group. The explanation will be given assuming that the item to be processed is the issuing company name (sender).

Figure 11(a) is a diagram similar to Figure 10(b) showing a partial pattern of the issuing company name (sender) in a registered document. Figure 11(b) is a scanned image, and the dashed rectangle represents a group of aligned text blocks. The document shown in the scanned image in Figure 11(b) is a document determined to be the same type of document as the registered document "0001", but shows an example in which the number of item rows in the table structure is increased compared to the registered document in Figure 7. Therefore, the index block 1101 of the issuing company name (sender) that should be estimated in the scanned image has shifted downward compared to the position of the index block 1002 of the issuing company name (sender) in the registered document.

Figure 11(c) shows one text block 1003 out of text blocks 1003-1005 representing a partial layout included in the partial pattern of the issuing company name, superimposed on the same coordinate system as the text blocks in the scanned image. The determination of the Y candidate positions will be explained with reference to Figure 11(c), focusing on text block 1003 in the partial pattern.

In Figure 11(c), search range 1100, represented by a dashed-dotted rectangle, represents the range to be searched to determine the group of Y candidate positions for the item to be processed. Text blocks 1101-1109, represented by a dashed-dotted rectangle, are the text blocks in the scanned image shown in Figure 11(b) whose rectangular centers are within search range 1100.

To determine the group of Y candidate positions, first one text block (text block 1003 in FIG. 11(c)) included in the partial layout is selected. The selected text block is then superimposed on the same coordinate system as the group of text blocks in the scanned image, and the difference in the Y position of the center of the rectangle with the text blocks in the scanned image within the search range (text blocks 1101 to 1109 in FIG. 11(c)) is calculated. The calculated difference is then voted for in the Y direction shift amount histogram. The range of the bins in the shift amount histogram can be any range.

Figure 12 is a diagram showing an example of a shift amount histogram in the Y direction. Figure 12 (a) is a shift amount histogram after voting for the difference amount of the Y position between the text block 1003 of the partial pattern in Figure 11 (c) and the text block 1102 of the scanned image. h indicates the upper limit of the absolute value of the search range in the Y direction from the reference. Voting is performed on position 1200 according to the difference amount in the Y direction between the text block 1003 and the text block 1102. Similarly, voting is performed according to the difference amount of the Y center between one text block included in the partial pattern and all text blocks in the search range of the scanned image. This voting is performed for all text blocks in the partial pattern. In other words, the difference amount of the Y center between the text blocks 1101 to 1109 in the search range is also calculated for the text blocks 1004 and 1005 of the partial pattern, and voted for in the shift amount histogram. Then, the shift amount histogram in the Y direction is completed. If the influence of noise is a concern, smoothing may be applied to the shift amount histogram generated in the Y direction.

Figure 12(b) shows the finally generated Y-direction shift amount histogram. After the generation of the shift amount histogram is completed, peak points are determined as shown at positions 1201 to 1206 in the histogram, and a group of Y candidate positions is determined based on the Y-direction shift amount corresponding to the bin of each peak point.

The search range 1100 for determining the Y candidate positions in FIG. 11(c) is automatically determined by a preset value based on the position of the index block of the partial pattern. The size of the search range may be a common range for all items, or may be determined according to the attributes of the item to be processed. For example, the index block of the title is often located at a fixed position in the document. Therefore, when the item to be processed is the title, the search range may be set narrower because the index block to be estimated from the search range is unlikely to be missed even if the search range is narrowed. By narrowing the search range, it is possible to prevent unnecessary candidate positions from being determined while suppressing the amount of calculation. On the other hand, the position of the index block whose item is the total amount may change up or down depending on the change in the number of item rows in the table structure in the document. Therefore, when the item to be processed is the total amount, the search range may be set wider up or down than other items.

In S804, the image processing unit 305 derives the degree of match between the partial layout of the partial pattern and the text blocks of the scanned image based on each of the Y candidate positions determined in S803.

Figure 13 shows the overlap state between a partial layout of an item to be processed and the layout of text blocks in the scanned image when the partial layout is placed at a certain position in the scanned image. Using Figure 13, we will explain how to derive the degree of matching between a partial layout and a group of text blocks in the scanned image.

In FIG. 13, the solid rectangles are text blocks 1003-1005 that make up the partial layout of the item being processed. The dashed rectangle represents the partial pattern range 1006. The dashed rectangles represent text blocks 1101, 1104-1106, and 1109 in the scanned image. The shaded areas 1309 and 1310 represent the areas where text blocks 1003-1005 in the partial layout overlap with the text blocks in the scanned image.

The degree of match between the partial layout and the text block of the scanned image (Score) is calculated using the following formula (1):

In the above formula (1), R represents all the text blocks that make up the partial layout, and N _R represents the total number of text blocks that make up the partial layout. In Fig. 13, R is text block 1003 to 1005, and N _R is 3.

Correlation(r) is the individual degree of correspondence of one text block r that constitutes a partial layout. The individual degree of correspondence Correlation(r) of text block r is derived using formula (2).

OverlappingQ is a set of text blocks in the scanned image that overlap with text block r. OverlapArea(r,q) is the area of the overlapping region between text block r and one of the text blocks in OverlappingQ, text block q. _{NOverlappingQ} represents the total number of OverlappingQ.

13, if r is text block 1003, OverlappingQ is only text block 1105 and OverlapArea(r,q) is area 1309. If r is text block 1005, OverlappingQ is only text block 1104 and OverlapArea(r,q) is area 1310. If r is text block 1004, there is no corresponding OverlappingQ, so _{NOverlappingQ} is 0 and Correlation(r) is 0.

Area_r is the area of text block r, and Area_q is the area of text block q.

When deriving the degree of matching using formula (1), the greater the number of text blocks in the scanned image and the larger the area of the text blocks, the larger the value of the individual degree of matching Collrelation(r) that is derived may be. Therefore, a penalty term PenaltyTerm may be added to the degree of matching Score, as shown in the following formula (1)'.

The penalty term PenaltyTerm in equation (1)' is derived using equation (3).

TotalArea_R is the total area of all text blocks that make up the partial layout.
In FIG. 13, it is the total area of text blocks 1003 to 1005 .

TotalArea_NonOverlappingQ is the total area of all the text blocks in the scanned image that exist within the partial pattern range and that do not overlap any of the text blocks that make up the partial layout. In the case of Figure 13, it is the total area of text blocks 1101, 1104, 1105, 1106, and 1109 that do not overlap with text blocks 1003 to 1005 out of the text blocks 1101, 1104, 1105, 1106, and 1109 in the partial pattern range 1006.

By providing a penalty term, it is possible to adjust the degree of match so that points are subtracted when a text block in the scanned image is present in an area where no text block constituting the partial layout in the partial pattern area 1006 was present. Therefore, even if there are few text blocks constituting the partial layout, the degree of match can be derived by utilizing information on areas within the partial pattern area where no text block constituting the partial layout is present. Note that the method of deriving the degree of match is not limited to derivation using the above formula, and it is sufficient if the degree of match with the partial layout can be determined.

In S804, the image processing unit 305 places a partial pattern (partial layout and partial pattern range) so that the index block is located at one of the Y candidate positions among the group of Y candidate positions determined in S803. The image processing unit 305 then scans the partial pattern in the X direction to derive the degree of match at each position. The image processing unit 305 performs this for all groups of Y candidate positions.

Figure 14 shows the processing of this step at one of the Y candidate positions determined in S803. In Figure 14(a), the solid rectangle represents text blocks 1003-1005 that make up the partial layout, and the dashed rectangle represents the partial pattern range 1006. The dashed rectangle represents text blocks 1101, 1105, and 1106 in the scanned image, and the shaded area represents the area where the text blocks in the partial layout and the text blocks in the scanned image overlap. Figure 14 shows the processing of this step proceeding in order from Figure 14(a) to (e), and shows how the partial pattern is scanned in the X direction (from left to right) within the search range and the degree of match at each position is derived. Similar processing is performed at each Y candidate position.

In S805, the image processing unit 305 determines the position at which the degree of match derived in S804 is greatest as the XY candidate position. For example, in the case of FIG. 14, the degree of match of the partial pattern (partial layout) is greatest at the position shown in FIG. 14(c). Therefore, the position of the text block 1003 indicating the index block included in the partial layout in FIG. 14(c) is determined as the XY candidate position.

In S806, the image processing unit 305 determines whether the degree of match at the XY candidate positions determined in S805 is equal to or greater than a predetermined threshold.

If the degree of match is equal to or greater than the threshold (YES in S806), in S807 the image processing unit 305 estimates the XY candidate position on the scanned image determined in S805 as the position of the text block (index block) of the item to be processed. The image processing unit 305 performs processing to estimate the index block of the item to be processed in the scanned image based on the estimated position.

For example, when the index block of an item to be processed in the registered document is shifted to a candidate XY position in the scanned image, it is determined whether the overlapping text block in the scanned image satisfies a specified condition. The specified condition is, for example, whether the overlap rate indicating the degree of overlap with the index block to be processed in the registered document is equal to or greater than a specified value, and whether the distance of the upper left coordinate with the index block to be processed in the registered document is within a certain range.

If it is determined that there is a text block that satisfies the specified condition (YES in S807), the process proceeds to S808. In S808, the image processing unit 305 estimates that the text block of the scanned image that is determined in S807 to satisfy the specified condition is a text block (index block) that contains a character string that indicates the item to be processed that was selected in S801.

If the degree of match is less than the threshold (NO in S806) or if it is determined that there is no corresponding text block (NO in S807), the process proceeds to S809. In S809, the image processing unit 305 determines that there is no text block in the scanned image that corresponds to the item to be processed selected in S801. For example, if a character string corresponding to the item to be processed is not written in a specified area in the scanned image, or if the position was incorrectly estimated in S804, a determination is made in S809.

In S810, the image processing unit 305 determines whether the process of estimating index blocks has been completed for all items registered in the index information of the registered document. If there are any unprocessed items, the process returns to S801.

If processing has been completed for all items, the process of this flowchart ends and proceeds to S508. In S508, the image processing unit 305 performs OCR processing on the index blocks of each estimated item and extracts character strings corresponding to each item as indexes.

As described above, in this embodiment, the index included in the scanned image is extracted by using a part of the layout of the text block. Therefore, according to this embodiment, even if the position of the index block included in the scanned image differs from that of the registered document due to a change in the contents of the input document, etc., the index can be extracted. In addition, in this embodiment, the type of input document is identified by document matching, and an extraction rule associated with the document type is used. Therefore, even in the process of estimating index blocks based on the partial layout of text blocks, erroneous extraction of indexes can be suppressed. Furthermore, in the document matching and index block estimation process, only the text blocks of the foreground region obtained as a result of preprocessing of the OCR process are used. Therefore, the index extraction process can be performed without incurring extra calculation costs.

<Embodiment 2>
In the first embodiment, a method for determining the partial pattern range based on a preset value has been described. However, if the partial pattern range is set too wide, the position of the index block cannot be estimated appropriately when the layout changes only around the index block. On the other hand, if the partial pattern range is small, the number of text blocks constituting the partial layout may be determined to be small, making it difficult to search for an area with a high degree of match in the scanned image. For this reason, in this embodiment, a method for determining the partial pattern range to an appropriate size will be described. Note that the present embodiment will be described mainly with respect to the differences from the first embodiment. The configuration and processing are the same as those of the first embodiment unless otherwise specified.

The number and layout of text blocks around an index block vary depending on the type of document. For this reason, in this embodiment, to determine the size of the partial pattern range, the area including the index block of the target item is gradually expanded and the number of text blocks that overlap with that area is counted. Then, when the number of overlapping text blocks reaches a certain number or more, the area is determined to be the partial pattern range of that item.

Figure 15 is a diagram for explaining a method of determining a partial pattern range in this embodiment. In Figure 15 (a), the solid line rectangle is the title index block 1000, and the dashed line rectangle is the area for determining the title partial pattern range. The areas are the initial area 1500, the second stage area 1501, and the maximum area 1502, respectively. Figure 15 (a) shows how the area for determining the partial pattern range in the title item is changed in stages. While gradually expanding the area from the initial area to the maximum area, text blocks are counted except for index blocks that overlap with the area. Then, when the counted text blocks reach a predetermined number or more, the area indicated by the dashed line rectangle is determined as the partial pattern range of the item. Note that the predetermined number is preferably 1 or more. In this embodiment, the predetermined number is described as 5.

The method for determining the partial pattern range in this embodiment will now be described in detail. First, the number of text blocks that overlap even slightly with the initial region 1500 is counted. In this case, there are no text blocks other than the index block 1000, so the process proceeds to the next step.

Next, the region is expanded, and text blocks that overlap even slightly with the second-stage region 1501 are counted. FIG. 15B shows the case where the region for determining the partial pattern range is the second-stage region 1501. As shown in FIG. 15B, text blocks 1001 and 1008 to 1013 overlap with the second-stage region 1501. Therefore, the number of text blocks that overlap with the second-stage region 1501 is counted as seven. It is then determined that the number of overlapping text blocks is equal to or greater than the predetermined number of five. Therefore, the position and size indicated by the second-stage region 1501 are determined as the partial pattern range of the title. Therefore, a layout consisting of the text blocks 1001 and 1008 to 1013, at least a portion of which is included in the partial pattern range, and the index block 1000 is determined as the partial layout of the title.

Alternatively, the number of surrounding text blocks varies depending on the item, and the areas where the layout of the text blocks changes little depending on the content written are different. For this reason, for example, the size of the partial pattern range may be made different depending on the attributes of the item. In other words, the size of the partial pattern may be set in advance depending on the attributes of the item.

When an item is a title, there is a characteristic that there are often no text blocks near the title text block. In addition, a title is characterized by being located at the top of a document where changes in the content of the document cause little change in the layout of the text block. For this reason, as shown in partial pattern range 1007 in Figure 10(c), if the item is the title of a document, an area that fits the entire image width in the X direction and approximately one-quarter of the image in the Y direction may be determined as the partial pattern range.

As described above, in this embodiment, the partial pattern range is determined according to the document. Therefore, the accuracy of the index block estimation process can be improved by using an appropriate partial pattern range according to the document.

<Embodiment 3>
In embodiment 1, a method was described in which the position at which the degree of match derived using a partial pattern is greatest is determined as the XY candidate position, and if the degree of match of the XY candidate position is equal to or greater than a predetermined threshold, the position of the index block of the item to be processed is estimated based on the XY candidate position.

However, an input document may contain multiple regions that contain text blocks whose layout is similar to the partial layout of a registered document. When an input document contains multiple regions whose partial layout is similar, the method of embodiment 1 may fail to estimate the index block of the item to be processed in the input document.

In this embodiment, therefore, a method for appropriately estimating the position of an index block in an input document will be described, even if the input document contains multiple areas similar to the partial layout of the item to be processed. Note that this embodiment will be described mainly focusing on the differences from embodiment 1. Parts that are not specifically mentioned have the same configuration and processing as embodiment 1.

Figure 16 is a flowchart for explaining the index block estimation process of S507 in this embodiment. Details of the index block estimation process in this embodiment will be explained with reference to the flowchart in Figure 16. S1600 to S1604 are the same as S800 to S804, so the explanation will be omitted.

In S1605, the image processing unit 305 determines the XY positions in the scanned image where the degree of match derived in S1604 is equal to or greater than a predetermined threshold. As a result of this step, there may be cases where multiple XY positions are not determined, but for convenience, the XY positions determined by this step are referred to as a group of XY candidate positions.

Figure 17 shows an example of a registered document in which there are multiple areas that are similar to a partial layout consisting of an index block and its surrounding blocks. Figure 17(a) shows an example of a registered document. Figure 17(b) shows a partial pattern when a text block 1705 containing a character string corresponding to the "Quotation Date" item in the registered document of Figure 17(a) is used as an index block. In Figure 17(b), the dashed-dotted rectangle indicates the partial pattern range 1700 of the "Quotation Date" item, and text blocks 1701 to 1706 represented by solid-line rectangles indicate text blocks that make up the partial layout of the "Quotation Date" item. In the explanation of the flowchart in Figure 16, the processing when "Quotation Date" is the item to be processed will be explained.

Figure 18 is a diagram for explaining an input document. Figure 18(a) is a diagram showing an input document, and in the explanation of this flowchart, it is assumed that the index block estimation process is performed on the scanned image obtained as a result of scanning this input document. Also, it is assumed that the document similar to the input document in Figure 18(a) is the registered document in Figure 17, which has been identified by the document matching in S504.

Figures 18(b) to (e) are diagrams in which the partial pattern of "Estimated Date" in Figure 17(b) is superimposed on an image showing the text blocks detected as a result of performing block selection processing on the scanned image of the input document in Figure 18(a). The rectangles in each of Figures 18(b) to (e) indicate partial patterns. That is, the solid-line rectangles are text blocks that make up the partial layout, and the dash-dotted-line rectangles are the range of the partial pattern.

The positions of the partial patterns shown in Figures 18(b) to (e) are the positions when the degree of match derived in S1604 is equal to or greater than a predetermined threshold. Therefore, among the text blocks represented by solid-line rectangles that make up the partial layout, the XY positions 1801 to 1804 of the index blocks are determined as a group of XY candidate positions as a result of the processing in this step.

In a document in which a simple text block arrangement is repeated, such as the input document shown in FIG. 18(a), if an index block exists among the repeated text blocks, multiple XY positions are determined where the degree of match is equal to or greater than the threshold. Therefore, for the input document shown in FIG. 18(a), the number of XY candidate position groups determined as a result of processing this step will be two or more.

In S1606, the image processing unit 305 switches processing depending on the number of XY candidate position groups determined in S1605. If the number of XY candidate position groups is 1, the process proceeds to S1610, and if the number of XY candidate position groups is 0, the process proceeds to S1612. The process of S1612 is the same as S809, so a description thereof will be omitted.

If the number of XY candidate position groups is two or more, proceed to S1607. In S1607, the image processing unit 305 acquires a similar position (group) that is a position in the registered document where the degree of match with the partial layout of the item to be processed is equal to or greater than a predetermined threshold.

The degree of match of the text block is derived by superimposing a partial layout included in the partial pattern of the item being processed onto a position in the registered document, and the XY position in the registered document where the degree of match is equal to or greater than a predetermined threshold is determined to be the "similar position." The degree of match between the text block in the registered document and the text block in the partial layout can be calculated in the same manner as in S1602 to S1604. In other words, instead of deriving the degree of match for the input document, the same procedure can be used to derive the degree of match for the registered document.

Figure 19 is a diagram for explaining similar positions within a registered document. Figure 19(a) is a diagram showing the same registered document as Figure 17(a). Figures 19(b) to (e) are each diagrams in which the partial pattern of "Estimated Date" from Figure 17(b) is superimposed on an image showing a text block detected as a result of performing block selection processing on the scanned image of the registered document from Figure 19(a). The rectangles in each of Figures 19(b) to (e) indicate partial patterns. That is, the solid-line rectangles are text blocks that make up the partial layout, and the dashed-dotted rectangles are the partial pattern ranges.

The positions of the partial patterns in Figures 19(b) to (e) are the respective positions when the derived degree of match is equal to or greater than a predetermined threshold. For this reason, the XY positions of the index blocks among the text blocks that make up the partial layout are determined as similar position groups 1901 to 1904. In this step, position information of the similar position group of the item to be processed is obtained. The similar position groups 1901 to 1904 also include the XY position of the index block 1705 at the time of registration shown in Figure 17(b), as in similar position 1902.

Note that it is not necessary to perform processing to determine similar positions within the registered document in S1607. For example, when registering a document, a partial pattern may be determined for each item, and then a group of similar positions may be determined, and information about the group of similar positions may be stored in advance as part of the extraction rules shown in FIG. 7. In other words, in S1607, it is sufficient to obtain the group of similar positions as one of the extraction rules for the stored item to be processed.

In S1608, the image processing unit 305 associates the group of similar positions in the registered document acquired in S1607 with the group of XY candidate positions in the input document determined in S1605. Specifically, for the group of similar positions sorted by Y position, the image processing unit 305 associates the group of XY candidate positions sorted under the same conditions as the group of similar positions in order starting from one side of the Y position, and then associates them in order starting from the other side of the Y position.

Figure 20 is a diagram to explain the processing of this step. The numbers in the table are the numbers indicating the symbols that indicate the positions within the document shown in Figure 18 or Figure 19.

Figure 20(a) is a diagram showing correspondence when the number of similar position groups and XY candidate position groups is the same as shown in Figures 18 and 19. Column 2001 is a group of similar positions sorted by Y position. Column 2002 is a group of XY candidate positions sorted by Y position, and is a group of XY candidate positions associated with the group of similar positions in column 2001 in order from the top of the Y position. Column 2003 is a group of XY candidate positions sorted by Y position, and is a group of XY candidate positions associated with the group of similar positions in column 2001 in order from the bottom of the Y position. In Figure 20(a), the group of XY candidate positions in column 2002 and the group of XY candidate positions in column 2003 are each associated with the same similar positions.

Fig. 20B is a diagram for explaining the method of matching in this step when the number of similar position groups is small compared to the number of XY position groups. For example, Fig. 20B shows the matching when the degree of match with the registered document when a partial pattern is superimposed on the position of the registered document shown in Fig. 19E is less than the threshold, and only similar position groups 1901 to 1903 are acquired in S1607. Column 2011 is a group of similar positions sorted by Y position. Column 2012 is a group of XY candidate positions that are matched in order from the top of the Y position to the group of similar positions in column 2011. Column 2013 is a group of XY candidate positions that are matched in order from the bottom of the Y position to the group of similar positions in column 2011. In Fig. 20B, the XY candidate position groups that correspond to the similar position groups are different between the matching from above and the matching from below.

Figure 20(c) is a diagram for explaining the method of matching in this step when the number of similar position groups is large compared to the number of XY candidate position groups. Figure 20(c) shows the matching when the degree of match with the input document when a partial pattern is superimposed on the position of the input document shown in Figure 18(e) is less than a threshold, and only XY positions 1801 to 1803 are determined as the XY candidate position group in S1605. Column 2021 is a group of similar positions sorted by Y position. Column 2022 is a group of XY candidate positions that are matched to the similar position group of column 2021 in order from the top of the Y position. Column 2023 is a group of XY candidate positions that are matched to the similar position group of column 2021 in order from the bottom of the Y position. Matching from above and matching from below will produce different results; matching from above will not find a group of XY candidate positions that correspond to similar position 1904, and matching from below will not find a group of XY candidate positions that correspond to similar position 1901.

In S1609, the image processing unit 305 determines one XY candidate position from the group of XY candidate positions determined in S1605 based on the result of the matching performed in S1608.

As shown in FIG. 20(a), the results of the matching performed in S1608 may match the results of the matching from above and the matching from below. In this case, from among the group of XY candidate positions, an XY position that is associated with a similar position that indicates the position of the index block at the time of registration is determined as one XY candidate position. In the example of FIG. 20(a), XY position 1802 that is associated with similar position 1902 that indicates the position of the index block is determined as one XY candidate position.

On the other hand, as shown in Figures 20(b) and (c), the results of the matching performed in S1608 may not match between the results of matching from above and the results of matching from below. In this case, first, the XY positions of the input document that are associated with the similar position indicating the position of the index block when matching is performed from above are determined. Furthermore, the XY positions of the input document that are associated with the similar position indicating the position of the index block when matching is performed from below are determined.

In the example of FIG. 20(b), XY position 1802 and XY position 1803 are determined to be associated with similar position 1902 indicating the position of the index block. In the example of FIG. 20(C), XY position 1802 and XY position 1801 are determined to be associated with similar position 1902. Then, of the two determined XY positions, the one with the higher degree of match derived in S1604 is determined as one XY candidate position in the XY candidate position group. Note that one XY position may be selected from one of two XY positions without using the degree of match. For example, two XY positions may be displayed and an instruction from the user may be received, and the user may remember for each item whether to use the correspondence from above or the correspondence from below.

When one XY candidate position is determined from the group of XY candidate positions, the process proceeds to S1610. In S1610, similar to the process of S807, the XY candidate position is estimated as the position of the index block to be processed, and the index block of the item to be processed is estimated from the text block of the scanned image. S1611 is the same as S808, and S1613 is the same as S810, so their explanations are omitted.

As described above, in this embodiment, when there are multiple XY candidate positions in an input document whose degree of match is equal to or greater than a threshold, one XY candidate position is determined after associating it with a group of similar positions in a registered document whose degree of match with the partial pattern is equal to or greater than a threshold. Therefore, the accuracy of the index block estimation process can be improved even when there are multiple areas in a document that are similar to a partial layout consisting of an index block and its surrounding text blocks.

<Other embodiments>
In the above embodiment, an example has been described in which the image forming apparatus 100 performs the processes of the steps in the flowchart of Fig. 4 by itself. Alternatively, all or part of these processes may be performed by another image processing apparatus on the system 105 having the functions of Fig. 3.

For example, the scan process is executed in the image forming device 100, and the scanned image is sent to the terminal 101 via the network. The terminal 101 may have a function similar to that of the image processing unit 305, and may execute the index extraction process in the terminal 101. In this case, the terminal 101 returns the index extraction result to the image forming device 100, and the image forming device 100 generates and sends a file based on the acquired index extraction result.

The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-mentioned embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions.

100 Image forming apparatus 305 Image processing unit 111 CPU

Claims (19)

a detection means for detecting text blocks in an input image;
A specifying means for specifying a registered document corresponding to the input image from among a plurality of registered documents;
a determining means for determining a text block in the input image corresponding to the item to be processed based on a partial layout defined in the specified registered document, the partial layout including a first text block corresponding to the item to be processed and at least one second text block existing in the vicinity of the first text block;
an acquisition means for acquiring a character string corresponding to the item to be processed in the input image by performing character recognition processing on the determined text block;
13. An image processing device comprising: The determining means is
The image processing device according to claim 1, characterized in that the determination is made by superimposing the partial layout at any position in a search range in the input image and deriving a degree of match based on the size of an area where a text block included in the partial layout overlaps with a text block in the input image. The determining means is
The image processing apparatus according to claim 2 , wherein the degree of match is derived within the search range, the search range being a predetermined area including a position in the input image corresponding to a position of the first text block in the specified registered document. The determining means is
4. The image processing apparatus according to claim 2, further comprising: deriving the vertical positions for determination based on a difference in vertical position between a text block included in the partial layout and a text block within the search range. The determining means is
The image processing apparatus according to claim 4 , wherein the degree of coincidence is derived at each position when the partial layout is superimposed in a horizontal direction on the group of vertical positions within the search range. The determining means is
The image processing device according to claim 2 , further comprising: deriving a position in the input image where the degree of matching is equal to or greater than a threshold and where the degree of matching is maximum as the position in the input image for making the determination. The determining means is
deriving candidate positions in the input image where the degree of match is equal to or greater than a threshold;
if the number of candidate positions is one, deriving the candidate position as a position in the input image for making the determination;
The image processing device according to claim 5, characterized in that when the number of candidate positions is two or more, a position in the identified registered document where the degree of similarity derived by the same method as the method of deriving the degree of similarity when a text block included in the partial layout is superimposed on any position in the identified registered document is equal to or greater than a threshold value is obtained as a similar position, and a position in the input image for making the decision is derived by matching the candidate positions with the similar positions. The determining means is
When the number of the candidate positions is two or more and the number of the candidate positions is the same as the number of the similar positions,
The image processing device according to claim 7, characterized in that, as a result of associating the candidate positions and the similar positions arranged under the same conditions in order from one side, the candidate position associated with the similar position corresponding to the first text block is derived as a position in the input image for making the determination. When the number of the candidate positions is two or more and the number of the candidate positions is different from the number of the similar positions,
As a result of associating the candidate positions and the similar positions arranged under the same conditions in order from one side, a first position indicated by the candidate position associated with the similar position corresponding to the first text block;
determining a second position indicated by the candidate position associated with the similar position corresponding to the first text block as a result of associating the candidate position and the similar position arranged under the same condition in order from the other side;
9. The image processing apparatus according to claim 7, wherein one of the first position and the second position, which satisfies a predetermined condition, is derived as a position in the input image for making the determination. The determining means is
10. The image processing device according to claim 6, further comprising: placing the first text block in the identified registered document at the position in the input image derived based on the degree of similarity; and, if a text block in the input image that overlaps with the placed text block satisfies a predetermined condition, determining that the overlapping text block is the text block corresponding to the item to be processed in the input image. The predetermined condition is:
11. The image processing apparatus according to claim 10, wherein the degree of overlap between the placed text block and the overlapping text block is equal to or greater than a predetermined value, and the distance between the vertices is within a certain range. a predetermined range is set based on the first text block in the specified registered document;
The degree of agreement is
The image processing device according to any one of claims 2 to 11, characterized in that, when the partial layout is superimposed on the input image, the degree of matching is adjusted to decrease as the area of text blocks included in the specified range in the input image that do not overlap with text blocks included in the partial layout increases. 13. The image processing device according to claim 1, wherein the partial layout includes the first text block in the identified registered document and the second text block included in a predetermined range based on the first text block in the identified registered document. The predetermined range is
The image processing device according to claim 12 or 13, characterized in that in the identified registered document, the region is based on the first text block and is an area including a predetermined number or more of text blocks. The identification means is
15. The image processing device according to claim 1, further comprising: a processor for processing the input image based on a layout of the input image and a layout of the text blocks in the input image; a processor for processing the input image and a layout of the text blocks in the input image; 16. The image processing apparatus according to claim 1, further comprising a registration unit that registers the input image as a new document when the specification unit cannot specify a registered document corresponding to the input image. The image processing device according to claim 1 , further comprising a setting unit that sets a property of the input image based on a character string corresponding to the item to be processed in the input image acquired by the acquisition unit. a detection step for detecting text blocks in the input image;
a step of identifying a registered document corresponding to the input image from among a plurality of registered documents;
a determining step of determining a text block in the input image corresponding to the item to be processed based on a partial layout defined in the identified registered document, the partial layout including a first text block corresponding to the item to be processed and at least one second text block existing in the vicinity of the first text block;
an acquisition step of acquiring a character string corresponding to the item to be processed in the input image by performing a character recognition process on the determined text block;
13. An image processing method comprising the steps of: A program for causing a computer to function as each of the means of an image processing device according to any one of claims 1 to 17.

Images