JP7790183B2 - Character matching device, program, and enclosure confirmation system - Google Patents

Description

The present invention relates to a character matching device, a program, and an enclosure verification system.

Traditionally, the task of enclosing multiple documents, such as notices and statements to be sent to the same person, into an envelope is common. Since documents contain information that can identify an individual, such as an address or name, accidentally enclosing documents with a different destination can lead to the leakage of personal information. Therefore, when enclosing documents, it is necessary to confirm that the documents being enclosed are all for the same person. There are various initiatives to assist in this confirmation process.
For example, some documents enclosed in the same envelope are printed with an optically readable code, such as a two-dimensional code or barcode, to confirm that they are addressed to the same person. However, there are cases where printing an optical code on a form is not possible due to reasons such as placing importance on design.

So, for example, "a system is provided which is at least comprised of a camera device for photographing forms, a form inspection device for inspecting forms to be enclosed in the same envelope, and a data storage device equipped with an inspection history database for storing the inspection history of forms, the form inspection device having a character recognition unit for character recognition of characters included in the image, and a form inspection unit for inspecting forms to be enclosed in the same envelope, and the form inspection unit performs an image acquisition process in which, when an image of a form number photographed by a user, consisting of at least a serial number and a branch number, is output from the camera device, the image of the form number output from the camera device is stored in memory, and The document inspection system is characterized by: when it is indicated that photographing of the document numbers for all documents to be inserted has been completed, the character recognition unit performs character recognition on all images of the document numbers stored in memory, and performs at least a cross-check to confirm that the serial numbers of all documents to be inserted in the same envelope match; and, once the document inspection process is complete, a history storage process is performed to store the images of the document numbers stored in memory in the inspection history database of the data storage device, in association with the results of the cross-check (see, for example, Patent Document 1).

Japanese Patent Application Laid-Open No. 2016-4365

Optical Character Recognition (OCR), for example, is used to recognize characters contained in images, including those described in Patent Document 1. However, there are limits to the accuracy of OCR, and there is a risk of misrecognition. There is a particular risk of misrecognition when the character to be recognized is surrounded by a frame or when other characters are present. There are also cases where hair, wrinkles in the paper, etc. are mistakenly recognized as numbers.

The present invention therefore aims to provide a character matching device, program, and enclosure verification system that are designed to compensate for the accuracy of OCR.

The present invention solves the above problems by the following means.
A first invention is a character matching device for matching multiple types of documents having the same identification character information, comprising: recognition processing means for performing character recognition processing on images obtained by photographing the documents; character matching means for comparing and matching character strings corresponding to the multiple documents; image similarity calculation means for comparing images of areas to be matched between the documents and calculating image similarity; determination means for determining whether the identification character information matches based on the matching result by the character matching means and the image similarity calculated by the image similarity calculation means; and determination result output means for outputting the determination result by the determination means to a display unit.
A second invention is the character matching device of the first invention, wherein the image similarity calculation means compares images of characters in the area to be matched between documents to calculate image similarity.
A third invention is a character matching device according to the first or second invention, further comprising an area specification means for specifying an area to be matched from the character string obtained by the recognition processing means, and the character matching means compares and collates each character string corresponding to the area to be matched specified by the area specification means.
A fourth invention is a character matching device according to the first or second invention, wherein the area to be matched is stored for each document, and the device is provided with a type recognition means for analyzing an image of the document to recognize the type of the document, and the character matching means compares and matches each character string corresponding to the area to be matched for each document based on the type of document recognized by the type recognition means.
A fifth invention is a character matching device according to any one of the first to fourth inventions, further comprising: a preprocessing means for preprocessing an image of the document; and a post-preprocessing recognition processing means for performing character recognition processing on the image preprocessed by the preprocessing means, wherein the character matching means performs matching using each character string corresponding to the plurality of documents obtained by the recognition processing means and each character string corresponding to the plurality of documents obtained by the post-preprocessing recognition processing means.
A sixth aspect of the present invention is the character matching device according to the fifth aspect of the present invention, wherein the preprocessing means performs a plurality of types of preprocessing on an image obtained by photographing the document.
A seventh invention is a character matching device according to any one of the first to sixth inventions, further comprising an output character string acquisition means for acquiring an output character string, and the determination result output means further outputs the output character string acquired by the output character string acquisition means and an image of each area to be matched.
An eighth aspect of the present invention is a program for causing a computer to function as any one of the character matching devices according to the first to seventh aspects of the present invention.
A ninth invention is an enclosure confirmation system including a character matching device of any one of the first to seventh inventions, and is equipped with a placement stand on which a plurality of the enclosures, which are documents, can be placed side by side, a photographing unit capable of photographing the placement stand, an image acquisition means for acquiring, via the photographing unit, a photographed image including an image of the plurality of documents placed on the placement stand, and an image extraction means for extracting an image of each document from the photographed image acquired by the image acquisition means.
A tenth invention is an enclosure confirmation system according to the ninth invention, which includes a weight detection unit provided in the placement area of each enclosure on the placement table, a weight memory unit that stores the weight of each type of enclosure, a weight confirmation means that confirms whether the weight of each enclosure detected by the weight detection unit matches the weight of each enclosure stored in the weight memory unit, and an alarm means that issues an alarm when the weight confirmation means confirms that the weights do not match.

The present invention provides a character matching device, program, and enclosure verification system that are designed to compensate for the accuracy of OCR.

1 is a diagram showing an overview of an enclosure confirmation system according to an embodiment of the present invention; 1 is a diagram showing functional blocks of an enclosure checking device according to an embodiment of the present invention; FIG. 2 is a diagram illustrating an example of a storage unit of the enclosure checking device according to the embodiment. 6 is a flowchart showing an enclosure checking process in the enclosure checking device according to the present embodiment. 10 is a flowchart showing a verification area specifying process in the enclosure check device according to the present embodiment. 10 is a flowchart showing a character matching process in the enclosure checking device according to the embodiment. FIG. 4 is a diagram showing an example of a screen output to a monitor according to the embodiment. 5A and 5B are diagrams for explaining the processing in the enclosure checking device according to the embodiment; FIG. 10 is a diagram showing an example of a determination result screen output to a monitor according to the embodiment. FIG. 10 is a diagram showing an example of a determination result screen output to a monitor according to the embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, this is merely an example, and the technical scope of the present invention is not limited to this example.
(Embodiment)
<Enclosure confirmation system 100>
FIG. 1 is a diagram showing an outline of an enclosure confirmation system 100 according to this embodiment.
The enclosure confirmation system 100 is used, for example, by a contracted service company that has received a request from a business partner in the process of enclosing multiple documents (enclosures) into envelopes to be sent to the business partner's customers, and is used to confirm the enclosing of multiple documents for the same person.

1, in the enclosure confirmation system 100, an operator P takes one sheet from each of stacks of documents 51 to 54 placed in a document storage area to the right of the operator P, and places the sheets on a tray 40 in front of the operator P in the order in which the documents are placed in the document storage area. In the example of FIG. 1, stacks of documents 51 to 54, in which four types of documents are stacked in the order in which they were printed, are placed in the document storage area.
Here, the documents are, for example, notices, statements, etc. sent to customers by business partners such as financial institutions such as banks and credit card companies, or insurance companies such as life insurance and non-life insurance companies. Each document is assigned key information (identification character information) for identifying the customer. The key information is, for example, at least a part of a cash card or credit card number, or an insurance policy number. Furthermore, the documents are printed based on data received from business partners, and the documents in the stacks 51 to 54 are all stacked in the same order. In other words, the stacks 51 to 54 are all in the order of customer X, customer Y, etc.

The table 40 is covered with a single-colored cloth or paper material, for example, green. If the table 40 is green, it has the advantage of being able to use a green background (background) during image processing, making it easier to extract images of objects placed on the table 40. Note that the table 40 may be blue instead of green. The table 40 also has placement areas 41 to 44 for placing documents, which are visible to the worker P. The worker P places the documents 51a to 54a in the placement areas 41 to 44 so that each document fits within each of the placement areas 41 to 44.

Then, when worker P operates switch 8 on the platform 40, camera 5 (photographing unit) photographs the platform 40. A confirmation result, including the captured image, as to whether it is of the same person is then displayed on monitor 4 (display unit). Switch 8 is an operating member for acquiring images via camera 5. Switch 8 is located on the front right side of platform 40 as seen by worker P, and can be operated by worker P's right hand. Switch 8 may also be located in a position where it can be operated with worker P's left hand. Camera 5 is pre-adjusted and positioned so that it can photograph platform 40.

If worker P is able to confirm from the confirmation results displayed on monitor 4 that the documents belong to the same person, he or she picks up documents 51a to 54a placed on table 40 again, stacks documents 51a to 54a in order, and places the documents in an empty envelope 55 in the envelope area to the left of worker P. Through this series of operations by worker P, multiple documents 51a to 54a for the same person are placed and sealed in a single envelope 55.

<Enclosure checking device 1>
Next, the functional blocks of the enclosure verification device 1 (character matching device) used in the enclosure verification system 100 to verify whether documents belong to the same person will be described.
FIG. 2 is a diagram showing functional blocks of the enclosure checking device 1 according to this embodiment.
FIG. 3 is a diagram showing an example of the storage unit 30 of the enclosure checking device 1 according to this embodiment.
The enclosure checking device 1 is a device for checking that a plurality of documents 51a to 54a placed on the placing table 40 shown in FIG. 1 are to be sent to the same person.
As shown in FIG. 2, the enclosure checking device 1 includes a control unit 10, a storage unit 30, a monitor 4, a camera 5, a speaker 6, weight sensors 71 to 74 (weight detection units), and a switch 8.

The control unit 10 is a central processing unit (CPU) that controls the entire enclosure checking device 1. The control unit 10 appropriately reads and executes the operating system (OS) and application programs stored in the storage unit 30, thereby cooperating with the above-mentioned hardware and performing various functions.
The control unit 10 includes a start acceptance unit 11, an image acquisition unit 12 (image acquisition means), an image cutting unit 13 (image cutting means), a matching area identification unit 14 (type recognition means, area identification means), a character recognition unit 15 (recognition processing means), a preprocessing unit 16 (preprocessing means), a post-preprocessing character recognition unit 17 (post-preprocessing recognition processing means), a character matching unit 18 (character matching means), an output character string acquisition unit 19 (output character string acquisition means), an image similarity calculation unit 20 (image similarity calculation means), a character determination unit 21 (determination means), a weight confirmation unit 22 (weight confirmation means), and a confirmation result output unit 23 (determination result output means, notification means).

The start receiving unit 11 receives an operation of the switch 8 by the worker P, thereby receiving a start of processing.
In response to the reception by the start reception unit 11 , the image acquisition unit 12 acquires a photographed image of the mounting table 40 via the camera 5 .
The image cutting unit 13 cuts out a document image (a photographed image of a document) that is an image of the document portion from the captured image acquired by the image acquisition unit 12. As described above, the mounting table 40 can detect the outline of the document, which is the document area, from the green screen background using color information, and cut out the document image. The image cutting unit 13 may also perform keystone correction on the cut-out document image.

The comparison area identification unit 14 recognizes the type of document by analyzing the document image cut out by the image cutout unit 13. The comparison area identification unit 14 may recognize the type of each document image using technology such as CNN (Convolutional Neural Network). Furthermore, instead of identifying documents by analyzing the document images, documents may be identified by previously associating the position of the document image relative to the entire captured image with the document type.
The verification area specification unit 14 then specifies a verification area (area to be verified) from the document image extracted by the image extraction unit 13. The verification area is an area that contains key information for identifying the customer to be verified. For example, the verification area specification unit 14 can specify the verification area by registering the identification area as a coordinate area for each document in advance.

The character recognition unit 15 obtains a character string by performing OCR processing (character recognition processing) on the collation area identified by the collation area identification unit 14.
The preprocessing unit 16 performs preprocessing on the matching region identified by the matching region identification unit 14. The preprocessing unit 16 can perform processing using a variety of different preprocessing techniques. Examples of preprocessing techniques include sharpening, brightness conversion, thinning, and color extraction. Sharpening refers to emphasizing the edges of an image, resulting in a sharp image. Brightness conversion refers to gamma correction, which can emphasize low-brightness areas. Thinning refers to the process of thinning a line in a binarized image so that only one pixel at the center of the line remains, and calculating the center line. Color extraction refers to the extraction of representative colors with high appearance frequencies, for example, by checking the histogram of the image. For example, if a red character image is present, the red character image can be extracted by color extraction.
The preprocessing character recognition unit 17 performs OCR processing on the collation region preprocessed by the preprocessing unit 16 to obtain a character string.

The character matching unit 18 compares the character strings obtained by the character recognition unit 15 with the character strings obtained by the preprocessing character recognition unit 17. For example, the character matching unit 18 may compare character strings obtained by the character recognition unit 15 for the same matching area, and may also compare character strings obtained by the preprocessing character recognition unit 17. Furthermore, for example, when multiple preprocessing methods are used, the character matching unit 18 may compare multiple character strings obtained by the preprocessing character recognition unit 17 for each preprocessing method, or may compare all character strings obtained by the preprocessing character recognition unit 17 regardless of the preprocessing method.

The output string acquisition unit 19 acquires an output string from multiple character strings. For example, the output string acquisition unit 19 may acquire a character string obtained by the character recognition unit 15 as the output string. Alternatively, the output string acquisition unit 19 may acquire, for example, the character string with the highest degree of match as a result of matching by the character matching unit 18 as the output string.

The image similarity calculation unit 20 compares the images of each character in the matching area identified by the matching area identification unit 14 across multiple documents to calculate the image similarity for each character. Image similarity can be calculated using techniques such as feature point matching using the AKAZE (Accelerated KAZE) algorithm or template matching. All of these techniques are based on the assumption that identical characters will have similar image features (structural information), and the same characters will have a high calculated image similarity.

The character determination unit 21 determines whether the key information matches based on the matching result by the character matching unit 18. The character determination unit 21 also determines whether the key information matches based on the matching result by the character matching unit 18 and the image similarity calculated by the image similarity calculation unit 20. The character determination unit 21 may determine that the key information matches, for example, if the image similarity calculated by the image similarity calculation unit 20 is equal to or greater than a threshold. Here, the threshold may be determined statistically, for example, based on the distribution of image similarities over the past month.

In the weight confirmation unit 22, weight sensors 71 to 74 detect the weight in response to acceptance by the start acceptance unit 11. Then, the weight confirmation unit 22 checks whether the detected weight of the document matches the weight of each document stored in the document storage unit 32 (weight storage unit) described later.
The confirmation result output unit 23 outputs the determination result by the character determination unit 21 to the monitor 4. The confirmation result output unit 23 may further add the output character string acquired by the output character string acquisition unit 19 and the image of each matching area to the determination result and output it.
Furthermore, when the weight confirmation unit 22 confirms that the weights do not match, the confirmation result output unit 23 notifies the user of the weight mismatch, for example, on the monitor 4. Note that when the confirmation result output unit 23 confirms that the weights do not match, it may also notify the user of the weight mismatch by sounding an alarm from the speaker 6, for example.
Details of each process will be described later.

The storage unit 30 is a storage area such as a hard disk or semiconductor memory element for storing programs, data, etc. required for the control unit 10 to execute various processes.
Here, a computer refers to an information processing device equipped with a control unit, a storage device, etc., and the enclosure checking device 1 is an information processing device equipped with a control unit 10, a storage unit 30, etc., and is included in the concept of a computer.
The storage unit 30 includes a program storage unit 31 , a document storage unit 32 , and an enclosing order storage unit 33 .

The program storage unit 31 is a storage area for storing various programs. The program storage unit 31 stores programs (not shown) for performing various functions executed by the control unit 10 of the enclosure check device 1.
The document storage unit 32 is a storage area that stores information about a document, including the document's weight. Fig. 3A shows example items in the document storage unit 32. The document storage unit 32 shown in Fig. 3A stores a document ID (IDentification), an image, a document name, key area coordinates, and a weight in association with each other.
The document ID is identification information that identifies the type of document, and a different ID is assigned to each of document A, document B, document C, . . .
The image is an image of the document, and the document name is the name of the document.
The key area coordinates are the coordinates of the key information within the document.
The weight is the weight per enclosed document, and is expressed in units of mg (milligrams), for example.

The enclosing order storage unit 33 is a storage area for storing, for each job requested by a client, the stacking order of documents to be enclosed and a document ID corresponding to the position on the placing table 40 in association with each other. Fig. 3B shows an example of items in the enclosing order storage unit 33. The enclosing order storage unit 33 shown in Fig. 3B stores, for each job ID, the stacking order, the location, and the document ID in association with each other.
The task ID corresponds to a request from a business partner, and one task ID is assigned to each request.
The stacking order is the order in which the enclosed documents are stacked.
The location has a name and a position on the table 40. The name is the item that the KEY is stacked on top of, for example, a document that can be seen through a window in an envelope. SUB1 to SUB3 are names corresponding to the stacking order.
The document ID is identification information that identifies the type of document.
Additionally, the storage unit 30 may include, for example, a customer information storage unit that stores customer information and task IDs, but detailed description thereof will be omitted.

1, the monitor 4 is provided, for example, at the rear side of the table 40 as seen from the worker P, in a position that is easily visible to the worker P who places documents on the table 40. The monitor 4 is a display device that functions as a display unit and is configured, for example, with a liquid crystal panel or the like.
The camera 5 is an imaging device and is provided vertically above the mounting table 40 with a lens (not shown) facing the mounting table 40, and captures an image of the mounting table 40.
The speaker 6 is an audio output device and is provided, for example, near the monitor 4 .
Weight sensors 71 to 74 are sensors provided on the loading platform 40 to detect weight. Weight sensors 71 to 74 detect the weight of documents placed on them, respectively. Weight sensor 71 is provided, for example, near the center of loading area 41. Like weight sensor 71, weight sensors 72 to 74 are also provided near the centers of loading areas 42 to 44, respectively.
The switch 8 is a button that the worker P operates.

<Processing Description>
Next, the process performed by the enclosure checking device 1 will be described.
FIG. 4 is a flowchart showing the enclosure checking process in the enclosure checking device 1 according to this embodiment.
FIG. 5 is a flowchart showing the verification area specifying process in the enclosure checker 1 according to this embodiment.
FIG. 6 is a flowchart showing the character matching process in the enclosure check device 1 according to this embodiment.
FIG. 7 is a diagram showing an example of a screen 60 output to the monitor 4 according to this embodiment.
FIG. 8 is a diagram for explaining the processing in the enclosure checking device 1 according to this embodiment.
9 and 10 are diagrams showing examples of a determination result screen 90 output to the monitor 4 according to this embodiment.

As a preliminary operation, for example, the worker P registers information about the document to be collated in the document storage unit 32 .
Next, the worker P places the documents of the various types to be enclosed in one envelope on the placing table 40 in order, as described with reference to FIG.

4, the control unit 10 (start reception unit 11) of the enclosure inspection device 1 determines whether or not it has received operation of the switch 8. When the worker P places documents corresponding to each area in the placement areas 41 to 44 of the placement table 40 and presses the switch 8, the control unit 10 determines that it has received operation of the switch 8. If it has received operation of the switch 8 (S11: YES), the control unit 10 proceeds to S12. On the other hand, if it has not received operation of the switch 8 (S11: NO), the control unit 10 remains in this process until it receives operation of the switch 8.
In S12, the control unit 10 performs a weight detection process. More specifically, in response to receiving a switch operation in the process of S11, the control unit 10 causes the weight sensors 71 to 74 to detect the weights, respectively. Then, the control unit 10 temporarily stores the detected weights in the memory unit 30.

In S13, the control unit 10 performs a matching area specification process.
The collation area specification process will now be described with reference to FIG.
5, the control unit 10 (image acquisition unit 12) captures an image of the table 40. In the example of FIG. 1, the captured image includes four different documents 51a to 54a.
In S<b>22 , the control unit 10 outputs the acquired photographed image to the monitor 4 .
FIG. 7 is an example of a screen 60 output to the monitor 4.
The screen 60 includes document areas 61 to 64 corresponding to the placement areas 41 to 44 shown in FIG.
The worker P can confirm by looking at the screen 60 of the monitor 4 that the camera 5 has taken an image by operating the switch 8 himself.

In S23, the control unit 10 (image cutout unit 13) cuts out a document image from the captured image. The control unit 10 may perform trapezoidal correction on the cut-out document image to make it rectangular.
In S24, the control unit 10 (matching area identification unit 14) identifies a matching area for each cut-out document image. The control unit 10 identifies the position of the matching area for each document by referring to the document storage unit 32, and then identifies the matching area. For example, image 81A shown in FIG. 8A is an image in which the matching area of document 51a has been identified, and image 81B shown in FIG. 8B is an image in which the matching area of document 52a has been identified. The control unit 10 then proceeds to S14 in FIG. 4.
In S14 of FIG. 4, the control unit 10 performs character matching processing.

The character matching process will now be described with reference to FIG.
In S31 of FIG. 6, the control unit 10 (character recognition unit 15) performs OCR processing on each collation area to obtain characters contained in each collation area.
Character string 81X shown in Fig. 8(A) is a character string obtained from image 81A of the collation area. Here, in Fig. 8(A), the character string in image 81A and character string 81X are identical. On the other hand, character string 81Y shown in Fig. 8(B) is a character string obtained from image 81B of the collation area. In Fig. 8(B), the character string in image 81B and character string 81Y are not identical. This is because when OCR processing was performed on image 81B, the line to the left of the number 9 was recognized as the number 1.
In this way, in OCR processing, erroneous recognition occurs depending on the accuracy of the OCR.

In S32 of FIG. 6, the control unit 10 (preprocessing unit 16) performs preprocessing on each matching region.
Images 82A to 85A shown in Fig. 8A are images obtained by performing sharpening, brightness conversion, thinning, and representative color extraction on image 81A, respectively. The same applies to images 82B to 85B shown in Fig. 8B.

In S33 of FIG. 6, the control unit 10 (post-preprocessing character recognition unit 17) performs OCR processing on each preprocessed collation area, and obtains the character string included in each collation area.
Character strings 82X to 85X shown in Fig. 8(A) are character strings obtained from images 82A to 85A of the matching region, respectively. Character strings 82Y to 85Y shown in Fig. 8(B) are character strings obtained from images 82B to 85B of the matching region, respectively.

6, the control unit 10 (character matching unit 18) matches each of the acquired character strings. More specifically, the control unit 10 matches the character strings acquired in the process of S31 for the same matching region.
The control unit 10 also compares the character strings obtained in the process of S33 with each other, and the control unit 10 (character determination unit 21) determines whether the key information matches.
If the comparison of the character strings obtained in the process of S31 shows that the character strings are the same, the control unit 10 determines that they match, and further compares the character strings obtained in the process of S33 to determine whether they match. On the other hand, if the comparison of the character strings obtained in the process of S31 shows that the character strings are not the same, the control unit 10 determines that they do not match.

Here, various methods can be used to compare the multiple character strings obtained by OCR processing after multiple preprocessing steps.
One method is to compare OCR results that have undergone the same preprocessing. In this case, the success rate of the comparison for each preprocessing is used as the confidence level, and if the confidence level is 100%, the control unit 10 determines that the results match. If the confidence level is other than 100%, the control unit 10 outputs the determination result and performs processing such as issuing an alert, as described below.

For example, the control unit 10 checks whether a character string recognized by OCR processing after sharpening of document 51a is identical to a character string recognized by OCR processing after sharpening of document 52a, which is a document by the same person. Similarly, the control unit 10 checks whether a character string recognized by OCR processing after brightness conversion of document 51a is identical to a character string recognized by OCR processing after brightness conversion of document 52a, which is a document by the same person. The control unit 10 then determines the number of matches out of the number of matches as the confidence level. The number of matches, i.e., if there are four types of preprocessing methods and three matches are identical, the confidence level is 3/4 (= 0.75).

Another method may be to simply compare the frequency of appearance of each number. The control unit 10, for example, tallies the number of times each number appears for each document. The control unit 10 then calculates the difference in the number of times each number appears for each number, and determines the average of the absolute values of the differences as the mismatch rate. If the mismatch rate is 0, the control unit 10 determines that there is a match. If the mismatch rate is other than 0, the control unit 10 performs processing such as issuing an alert when outputting the determination result, as described below.
The above method is merely an example, and other methods may be used.

6, the control unit 10 (output character string acquisition unit 19) acquires an output character string. The control unit 10 can set the output character string to, for example, the character string of the document of the KEY acquired in the process of S31.
In S36, the control unit 10 (image similarity calculation unit 20) compares the image for each character in each matching region and calculates the image similarity.
In S37, the control unit 10 (character determination unit 21) determines whether the key information matches based on the calculated image similarity and a threshold value. If the image similarity is equal to or greater than the threshold value, the control unit 10 determines that the key information matches. Thereafter, the control unit 10 proceeds to S15 in FIG. 4.

In S15 of Fig. 4, the control unit 10 (weight confirmation unit 22) performs a weight confirmation process. The control unit 10 checks whether the weights corresponding to the weight sensors 71 to 74 temporarily stored in the memory unit 30 in the process of S12 match the weights stored in the document memory unit 32. The control unit 10 may consider the weights to match even if there is a slight error in the weights (for example, an error of about 10% from the weight stored in the document memory unit 32). This is because the weight confirmation is primarily used to detect the incorrect placement of multiple sheets.
In S<b>16 , the control unit 10 (confirmation result output unit 23 ) outputs the determination result to the monitor 4 .

9 shows an example of a determination result screen 90 output to the monitor 4. The determination result screen 90 may be output as a pop-up screen superimposed on the screen 60 of FIG.
The judgment result screen 90 includes a key information output section 91 , matching area output sections 92 to 95 for each document, and a result output section 96 .
The key information output section 91 is an area for outputting output character strings. In the example of Fig. 9, two pieces of key information are extracted, and two output character strings are output.
The collation area output sections 92 to 95 are areas that output the collation areas of the documents 51a to 54a.
The result output section 96 is an area that displays the collation result in an easy-to-understand manner at a glance, and displays the collation result using, for example, "◯", "×", "Δ" or the like.
The message output section 97 is an area for outputting a message.

The determination result screen 90 shown in Fig. 9 is an example that is output when the determination result is a match. Therefore, a mark "o" indicating a match is output to the result output unit 96. Here, a match can be, for example,
(a) The character string obtained by simply performing OCR processing on the collation area matches,
(b) The character strings obtained by OCR processing after preprocessing match,
(c) It is determined to be a match based on image similarity,
(d) weight-matched;
This refers to the case.
As shown in the judgment result screen 90 of FIG. 9, when the judgment result is a match, no particular message is output to the message output section 97, and the section remains blank.

On the other hand, the judgment result screen 90-2 shown in Figure 10 is an example output when the judgment result is a mismatch. The example in Figure 10 shows a case where the comparison area output unit 93-2 shows a mismatch. The comparison area output unit 93-2 has its corresponding column colored to clearly indicate a mismatch. In addition, an "x" mark indicating a mismatch is output to the result output unit 96-2. Here, a mismatch refers to, for example, a case where the character strings obtained by simply performing OCR processing on the comparison area do not match, or a case where the weights do not match. A weight mismatch also applies when multiple forms are stacked or when there are not enough documents to be placed.

The mismatched parts are output as a message in the message output section 97-2 of the judgment result screen 90-2. In addition, a window 98 is also output on the judgment result screen 90-2. The window 98 prevents the operator P from proceeding to the next step unless the operator P inputs the release key, and prompts the operator P to confirm.
In addition, if the weights do not match, in order to distinguish this from a case where the character strings do not match, the control unit 10 (confirmation result output unit 23) may alert the worker P, for example, by emitting a warning sound from the speaker 6.

Although not shown, even if the character strings obtained by simply performing OCR processing on the comparison area match and the weights match, if the character strings obtained by performing OCR processing after preprocessing do not match, or if a mismatch is determined based on image similarity, the control unit 10 outputs a determination result screen in which a "△" mark is output to the result output unit 96 of the determination result screen 90. Then, for example, a window 98 like the determination result screen 90-2 in Figure 10 described above is output, prompting the worker P to confirm visually, etc.

Then, in S17 of FIG. 4, the control unit 10 determines whether to end the process. For example, once confirmation of the enclosed documents has been completed, the control unit 10 determines that the process should be ended when worker P performs an operation to end the process (not shown). If the process should be ended (S17: YES), the control unit 10 ends the process. On the other hand, if the process should not be ended (S17: NO), the control unit 10 moves the process to S11 and remains in the process until worker P operates switch 8.

As described above, the enclosure confirmation system 100 of this embodiment has the following advantages.
(1) The enclosure confirmation device 1 extracts a matching area containing key information from a document image, performs OCR processing on the matching area to obtain a character string, performs preprocessing on the matching area to remove noise, and then performs OCR processing on the matching area to obtain a character string, and compares the obtained character strings. The enclosure confirmation device 1 also compares the images of the matching areas between documents to calculate image similarity. The enclosure confirmation device 1 then determines whether the key information matches based on the matching result obtained through OCR processing and the calculated image similarity.
Therefore, by using image similarity, it is possible to supplement the accuracy of OCR and prevent documents from being erroneously enclosed.

(2) The enclosure check device 1 compares the images of each character in the matching area between the documents and calculates the image similarity for each character.
Therefore, by using the image similarity for each character, it is possible to calculate the similarity in smaller units and use it for matching.

(3) The enclosure check device 1 stores a matching area for each document, and by recognizing the document type from the document image, the enclosure check device 1 can obtain a matching area corresponding to the recognized document type from the document image.The enclosure check device 1 then compares and verifies the character strings corresponding to the matching area.
Therefore, the enclosure check device 1 can obtain a matching area by determining the type of document.

(4) The enclosure verification device 1 performs preprocessing on the document image and then performs OCR processing to obtain a character string, and compares the obtained character string with a character string obtained by performing OCR processing before performing preprocessing.
Therefore, the enclosure verification device 1 can not only simply use the OCR processed data to determine whether the key information matches, but can also use the OCR processed data after pre-processing to make the determination, and by adding measures to compensate for the accuracy of the OCR, it is possible to prevent documents from being enclosed incorrectly.

(5) The enclosure verification device 1 compares the acquired character strings by performing OCR processing on the comparison area and comparing character strings obtained by performing OCR processing after performing preprocessing to remove noise, and determines whether the key information matches based on the results of each comparison.
Therefore, matching between character strings obtained by performing OCR processing after preprocessing to remove noise can be used to compensate for matching errors caused by matching between character strings obtained by performing OCR processing on the matching area.
In particular, the risk is greatest when the character strings obtained by OCR processing match even though the key information does not match. Even in such cases, the risk of a mismatch in the key information can be reduced by using the results of matching the character strings obtained by OCR processing after performing preprocessing to remove noise.

(6) The enclosure verification device 1 uses multiple preprocessing methods, allowing for preprocessing that is suited to the characteristics of the image in the matching area. This improves the accuracy of the matching process, which may otherwise determine that key information matches when it actually does not.

(7) The enclosure check device 1 obtains key information from the obtained character string, and outputs a determination result that further includes the key information and an image of the matching area to the monitor 4.
Therefore, information can be provided to the worker P in a display format that allows the worker P to easily visually check whether the key information matches.

(8) The apparatus is provided with a table 40 on which a plurality of documents can be placed side by side, and a camera 5 capable of photographing the table 40, and images of the plurality of documents placed on the table 40 are extracted from the image photographed by the camera 5.
Therefore, by simply placing documents to be enclosed in the same envelope side by side on the table 40, the key information can be verified.

(9) Weight sensors 71 to 74 are provided in the document placement areas of the placement table 40, and it is confirmed whether the weight of each document detected by the weight sensors 71 to 74 matches the weight of each document stored in the document memory unit 32, which stores the weight of each of multiple documents.If it is confirmed that the weights do not match, an alert is issued.
Therefore, documents to be enclosed in the same envelope can be confirmed by weight, and it can be determined whether documents to be enclosed in the next envelope have been mixed in, for example.

Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments. Furthermore, the effects described in the embodiments are merely a list of the most favorable effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments. The above-described embodiments and the modified forms described below can also be used in appropriate combinations, but detailed explanations will be omitted.

(Modified form)
(1) In the present embodiment, the enclosure confirmation device compares character strings obtained by performing OCR processing on the comparison area, compares character strings obtained by performing OCR processing on the comparison area after preprocessing, and makes a determination based on the image similarity for each character calculated by comparing images of each character in the comparison area across multiple documents. However, this is not limited to this. For example, the enclosure confirmation device may only compare character strings obtained by performing OCR processing on the comparison area after preprocessing, and omit the determination based on the image similarity for each character calculated by comparing images of each character in the comparison area across multiple documents. Alternatively, the enclosure confirmation device may only compare character strings obtained by performing OCR processing on the comparison area after preprocessing, and omit the determination based on the image similarity for each character calculated by comparing images of each character in the comparison area across multiple documents. In either case, misinsertion of documents can be prevented by not only comparing character strings obtained by performing OCR processing on the comparison area but also using other processes that supplement the accuracy of OCR.

(2) In this embodiment, an example has been described in which all documents contain key information, but this is not limiting. The present invention can also be used when documents containing the same content, such as pamphlets sent to all customers, are partially enclosed in the same envelope. In this case, OCR verification can be omitted for the portions of the document that do not contain the relevant key information, and detection by the weight sensor can be performed.
In this way, it is possible to check whether the enclosed items are correct, including the order in which they are enclosed, even when documents such as pamphlets are enclosed.

(3) In the present embodiment, the verification area is registered in advance, but this is not limiting. For example, the enclosure check device may identify a verification area from a character string obtained by OCR of a document image, and compare the character string corresponding to the identified verification area.
In this way, the enclosure checker does not need to register the verification area in advance, and can obtain the verification area from the character string included in the document.

(4) In this embodiment, four different types of pre-processing have been described, but the pre-processing is not limited to this. It is not necessary to use all four types of pre-processing, and it is sufficient to use one or more types of pre-processing.
(5) In this embodiment, no particular explanation has been given regarding the system configuration of the enclosed item verification device. However, the device may be a standalone device configured to perform matching processing independently within the terminal, or the device may be configured to have a control unit and memory unit in a server and perform processing on the server.

1 Enclosure checking device 4 Monitor 5 Camera 6 Speaker 8 Switch 10 Control unit 11 Start acceptance unit 12 Image acquisition unit 13 Image extraction unit 14 Matching area specification unit 15 Character recognition unit 16 Preprocessing unit 17 Preprocessed character recognition unit 18 Character matching unit 19 Output character string acquisition unit 20 Image similarity calculation unit 21 Character determination unit 22 Weight confirmation unit 23 Confirmation result output unit 30 Memory unit 32 Document memory unit 33 Enclosure order memory unit 40 Placement table 41, 42, 43, 44 Placement area 51a, 52a, 53a, 54a Document 55 Envelope 60 Screen 61, 62, 63, 64 Document area 71, 72, 73, 74 Weight sensor 90, 90-2 Determination result screen 100 Enclosed item confirmation system P Worker

Claims (9)

A character matching device for matching a plurality of types of documents having the same identification character information,
a recognition processing means for performing character recognition processing on the image of the document;
a pre-processing means for pre-processing the image of the document;
a post-preprocessing recognition processing means for performing character recognition processing on the image preprocessed by the preprocessing means;
a character matching means for matching each character string corresponding to the plurality of documents obtained by the recognition processing means with each character string corresponding to the plurality of documents obtained by the preprocessing recognition processing means ;
an image similarity calculation means for comparing images of regions to be matched between documents and calculating image similarity;
a determination means for determining whether the identification character information matches based on the comparison result by the character comparison means and the image similarity calculated by the image similarity calculation means;
a determination result output means for outputting the determination result by the determination means to a display unit;
A character matching device comprising: 2. The character matching device according to claim 1,
The image similarity calculation means is a character matching device that compares images of characters in the area to be matched between documents to calculate image similarity. 3. The character matching device according to claim 1,
an area specifying means for specifying an area to be matched from the character string obtained by the recognition processing means;
The character matching means performs matching by comparing character strings corresponding to the areas to be matched identified by the area identifying means. 3. The character matching device according to claim 1,
The area to be matched is stored for each document,
a type recognition means for analyzing an image of the document and recognizing the type of the document;
The character matching means compares and matches each character string corresponding to the area to be matched for each document based on the type of document recognized by the type recognition means. 5. The character matching device according to claim 1 ,
The preprocessing means performs a plurality of types of preprocessing on the image of the document. 6. The character matching device according to claim 1,
an output character string acquisition means for acquiring an output character string;
The determination result output means further outputs the output character string acquired by the output character string acquisition means and an image of each area to be matched. A program for causing a computer to function as the character matching device according to any one of claims 1 to 6 . An enclosure confirmation system including the character matching device according to any one of claims 1 to 6 ,
a placement table on which a plurality of the document enclosures can be placed side by side;
an imaging unit capable of imaging the mounting table;
an image acquisition unit that acquires, via the photographing unit, photographed images including images of the plurality of documents placed on the document table;
an image cutting means for cutting out an image of each document from the photographed image acquired by the image acquisition means;
An enclosure confirmation system comprising: 9. The enclosure confirmation system according to claim 8 ,
a weight detection unit provided in a placement area of each enclosure on the placement table;
a weight storage unit for storing the weight of each type of enclosed item;
a weight confirmation means for confirming whether the weight of each enclosure detected by the weight detection unit matches the weight of each enclosure stored in the weight memory unit;
a notification means for notifying when the weight confirmation means confirms that the weights do not match;
An enclosure confirmation system comprising: