JP7562292B2 - Information processing device, server, system, information processing method, and program - Google Patents

Description

This disclosure relates to a technology for extracting character strings corresponding to specified items based on the results of OCR processing.

There are tasks in which users input specific information written on paper forms into a computer. There are also input support systems that assist with this input task. In input support systems, character recognition processing is performed on the image data obtained by reading the form with a scanner, and the information required for input is extracted from the character recognition results.

Patent document 1 describes a technology that allows a user to specify an area on an image where the information required for input is displayed, thereby correcting the character recognition results.

JP 2000-3403 A

However, with the technology of Patent Document 1, in order to correct the character recognition results, it is necessary to use an input device such as a mouse to specify the area on the image where the information required for input is displayed, which requires cumbersome operations for the user. In particular, when the screen size of the input device is small, such as a touch panel installed in an MFP or a mobile terminal, operations such as pinching to enlarge the screen and swiping to scroll the screen are necessary, making the user's operations even more cumbersome.

The technology disclosed herein aims to reduce the amount of user operations required when performing processing based on character recognition results.

The information processing device according to the present disclosure is characterized in having an acquisition means for acquiring an extraction result including a processing result including a character string in an image obtained by performing character recognition processing on image data of an image and information on a character area in which the character string is recognized, and information on a character string and a character area in the image corresponding to each of predefined specified items, output by inputting the processing result into a learning model trained to output information on a character string and a character area corresponding to each of the predefined items ; a display control means for displaying a character string corresponding to each of the predefined items on a display unit based on the extraction result; a receiving means for receiving an instruction from a user to correct a character string corresponding to any of the predefined items displayed on the display unit; a correction means for determining a character area corresponding to an item instructed to be corrected by the user based on the corrected character string and the processing result, and correcting the extraction result based on the determination; and an output means for outputting the extraction result corrected by the correction means as data for the learning model to learn from.

The technology disclosed herein can reduce the amount of user operations required when performing processing based on character recognition results.

FIG. 1 is a diagram illustrating a configuration of an input support system. FIG. 2 is a diagram illustrating an example of a hardware configuration of a key-value extraction server. FIG. 2 illustrates an example of a hardware configuration of an MFP. FIG. 2 is a diagram illustrating an example of a functional configuration of a key-value extraction server. FIG. 2 illustrates an example of a functional configuration of an MFP. 1 is an example of an image of a document read by a scanner. 13 is an example of a key-value extraction result. FIG. 11 is a sequence diagram showing a process for determining a character area corresponding to a key. FIG. 4 is a diagram showing a screen displayed on an operation panel. 1 is an example of an image of a document read by a scanner. 13 is an example of a key-value extraction result. 1 is an example of an image of a document read by a scanner. 13 is an example of a key-value extraction result. 13 is an example of a corrected key-value extraction result. 11A to 11C are diagrams illustrating transitions of screens displayed on an operation panel. FIG. 2 is a diagram illustrating an example of a functional configuration of a key-value extraction server. FIG. 11 is a sequence diagram showing a process for determining a character area corresponding to a key. 13 is an example of a key-value extraction result. 11A to 11C are diagrams illustrating transitions of screens displayed on an operation panel. FIG. 11 is a sequence diagram showing a process for determining a character area corresponding to a key.

Below, the form for implementing the present embodiment will be described with reference to the drawings. Note that the embodiment does not limit the technology of the present disclosure, and all configurations described in the embodiment are not necessarily essential to the means for solving the problem. In addition, the following embodiment describes a preferred embodiment, but is not limited to the following embodiment, and various modifications and changes are possible within the scope of the gist.

<Embodiment 1>
[System configuration]
Fig. 1 is a diagram showing the overall configuration of an input support system according to this embodiment. The input support system 100 will be described with reference to Fig. 1. The input support system 100 of this embodiment will be described as a system for performing input support by inputting required data on behalf of a user to an accounting server 103 that provides a service (accounting service) that processes accounting operations of a company. Note that input support by the input support system is not limited to accounting services, and can be used as a system that supports general form input operations.

The MFP (Multifunction Peripheral) 300 is an image forming device that also functions as an information processing device that can be connected to a network. The MFP 300 has a scanning function that converts paper documents into image data.

The key-value extraction server 200 is a server that functions as an information processing device that extracts information (value), such as character strings corresponding to specific items (keys), from a scanned image of a paper document. In this embodiment, the items include items required for accounting services. For example, the key-value extraction server 200 extracts the character string "2019/06/12", which is a value corresponding to the "Date" item, from an image of an invoice.

The accounting server 103 is a server that provides accounting services that perform accounting processing by inputting information on specific forms such as invoices. The MFP 300, the key value extraction server 200, and the accounting server 103 are connected via the Internet 104.

[Hardware configuration]
FIG. 2 is a block diagram showing the basic hardware configuration of the key-value extraction server 200 and the accounting server 103.

The CPU (Central Processing Unit) 201 is a unit that executes various programs and realizes various functions. The RAM (Random Access Memory) 202 is a unit that stores various information. The RAM 202 is also used as a temporary working memory area for the CPU 201. The ROM (Read Only Memory) 203 is a unit that stores various programs, etc. For example, the CPU 201 loads a program stored in the ROM 203 into the RAM 202 and executes the program.

The external storage unit 207 is composed of, for example, a flash memory, a hard disk drive (HDD), or a solid state disk (SSD), and stores various programs. The CPU 201 executes processing based on the programs stored in the external storage unit. This realizes the functional configuration constituting the key-value extraction server 200 as shown in FIG. 4 and the processing of each step in the sequence described below.

The Input/Output interface 204 is a unit that provides an interface with output devices such as a display device, and input devices such as a keyboard and a mouse.

The NIC (Network Interface Card) 205 is a unit for connecting the key-value extraction server 200 and the accounting server 103 to a network. The above-mentioned units are configured to be able to send and receive data via the bus 206.

Figure 3 is a block diagram showing the basic hardware configuration of MFP 300. CPU 301, RAM 302, ROM 303, and NIC 304 in Figure 3 are similar to CPU 201, RAM 202, ROM 203, and NIC 205 in Figure 2, respectively, and therefore will not be described.

The scanner 305 is an input unit that converts paper documents into image data. The printer engine 306 is an output unit that prints the image data. The operation panel 307 is an input unit (input section) that accepts touch operations from the user, and is an output unit (display section) that displays information about the MFP 300 on a display. The above-mentioned units are configured to be able to send and receive data via the bus 308.

[Functional configuration of the key-value extraction server]
Fig. 4 is a diagram showing the functional configuration of the key-value extraction server 200. The functions of the key-value extraction server 200 will be described with reference to Fig. 4.

The image data acquisition unit 401 acquires image data of a scanned image obtained by reading an original from the MFP 300 via the NIC 205. The OCR unit 402 is a character recognition processing unit that identifies a character area, which is an area containing a character string in the image indicated by the image data, and performs character recognition processing (OCR processing) on the character area. As a result of the character recognition processing, the OCR unit 402 outputs at least the position information of the character area in the image and the character string contained in the character area.

The key-value extraction unit 403 performs processing to extract values (character strings) corresponding to keys (items) contained in the results of the character recognition processing by the OCR unit 402. The keys (items) are defined in advance according to the input support performed by the input support system 100. The values (character strings) corresponding to these keys (items) are also called key values. The key-value extraction unit 403 extracts key values using the results of the character recognition processing and extraction rules 409.

The extraction rule 409 of this embodiment will be described as a learning model that is machine-learned to output character strings corresponding to each item based on the characteristics of the processing result of the character recognition process. The characteristics of the processing result of the character recognition process include (1) characteristics obtained from the character string that has been recognized, (2) rectangular information of the character area, and (3) characteristics obtained from the character strings surrounding the character string.

In order for the key-value extraction unit 403 to extract character strings and character areas corresponding to specific items, the extraction rules 409 are input with the character areas that are the results of the character recognition process by the OCR unit 402 and the character strings recognized in the character areas. As a result, the extraction rules 409 are trained to output character strings and character areas corresponding to each item. Therefore, the key-value extraction unit 403 can extract character strings and character areas corresponding to specific items based on the output results. Alternatively, the extraction rules 409 may be trained to output values related to the character strings and character areas corresponding to the items, and the key-value extraction unit 403 may extract character strings and character areas corresponding to the items based on the output values.

The extraction rule 409 is, for example, a learning model generated by linking input data and teacher data and training them using an algorithm such as SVM (Support Vector Machine). Note that it is not limited to SVM, and a decision tree or neural network algorithm may also be used.

The key-value extraction unit 403 also generates a key-value extraction result, which is data in which each item is associated with a character string and a character area corresponding to the item, as an extraction result. The key-value extraction result transmission unit 405 transmits the processing result of the character recognition process performed by the OCR unit 402 and the key-value extraction result to the MFP 300. Details of the key-value extraction result will be described later.

In this embodiment, if the key-value extraction unit 403 erroneously extracts a character string or character area corresponding to an item, the user can instruct the MFP 300 to correct the character string and character area to the correct one corresponding to the item. The correction result acquisition unit 406 acquires data (correction result) from the MFP 300 indicating the correct character string and character area corresponding to the item corrected based on the user's instruction.

The learning unit 407 learns the extraction rule 409 based on the correction result acquired by the correction result acquisition unit 406. The learning unit 407 can further learn the extraction rule 409 in this embodiment so that the extraction rule 409 becomes suitable for the work of each user who uses the input support system 100.

For example, the learning unit 407 inputs the results of character recognition processing performed by the OCR unit 402 as input data into the extraction rules 409, and causes them to output values indicating a character string and character area corresponding to a certain item. The learning unit 407 then calculates the amount of deviation between the output result and the value indicating the actual correct character string and character area corresponding to that item. The learning unit 407 updates parameters and the like in the extraction rules 409 so as to reduce the amount of deviation. Data indicating the actual correct character string and character area corresponding to an item is called teacher data. Also, updating the parameters of the extraction rules 409 using data for learning, such as teacher data, is called learning.

In this embodiment, in the learning by the learning unit 407, data in which specific items are associated with the character areas and character strings corresponding to each item, which is the correction result acquired by the correction result acquisition unit 406, may be used as training data. Therefore, if the extraction rule 409 erroneously outputs a character string or character area corresponding to a certain item, the extraction rule 409 can be trained to output the correct character string and character area.

Note that extraction rule 409 is not limited to a learning model. For example, extraction rule 409 may also be a table in which the coordinates and size of character areas corresponding to each item are associated with each type of form. In this case, a correction unit (not shown) corrects the coordinates and size values of the character area containing a character string indicating a specific item based on the correction result acquired by correction result acquisition unit 406. In this way, processing is performed so that processing for extracting character strings corresponding to items from the character strings obtained by character recognition processing is performed correctly.

The area determination unit 404 determines (specifies) the correct character area corresponding to the item based on the character string corresponding to the item corrected by the user and the position in the scanned image specified by the user, and corrects the key-value extraction result. Details will be described later.

The key-value extraction result transmission unit 405 transmits the key-value extraction result to the MFP 300. The key-value data transmission unit 408 transmits each item and the character string corresponding to the item to the accounting server 103 via the NIC 205.

The functions of each part in FIG. 4 are realized by the CPU of the key-value extraction server 200 expanding the program code stored in the ROM into the RAM and executing it. Alternatively, some or all of the functions of each part in FIG. 4 may be realized by hardware such as an ASIC or electronic circuit.

[Functional configuration of MFP]
Fig. 5 is a diagram showing the functional configuration of the MFP 300. The functions of the MFP 300 will be described with reference to Fig. 5.

The UI unit 501 functions as a display control unit that causes the operation panel 307 to display a screen for providing a predetermined notification to the user, and as a reception unit that receives input from the user via the operation panel 307.

The scan execution unit 502 converts data obtained by reading an original document with the scanner 305 into image data. The image data transmission unit 503 transmits the image data of the original document to the key-value extraction server 200 via the NIC 304.

The key-value extraction result acquisition unit 504 acquires the key-value extraction result from the key-value extraction server 200 via the NIC 304.

The area determination unit 506 obtains the corrected character string corresponding to the item instructed to be corrected by the user via the UI unit 501, and determines the correct character area corresponding to that item based on the obtained corrected character string. The area determination unit 506 corrects the key-value extraction result by reflecting the determined character area in the key-value extraction result. Details will be described later.

The modification result transmission unit 505 transmits (outputs) the modified key-value extraction result based on the modification made by the user via the UI unit 501 to the key-value extraction server 200 via the NIC 304. The modified key-value extraction result is used as data for learning by the learning unit 407.

In this way, the MFP 300 of this embodiment also functions as an information processing device that performs predetermined processing based on the key-value extraction results sent from the key-value extraction server 200.

The functions of each part in FIG. 5 are realized by the CPU of the MFP 300 expanding program code stored in the ROM into the RAM and executing it. Alternatively, some or all of the functions of each part in FIG. 5 may be realized by hardware such as an ASIC or electronic circuit.

[Key-value extraction results]
Fig. 6 is a diagram showing an example of an image represented by image data of a paper document scanned by the scan execution unit 502 of the MFP 300. As shown in Fig. 6, the image is image data of a scanned image obtained by reading a paper document (original) on which information to be input to the accounting server 103, such as an estimate, is written.

Figure 7 shows a key-value extraction result 700 that includes character regions and character strings recognized as a result of character recognition processing, and values indicating that the recognized character regions and character strings correspond to predefined items.

In the key-value extraction result 700, data in each column that holds a character area (column 701), a character recognition result indicating a character string recognized in the character area (column 702), and a label indicating the name of the item (column 703) are stored in association with each other on a record-by-record (row-by-row) basis. A record is also generated for each character area.

The character area in column 701 is a column that holds position information for character areas, which are rectangular areas in an image that contain a character string. The position information holds coordinate values indicating the position of the character area and values indicating the size of the character area. The coordinate system for the image uses a coordinate system whose origin is at the top left, with the vertical direction extending in the y direction and the horizontal direction extending in the x direction. From the left, column 701 holds the x coordinate, the y coordinate, the x size (width) value, and the y size (height) value.

Column 702 is a column that holds character strings recognized as a result of OCR processing on a character area. The information held in columns 701 and 702 is the processing result of character recognition processing by OCR unit 402 on image data.

Column 703 is a column for holding values indicating the name of an item (key). The value indicating the name of each item is called a label. Items in the accounting system are items required by the accounting server 103, and in this embodiment, the items are described as five items: the title of the scanned document, the billing telephone number, the document management number, the document issue date, and the total amount. Thus, in this embodiment, the key-value extraction result is described as including the processing result of the character recognition process, but the processing result of the character recognition process may be generated as separate data.

In column 703, the record that holds the character string and character area corresponding to the item indicating the document title is assigned the value "Title" (label). The billing telephone number item is assigned the value "Telephone number". The document management number item is assigned the value "Number". The document issue date item is assigned the value "Date". The total amount item is assigned the value "Amount".

The key-value extraction unit 403 inputs the character area in column 701 and the character string in column 702 into the extraction rule 409, and outputs the character string and character area corresponding to each item. Based on the output result of the extraction rule 409, the key-value extraction unit 403 assigns a label to indicate the name of the corresponding item to the records corresponding to each item among the records having the character string and character area in the processing result of the character recognition process. In this way, the key-value extraction result 700 is generated. The key-value extraction result 700 is transmitted to the MFP 300.

The character strings resulting from character recognition in rows 704 to 708 of the key-value extraction result 700 in FIG. 7 are character strings extracted as character strings (values) corresponding to items (keys). For this reason, columns 703 in rows 704 to 708 are given labels indicating that they are items to be extracted.

The key-value extraction result 700 shown in FIG. 7 is an example in which the key-value extraction unit 403 correctly extracts a key-value, and the correct label is stored in column 703. However, the label in column 703 may be assigned to a record other than the record corresponding to the item. In other words, the extraction rule 409 may erroneously output a character string and character area corresponding to an item. For example, if the telephone number of the billing destination is set as an item, it is conceivable that a telephone number other than the billing destination will be output as a character string corresponding to that item. In this case, the user can input and correct the character string corresponding to the item. The user can also touch the screen to indicate the correct character area corresponding to the item.

If there is training data including the correct character string corresponding to the item and the correct character area corresponding to the item, the learning unit 407 can have the extraction rule 409 learn and update the extraction rule 409. In this way, by having the extraction rule 409 learn, it is possible to update the extraction rule 409 so that the correct key value is extracted. However, the screen size of the operation panel 307 provided on the MFP 300 may be small. For this reason, when the user tries to indicate the correct character area via the operation panel 307, the user's operation may become cumbersome.

In this embodiment, therefore, a form is described in which a character area corresponding to an item input by a user is determined based on the correct character string corresponding to that item, and the determined character area is output as data for learning. In addition, a form is described in which, when a user touches the operation panel 307 to indicate a character area that contains a character string corresponding to an item, the correct character area corresponding to the item is determined even if the touch position is not accurate.

[Character extraction and correction processing]
Fig. 8 is a sequence diagram showing the processing in the input support system of this embodiment. The processing in each device shown in the sequence diagram of Fig. 8 is performed by the CPU of each device expanding the program code stored in the ROM into the RAM and executing it. In addition, some or all of the functions of the steps in Fig. 8 may be realized by hardware such as an ASIC or an electronic circuit. Note that the symbol "S" in the explanation of each process means that it is a step in the flow. For convenience of explanation, the user's actions are also explained with the letter "S" attached.

In S801, the user places an original (paper document) on the scanner 305 of the MFP 300 and issues an instruction to start scanning. The user's operation in this step triggers the start of processing in the input support system.

In S802, the scan execution unit 502 of the MFP 300 scans the paper document and generates image data of the paper document. The image data transmission unit 503 transmits the generated image data to the key-value extraction server 200.

In S803, the image data acquisition unit 401 acquires the image data sent from the MFP 300, and the OCR unit 402 performs character recognition processing on the image data.

In S804, the key-value extraction unit 403 extracts character strings and character regions corresponding to each predefined item based on the results output by inputting the results of the character recognition process to the extraction rule 409. The key-value extraction unit 403 then assigns a label indicating the name of the item to records that contain character regions and character strings corresponding to the items among the records included in the results of the character recognition process, thereby generating the key-value extraction result 700 shown in FIG. 7. The key-value extraction result transmission unit 405 transmits the generated key-value extraction result 700 to the MFP 300.

In S805, the key-value extraction result acquisition unit 504 of the MFP 300 acquires the key-value extraction result 700, which includes the processing result of the character recognition process. In addition, the UI unit 501 of the MFP 300 displays each item and each character string corresponding to the item on the operation panel 307 based on the acquired key-value extraction result.

Figure 9 shows a screen 901 displayed on the operation panel 307 of the MFP 300 in this step. Screen 901 is displayed so that the user can confirm and modify the key value extraction results. Value display fields 902-906 are display fields that display character strings corresponding to the items displayed in the key display fields 912-916.

In S806, the user checks the extracted strings corresponding to each item on screen 901 displayed on the operation panel 307 of the MFP 300, and corrects any errors in the strings.

When the user touches any of the value display fields 902 to 906 on screen 901 displayed on the operation panel 307 of the MFP 300, screen 901 switches to edit mode. In edit mode, the user can correct the string in the value display field by inputting the correct string for the item in the value display field that the user touched. In other words, the user can specify the item to be corrected by touching the value display field to be corrected. When the user presses the register button 907 after inputting the corrections, the UI unit 501 accepts the registration of the corrected string.

For example, on screen 901, "098-765-4321" is displayed as the extraction result as the value of "phone number" displayed in key display field 916. As shown in rectangle 605 in FIG. 6, the character string of the billing destination phone number that should be correctly extracted is "0123-45-6789". In this case, the user can touch value display field 906 and then input "0123-45-6789" to correct the character string corresponding to the item.

The cause of errors in the extraction of key values that users need to correct may be that the character area corresponding to the item is incorrectly extracted. Alternatively, the cause of errors in the extraction of key values that users need to correct may be that the character area corresponding to the item is correctly extracted, but the character recognition for that character area is incorrect.

In S807, the area determination unit 506 of the MFP 300 determines whether the difference between the character string resulting from character recognition corresponding to the item before correction and the corrected character string is within a predetermined range (acceptable range). In this embodiment, if the edit distance (Levenshtein distance) between the character string corresponding to the item before correction extracted by the key-value extraction unit 403 and the corrected character string corrected by the user is less than or equal to a threshold, it is determined that the difference is within the predetermined range. The threshold is, for example, 1.

For example, if the pre-correction character string corresponding to a certain item extracted by the key-value extraction unit 403 in S804 is "0123-45-6788" and the corrected character string corrected by the user is "0123-45-6789", the edit distance will be 1. Therefore, if the threshold value is 1, the difference between the pre-correction character string and the corrected character string corresponding to the item is determined to be within a predetermined range.

If the difference in the character strings is within a predetermined range (YES in S807), the cause of the error in extracting the key value is likely to be that the character area corresponding to the item was correctly extracted, but the character recognition process for the character area was incorrect. For this reason, the process proceeds to S815 without determining the correct character area corresponding to the item, which will be described later.

If the difference in the character strings is not within the specified range (NO in S807), the cause of the error in key-value extraction is believed to be that the character area corresponding to the item was erroneously extracted. In this case, the process proceeds to S808, where the area determination unit 506 determines a character string that is the same as the corrected character string from the processing results of character recognition processing on the entire image data, and determines the character area in which this character string is recognized as the correct character area corresponding to the item. In this embodiment, since the processing results of character recognition processing are included in the key-value extraction results, the area determination unit 506 determines the correct character area corresponding to the item from the key-value extraction results. Note that "same" here is not limited to characters that are exactly the same, but also includes characters that can be considered to be the same.

Figure 10 is a diagram showing an example of a scanned image of a paper document different from that shown in Figure 6, which is a scanned image scanned by the scan execution unit 502 of the MFP 300 in S802. Figure 11 is a key-value extraction result different from that shown in Figure 7, showing the result of performing character recognition and key-value extraction on the image data of Figure 10. The processing of S808 will be described using Figures 10 and 11.

Rows 1102 to 1106 in FIG. 11 are records containing the results of the character recognition process (character regions and character recognition results) to which labels indicating items have been added. Dotted rectangles 1001 to 1005 in FIG. 10 indicate extracted character regions corresponding to items. However, the key-value extraction results in FIG. 11 contain an extraction error by the key-value extraction unit 403. Specifically, the record in row 1106 holds a telephone number label as a character region and character string corresponding to the item labeled "telephone number", but the labeling is incorrect. For this reason, it is assumed that the user inputs "0123-45-6789", which is the correct character string corresponding to "telephone number", in S806. It is also assumed that the corrected character string "0123-45-6789" input by the user in S807 and the erroneously extracted character recognition result "098-765-4321" in row 1106 are not determined to be within the acceptable range, and therefore the process transitions to S808.

In S808, the area determination unit 506 derives the edit distance between the corrected string corrected by the user and all character strings stored in the character recognition results in column 1108. If there is one character string in the character recognition results whose edit distance is equal to or less than a predetermined value, the area determination unit 506 considers that the character string equal to or less than the predetermined value and the corrected string are the same, and determines that the correct character area corresponding to the item has been determined. The predetermined value is, for example, 1.

Column 1101 in FIG. 11 is a column that holds the edit distance between the character string of the character recognition result for each row and the character string "0123-45-6789" corrected by the user. In FIG. 11, the only character in column 1101 with an edit distance of 1 or less is the character recognition result held in row 1107. In other words, there is only one character string with an edit distance of 1 or less, which is the predetermined value. Therefore, in the example of FIG. 11, the correct character area corresponding to the "phone number" item can be determined to be the character area in row 1107. Dotted rectangle 1006 in FIG. 10 is the character area corresponding to the character area in row 1107 in FIG. 11.

In this manner, in this embodiment, if a character area corresponding to an item is erroneously extracted, the user does not need to search for and specify the correct character area from the key-value extraction results or the processing results of the character recognition process in order to correct it to the correct character area corresponding to the item. When the user inputs the correct character string corresponding to the item, the area determination unit 506 can determine the character area corresponding to the item based on the corrected character string.

In S809, it is determined whether a character area corresponding to the item has been determined based on the corrected character string entered by the user in S808. For example, if there are multiple character strings whose edit distance from the corrected character string is less than or equal to a predetermined value, or if there are no character strings whose edit distance from the corrected character string is less than or equal to a predetermined value, the correct character area corresponding to the item specified by the user to be corrected (the item extracted in error) is not determined in S808. In this case, it is determined that the correct character area has not been determined.

If the correct character area has not been determined (S809 is NO), in S810, the UI unit 501 causes the operation panel 307 to display a preview image based on the image data of the scanned document. The UI unit 501 then prompts the user to touch the preview image on the operation panel 307 to indicate the correct character area that corresponds to the item that the user has instructed to be corrected.

In S811, the user touches the preview screen displayed on the operation panel 307 of the MFP 300 to indicate the position of the correct character area for the item that the user has instructed to be corrected, in which the corrected character string is written.

In S812, the UI unit 501 of the MFP 300 receives a touch by the user and detects the touch position on the operation panel 307. The UI unit 501 then displays screen 1505 (see FIG. 15) on the operation panel 307 and receives the user's press of the register button 907. In screen 1505 that is displayed at this time, the corrected character string is displayed in the value display field corresponding to the item that the user instructed to be corrected.

The correction result transmission unit 505 converts the touch position on the operation panel 307 touched by the user into touch position coordinates mapped to the image coordinates, and determines the touch position coordinates. The correction result transmission unit 505 transmits the corrected character string input by the user in S806 and the touch position coordinates to the key-value extraction server 200.

In S813, the correction result acquisition unit 406 of the key-value extraction server 200 acquires the touch position coordinates and the corrected character string and its item sent from the MFP 300. The area determination unit 404 of the key-value extraction server 200 determines the correct character area corresponding to the item instructed to be corrected by the user based on the touch position coordinates and the corrected character string.

Figure 12 is a scan image scanned by the scan execution unit 502 of the MFP 300 in S802, and is a diagram showing an example of a scan image of a paper document different from those shown in Figures 6 and 10. Figure 13 is a table showing the results of performing character recognition and key-value extraction on the image data of Figure 12, and is different from those shown in Figures 7 and 11. The processing from S810 to S813 will be explained using Figures 12 and 13.

Rows 1302 to 1306 in FIG. 13 are records containing the results of character recognition processing in which a label indicating the item is assigned as a string (character area) corresponding to a specific item. However, the "phone number" in column 1312 is not assigned to the correct string (character area).

For this reason, suppose that the key-value extraction unit 403 extracted the character string in the "phone number" field as "098-765-4321", but the user corrected the character string in the "phone number" field to "0123-45-6789". Column 1311 in FIG. 13 holds the edit distance between the character string "0123-45-6789" corrected by the user's input and the character string resulting from character recognition. In the example in FIG. 13, there are two character strings with an edit distance of 1 or less, so in S808 the region determination unit 506 was unable to determine the correct character region corresponding to the phone number field. For this reason, the process transitions to S810.

Touch position 1210 shown in FIG. 12 is the position touched by the user in S811, and the touch position coordinates are determined to be (x, y) = (1110, 493) in S812.

In S813, the area determination unit 404 of the key-value extraction server 200 derives the distance between the touch position coordinates and the character areas of all rows of the table of the key-value extraction results. Among the character areas where the distance is equal to or less than a predetermined value, the area determination unit 404 determines the character area in which the character string with the smallest edit distance from the corrected character string is recognized as the character area corresponding to the item instructed to be corrected by the user. The predetermined value is, for example, 100.

Column 1301 in FIG. 13 is a column that holds the distance between the position of a character area and the touch position in the image touched by the user, and shows the distance from the character area when the touch position coordinates are (1110, 493). Rows 1307 to 1310 are records that have character areas whose distance from the touch position coordinates is a predetermined value of 100 or less. Of these, the area determination unit 404 determines that the character area in row 1308, which has the smallest edit distance in column 1311, is the correct character area corresponding to the "phone number" item, which is the item instructed to be corrected by the user. Note that if there are two or more smallest distances, the character area with the smallest distance may be determined to be the correct character area.

In this way, in this embodiment, when a user touches an image to indicate the correct character area that corresponds to an item that the user has instructed to correct, the character area that corresponds to that item is determined taking into consideration not only the touch position coordinates but also the edit distance with the corrected character string.

For example, in FIG. 13, the character area with the smallest distance from the touch position coordinates is the character area of line 1307 where the character string "TEL:" is recognized, but this is not the character area that corresponds to the item. If the character area corresponding to the item is determined simply based on the touch position coordinates in this way, there is a risk that the correct character area cannot be determined if the user cannot touch accurately, such as when the screen is small. According to this embodiment, the character area corresponding to the item can be determined even in such cases where the user cannot touch accurately.

In S814, the area determination unit 404 determines whether the correct character area corresponding to the item was determined in S813.

If the correct character area is determined in S813 (YES in S814), or if the correct character area is determined in S808 (YES in S809), proceed to S815.

In S815, the learning unit 407 uses data indicating the position information of the correct character area corresponding to the item determined by the area determination units 404 and 506 as training data to train the extraction rules 409. The extraction rules 409 are updated by extracting and learning the characteristics of the character recognition results contained in the training data.

If the correct character area corresponding to the item specified by the user in S808 or S813 can be determined, the area determination unit 404, 506 corrects the key-value extraction results so that a label is assigned to the record containing the correct character area.

Figure 14 is a diagram showing the key-value extraction result after the label in the key-value extraction result in Figure 11 has been modified by MFP 300 based on the processing result of S808. The area determination unit 506 has determined that the character area in which "phone number" is the item is the character area in row 1107. Therefore, the area determination unit 506 modifies the key-value extraction result so that "phone number" is assigned to row 1107 in column 1401 as shown in Figure 14.

Similarly, when a character area corresponding to an item is determined in S813, the area determination unit 404 corrects the key-value extraction result so that "phone number" in column 1312 in the key-value extraction result of FIG. 13 is assigned to row 1308. In this way, the area determination units 404 and 506 also have the function of correcting the key-value extraction result by reflecting the determined correct character area in the key-value extraction result.

In the learning in S815, for example, the corrected key-value extraction result as shown in FIG. 14 is used as data for learning. The corrected key-value extraction result as shown in FIG. 14 may be used as is as training data for learning, or the corrected key-value extraction result may be processed and used as training data or input data for learning.

If the correct character area is not determined in S814 (NO in S814), proceed to S816. For example, if there is no character area whose distance from the touch position coordinates is within a predetermined value, it is determined in S814 that the character area could not be determined. In this case, since the correct character area corresponding to the item specified by the user for correction could not be determined, proceed to S816 without proceeding to the learning step in S815. This makes it possible to prevent data that includes an incorrect character area corresponding to an item from being used as learning data.

In S816, the key-value data transmission unit 408 transmits data on the items and the character strings corresponding to each item to the accounting server 103. The character recognition result or the key-value extraction result with the label corrected may be transmitted to the accounting server 103, or records with labels may be extracted from the corrected key-value extraction result and only the data to be used by the accounting server 103 may be transmitted.

Figure 15 shows the transition of screens displayed on the operation panel 307 of the MFP 300 at each step of the flow in Figure 8.

Screen 1501 is the same as screen 901 that is displayed as a result of the processing of S805. Screen 1502 shows the screen after the telephone number character string on screen 1501 has been corrected by user operation 1506 in S806.

Screen 1503 shows a preview image of the image data of a scanned document, which is displayed in S810 when the correct character area cannot be determined from the character string entered by the user. Screen 1504 shows how the correct character area corresponding to the item instructed to be corrected is touched by user operation 1507 in S811. Screen 1505 is the screen displayed after S811, and shows how the register button 907 is pressed by user operation 1508.

The arrows in FIG. 15 indicate screen transitions. Screen 1501 transitions to screen 1502, and if a YES decision is made in S807 or S809, screen 1502 transitions to screen 1505. If a NO decision is made in S807 and S809, screen 1502 transitions to screen 1503. Screen 1503 transitions to screen 1504, and screen 1504 transitions to screen 1505.

The sequence diagram in Figure 8 describes the flow when the user modifies the key-value. For example, if the user does not modify the key-value in S806, the key-value is correctly extracted. Even in this case, the key-value extraction result may be used as data for learning to allow the extraction rule 409 to learn.

As described above, according to this embodiment, the correct character area corresponding to an item input by the user can be determined based on the corrected character string corresponding to that item. This eliminates the need for the user to perform the cumbersome operation of inputting a character area corresponding to the item, thereby improving usability.

Even when a user touches the screen to specify the correct character area corresponding to a specific item, the correct character area corresponding to the item is determined based on the character string corrected by the user in addition to the touch position on the document image touched by the user. This makes it possible to determine the correct character area corresponding to the item even if the touch position is not accurate.

The description has been given assuming that the area determination unit 506 of the MFP 300 determines the correct character area corresponding to the item instructed to be corrected by the user in S808, and the area determination unit 404 of the key-value extraction server 200 determines the correct character area corresponding to the item in S813. Alternatively, only one of the area determination units may perform the process of determining the correct character area corresponding to the item in both steps S808 and S813. For example, the area determination unit 506 of the MFP 300 may determine the character area corresponding to the item not only in S808 but also in S813. Alternatively, the area determination unit 404 of the key-value extraction server 200 may determine the character area corresponding to the item not only in S813 but also in S808. Furthermore, the area determination unit 404 of the key-value extraction server 200 may perform the process of determining the correct character area corresponding to the item when position information of the correct character area corresponding to the item cannot be obtained from the MFP 300.

In this embodiment, the image data on which character recognition processing is performed has been described as being a scanned image. In addition, for example, the image data on which character recognition processing is performed may be an image of a document generated as electronic data.

<Embodiment 2>
In the first embodiment, the MFP 300 performs character recognition processing on all character areas of the image data obtained by scanning, and a key value is extracted based on the results. However, for reasons such as reducing the processing load of the character recognition processing, character recognition processing may be performed on only some character areas of the image data, and a key value may be extracted based on the results of the character recognition processing. For this reason, in this embodiment, a method for determining a correct character area corresponding to the item to be corrected based on a character string corrected by the user in such a case will be described. This embodiment will be described mainly with respect to the differences from the first embodiment. The configuration and processing are the same as those of the first embodiment unless otherwise specified.

[Functional configuration of the key-value extraction server]
16 is a diagram showing the functional configuration of the key-value extraction server 200 of this embodiment. The same components as those in the first embodiment are denoted by the same reference numerals and will not be described. The key-value extraction server 200 of this embodiment further includes a character region detection unit 1601.

The character area detection unit 1601 performs processing to detect character areas present in the scanned image represented by the image data. The character area detection method may be a known method such as extracting a rectangular area assumed to be a character from an image binarized using a certain threshold value.

In addition, the key-value extraction unit 1602 of this embodiment estimates the form type of the scanned image from the results output by inputting the position information of the character area obtained as a result of processing by the character area detection unit 1601 into a learning model (not shown). The learning model is a learning model that has been machine-trained to output the form type of the image based on the position and size characteristics of the character area in the image. As another example, the key-value extraction unit 403 may estimate the type of form by calculating the degree of agreement between the layout of the character area for each type of form stored in the database and the layout of the character area obtained as a result of processing by the character area detection unit 1601.

Then, the key-value extraction unit 1602 of this embodiment uses the type of form to extract a character area that contains a key-value, which is a character string corresponding to a key (item).

The extraction rules 1603 of this embodiment will be described as a learning model that is trained to output information that can identify the character areas containing the character strings of each item, based on the type of form and the characteristics of the character areas in the image. The key-value extraction unit 1602 inputs the type of form and information such as the position of the character area of the scanned image obtained as a result of processing by the character area detection unit 1601 into the extraction rules 1603, and extracts the character areas of each item in the scanned image based on the output result.

Alternatively, extraction rules 1603 may be a database in which position information such as coordinates indicating the position and size of the character area of each item is stored for each type of form. In that case, for example, key-value extraction unit 1602 may extract a character area in the scanned image that is located at the same position as or near the position of the character area of a certain item stored in extraction rules 1603 as the character area corresponding to that item.

Alternatively, extraction rule 1603 may be a learning model that is trained to output information identifying the character area of each item based only on the characteristics of the character area in the image. In this case, estimation of the type of form does not need to be performed. In other words, key-value extraction unit 1602 may input information such as the position of the character area of the scanned image obtained as a result of processing by character area detection unit 1601 into extraction rule 1603, and extract the character area of each item in the scanned image based on the output result.

The OCR unit 402 then performs character recognition processing on the character regions of each item extracted by the key-value extraction unit 1602. The resulting character strings are then output as character strings corresponding to each item.

[Character extraction and correction processing]
17 is a sequence diagram showing the process in the input support system of this embodiment. Steps S1701 to S1702 are the same as steps S801 to S802, so the description will be omitted.

In S1703, the image data acquisition unit 401 acquires the image data of the scanned image sent from the MFP 300, and the character area detection unit 1601 detects the character area of the acquired image data.

In S1704, the key-value extraction unit 1602 estimates the document type of the image data based on the information of the detected character area. Then, in S1705, the key-value extraction unit 1602 uses the extraction rules 1603 to extract character areas corresponding to each predefined item.

In S1706, the OCR unit 402 performs character recognition processing only on the character areas corresponding to each item detected in S1705, and extracts key values, which are character strings for each item. In this embodiment, the character recognition processing for extracting key values is performed on some of the character areas. This makes it possible to reduce the processing load when extracting key values.

Then, the key-value extraction unit 1602 assigns a label indicating the name of each item to records (rows) that contain character regions and character strings corresponding to each item, and generates the key-value extraction result 1800 shown in FIG. 18. The key-value extraction result transmission unit 405 outputs the generated key-value extraction result 1800 to the MFP 300.

The key-value extraction result 1800 in FIG. 18 is the result of performing the processes of S1702 to S1706 on the form image in FIG. 6. The contents stored in columns 1801 to 1803 are the same as those in columns 701 to 703. The character strings of the character recognition results in rows 1804 to 1808 are character strings (key values) extracted as character strings corresponding to items. For this reason, a label is added to column 1803 in rows 1804 to 1808 to indicate that it is an item to be extracted.

Also, as shown in FIG. 18, in this step, character recognition processing is not performed on character areas other than those corresponding to items, so the character recognition results for column 1802 in rows other than rows 1804 to 1808 contain "-", indicating that there is no character string.

Steps S1707 to S1709 are similar to steps S805 to S807, so the explanation will be omitted.

In S1709, if the difference between the character string corrected by the user and the character string of the item to be corrected in the initial key-value extraction result is within a predetermined range (YES in S1709), proceed to S1713. In other words, in this case, the character area corresponding to the item was correctly extracted in S1705. Therefore, proceed to the learning step of S1713 without determining the correct character area corresponding to the item to be corrected instructed by the user, as described below.

In S1709, if the difference between the character string corrected by the user and the character string of the item to be corrected in the original key-value extraction result is not within a predetermined range (NO in S1709), proceed to S1710. In other words, in this case, it is considered that the character area corresponding to the item to be corrected was erroneously extracted in S1705. Therefore, a process is performed to determine the correct character area corresponding to the item to be corrected. The correction result transmission unit 505 transmits the corrected character string entered by the user in S1708 to the key-value extraction server 200.

In S1710, the correction result acquisition unit 406 acquires the character string of the correction result sent from the MFP 300. Then, the OCR unit 402 performs character recognition processing on all character areas of the scanned image detected in S1703.

In this embodiment, character recognition processing is performed on all character areas only if it is determined in S1709 that the difference between the character string corrected by the user and the character string of the item to be corrected in the key-value extraction result is not within a predetermined range. As a result, the key-value extraction result 1800 is updated so that the character string obtained as a result of character recognition in this step is held in the location where "-" was held in the character recognition result of column 1802 in the key-value extraction result 1800 in FIG. 18.

In S1711, the area determination unit 404 of the key-value extraction server 200 calculates the edit distance between the character strings of all the character recognition results recognized in S1710 and the character string corrected by the user. The area determination unit 404 then determines the character area of the item to be corrected based on the edit distance. For example, if there is only one character recognition result with an edit distance of 1 or less, it is determined that the correct character area has been determined.

For example, suppose that the image in FIG. 10 is the scanned image obtained by scanning with the scan execution unit 502 of the MFP 300 in S1702. Also suppose that the user has corrected the character string in the telephone number field, which is the field to be corrected, to "0123-45-6789." In this case, the edit distance between each character string obtained as a result of character recognition processing of all character areas detected from the scanned image in FIG. 10 and the corrected character string "0123-45-6789" is calculated.

Figure 11 shows the key-value extraction result in which the character string obtained by performing character recognition processing on all character areas is added to the character recognition result in column 1108, and the edit distance described above is added to column 1101. In Figure 11, the only character string with an edit distance of 1 or less is "0123-45-6789" stored in the character recognition result in column 1108 in row 1107. Therefore, in Figure 11, the character area in row 1107 is determined to be the correct character area.

In S1712, the area determination unit 404 determines whether the correct character area corresponding to the item to be corrected in S1711 has been determined. If the correct character area has been determined in S1711 (YES in S1712), the process proceeds to S1713.

In S1713, the learning unit 407 updates the learning model by extracting and learning the characteristics of the character area determined in S1711. Alternatively, if the extraction rule 1603 is a database, the extraction rule 1603 is modified and updated so that the position information of the character area determined by the area determination unit 404 is associated with the character area corresponding to the item to be modified.

If the correct character area is not determined in S1711 (NO in S1712), the process does not proceed to S1713, which is a learning step, but proceeds to S1714. This prevents data that contains an incorrect character area corresponding to an item from being used as learning data. The details of step S1714 are the same as those of S816, so a description thereof will be omitted.

Figure 19 shows the transition of screens displayed on the operation panel 307 of the MFP 300 at each step of the flow in Figure 17. Screen 1901 is the screen displayed as a result of the processing of S1707. Screen 1902 shows the screen after the telephone number character string on screen 1901 has been corrected by user operation 1903 in S1708. Screen 1904 is the screen displayed after S1708, and shows the state in which the register button is pressed by user operation 1905. The arrows in Figure 19 show the screen transitions. Screen 1901 transitions to screen 1902, and screen 1902 transitions to screen 1904.

As described above, according to this embodiment, even when extracting key values by performing character recognition processing on some character areas, it is possible to determine the correct character area corresponding to the item to be corrected based on the corrected character string entered by the user. This eliminates the need for the user to perform the cumbersome operation of inputting the character area corresponding to the item, thereby improving usability.

In the above description, the key-value extraction server 200 detects character areas and performs character recognition processing, but the MFP 300 may have functions similar to those of the character area detection unit and OCR unit. In that case, the character area detection and character recognition processing described in the sequence diagram above may be performed by the MFP 300.

<Embodiment 3>
In the third embodiment, as in the second embodiment, a method for determining the correct character area of an item to be corrected when extracting a key value based on the result of character recognition processing executed only on a part of the character area of a scanned image will be described. However, in the third embodiment, a method for determining the correct character area corresponding to the item to be corrected using the position where the user touched the screen will be described. This embodiment will be described focusing on the differences from the second embodiment. The configuration and processing are the same as in the second embodiment unless otherwise specified.

Figure 20 is a sequence diagram showing the processing in the input support system of this embodiment. S2001 to S2009 are the same as S1701 to S1709, so the explanation is omitted.

In this embodiment, if the difference between the character string corrected by the user and the character string of the item to be corrected in the key-value extraction result is not within a predetermined range (NO in S2009), the processes of S2010 to S2015 are performed. The processes of S2010 to S2012 are the same as the processes of S810 to S812.

That is, in S2010, the UI unit 501 displays a preview image on the operation panel 307 based on the image data of the scanned document (document). The UI unit 501 then prompts the user to touch the preview image on the operation panel 307 to indicate the correct text area that corresponds to the item to be corrected as instructed by the user.

In S2011, the user touches the preview screen displayed on the operation panel 307 of the MFP 300 to indicate the position of the correct character area for the item that the user has instructed to be corrected, in which the corrected character string is written.

In S2012, the UI unit 501 of the MFP 300 accepts a touch by the user and detects the touch position on the operation panel 307. The correction result transmission unit 505 converts the touch position on the operation panel 307 touched by the user into touch position coordinates mapped to the image coordinates and determines the touch position coordinates. The UI unit 501 then displays screen 1505 on the operation panel 307 and accepts the user pressing the register button. The correction result transmission unit 505 transmits the corrected character string entered by the user and the touch position coordinates to the key-value extraction server 200.

In S2013, the OCR unit 402 of the key-value extraction server 200 performs character recognition processing on all character areas detected in S2003.

The processing of S2014 is the same as S813. That is, in S2014, the correction result acquisition unit 406 of the key-value extraction server 200 acquires the touch position coordinates, the corrected character string, and the item to be corrected, all sent from the MFP 300. The area determination unit 404 of the key-value extraction server 200 derives the distance between the touch position coordinates and the character areas of all lines of the key-value extraction result. Among the character areas whose distance is equal to or less than a predetermined value, the area determination unit 404 determines the character area in which the character string with the smallest edit distance from the corrected character string is recognized as the correct character area corresponding to the item to be corrected. The predetermined value is, for example, 100.

For example, touch position 1210 shown in FIG. 12 is the position touched by the user in S2011, and the touch position coordinates are determined to be (x, y) = (1110, 493) in S2012.

Figure 13 shows the results of performing character area detection, character recognition processing, and key-value extraction on the image data in Figure 12. It is assumed that the label "phone number" in column 1312 is not assigned to the correct character area. Column 1311 in Figure 13 holds the edit distance between the character string "0123-45-6789" corrected by the user's input and the character string resulting from character recognition in Figure 13.

Column 1301 in FIG. 13 is a column that holds the distance between the position of a character area and the touch position in the image touched by the user, and shows the distance from the character area when the touch position coordinates are (1110, 493). Rows 1307 to 1310 are records that have character areas whose distance from the touch position coordinates is a predetermined value of 100 or less. Of rows 1307 to 1310, the area determination unit 404 determines that the character area in row 1308, which has the smallest edit distance in column 1311, is the correct character area that corresponds to the "phone number" field, which is the field that the user has instructed to correct.

Steps S2015 to S2017 are the same as steps S1712 to S1714, so the explanation will be omitted.

Note that this embodiment may be combined with the method of embodiment 2. For example, in the sequence diagram of FIG. 17 for embodiment 2, if the correct character area cannot be determined in S1711, steps S2010 to S2012 may be performed, the character recognition process of S2013 may be skipped, and the processes of S2014 to S2017 may be performed.

As described above, according to this embodiment, even when extracting key values by performing character recognition processing on some character areas, the correct character area corresponding to an item can be determined based on the corrected character string entered by the user. This eliminates the need for the user to perform the cumbersome operation of entering a character area corresponding to an item, thereby improving usability.

<Other embodiments>
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

300 MFP
501 UI unit 504 Key-value extraction result acquisition unit 506 Area determination unit 505 Correction result transmission unit

Claims (17)

an acquisition means for acquiring an extraction result including a processing result including information on a character string in an image and a character area in which the character string is recognized, obtained by subjecting image data of an image to character recognition processing, and information on a character string and a character area in the image corresponding to each of the predetermined items, the information being output by inputting the processing result into a learning model trained to output information on a character string and a character area corresponding to each of the predetermined items;
a display control means for displaying character strings corresponding to the respective predetermined items on a display unit based on the extraction result;
a receiving means for receiving, from a user, an instruction to correct a character string corresponding to any one of the predetermined items displayed on the display unit;
a correction means for determining a character area corresponding to an item for which a user has instructed correction based on the corrected character string and the processing result, and correcting the extraction result based on the determination;
an output means for outputting the extraction result corrected by the correction means as data for learning the learning model;
13. An information processing device comprising: The correction means is
The information processing device according to claim 1, characterized in that, among the character areas in the image recognized by the character recognition process, a character area in which a character string that is the same as the corrected character string is recognized is determined to be the character area corresponding to the item instructed by the user to be corrected. The correction means is
The information processing device according to claim 1 or 2, characterized in that, when there is one character string among the character strings recognized in the character recognition process, the edit distance from the corrected character string being equal to or less than a predetermined value, the character region in which the character string having an edit distance equal to or less than the predetermined value is recognized among the character regions in the image recognized in the character recognition process is determined to be the character region corresponding to the item instructed to be corrected by the user. The information processing device according to claim 3, characterized in that the predetermined value is 1. The correction means is
5. The information processing device according to claim 1, wherein when a difference between a character string extracted based on the output result of the learning model corresponding to an item that the user has instructed to be corrected and the corrected character string is within a predetermined range, the character area in the extraction result corresponding to the item that the user has instructed to be corrected is not corrected. The information processing device according to claim 5 , wherein the case where the character string is within the predetermined range is a case where an edit distance between a character string extracted based on an output result of the learning model and the corrected character string is 1 or less. a second receiving means for receiving a touch position where the user touches the display unit as a position in the image corresponding to the item that the user has instructed to be corrected;
The information processing device according to any one of claims 1 to 6, further comprising: a second correction means for determining the character area corresponding to the item that the user has instructed to be corrected based on the touch position, the corrected character string, and the processing result, and correcting the extraction result based on the determination. The second correction means includes:
The information processing device according to claim 7, characterized in that a distance is derived between a position in the image based on the touch position and a position of each character area in the image recognized by the character recognition process, and among the character areas where the distance is within a predetermined range, a character area in which a character string having a smallest edit distance with the corrected character string is recognized by the character recognition process is determined to be the character area corresponding to the item instructed to be corrected by the user. when the correction means cannot determine a character area corresponding to the item instructed to be corrected by the user, the second correction means determines a character area corresponding to the item instructed to be corrected by the user;
The information processing device according to claim 7 or 8, characterized in that, when the second correction means determines a character area corresponding to an item that the user has instructed to be corrected, the output means outputs the extraction result corrected based on the determination by the second correction means as data for learning by the learning model. The output means includes:
10. The information processing apparatus according to claim 9, wherein, when the second correction means is unable to determine a character area corresponding to an item instructed by the user to be corrected, the data for learning is not output. The information processing apparatus according to claim 1 , further comprising an image data transmission means for transmitting image data of the image to a server that performs the character recognition process, the image being a scanned image obtained by scanning an original document. the extraction result acquired by the acquiring means is a table in which character strings in the image obtained by performing the character recognition process on the image data of the image, character regions in the image where the character strings are recognized, and values indicating which of the character strings corresponds to each of the predetermined items are associated with each other;
12. The information processing apparatus according to claim 1, wherein, when the table is modified, the output means outputs the table reflecting the modification as the data for the learning. character recognition means for performing character recognition processing on image data of an image;
a transmitting means for transmitting to the information processing device according to any one of claims 1 to 12, the extraction result including a processing result including information on a character string in the image obtained by the character recognition processing by the character recognition means and a character area in which the character string is recognized, and information on a character string and a character area in the image corresponding to each of the predetermined items output by inputting the processing result into the learning model;
A learning means for acquiring data for the learning model from the information processing device and training the learning model based on the data for learning;
A server comprising: a character string after correction in the item instructed by the user to be corrected, and a touch position where the user touched the display unit as a position in the image corresponding to the item instructed by the user to be corrected;
A correction result acquisition means for acquiring the result;
and a third correction means for determining a character area corresponding to an item for which the user has instructed to correct, based on the touch position, the corrected character string, and the processing result, and correcting the extraction result based on the determination.
The learning means includes:
The server according to claim 13 , characterized in that, when the data for learning is not output from the information processing device, the learning model is trained based on the extraction result corrected by the third correction means. A system including an information processing device and a server,
The server,
character recognition means for performing character recognition processing on image data of an image;
a transmitting means for transmitting to the information processing device an extraction result including a processing result including information on a character string in the image obtained by character recognition processing by the character recognition means and a character area in which the character string is recognized, and information on a character string and a character area in the image corresponding to each of the predetermined items, which is output by inputting the processing result into a learning model trained to output information on a character string and a character area corresponding to each of the predetermined items defined in advance;
having
The information processing device
an acquisition means for acquiring the extraction result transmitted from the transmission means ;
a display control means for displaying character strings corresponding to the respective predetermined items on a display unit based on the extraction result;
a receiving means for receiving, from a user, an instruction to correct a character string corresponding to any one of the predetermined items displayed on the display unit;
a correction means for determining a character area corresponding to an item for which a user has instructed correction based on the corrected character string and the processing result, and correcting the extraction result based on the determination;
an output means for outputting the extraction result corrected by the correction means as data for learning the learning model;
The server ,
Further comprising a learning means for learning the learning model based on data for learning the learning model output from the output means of the information processing device.
A system characterized by: an acquisition step of acquiring an extraction result including a processing result including information on a character string in an image and a character area in which the character string is recognized, obtained by performing character recognition processing on image data of the image, and information on a character string and a character area in which the character string is recognized , in the image corresponding to each of the predetermined items, which is output by inputting the processing result into a learning model trained to output information on a character string and a character area corresponding to each of the predetermined items;
a display control step of displaying character strings corresponding to the respective predetermined items on a display unit based on the extraction result;
a receiving step of receiving, from a user, an instruction to correct a character string corresponding to any one of the predetermined items displayed on the display unit;
a correction step of determining a character area corresponding to an item for which a user has instructed correction based on the corrected character string and the processing result, and correcting the extraction result based on the determination;
an output step of outputting the extraction result corrected in the correction step as data for learning by the learning model;
13. An information processing method comprising: A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 12 .

Images