JP7767809B2 - Information processing system and program - Google Patents

Description

The present invention relates to an information processing system and a program.

Patent document 1 discloses a process in which, if the character recognition process for a document is unsuccessful and the image is unclear, the image management server issues a scan ticket to the image processing device with high-resolution scanning already set.

Japanese Patent Application Laid-Open No. 2019-197596

By analyzing the images obtained by reading a plurality of recording media, the information written on each recording medium can be obtained.
When multiple recording media are read at once, the images of the individual recording media included in the read image become small, which may reduce the accuracy of obtaining information written on the recording media.
The object of the present invention is to improve the accuracy of acquiring information recorded on recording media compared to acquiring information recorded on individual recording media by analyzing only the scanned image obtained by scanning multiple recording media at once.

The invention described in claim 1 is an information processing system having a processor, which identifies a condition-satisfying image, which is a medium image that represents the recording medium and satisfies specific conditions, from among the read images obtained by reading a plurality of recording media with an image reading device, and outputs information that enables at least a portion of a corresponding recording medium, which is a recording medium among the plurality of recording media that corresponds to the condition-satisfying image, to be read at a size larger than the size in the condition -satisfying image and to be read by another device other than the image reading device .
The invention described in claim 2 is an information processing system described in claim 1, in which the processor outputs information that enables a specific portion of the corresponding recording medium to be read at a size larger than the size in the condition-satisfying image and to be read by the other device .
The invention described in claim 3 is an information processing system described in claim 2, in which the processor outputs information that enables the portion of the compatible recording medium on which characters are written to be read at a size larger than the size in the condition-satisfying image and to be read by the other device .
The invention described in claim 4 is an information processing system described in claim 2, in which the processor outputs information that enables parts of the compatible recording medium from which information could not be acquired and/or parts from which the acquired information does not satisfy predetermined conditions to be read at a size larger than the size in the condition-satisfying image and to be read by the other device .
A fifth aspect of the present invention is the information processing system according to the first aspect, wherein the other device is a portable terminal device carried by the user.
The invention described in claim 6 is an information processing system described in claim 1, in which the processor outputs information that causes information about the compatible recording medium to be displayed on a display unit referenced by a user who performs an operation to enable at least a portion of the compatible recording medium to be read in the larger size.
The invention described in claim 7 is an information processing system described in claim 6 , in which the processor outputs information that causes information indicating the location of at least a portion of the compatible recording medium to be displayed on the display unit.
The invention described in claim 8 is an information processing system described in claim 1, in which the processor further outputs information that causes the scanned image containing multiple media images to be displayed on a display unit provided in the other device .
The invention described in claim 9 is an information processing system described in claim 8, in which the processor outputs information that causes the scanned image, to which an image indicating which of the multiple medium images contained in the scanned image is the condition-satisfying image, to be displayed on the display unit provided in the other device.
The invention described in claim 10 is an information processing system described in claim 1, in which the processor outputs information that causes information obtained from the image obtained by reading at the large size to be displayed on a display unit provided on the other device .
The invention described in claim 11 is an information processing system described in claim 1, in which the processor displays, on a display unit provided in the image reading device that reads the multiple recording media, the written information, which is information written on the compatible recording medium and which is obtained by the other device analyzing the reading results obtained by reading the compatible recording medium by the other device.
The invention described in claim 12 is a program for enabling a computer to realize the following functions: a function of identifying a condition-satisfying image, which is a medium image that represents the recording medium and satisfies specific conditions, from among the read images obtained by reading a plurality of recording media with an image reading device; and a function of outputting information that enables at least a portion of a corresponding recording medium, which is a recording medium among the plurality of recording media that corresponds to the condition-satisfying image, to be read at a size larger than the size in the condition-satisfying image and to be read by another device other than the image reading device .

According to the invention of claim 1, the accuracy of obtaining information recorded on recording media can be improved compared to when obtaining information recorded on individual recording media by analyzing only the read images obtained by reading multiple recording media at once.
According to the second aspect of the present invention, a specific portion of the compatible recording medium can be read at a size larger than the size of the condition satisfying image.
According to the third aspect of the present invention, the portion of the compatible recording medium on which characters are written can be read at a size larger than the size of the condition satisfying image.
According to the invention of claim 4, it is possible to make it possible to read portions of the compatible recording medium from which information could not be acquired and/or portions from which the acquired information does not satisfy predetermined conditions at a size larger than the size of the condition-satisfying image.
According to the invention of claim 5 , at least a portion of the compatible recording medium can be read at a size larger than the size of the condition-satisfying image and can be read by a portable terminal device carried by the user.
According to the invention of claim 6 , information about the compatible recording medium can be displayed on a display unit referenced by a user who performs an operation to read at least a portion of the compatible recording medium in a large size.
According to the seventh aspect of the present invention, information indicating the position of at least a part of the compatible recording medium can be displayed on the display unit.
According to the eighth aspect of the present invention, the scanned image including a plurality of medium images can be displayed on a display unit provided in another device .
According to the invention of claim 9 , a scanned image to which an image indicating which of the multiple medium images contained in the scanned image is a condition-satisfying image can be added can be displayed on a display unit provided in another device.
According to the tenth aspect of the present invention, information obtained from an image obtained by scanning in a large size can be displayed on a display unit provided in another device .
According to the invention of claim 11, the written information, which is information written on a compatible recording medium, and which is obtained by another device analyzing the reading results obtained by reading the compatible recording medium by another device, can be displayed on a display unit provided in an image reading device that has read multiple recording media.
According to the invention of claim 12, the accuracy of obtaining information recorded on recording media can be improved compared to when obtaining information recorded on individual recording media by analyzing only the read images obtained by reading multiple recording media at once.

FIG. 1 is a diagram showing the overall configuration of an information processing system. FIG. 2 illustrates an example of a hardware configuration of a management server. FIG. 2 is a diagram illustrating an example of the hardware configuration of a user terminal. FIG. 2 is a diagram illustrating functional units of the image forming apparatus. FIG. 2 is a front view of the image forming apparatus. FIG. 2 is a sequence diagram showing the flow of processing executed in the information processing system of the present embodiment. 10A and 10B are diagrams illustrating examples of displays on a display unit of an image forming apparatus. 10A and 10B are diagrams illustrating examples of displays on a display unit of an image forming apparatus. FIG. 10 is a diagram showing an example of a display on a user terminal. FIG. 10 is a diagram showing a screen displayed on a user terminal when a complementary selection image is selected by a user. FIG. 10 is a diagram showing a screen displayed on a user terminal after a user logs in. 10 is a diagram showing the display unit of the user terminal after one condition-satisfying image from among a plurality of condition-satisfying images has been selected by the user. FIG. FIG. 10 is a diagram showing a screen displayed on the display unit of the user terminal after re-photographing of the compatible recording medium has been performed. FIG. 10 is a diagram showing an example of a display on a user terminal. 10A and 10B are diagrams illustrating examples of displays on a display unit provided in the image forming apparatus.

Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.
FIG. 1 is a diagram showing the overall configuration of an information processing system 1 according to this embodiment.
The information processing system 1 is provided with a management server 300 as an example of an information processing device. Furthermore, an external server 400 is provided to which information obtained by the management server 300 is transmitted. The information processing system 1 is also provided with a user terminal 500 owned by each user, and an image forming device 600 having an image reading function.
Although not shown in FIG. 1, a plurality of user terminals 500 and a plurality of image forming apparatuses 600 are provided.

The user terminal 500 is, for example, a smartphone.
The user terminal 500 may be any device other than a smartphone that has a camera (capability to take photographs and read images), and may be a tablet terminal with a camera, a camera with communication capabilities, or a game console with a camera and communication capabilities.
In this embodiment, the user terminal 500, the image forming apparatus 600, and the management server 300 are interconnected via a communication line 310 such as an Internet line. The management server 300 and the external server 400 are also interconnected via a communication line 320 such as an Internet line.

FIG. 2 is a diagram showing an example of the hardware configuration of the management server 300.
The management server 300 has a control unit 101 that controls the operation of the entire device, a hard disk drive 102 as an example of an information storage device that stores information, and a network interface 103 that realizes communication via a LAN (Local Area Network) cable or the like.

The control unit 101 has a CPU (Central Processing Unit) 111 as an example of a processor, a ROM (Read Only Memory) 112 in which programs are stored, and a RAM (Random Access Memory) 113 used as a work area.
The CPU 111 may be a multi-core processor. The ROM 112 may be a rewritable non-volatile semiconductor memory. The control unit 101 is a so-called computer.
In this embodiment, the CPU 111 executes a program stored in the ROM 112 or the like to perform the processing described below.

The hard disk drive 102 is a device that reads and writes data from a nonvolatile storage medium, which is a disk-shaped substrate coated with a magnetic material on its surface, although the nonvolatile storage medium may also be a semiconductor memory or a magnetic tape.
In addition, the management server 300 may also include input devices such as a keyboard and a mouse, and a display device such as a liquid crystal display, as needed.
The control unit 101, hard disk drive 102, and network interface 103 are connected via a bus 104 and a signal line (not shown).

Here, the program executed by CPU 111 can be provided to management server 300 in a state where it is stored on a computer-readable recording medium such as a magnetic recording medium (magnetic tape, magnetic disk, etc.), an optical recording medium (optical disk, etc.), a magneto-optical recording medium, or a semiconductor memory.
Furthermore, the program executed by the CPU 111 may be provided to the management server 300 using a communication means such as the Internet.

1 shows functional units implemented by the CPU 111 of the management server 300. Note that Fig. 1 shows functional units related to reading receipts.
As shown in FIG. 1, the management server 300 includes a receipt registration unit 351 , a failure location identification unit 352 , and an information acquisition unit 353 .
The receipt registration unit 351 performs processes related to receipts, such as registering information obtained from receipts (described later) in the external server 400 .
The failure location identifying unit 352 identifies the location on the receipt where acquisition of written information failed.

The information acquisition unit 353 has an image separation unit 353A, an image rotation unit 353B, and an OCR unit 353C, and analyzes images obtained by reading multiple receipts (described below) and extracts information contained in each receipt.
Here, the receipt registration unit 351, the failure location identification unit 352, and the information acquisition unit 353 are realized by the CPU 111 of the management server 300 executing programs stored in the ROM 112 or the like.

In this embodiment, each process executed by the management server 300, which will be described below, is performed by the CPU 111, which is an example of a processor. In other words, in this embodiment, the CPU 111, which is an example of a processor, executes a program stored in the ROM 112 or the like, thereby performing the process described below.
In other words, in this embodiment, the CPU 111, which is an example of a processor, executes a program stored in the ROM 112 or the like to realize the respective functional units shown in Fig. 1, such as the receipt registration unit 351, the failure location identification unit 352, and the information acquisition unit 353. Then, in this embodiment, the processes described below are executed by these respective functional units.

Fig. 3 is a diagram showing an example of the hardware configuration of the user terminal 500. The configuration shown in Fig. 3 assumes that the user terminal 500 is a smartphone.
The user terminal 500 has a control unit 201 that controls the operation of the entire device, a memory card 202 that stores various data, various communication interfaces 203 that comply with wireless communication standards, an input device 204 such as a touch sensor, a display unit 205 composed of a liquid crystal display, an organic EL (Electro Luminescence) display, etc., a GPS (Global Positioning System) sensor 206, and a camera 207.

The control unit 201 includes a CPU 211 as an example of a processor, a ROM 212 storing programs, and a RAM 213 used as a work area. The CPU 211 may be a multi-core processor. The ROM 212 may be a rewritable non-volatile semiconductor memory.
The communication interface 203 is, for example, an interface used for connection to a mobile communication system or an interface used for connection to a wireless LAN.

The GPS sensor 206 is a sensor that receives radio waves from GPS satellites and measures the position of the user terminal 500. The latitude, longitude, and altitude information output from the GPS sensor 206 indicates the current position of the user terminal 500. The GPS sensor 206 may be compatible with an indoor positioning system.
The camera 207 is configured with a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS), and captures images of objects located around the user terminal 500 .

4 is a diagram showing functional units of the image forming apparatus 600. FIG.
The image forming apparatus 600 of this embodiment has a function of reading an image on a recording medium such as a receipt, and can be regarded as an image reading apparatus.
As shown in FIG. 4, each functional unit constituting an image forming apparatus 600 of this embodiment is connected to a bus 601 and exchanges data via this bus 601 .

The display unit 606 displays an image and notifies information to the operator who operates the image forming apparatus 600. The display unit 606 is formed in a rectangular shape as shown in Fig. 5. The display unit 606 is configured by, for example, a liquid crystal display or an organic EL display, and notifies information to the user who operates the image forming apparatus 600.
The operation accepting unit 607 (see FIGS. 4 and 5) accepts operations from the user. Specifically, the operation accepting unit 607 accepts operations by the user on the display content displayed on the display unit 606.

Here, the display unit 606 and the operation acceptance unit 607 are configured by, for example, a touch panel display, and the display unit 606 and the operation acceptance unit 607 are arranged in an overlapping state.
In this embodiment, the operation acceptance unit 607 is configured by a touch panel display, but the operation acceptance unit 607 may be configured by a pointing device such as a mouse.

The display unit 606 (see FIG. 4) is controlled by a control unit 660. Furthermore, in this embodiment, a user's operation on the display content on the display unit 606 is received by an operation receiving unit 607.
In this embodiment, in response to a user operation accepted by the operation accepting unit 607, a process corresponding to this operation is executed.

The image reading unit 608, which is an example of a reading means, is configured by a so-called scanner device, reads an image on a set recording medium, and generates a read image (image data) of the recording medium.
In this embodiment, as will be described later, a plurality of receipts are read by the image reading unit 608. In other words, a plurality of recording media are read by the image reading unit 608.

The image forming unit 609 forms a toner image on a sheet of paper according to image data using, for example, an electrophotographic method. Note that the image forming unit 609 may also form an image using other methods, such as an inkjet head method.
The communication unit 610 is connected to the communication line 310 (see FIG. 1) and functions as a communication interface for communicating with the management server 300 .

The image processing unit 611 performs image processing such as color correction and gradation correction on the image represented by the image data.
The storage unit 605 is configured by a hard disk device or the like, and stores, for example, data received by the communication unit 610 and scanned images (image data) generated by the image scanning unit 608 .
The control unit 660 controls each unit of the image forming apparatus 600. The control unit 660 is configured with a CPU (Central Processing Unit) 602 as an example of a processor, a ROM (Read Only Memory) 603, and a RAM (Random Access Memory) 604.
The ROM 603 stores a program executed by the CPU 602. The CPU 602 reads out the program stored in the ROM 603 and executes the program using the RAM 604 as a work area.

Here, the program executed by CPU 602 can be provided to image forming apparatus 600 in a state where it is stored on a computer-readable recording medium such as a magnetic recording medium (such as a magnetic tape or a magnetic disk), an optical recording medium (such as an optical disk), a magneto-optical recording medium, or a semiconductor memory.
The program executed by the CPU 602 may be downloaded to the image forming apparatus 600 using a communication means such as the Internet.

In this embodiment, the term "processor" refers to a processor in a broad sense, and includes general-purpose processors (e.g., CPU: Central Processing Unit, etc.) and dedicated processors (e.g., GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, programmable logic device, etc.).
Furthermore, the operations of the processors may not only be performed by a single processor, but may also be performed by multiple processors located at physically separate locations working together. The order of the operations of the processors is not limited to the order described in this embodiment, and may be changed.

6 is a sequence diagram showing the flow of processing executed in the information processing system 1 of this embodiment. The flow of processing executed in this embodiment will be described below with reference to this sequence diagram.
In this embodiment, first, a user places a plurality of receipts, which are an example of a plurality of recording media, in the image reading unit 608 (see FIG. 4) of the image forming apparatus 600. The image reading unit 608 is provided with a platen made of transparent glass, and the plurality of receipts are placed in a line on this platen.

Next, the user presses a start button (not shown) provided on the image forming device 600 to instruct the device to start reading the multiple receipts (step S101). In other words, the user instructs the device to start reading the information written on the multiple receipts.
As a result, the image reading unit 608, which is an example of a reading means, reads the multiple receipts (step S102) and obtains read images.
More specifically, the image reading unit 608 obtains a read image that includes a plurality of receipt images, each of which represents a receipt.

In other words, the image representing each receipt is an example of a medium image, and the image reading unit 608 obtains a read image that includes a plurality of medium images.
In this embodiment, the scanned image is then transmitted to the management server 300 (step S102), and the management server 300 acquires the scanned image.
Next, the management server 300 acquires written information, which is information written on each receipt, from the scanned image.

Specifically, first, the receipt registration unit 351 extracts rectangular images (hereinafter referred to as "rectangular images") contained in the scanned image from the scanned image (step S103).
Next, the image separating unit 353A of the information acquiring unit 353 separates each rectangular image from other images located around the rectangular image (step S104). Next, the image rotating unit 353B rotates each rectangular image to make each rectangular image upright (step S105).

Next, the OCR unit 353C performs OCR (Optical Character Recognition) processing (step S106) to obtain information contained in each of the erect rectangular images.
In other words, the OCR unit 353C acquires written information, which is information written on each upright receipt.
Specifically, the OCR unit 353C acquires text information written on the receipt, such as the date, amount, etc., as written information written on each receipt. In this embodiment, the information written on the receipt is acquired for each receipt.

Then, in this embodiment, the information acquisition unit 353 outputs each image of the receipt (rectangular image in an upright state) (hereinafter referred to as a "medium image") in association with the above-mentioned written information acquired from each of these medium images.
In this embodiment, there may be cases where the OCR unit 353C fails to acquire the written information, and the information acquisition unit 353 complements the portion where the acquisition of the written information failed with a predetermined default value.
In other words, for a portion where the information acquisition unit 353 was unable to acquire description information, the information acquisition unit 353 outputs a predetermined default value as description information. In this case, description information different from the original description information is output.

More specifically, if the printing on the receipt is faint or if the receipt is curved or folded, it becomes difficult to read the receipt.
In this case, a situation occurs in which it becomes impossible to obtain the description information, and in this case, as described above, the information acquisition unit 353 complements the information with a predetermined default value for the part where it was unable to obtain the description information, and outputs this default value.

In this embodiment, the failure part identification unit 352 extracts an image of a part where the OCR unit 353C failed to acquire written information (hereinafter, "failure part image"). Then, in this embodiment, the information acquisition unit 353 outputs the extracted failure part image in association with a medium image, which is an image representing each receipt.
As a result, in this embodiment, the medium image, the written information, and the image of the failure portion are output from the information acquisition unit 353 (step S107).

For receipts for which acquisition of the described information has failed, the medium image, the described information, and an image of the failed part are output from the information acquisition unit 353. In this case, the described information is the default value described above.
Furthermore, for receipts for which the described information has been successfully acquired, a medium image and the described information are output from the information acquisition unit 353. In this case, the described information is not the default value described above, but information acquired by the OCR unit 353C.

Thereafter, in this embodiment, the receipt registration unit 351 determines whether the OCR process was successful (step S108).
More specifically, the receipt registration unit 351 determines whether or not the information output from the information acquisition unit 353 includes an image of the failure location, for example.
If the receipt registration unit 351 finds that the image of the failed portion is included, it determines that the OCR process on the receipt from which the image of the failed portion was obtained has failed.
If the receipt registration unit 351 determines that the OCR processing has failed, in other words, that it has been unable to obtain the written information, it outputs information about the receipt for which it has been unable to obtain the written information to the image forming device 600.

More specifically, in this case, the receipt registration unit 351 outputs to the image forming device 600 the medium image, the description information (default value), and the image of the failure area for the receipt for which the description information could not be obtained (step S109).
Here, the image of the failed portion is an example of information about a portion of a receipt where an attempt was made to obtain written information, among the portions of the receipt where the written information could not be obtained.

In this embodiment, the receipt registration unit 351 outputs, to the image forming apparatus 600, information about the location where the receipt registration unit 351 attempted to obtain the written information.
Here, this image forming device 600 is the image forming device 600 that read the above-mentioned multiple receipts from which the written information was obtained. In this embodiment, information such as images of the failure areas is output to this image forming device 600 that read the receipts.

In this embodiment, the medium image and written information are also sent to the image forming apparatus 600 together with the image of the failure portion.
That is, in this embodiment, as information about a receipt for which the description information could not be obtained, in addition to an image of the failed portion, a medium image and the description information are output.
The medium image is an image that represents the entire receipt. The written information in this case is the default value as described above. In other words, the written information in this case is a predetermined fixed value.

In this embodiment, the display unit 606 (see FIG. 5) of the image forming apparatus 600 then displays information about the receipt for which the information could not be obtained (step S110).
Specifically, in this embodiment, the display unit 606 displays a medium image, the written information, and an image of the failure area as information about the receipt for which the written information could not be obtained.

FIG. 7 is a diagram showing an example of a display on the display unit 606 of the image forming apparatus 600. As shown in FIG.
As described above, the display unit 606 of this embodiment displays the medium image G1, written information G2, and failure location image G3.
Here, the medium image G1 is a scanned image obtained by scanning the receipt with the image forming device 600, and is an image representing the entire receipt. In other words, the medium image G1 is a rectangular image separated from the scanned image by the image separating unit 353A.

Furthermore, as described above, the medium image G1 is an image of a receipt from which the written information could not be acquired.
In addition, in this embodiment, the image indicated by the symbol 7A shows the failure area image G3, which shows an image of the part of the receipt where the written information G2 (the original correct written information) could not be obtained.

Here, the portion of written information G2 displayed in FIG. 7 is not an image, but is text information obtained by the OCR unit 353C, or a default value obtained by the above-mentioned complementation.
More specifically, in this example, the information "Total,""Date of receipt," and "Month/Day" in the description information G2 are text information obtained by the OCR unit 353C. Also, in this example, the information "¥10,000" in the description information G2 is the default value obtained by the above-mentioned interpolation.

In this example, written information G2 obtained by complementing with default values is displayed in the display area of the failure location image G3.
In this example, the display area of the failure area image G3 displays "Total" as an example of amount information and "Date of receipt" as an example of date information, but in this example, the amount "￥10,000" displayed in the "Total" section is not the correct value, but is the default value as described above.
In this embodiment, an example has been described in which a default value is added and the default value is displayed, but the default value may not be displayed and the area where the default value is displayed may be left blank.

Furthermore, the display unit 606 displays a screen G4 (hereinafter referred to as "input screen G4") as part of the screen, which allows the user to input information written on the receipt.
In other words, the display unit 606 of the image forming apparatus 600 displays an input screen G4 for the user to input the correct written information G2.

In this embodiment, the user can input the correct information written on the receipt by operating this input screen G4. In other words, in this embodiment, the user can modify the default values obtained by the management server 300.
Specifically, in this example, as described above, the amount is incorrect, and the user can operate the display unit 606 to input information regarding the correct amount.

In the present embodiment, as described above, the display unit 606 of the image forming apparatus 600 displays the failure location image G3 as an example of information about the location where the description information G2 was attempted to be acquired.
Incidentally, the information about the location where the description information G2 was attempted to be obtained is not limited to this failure location image G3.

Other examples of information about the location where the written information G2 was attempted to be obtained include, for example, "The amount column of the receipt titled XX could not be read."
Other information that may be given includes, for example, "The date column of the third receipt from the top of the lined-up receipts could not be read."
The information about the location where the description information G2 was attempted to be obtained is not limited to image information such as the failed location image G3, but may also be text information as described above.

The success or failure of reading each receipt can be determined by, for example, notifying the user of the reading results and allowing the user to make a decision. However, in this case, the user must determine the success or failure of each receipt, which is time-consuming.
In contrast, in this embodiment, the parts where reading failed are identified and then notified to the user, which eliminates the need for the user to determine the success or failure of every receipt, reducing the burden on the user.

In this embodiment, as shown in FIG. 7, a selection button 800 "Rescan with mobile" is displayed on the display unit 606 of the image forming apparatus 600.
In this embodiment, when this selection button 800 is selected by the user, the display unit 606 of the image forming device 600 displays the information shown in FIG. 8 (a diagram showing an example of a display on the display unit 606 of the image forming device 600).
In the display shown in Fig. 8, the area indicated by the reference symbol 8A displays the method for reading the receipt using the user terminal 500. Also, in the display shown in Fig. 8, an input image 606X is displayed for the user to input that reading of the receipt by the user terminal 500 has been completed.

In this embodiment, when the selection button 800 shown in FIG. 7 is selected by the user, information indicating that the selection button 800 has been selected is transmitted to the user terminal 500 via the management server 300 .
Accordingly, in this embodiment, in addition to the display shown in FIG. 8 being displayed on the image forming device 600, the display shown in FIG. 9 (a diagram showing an example of a display on the user terminal 500) is displayed on the user terminal 500, which is an example of another reading device.

In this display shown in FIG. 9, the message "Failed to read receipt" is displayed at the top of the display unit 205.
Furthermore, a plurality of selection images 205A, which are images selected by the user, are displayed on the display unit 205. Each of the selection images 205A corresponds to an application installed on the user terminal 500.

When the user selects an application installed on the user terminal 500, the user selects this selection image 205A.
In this embodiment, one of the selection images 205A is a selection image 205B (hereinafter referred to as "complementary selection image 205B") that corresponds to a complementary application, which is an application used to complement receipt information.

FIG. 10 is a diagram showing a screen displayed on the user terminal 500 when the complementary selection image 205B shown in FIG. 9 is selected by the user.
In this embodiment, when the user selects the complementary selection image 205B shown in FIG. 9, the complementary application is started, and the screen shown in FIG. 10 is displayed on the display unit 205 of the user terminal 500.
The screen shown in FIG. 10 is a login screen used for user login.

This login screen has a user ID input field 91 for inputting a user ID and a password input field 92 for inputting a password.
The user enters his/her user ID and password to log in. If the user has already logged in, the display of the screen shown in Fig. 10 is omitted.
Additionally, if the user has already entered a user ID and password into the image forming device 600, login may be performed on the user terminal 500 if these user ID and password match the user ID and password that the user enters by operating the user terminal 500.

FIG. 11 shows a screen displayed on the user terminal 500 after the user logs in. As shown in FIG.
In this embodiment, when a user logs in, the screen shown in FIG. 11 is displayed on the display unit 205 of the user terminal 500.
On this screen shown in FIG. 11, the method of using the complementary application selected by the user is displayed in the upper part of the screen, in a location indicated by reference numeral 11A.

Furthermore, on this screen shown in FIG. 11, an update button 93 for this screen is displayed at the location indicated by reference numeral 11B.
If information about the receipt that failed to be read is not displayed in the area indicated by reference symbol 11C in Fig. 11, the user presses the update button 93. This updates the screen on the user terminal 500, and information about the receipt that failed to be read is displayed as indicated by reference symbol 11D in Fig. 11.
Furthermore, on this screen shown in FIG. 11, date and time information indicating the date and time when the receipt reading failed is displayed in the area indicated by reference numeral 11E.

Furthermore, on this screen, the scanned image 94 including the medium image G1, which is an image of a receipt, is displayed in the area indicated by the reference numeral 11C.
In other words, this screen displays a scanned image 94 including a plurality of medium images G1, which are images of a plurality of receipts scanned by the image scanning unit 608 (see FIG. 4).

In this embodiment, in step S102 of FIG. 6, the scanned image 94 is transmitted from the image forming apparatus 600 to the management server 300, and thereafter the management server 300 holds this scanned image 94.
In this embodiment, the scanned image 94 held by the management server 300 is transmitted to the user terminal 500, and the scanned image 94 is displayed on the user terminal 500 as shown in FIG.

In this embodiment, when the selection button 800 shown in FIG. 7 is selected by the user, information indicating that the selection button 800 has been selected by the user is transmitted from the image forming apparatus 600 to the management server 300 .
In response to this, in this embodiment, a scanned image 94 is sent from the management server 300 to the user terminal 500, and as shown by symbol 11D in Figure 11, this scanned image 94, which includes multiple medium images G1, is displayed on the display unit 205 of the user terminal 500.

More specifically, in this embodiment, when the selection button 800 shown in Figure 7 is selected by the user, the CPU 111 of the management server 300 (see Figure 2) outputs information that causes a scanned image 94 containing multiple medium images G1 to be displayed on the display unit 205 of the user terminal 500.
As a result, in this embodiment, the scanned image 94 is sent from the management server 300 to the user terminal 500, and the scanned image 94 containing multiple medium images G1 is displayed on the display unit 205 of the user terminal 500, as shown in Figure 11.

In addition, the above-mentioned "information for causing the scanned image 94 to be displayed on the display unit 205 of the user terminal 500" output by the CPU 111 of the management server 300 may be explicit information that specifically instructs that the scanned image 94 be displayed on the user terminal 500, or it may be information that is not this explicit information.
An example of information that is not explicit information is information indicating that the selection button 800 (see FIG. 7) has been selected.

When the CPU 111 of the management server 300 outputs “information for displaying the scanned image 94 on the display unit 205 of the user terminal 500 ,” the management server 300 transmits the scanned image 94 to the user terminal 500 .
At this time, the management server 300 transmits the scanned image 94 to the management server 300, to which an image indicating the condition-satisfying image has been added, as will be described later.

In addition, when the CPU 111 of the management server 300 outputs "information for causing the scanned image 94 to be displayed on the display unit 205 of the user terminal 500," the management server 300 transmits this "information for causing the scanned image 94 to be displayed on the display unit 205 of the user terminal 500" to the user terminal 500.
This "information that causes the scanned image 94 to be displayed on the display unit 205 of the user terminal 500" may be explicit information, as described above, or may be non-explicit information.

When the user terminal 500 receives this "information for displaying the scanned image 94 on the display unit 205 of the user terminal 500," the user terminal 500 performs a display process for the scanned image 94.
The "information that causes the scanned image 94 to be displayed on the display unit 205 of the user terminal 500" may be any information that results in the scanned image 94 being displayed on the display unit 205 of the user terminal 500, and is not limited to the explicit information described above.

In this embodiment, when the scanned image 94 is transmitted to the user terminal 500, the CPU 111 of the management server 300 identifies, from the scanned image 94, a condition satisfying image 95 that is a medium image G1 that satisfies a specific condition.
Then, the CPU 111 of the management server 300 associates the identified condition satisfying image 95 with an image indicating that it is a condition satisfying image 95 .

Next, the CPU 111 of the management server 300 transmits the scanned image 94 in a state in which an image indicating that the image is a condition satisfying image 95 is associated with the condition satisfying image 95 to the user terminal 500 .
Here, the "condition satisfying image 95" refers to an image of a receipt for which the described information could not be acquired. In other words, the "condition satisfying image 95" here refers to the medium image G1 for which the above-mentioned default value has been output.

11 includes a condition satisfying image 95. More specifically, in this embodiment, the image denoted by reference numeral 11G is the condition satisfying image 95.
In this embodiment, to inform the user of the condition satisfying image 95, an image 96 indicating that it is the condition satisfying image 95 is added to the scanned image 94. As a result, the image 96 indicating that it is the condition satisfying image 95 is displayed on the display unit 205 of the user terminal 500.
In this embodiment, an image 97 of a rectangular frame surrounding the condition satisfying image 95 (hereinafter referred to as a “frame image 97 ”) is displayed as an image 96 indicating the condition satisfying image 95 .

In this embodiment, a frame image 97 , which is an image 96 indicating the condition satisfying image 95 , is added to the scanned image 94 in this manner.
As a result, in this embodiment, as shown in FIG. 11, the scanned image 94 to which the frame image 97, which is an image 96 indicating the condition satisfying image 95, is added is displayed on the display unit 205 referred to by the user.
This allows the user accessing the user terminal 500 to identify the condition-satisfying image 95 .

In this embodiment, the CPU 111 of the management server 300 outputs information that causes the display unit 205 of the user terminal 500 to display the scanned image 94 to which the image 96 indicating the condition satisfying image 95 has been added.
As mentioned above, this information may be explicit or non-explicit.
As in the above, the implicit information may include, for example, information indicating that the selection button 800 has been selected.

When the CPU 111 of the management server 300 outputs information to display the scanned image 94 to which an image 96 indicating the condition-satisfying image 95 has been added, for example, the image 96 indicating the condition-satisfying image 95 is added to the scanned image 94 in this management server 300.
In this embodiment, as described above, a rectangular frame image 97 is added to the scanned image 94 as an image 96 indicating the condition satisfying image 95 .
Then, the scanned image 94 to which the image 96 showing this condition-satisfying image 95 has been added is transmitted from the management server 300 to the user terminal 500 .

As a result, as shown in FIG. 11, the display unit 205 of the user terminal 500 displays a scanned image 94 to which a frame image 97, which is an example of an image 96 indicating a condition satisfying image 95, has been added.
Here, the image 96 representing the condition satisfying image 95 is a rectangular frame image 97, but the image 96 representing the condition satisfying image 95 is not limited to this.
Image 96 indicating condition satisfying image 95 may be, for example, an image having a star shape or the like, and this image may be displayed in association with condition satisfying image 95. Image 96 indicating condition satisfying image 95 may also be, for example, text information, and this text information may be displayed in association with condition satisfying image 95.

FIG. 12 is a diagram showing the display unit 205 of the user terminal 500 after one condition satisfying image 95 has been selected by the user from the plurality of condition satisfying images 95 displayed in FIG.
In the display shown in Figure 12, a display is made so that at least a portion of the corresponding recording medium, which is the recording medium corresponding to the condition-satisfying image 95 selected by the user, can be read at a size larger than the size of this condition-satisfying image 95.

Here, the "corresponding recording medium" refers to the actual recording medium that corresponds to the condition satisfaction image 95 selected by the user and is represented by this condition satisfaction image 95, among the multiple actual recording media (receipts) set in the image reading unit 608.
In this embodiment, an example has been described in which the display shown in Figure 12 is displayed when one condition-satisfying image 95 is selected.However, in this embodiment, even if the user selects button 11J displayed at the bottom of the screen in the display shown in Figure 11, the display shown in Figure 12 is also displayed, and one condition-satisfying image 95 is displayed.
In addition, in this embodiment, if the user performs a swipe operation on the screen while the display shown in Figure 11 is being displayed, the screen is switched and one condition satisfaction image 95 is displayed.

In this embodiment, when the selection button 800 (see Figure 7) is selected, the CPU 111 of the management server 300 outputs information that enables at least a portion of the corresponding recording medium corresponding to the condition-satisfying image 95 to be read at a size larger than the size in this condition-satisfying image 95.
In this embodiment, this information is sent to the user terminal 500 and is received by the user terminal 500. In response to this, in this embodiment, the user terminal 500 displays the information shown in FIG.

As a result, in this embodiment, the user terminal 500 is used to read at least a portion of the corresponding recording medium corresponding to the condition satisfaction image 95 at a size larger than the size of the condition satisfaction image 95 .
In this embodiment, when the above-mentioned "information to enable the image to be read at a size larger than the size in the condition-satisfying image 95" is output, the CPU 211 of the user terminal 500 (see Figure 3) displays the image shown in Figure 12 in accordance with this information.

Specifically, upon receiving this information, the CPU 211 of the user terminal 500 displays on the display unit 205 a message prompting the user to perform an operation to read at least a portion of the compatible recording medium at a size larger than the size in the condition satisfaction image 95.
In this embodiment, the user then performs an operation to read at least a portion of the compatible recording medium at a size larger than the size of the condition satisfying image 95 while referring to the display shown in FIG. 12 (step S111 in FIG. 6).

In this embodiment, on the display unit 205 of the user terminal 500, as shown by reference numeral 12A in Figure 12, a rectangular frame image (hereinafter referred to as "frame image 511") surrounding the amount entry field and date entry field of the condition satisfaction image 95 is displayed.
Furthermore, in this embodiment, as shown in Figure 12, the display unit 205 of the user terminal 500 has a display area 512 at the top of the display unit 205 for images captured by the camera 207 (see Figure 3), and the image captured by the camera is displayed within this display area 512.
The camera 207 is provided on the surface of the user terminal 500 opposite to the surface on which the display unit 205 is provided.

In this embodiment, a frame image 513, which is an image of a rectangular frame, is displayed within the display area 512.
The user who operates the user terminal 500 operates the user terminal 500 so that the amount entry field and date entry field of the actual receipt, which is the corresponding recording medium, fit inside this rectangular frame image 513.
In this embodiment, the shape of the frame image 513 displayed in the display area 512 is similar to the shape of the frame image 511 displayed on the condition satisfaction image 95 .

In this embodiment, the user takes a picture of the compatible recording medium while referring to the display unit 205 of the user terminal 500 shown in FIG.
Specifically, the user positions the amount entry field and date entry field of the corresponding recording medium inside the frame image 513 displayed in the display area 512, and then photographs the amount entry field and date entry field.
More specifically, the user positions the price entry field and date entry field of the corresponding recording medium inside the frame image 513, and then presses the start shooting button 515 displayed at the bottom of the screen.

As a result, in this embodiment, the amount entry field and date entry field that are part of the compatible recording medium are read in a size larger than the amount entry field and date entry field in the condition satisfaction image 95 .
In this embodiment, the amount entry field and the date entry field, which are part of the compatible recording medium, are photographed in close-up by the user terminal 500. As a result, the amount entry field and the date entry field are read in a size larger than the amount entry field and the date entry field in the condition satisfaction image 95.

In the display in FIG. 12, one condition satisfying image 95 selected by the user is displayed.
In this embodiment, the corresponding recording medium, which is the actual recording medium corresponding to this condition satisfying image 95 shown in FIG. 12, is re-read by the user terminal 500 operated by the user.
In other words, the actual receipt corresponding to one condition satisfaction image 95 shown in FIG. 12 is re-read by the user terminal 500 operated by the user.

In this embodiment, as described above, the CPU 111 of the management server 300 outputs information that allows at least a portion of the compatible recording medium to be read at a size larger than the size of the condition satisfying image 95 .
As above, the "information that allows at least a portion of the compatible recording medium to be read at a size larger than the size in the condition-satisfying image 95" may be explicit information, or may be non-explicit information such as information indicating that the selection button 800 has been selected.

An example of explicit information is instruction information that instructs the document to be read at a large size.
An example of instruction information that instructs that the image be read at a larger size is control information for controlling a mechanism provided in an image reading functional unit, and includes information that instructs that the image be read at a larger size.

In this embodiment, the user terminal 500 is used to manually re-photograph the compatible recording medium, but the re-photographing of the compatible recording medium may be performed automatically.
In this case, when control information including information instructing that the image be read at a larger size is output, the focal length, angle of view, etc. are automatically changed and the image is read at a larger size.
Another example of explicit information for enabling reading at a larger size is, for example, "control information that causes the display unit 205 of the user terminal 500 to display the information, 'Please take a close-up of the compatible recording medium with the user terminal 500.'"

Similarly to the above, examples of information that is not explicit information include information indicating that the selection button 800 has been selected.
In this case, for example, the CPU 111 of the management server 300 outputs to the user terminal 500 information indicating that the selection button 800 has been selected.
When the user terminal 500 receives this information indicating that the selection button 800 has been selected, the user terminal 500 executes a predetermined process. If the predetermined process includes a process for displaying the screen shown in FIG. 12, the screen shown in FIG. 12 will be displayed.
In this case, at least a part of the compatible recording medium is read in a size larger than the size of the condition satisfying image 95 by the user terminal 500 operated by the user who has referred to the screen shown in FIG.

In this embodiment, the CPU 111 of the management server 300 outputs information that causes a specific portion of the compatible recording medium to be read at a size larger than the size of the condition satisfying image 95 .
More specifically, in this embodiment, the CPU 111 of the management server 300 outputs information that enables the portion of the compatible recording medium from which the description information could not be obtained to be read at a size larger than the size in the condition satisfaction image 95.

In other words, the CPU 111 of the management server 300 outputs information that allows the portion of the compatible recording medium on which characters are written to be read at a size larger than the size of the condition satisfaction image 95 .
In other words, the CPU 111 of the management server 300 outputs information that causes the portion of the compatible recording medium from which the failure portion image G3 was obtained to be read at a size larger than the size of the condition satisfaction image 95 .

In response to this, in this embodiment, the CPU 111 of the management server 300 adds a frame image 511 (see FIG. 12) to the condition satisfaction image 95, surrounding the amount entry field and the date entry field of the condition satisfaction image 95.
In this embodiment, the condition-satisfying image 95 with this frame image 511 added is transmitted from the management server 300 to the user terminal 500, and the condition-satisfying image 95 with this frame image 511 added is displayed on the user terminal 500.
As a result, the user terminal 500 operated by the user takes close-up photographs of the amount entry field and the date entry field of the corresponding recording medium, and the amount entry field and the date entry field are read in a size larger than the size in the condition satisfaction image 95.

In this embodiment, an example has been described in which a portion of a compatible recording medium is read by the user terminal 500, but this is not limited to this, and the entire compatible recording medium may also be read by the user terminal 500.
Specifically, in this case, for example, a message "Please retake a photo on a compatible recording medium" is displayed on the display unit 205 of the user terminal 500.

In this case, the user operates the user terminal 500 so that all parts of the compatible recording medium are displayed on the display unit 205 of the user terminal 500. More specifically, the user operates the user terminal 500 so that all parts of the compatible recording medium fit within the display area 512 (see FIG. 12 ).
Then, the user photographs the compatible recording medium with all parts of the compatible recording medium displayed within the display area 512 .
As a result, in this case too, a closer image of the corresponding recording medium is taken than when the corresponding recording medium is read by the image reading unit 608, and the corresponding recording medium is read at a size larger than the size of the corresponding recording medium in the condition satisfaction image 95.

12 is displayed on the user terminal 500, which is a device that is different from the image forming device 600 that also functions as an image reading device. In this embodiment, the user who has seen this display operates the user terminal 500 to re-photograph the corresponding recording medium.
In this embodiment, the CPU 111 of the management server 300 outputs information that enables at least a portion of the corresponding recording medium to be read at a size larger than the size in the condition-satisfying image 95 and to be read by a user terminal 500, which is an example of a device other than an image reading device.

This information is not particularly limited and may be explicit information as described above, or non-explicit information such as information indicating that the selection button 800 has been selected.
When the CPU 111 of the management server 300 outputs this information, the display shown in Fig. 12 is displayed on the display unit 205 of the user terminal 500, which is a device different from the image forming device 600. Then, the user uses the user terminal 500 to take a close-up image of the corresponding recording medium.

The user terminal 500 is a portable terminal device carried by the user, and the display shown in FIG. 12 is displayed on this portable terminal device.
The user then photographs the compatible recording medium by operating the portable user terminal 500. In other words, the user operates the portable user terminal 500 to take a close-up shot of the compatible recording medium.

When the portable user terminal 500 is used as in this embodiment, the optical path length can be made shorter, and close-up photography of the corresponding recording medium can be performed.
In this embodiment, the optical path length when the user terminal 500 reads the compatible recording medium (receipt) is shorter than the optical path length when the image reading unit 608 reads the compatible recording medium.
In other words, in this embodiment, the length of the optical path that the reflected light from the compatible recording medium (receipt) set in the image reading unit 608 takes when heading towards the light receiving element (not shown) provided in the image reading unit 608 is shorter than the length of the optical path that the reflected light from the compatible recording medium takes when heading towards the light receiving element provided in the user terminal 500.

In this case, a close-up image of the compatible recording medium is taken, and the size of the image of the compatible recording medium becomes larger.
Here, in this embodiment, as described above, at least a portion of the corresponding recording medium is read at a size larger than the size of this portion in the condition-satisfying image 95, but this "read at a larger size" can also be said to be a state in which the optical path length at the time of reading is read in a state that is smaller than the optical path length at the time of reading by the image reading unit 608.
If the size of the image of the compatible recording medium can be increased as in this embodiment, the accuracy of obtaining the information written in the amount entry field and date entry field of the compatible recording medium can be improved.

12 is the display unit 205 that is referred to by a user who performs an operation to read at least a portion of the compatible recording medium in a large size. In this embodiment, information about the compatible recording medium is displayed on the display unit 205 that is referred to by the user.
In this embodiment, the CPU 111 of the management server 300 outputs information that causes the display unit 205, which is referenced by the user, to display information about the compatible recording medium.
As mentioned above, this information may be explicit information that specifically instructs the display of information about the compatible recording medium, or it may be non-explicit information, such as information indicating that the selection button 800 has been selected.

When information that allows information about the compatible recording medium to be displayed is output, in this embodiment, a condition satisfaction image 95 is displayed on the display unit 205 of the user terminal 500, as shown in Figure 12.
In addition, when information that displays information about compatible recording media is output, a rectangular frame image 511, which is information indicating which part of the compatible recording media should be photographed, is displayed on the display unit 205 of the user terminal 500.
In other words, the display unit 205 displays information about the compatible recording medium, such as an image of the entire receipt, which is the recording medium that failed to be read, and information indicating which part of the receipt, which is the recording medium that failed to be read, should be re-photographed.

In this embodiment, as indicated by reference numeral 12A in FIG. 12, a rectangular frame image 511 is displayed on the display unit 205 of the user terminal 500 as information indicating the area where re-photographing is required.
In other words, in this embodiment, the rectangular frame image 511 is displayed as information indicating the position of a part of the compatible recording medium where re-photographing is performed by the user terminal 500 .
More specifically, in this embodiment, a frame image 511 that surrounds the amount entry field and the date entry field of the condition satisfaction image 95 is displayed at the location where the amount entry field and the date entry field are displayed.

In this embodiment, the CPU 111 of the management server 300 outputs information that causes the information indicating the position of this part to be displayed on the display unit 205 .
As mentioned above, this information may be explicit information that specifically instructs the display of information indicating the position of this portion on the display unit 205, or it may be information other than explicit information, such as information indicating that the selection button 800 has been selected.
When information is output to cause information indicating a partial position to be displayed on the display unit 205, a frame image 511 is added to the condition-satisfying image 95, and this condition-satisfying image 95 with the frame image 511 added is displayed on the display unit 205 of the user terminal 500.

In this embodiment, the CPU 111 of the management server 300 adds a frame image 511 to the condition satisfaction image 95 so that the amount field and date entry field of the condition satisfaction image 95 for which information could not be obtained are surrounded by the frame image 511.
Then, the CPU 111 of the management server 300 transmits the condition satisfying image 95 to which the frame image 511 has been added to the user terminal 500 .
As a result, in this embodiment, condition satisfaction image 95 with frame image 511 added thereto is displayed on display unit 205 of user terminal 500. In other words, in this embodiment, condition satisfaction image 95 with information indicating the position of the above-mentioned part added thereto is displayed on display unit 205 of user terminal 500.

The "information indicating a part of the position" is not limited to the rectangular frame image 511 described above, and may be text information such as "Please take another photo of the amount field and date entry field."
Furthermore, the "information indicating a partial position" does not necessarily have to be displayed superimposed on the condition satisfaction image 95, but may be displayed in a location outside the condition satisfaction image 95. Specifically, for example, text information such as "Please take another photo of the amount field and the date entry field" may be displayed in a location outside the condition satisfaction image 95.

FIG. 13 shows a screen displayed on the display unit 205 of the user terminal 500 after re-photographing of the compatible recording medium has been performed.
In this embodiment, when a user operates the user terminal 500 and reads at least a portion of the compatible recording medium in a large size, information obtained from the image obtained by reading in this large size is displayed on the display unit 205 of the user terminal 500.
In other words, in this embodiment, when a user uses the user terminal 500 to photograph the amount entry field and date entry field of a compatible recording medium, the information written in the amount entry field and date entry field is displayed on the display unit 205 of the user terminal 500.

Specifically, in this embodiment, when the user takes a picture of the amount entry field and the date entry field, the image obtained by taking the picture is analyzed by the CPU 211 of the user terminal 500 .
The CPU 211 of the user terminal 500 performs the same processing as the OCR processing and the like performed by the management server 300, and acquires the information entered in each of the amount entry field and date entry field.
More specifically, in this embodiment, the above-mentioned complementary application performs processing similar to the above-mentioned OCR processing, etc., and obtains the information entered in each of the amount entry field and date entry field (see step S112 in Figure 6).

In this embodiment, as indicated by reference numeral 13A in Fig. 13, the information acquired from the compatible recording medium is displayed on the display unit 205 of the user terminal 500. In other words, the information acquired by re-photographing the receipt, which is the compatible recording medium, is displayed on the display unit 205 of the user terminal 500.
More specifically, the display unit 205 of the user terminal 500 displays the information written in the amount entry field and the date entry field of the compatible recording medium.

In this embodiment, when the selection button 800 shown in Figure 7 is selected by the user, the CPU 111 of the management server 300 outputs information to the user terminal 500, which information is obtained by re-photographing the corresponding recording medium and is displayed on the display unit 205 of the user terminal 500.
In other words, the CPU 111 of the management server 300 outputs information to the user terminal 500 so that the information obtained from the image obtained by reading it at a large size is displayed on the display unit 205 referenced by the user.
As mentioned above, this information may be explicit information that specifically instructs the display, or it may be non-explicit information, such as information indicating that the selection button 800 has been selected by the user.

In this embodiment, the information obtained by re-photographing the compatible recording medium is displayed on the display unit 205 of the user terminal 500. When the user terminal 500 receives this information, the display shown in FIG. 13 is displayed on the user terminal 500 after the compatible recording medium is re-photographed.
In the display shown in FIG. 13, the information entered in each of the amount entry field and the date entry field, which is obtained by re-photographing using the user terminal 500, is displayed.

13, information obtained by re-imaging the compatible recording medium is displayed in the middle section of the display unit 205. Specifically, as shown by the reference numeral 13A, information about the amount and information about the date obtained by analyzing the images of the amount entry field and the date entry field is displayed in the middle section of the display unit 205.
Furthermore, an image obtained by the above-mentioned re-photographing is displayed in a field indicated by reference numeral 13B at the top of the display unit 205 (not shown in FIG. 13). Specifically, an image obtained by photographing a part of the compatible recording medium is displayed.

In addition, as shown by reference symbol 13C, the display unit 205 of the user terminal 500 asks whether or not to supplement the information originally obtained about the compatible recording medium (receipt) with information obtained by re-photographing the compatible recording medium.
Also, the display unit 205 of the user terminal 500 displays a selection item 580 for receiving a response to this inquiry from the user.
Specifically, as selection items 580, a selection item 581 of "Yes" for accepting completion from the user and a selection item 582 of "Take Again" for accepting re-photographing of the corresponding recording medium from the user are displayed.

In this embodiment, for example, if the user selects the "Yes" option 581, the information originally obtained about the compatible recording medium is supplemented with new information obtained by re-photographing (details will be described later).
Specifically, in this case, new information about the compatible recording medium obtained by re-shooting is transmitted to the external server 400 via the management server 300 (see FIG. 1).
Then, the information about the amount and the information about the date stored in the external server 400 is replaced with the new information about the amount and the new information about the date obtained by re-photographing.
On the other hand, if the user selects the option 582 to "take another picture," the screen shown in Fig. 12 is displayed again on the user terminal 500. The user then refers to this screen again to take another picture of the corresponding recording medium.

In FIG. 12, one of the three condition satisfying images 95 corresponding to the three compatible recording media for which the written information G2 could not be obtained is displayed.
In this embodiment, as shown in FIG. 11, there are three condition satisfying images 95, and in FIG. 12, one of the three condition satisfying images 95 is displayed.

In the display shown in Figure 12, a screen switching button 588 is provided, and when the user operates this screen switching button 588, the screen is switched and the second condition satisfaction image 95 or the third condition satisfaction image 95 is displayed on the display unit 205.
In this embodiment, the screen is also switched when the user performs a so-called swipe operation on the screen shown in FIG.

The user operates the user terminal 500 to photograph a second corresponding recording medium corresponding to the second condition satisfying image 95 and a third corresponding recording medium corresponding to the third condition satisfying image 95 .
Specifically, the user takes close-up photographs of the amount entry field and the date entry field of the second compatible recording medium and the third compatible recording medium, in the same manner as described above.
As a result, in this case, new information about the amount and new information about the date are also acquired for the second and third compatible recording media.

In addition, in the display shown in Figure 12, information indicating which corresponding recording medium the condition-satisfying image 95 displayed on the display unit 205 corresponds to is displayed in the location indicated by symbol 12C.
12 also displays an enlargement selection button 589 that the user can select if they wish to enlarge the display. In this embodiment, when the user selects this enlargement selection button 589, the amount entry field and date entry field of the condition satisfaction image 95 are enlarged and displayed.

Note that the screen shown in Figure 13 has been described as an example in which the selection item 581 "Yes" is displayed, but if there are a second compatible recording medium and a third compatible recording medium, as in this embodiment, a different display may be made.
Specifically, for example, instead of the "Yes" selection item 581, a button displaying the text information "Next Receipt" or the like may be displayed as a switch button for switching to a screen for photographing other compatible recording media.
When this switching button is selected, the screen is switched and information about the second or third condition satisfying image 95 is displayed on the display unit 205 .

In this embodiment, when the user terminal 500 photographs all compatible recording media to be re-photographed, the screen shown in Figure 14 (a diagram showing an example display on the user terminal 500) is displayed on the display unit 205 of the user terminal 500.
On this screen, a send button 591 for sending the results of the re-photographing to the management server 300 is displayed.
This screen also displays information indicating that re-imaging has been completed, as indicated by reference numeral 14A. This screen also displays information indicating that the user needs to press the send button and proceed to operate the image forming device 600, as indicated by reference numeral 14A.

In this embodiment, when the user presses the send button 591, the above information obtained by re-photographing (hereinafter referred to as "re-photographing information") is sent from the user terminal 500 to the management server 300 (step S113 in Figure 6).
Specifically, the information entered in the amount entry field and the date entry field, which is acquired based on the re-photographing by the user terminal 500, is sent to the management server 300.

Next, the user who presses the send button 591 refers to the screen shown in FIG.
The user then selects the input image 606X displayed on this screen. This input image 606X is an image that allows the user to input that the user terminal 500 has finished reading the receipt.
When the user selects the input image 606X, the information displayed on the display unit 606 provided in the image forming apparatus 600 is updated. Specifically, the screen shown in FIG. 8 is updated.

In this embodiment, the user selects the input image 606X and instructs updating of the information (see step S114 in FIG. 6 ), thereby updating the information displayed on the display unit 606 provided in the image forming apparatus 600.
Specifically, in this embodiment, when the input image 606X is selected by the user, the image forming apparatus 600 accesses the management server 300 and acquires the re-photographing information (step S115).

As a result, in this embodiment, the description information G2 obtained by re-photographing is displayed on the display unit 606 of the image forming device 600, as shown in Figure 15 (a diagram showing an example of the display on the display unit 606 provided on the image forming device 600) (step S116 in Figure 6).
In this embodiment, the user can also check the written information G2 obtained by taking a close-up photograph of the actual receipt using the user terminal 500 on the display unit 606 of the image forming apparatus 600.

Thereafter, in this embodiment, the user presses the confirm button 595 displayed on the display unit 606 (see FIG. 15) (step S117 in FIG. 6).
As a result, a confirmation notification is sent from the image forming apparatus 600 to the management server 300 to notify the confirmation (step S118). Then, in this embodiment, the management server 300 sends the description information G2 to the external server 400 (see FIG. 1) (step S119 in FIG. 6).

Then, in the external server 400, the information originally obtained about the corresponding recording medium (receipt) is supplemented with the written information G2 obtained by re-photographing.
Specifically, in this embodiment, the information about the amount and the information about the date stored in the external server 400 is replaced with new information about the amount and new information about the date obtained based on the re-photographing.

Regarding re-reading of the compatible recording medium, it is also possible that the user sets the compatible recording medium in the image forming apparatus 600 again without using the user terminal 500, and re-acquires the written information G2.
In other words, it is possible to re-acquire the described information G2 by setting the receipt from which the described information G2 could not be acquired back into the image forming device 600. In this case, however, the situation where the described information G2 cannot be acquired may occur again.

In contrast to this, in this embodiment, the image obtained by re-reading at a larger size is analyzed to obtain the written information.
This increases the possibility of obtaining the information recorded on the compatible recording medium, compared to when the compatible recording medium is set in the image forming apparatus 600 again and read from the compatible recording medium.

The user terminal 500 is a reading device that can read at a higher resolution than the reading resolution of the image reading unit 608 provided in the image forming device 600 .
In this embodiment, the compatible recording medium is read by the user terminal 500, which has a higher reading resolution than the image reading unit 608 of the image forming apparatus 600. In other words, in this embodiment, the actual receipt is read by the user terminal 500, which has a higher reading resolution than the image reading unit 608.

In recent years, there are also smartphones and the like that have a reading resolution higher than the reading resolution of the image reading unit 608 provided in the image forming apparatus 600 .
In this embodiment, by using a user terminal 500 configured as a smartphone or the like, it is possible to read a compatible recording medium with a reading resolution higher than that of the image reading unit 608 .

Furthermore, as described above, the user terminal 500 can be used to take close-up pictures of compatible recording media.
The user terminal 500 can approach the object to be photographed, and in this embodiment, can take a close-up photograph of the object to be photographed.
As a result, in this embodiment, as described above, the failed portion of the receipt, that is, the portion where the information could not be read, can be photographed in an enlarged state. In other words, in this embodiment, the failed portion can be photographed in an enlarged state compared to when the image forming device 600 reads the portion.

(others)
In the above description, the case where information about a receipt for which the written information G2 could not be acquired is output to the image forming device 600 or the user terminal 500 has been described.
However, not limited to this, information about receipts in which the written information G2 acquired from the receipt (hereinafter sometimes referred to as "acquired written information G2") does not satisfy predetermined conditions may also be output to the image forming device 600 or the user terminal 500.
In other words, information about receipts for which written information G2 has been obtained from the receipt but the written information G2 does not satisfy predetermined conditions may be output to the image forming device 600 or the user terminal 500.

It is also possible that the multiple receipts may include both receipts for which the written information G2 could not be obtained and receipts for which the obtained written information G2 does not meet the predetermined conditions.
In this case, both information about receipts for which the description information G2 could not be obtained, and information about receipts for which the obtained description information G2 does not satisfy the predetermined conditions may be output to the image forming device 600 or the user terminal 500.

Here, when outputting information about a receipt in which the acquired description information G2 does not satisfy the predetermined conditions, similar to the above, for example, the medium image G1, the description information G2 (for example, the description information G2 supplemented with a default value), and the failure area image G3 (information about the location where the acquired description information G2 is located) are output to the image forming device 600 or the user terminal 500.
It should be noted that the description information G2 itself obtained by the OCR unit 353C may be output to the image forming apparatus 600 or the user terminal 500, rather than the description information G2 supplemented with default values.

In addition, when outputting information about a receipt in which the acquired written information G2 does not satisfy predetermined conditions, for example, a medium image G1 and an image of the failure area G3 may be output to the image forming device 600 or the user terminal 500.
In this case, the description information G2 is not displayed on the image forming apparatus 600 or the user terminal 500, and the area where the description information G2 should be displayed is left blank.

Here, "the written information G2 could not be obtained" refers to a state in which information such as numerical values or character strings could not be obtained, even though such information would normally be obtained.
For example, as shown in Figure 7, receipts may include information such as "total" and "date of receipt." If information about the "total" and "date of receipt" is obtained but numerical information cannot be obtained from the area next to this information, this corresponds to "written information G2 could not be obtained."

Furthermore, "the acquired description information G2 does not satisfy the predetermined conditions" refers to a state in which the description information G2 has been acquired, but the acquired description information G2 does not satisfy the predetermined conditions.
For example, if a numerical value is acquired as description information G2, and the number of digits of this numerical value exceeds a predetermined expected number of digits, or is smaller than the predetermined expected number of digits, this corresponds to "the acquired description information G2 does not satisfy the predetermined conditions."
Also, for example, if information other than numbers is obtained in a situation where a numerical value should normally be obtained, this also falls under the category of "the obtained written information G2 does not satisfy the predetermined conditions."

In addition, in this embodiment, as shown in Figures 7 and 11, an example is described in which information obtained by the image reading unit 608 of the image forming device 600, such as the medium image G1, is displayed on both the image forming device 600 and the user terminal 500.
Incidentally, the display on both the image forming apparatus 600 and the user terminal 500 is not essential, and the display on the image forming apparatus 600 may be omitted.
If reading of a recording medium such as a receipt fails, the display shown in FIG. 7 may not be displayed, and the screen shown in FIG. 9 may be displayed first on the user terminal 500.
In this case, when the user selects the complementary selection image 205B corresponding to the complementary application, the screen shown in FIG. 10 is displayed, and then the screens shown in FIG. 11 and subsequent figures are displayed in sequence.

Additionally, machine learning may be performed to improve the accuracy of reading receipts.
Specifically, machine learning may be performed sequentially based on the image of the failed area, the correct information subsequently entered by the user, and the information obtained by subsequent re-photographing, to improve the accuracy of the reading.
More specifically, each time the user inputs correct information, machine learning may be performed based on the image of the failed area and this correct information to improve the accuracy of reading.
Alternatively, each time a re-image is taken, machine learning may be performed based on the image of the failed area and the information obtained by the re-image, thereby improving the accuracy of the reading.
More specifically, based on the results of machine learning, for example, the parameters used when reading a receipt may be changed to improve the accuracy of reading.

1...information processing system, 94...read image, 95...condition-satisfying image, 97...frame image, 111...CPU, 205...display unit, 500...user terminal, 600...image forming device

Claims (12)

An information processing system including a processor,
The processor:
Identifying a condition-satisfying image, which is an image representing the recording medium and satisfies a specific condition, from among read images obtained by reading a plurality of recording media with an image reading device;
outputting information for reading at least a part of a corresponding recording medium, which is a recording medium among the plurality of recording media that corresponds to the condition satisfying image, at a size larger than the size of the condition satisfying image and for reading the corresponding part by a device different from the image reading device ;
Information processing system. The processor:
outputting information that allows the specific portion of the corresponding recording medium to be read at a size larger than the size in the condition-satisfying image and to be read by the other device ;
The information processing system according to claim 1 . The processor:
outputting information that enables the portion of the compatible recording medium on which characters are written to be read at a size larger than the size of the condition satisfying image and that enables the portion to be read by the other device ;
The information processing system according to claim 2 . The processor:
outputting information that enables a portion of the compatible recording medium from which information could not be acquired and/or a portion from which acquired information does not satisfy a predetermined condition to be read at a size larger than the size of the condition-satisfying image and to be read by the other device ;
The information processing system according to claim 2 . 2. The information processing system according to claim 1 , wherein the other device is a portable terminal device carried by a user. The processor:
outputting information for displaying information about the compatible recording medium on a display unit referenced by a user who performs an operation to read at least a portion of the compatible recording medium in the large size;
The information processing system according to claim 1 . The processor:
outputting information that causes information indicating the location of at least a portion of the compatible recording medium to be displayed on the display unit;
The information processing system according to claim 6 . The processor:
and further outputting information for displaying the scanned image including the plurality of medium images on a display unit provided in the other device .
The information processing system according to claim 1 . The processor:
outputting information for displaying the read image, to which an image indicating which of the plurality of medium images included in the read image is the condition-satisfying image, on the display unit provided in the other device;
The information processing system according to claim 8 . The processor:
outputting information to a display unit provided in the other device so that information obtained from the image obtained by scanning at the large size is displayed;
The information processing system according to claim 1 . The processor:
The written information, which is information written on the compatible recording medium and is obtained by the other device analyzing the reading results obtained by reading the compatible recording medium by the other device, is displayed on a display unit provided in the image reading device that reads the multiple recording media.
The information processing system according to claim 1 . A function of identifying a condition-satisfying image, which is an image representing the recording medium and satisfies a specific condition, from among images obtained by reading a plurality of recording media with an image reading device;
a function of outputting information that allows at least a portion of a corresponding recording medium, which is a recording medium among the plurality of recording media that corresponds to the condition satisfying image, to be read at a size larger than the size of the condition satisfying image and to be read by a device other than the image reading device ;
A program to make the above happen on a computer.