JP7690273B2 - Information processing device, information processing method, and program - Google Patents

Description

The present disclosure relates to an information processing device, an information processing method, and a program.

In recent years, with the development of network technology and virtualization technology, it has become possible to use various functions by accessing services on a network, known as cloud services, without installing software on an information processing device such as a PC. One example of such a cloud service is cloud storage, which allows the creation, editing, viewing, and sharing of electronic documents on a network. Some cloud storage services also allow the viewing of editing history, the restoration of any version, and the extraction of differences between any versions.

Furthermore, files managed on the cloud can be viewed and edited regardless of the location where the user is working or the type of terminal device used for the work, as long as the cloud storage can be accessed via a network. Due to these characteristics, files managed on the cloud can also be shared in real time with other users in remote locations, for example.

On the other hand, even in recent years, confirmation and revision of the contents of electronic documents are often performed based on a printout of the contents of the electronic document on a recording medium such as paper. Against this background, various printers and MFPs (Multifunction Peripherals) that can print electronic documents managed in cloud storage by linking with the cloud storage have been proposed in recent years. For example, Patent Document 1 discloses an example of an image processing device that can print electronic documents managed in cloud storage.

JP 2017-76244 A

In a situation where the contents of an electronic document are revised or proofread by hand using a printed copy, one method of reflecting the results of the revision or proofreading on the electronic document is to directly edit the electronic document on cloud storage, for example. However, this method requires that the contents that were once written by hand be electronically transcribed again using a keyboard or the like, which can be cumbersome.
It is also possible to import revised or proofread printed matter as electronic data by using the scanning function of a scanner or MFP. However, in this case, the results of scanning the printed matter are imported as image data, so even if the image data is used, it is difficult to automatically reflect the results of revision or proofreading of the printed matter in the text data of the electronic document.

In consideration of the above problems, the present invention aims to make it possible to reflect the results of proofreading in an electronic document in a more suitable manner, even in situations where an electronic document is proofread by adding edits to a printed matter on which the contents of the electronic document are printed.

The information processing device according to the present invention includes a scanner unit that reads a printed matter to which information indicating the location on cloud storage where an original electronic document is stored is attached as an image to generate a first image, and an operation unit that displays a screen for receiving from the user a first instruction specifying whether or not to reflect a correction instruction written on the printed matter in the original electronic document stored in the location indicated by the information, and a second instruction specifying a cloud storage to which the first image is to be sent from a list of cloud storages available to the user, the information processing device including: a receiving unit that receives the first image, the first instruction, and the second instruction; and, when the first instruction received by the receiving unit specifies that the correction instruction should not be reflected in the original electronic document and the second instruction specifies the cloud storage to which the first image is to be sent, a transmitting unit that transmits the first image received by the receiving unit to the cloud storage specified in the second instruction; and when the instruction specifies that the correction instructions should be reflected in the original electronic document, and the original electronic document can be obtained from a location on the cloud storage based on information indicating a location on the cloud storage where the original electronic document is stored, obtained by analyzing the first image received by the receiving means, the device is characterized in that it comprises: a generation means for generating a second image by rendering the original electronic document and generating a difference image by extracting a difference between the first image received by the receiving means and the generated second image; an identification means for recognizing the correction instructions indicated by a proofreading symbol included in the difference image and text information in a natural language included in the difference image by performing an analysis process and an OCR process on the difference image generated by the generation means, and further for identifying a character string in the electronic document that is to be reflected with the correction instructions based on the coordinates of the proofreading symbol; and an execution means for executing a correction process based on the correction instructions on the character string in the electronic document identified by the identification means.

According to the present invention, even in a situation where an electronic document is proofread by adding revisions to a printed matter on which the contents of the electronic document are printed, it is possible to reflect the results of the proofreading in the electronic document in a more suitable manner.

FIG. 1 illustrates an example of a system configuration of an information processing system. FIG. 2 illustrates an example of a hardware configuration of an MFP. FIG. 2 is a diagram illustrating an example of a software configuration of the MFP. FIG. 2 illustrates an example of a software configuration of an MFP server. FIG. 11 is a diagram showing an example of a menu screen. FIG. 13 is a diagram showing an example of a cloud print screen. FIG. 13 is a diagram showing an example of a cloud scan screen. FIG. 13 is a diagram showing an example of a login screen. FIG. 11 is a diagram showing an example of a menu screen. FIG. 13 is a diagram showing an example of a history confirmation screen. FIG. 13 is a diagram showing an example of a setting screen. 11 is a flowchart showing an example of processing of an information processing system. 10 is a flowchart showing another example of the processing of the information processing system. 10 is a flowchart showing another example of the processing of the information processing system.

A preferred embodiment of the present disclosure will be described in detail below with reference to the accompanying drawings. Note that in this specification and drawings, components having substantially the same functional configuration are designated by the same reference numerals to avoid redundant description.

<System Configuration>
An example of a system configuration of an information processing system according to the present embodiment will be described with reference to Fig. 1. The information processing system according to the present embodiment includes an MFP 100, a terminal device 113, an MFP server 120, and a cloud storage 121. The MFP 100, the terminal device 113, the MFP server 120, and the cloud storage 121 are connected to each other via a predetermined network.
1, the MFP 100 and the terminal device 113 are connected via a LAN 110. The MFP 120 and the cloud storage 121 are connected to the Internet 112. In addition, the communication via the LAN 110 and the communication via the Internet 112 are mediated by a gateway 111. This allows the MFP 100 and the terminal device 113 connected to the LAN 110 to access the MFP server 120 and the cloud storage 121.

Note that the configuration shown in FIG. 1 is merely an example, and the network configuration is not limited as long as it is possible to interconnect the MFP 100, the terminal device 113, the MFP server 120, and the cloud storage 121. In addition, the network connecting the various components described above, such as the LAN 110 and the Internet 112, may include multiple networks, and the type of some of the multiple networks may be different from the type of the other networks.

The MFP 100 includes a controller unit 101 , an operation unit 102 , a printer unit 103 , and a scanner unit 104 .
The operation unit 102 is a schematic representation of an interface for receiving instructions from a user for the MFP 100 and presenting various pieces of information to the user.
The printer unit 103 prints an image corresponding to electronic data (e.g., an electronic document, etc.) on a recording medium such as paper (hereinafter also referred to as a "sheet"), and outputs the printed matter with the image printed on the recording medium to a specified paper output destination.
The scanner unit 104 reads a printed material and converts the result of the reading (in other words, the result of optically reading the printed material) into an image signal to generate image data.
The controller unit 101 is connected to an operation unit 102, a printer unit 103, and a scanner unit 104, and controls the operations of these units, thereby achieving various functions of the MFP 100 (i.e., functions as a multifunction peripheral).

The terminal device 113 is a schematic representation of a print client that issues print instructions to the MFP 100 using a print protocol such as the Line Printer Remote (LPR) protocol. The terminal device 113 may also provide a user interface, such as a web browser, that accesses other devices (e.g., the MFP server 120 or cloud storage 121) via a network and enables the user to operate the other devices. The terminal device 113 may be realized, for example, by a PC (Personal Computer) or the like.

The gateway 111 corresponds to a so-called network router that mediates communication between a plurality of devices connected to different networks (in other words, relays communication).
The MFP server 120 is a server that executes various processes related to improving the functionality of the MFP 100 .
The cloud storage 121 is a so-called file sharing server that transmits and receives files (data) to and from the MFP 100 using a predetermined protocol such as HTTP (Hyper Text Transfer Protocol).

<Hardware Configuration>
An example of the hardware configuration of the MFP 100 will be described with reference to FIG. 2, focusing in particular on the configuration of the controller unit 101.
The controller unit 101 of the MFP 100 includes a central processing unit (CPU) 201, a dynamic random access memory (DRAM) 202, and an I/O controller 203. The controller unit 101 also includes a network I/F 204, a serial advanced technology attachment (SATA) 205, a panel I/F 206, a printer I/F 207, and a scanner I/F 208. The controller unit 101 may also include a universal serial bus (USB) 212.

The CPU 201 and the DRAM 202 are connected via a bus. The DRAM 202 is used as a working area for temporarily storing program data representing arithmetic instructions, data to be processed, and the like, in the process of the CPU 201 performing various arithmetic operations.
The CPU 201 and the I/O controller 203 are connected via a bus. The I/O controller 203 controls input/output to and from various devices in accordance with instructions from the CPU 201.

The I/O controller 203 is connected to a SATA I/F 205, which is connected to a storage device such as a Flash ROM 211. The Flash ROM 211 stores programs that the CPU 201 uses to realize the functions of the MFP 100, as well as various data such as document files.

Furthermore, a network I/F 204 is connected to the I/O controller 203, and a communication device for realizing communication with a predetermined network, such as a wired LAN device 210, is connected to the network I/F 204. The CPU 201 controls a communication device, such as the wired LAN device 210, via the network I/F 204, thereby realizing communication with other devices via a network, such as the LAN 110.

The I/O controller 203 is also connected to a panel I/F 206. The CPU 201 controls the operation of the operation unit 102 via the panel I/F 206, thereby accepting instructions from a user via the operation unit 102 and presenting various information to the user via the operation unit 102.

A printer I/F 207 is also connected to the I/O controller 203. The CPU 201 controls the operation of the printer unit 103 via the printer I/F 207, thereby realizing a process of printing an image on a recording medium (sheet) such as paper and outputting the printed matter to a predetermined paper output destination.
A scanner I/F 208 is also connected to the I/O controller 203. The CPU 201 controls the operation of the scanner unit 104 via the scanner I/F 208 to realize a process of reading an original (printed material) and converting the result of the reading into an image signal to generate image data.

A USB I/F 212 may also be connected to the I/O controller 203. The CPU 201 controls the operation of various devices connected to the USB I/F 212 via the USB I/F 212. An example of a device connected to the USB I/F 212 is an IC card reader 213. In this case, the CPU 201 controls the operation of the IC card reader 213 via the USB I/F 212 to realize, for example, processing related to user authentication using an IC card.

The CPU 201 executes processes related to the realization of various functions of the MFP 100 by controlling the operation of various devices as exemplified above. Here, an example of control when realizing a copy function is described below.

The CPU 201 reads out predetermined program data from the flash ROM 211 via the SATA I/F 205, and loads the data in the DRAM 202. The CPU 201 controls the operation of the operation unit 102 via the panel I/F 206 in accordance with the program loaded in the DRAM 202, thereby accepting a copy-related instruction from a user via the operation unit 102.
When the CPU 201 receives an instruction related to copying, it controls the operation of the scanner unit 104 via the scanner I/F 208 to cause the scanner unit 104 to read an original document and obtain image data (electronic data) according to the result of the reading. The CPU 201 stores the obtained image data in a predetermined storage area (e.g., DRAM 202).
The CPU 201 transfers the image data stored in a predetermined storage area to the printer unit 103 via the printer I/F 207, and instructs the printer unit 103 to execute processing related to printing based on the image data. In response to this instruction, the printer unit 103 prints an image corresponding to the image data on a recording medium (sheet) such as paper, and ejects the printed matter corresponding to the printing result to a predetermined paper ejection destination.

<Software configuration>
An example of the software configuration of the information processing system according to the present embodiment will be described below with reference to FIGS. 3 and 4, focusing in particular on the MFP 100 and the MFP server 120. FIG.

(Software configuration of MFP 100)
First, an example of the software configuration of the MFP 100 will be described with reference to FIG.
Components related to realizing the functions of the MFP 100 include, for example, an operation control unit 301, a data storage unit 302, a job control unit 303, an image processing unit 304, a print processing unit 305, and a reading processing unit 306. Components related to realizing the functions of the MFP 100 include a network control unit 307, a TCP/IP control unit 308, an authentication control unit 309, and an MFP server communication unit 310. The processing of each of the series of components shown in Fig. 3 is realized, for example, by the CPU 201 expanding a program stored in a storage area such as the Flash ROM 211 into a working area such as the DRAM 202 and executing the program.

The operation control unit 301 displays a screen image on the operation unit 102 for accepting instructions from the user and presenting various information to the user, and controls the detection of user operations on the screen image and the execution of processes linked to components such as buttons displayed on the screen.

The data storage unit 302 stores various data in a storage area such as the Flash ROM 211, and executes processes related to reading out various data from the storage area. As a specific example, a case will be considered where a user issues an instruction to change device settings related to the operation of the MFP 100 via the operation unit 102. In this case, the operation control unit 301 recognizes the content of the instruction from the user accepted by the operation unit 102, and requests the data storage unit 302 to store the setting values of the device settings according to the content of the instruction. Upon receiving the request from the operation control unit 301, the data storage unit 302 stores the setting values according to the instruction from the user in a specified storage area (e.g., the Flash ROM 211, etc.) as the setting values of the device settings.

A job control unit 303 performs various controls related to the execution of a job.
The image processing unit 304 processes the image data into data of a desired format (for example, data in a format according to the intended purpose) in accordance with instructions from the job control unit 303 .
The print processing unit 305 controls the operation of the printer unit 103 via the printer I/F 207 in accordance with instructions from the job control unit 303, and performs various processes related to printing images on a recording medium (sheet) such as paper, and outputting printed matter to a specified paper output destination.
The reading processing unit 306 controls the operation of the scanner unit 104 via the scanner I/F 208 in accordance with instructions from the job control unit 303 , and executes various processes related to reading a document placed on the reading surface of the scanner unit 104 .

The network control unit 307 controls network-related settings such as IP addresses at specific times, such as when the system is started or when a setting change is detected, in accordance with setting values stored in a specific storage area, thereby controlling settings related to the operation of components related to communication, such as the TCP/IP control unit 308.
A TCP/IP control unit 308 executes various processes for transmitting and receiving network packets to and from other devices through the network to which the MFP 100 is connected via the network I/F 204 .

The authentication control unit 309 executes processing related to authentication of a user based on a predetermined authentication method. For example, the authentication control unit 309 may authenticate a user who uses the MFP 100 by comparing information read from an IC card by the IC card reader 213 with data related to the user stored in a predetermined storage area by the data storage unit 302.

The MFP server communication unit 310 causes the network control unit 307 to control communication with the MFP server 120 according to instructions from the job control unit 303.

Here, an example of the operation of each component when implementing a copy function will be described below.
For example, when the operation control unit 301 receives an instruction to start copying from a user, it instructs the job control unit 303 to start processing to realize a copy function. The job control unit 303 instructs the reading processing unit 306 to read an original document, and obtains image data generated based on an image signal output according to the result of the reading. In the following description, for convenience, the image according to the result of reading the original document is also referred to as a "scanned image."
The job control unit 303 instructs the image processing unit 304 to convert the acquired image data into data in a format suitable for printing the scanned image.
Then, the job control unit 303 instructs the print processing unit 305 to print based on the image data whose format has been converted by the image processing unit 304. In response to this instruction, the print processing unit 305 causes the printer unit 103 to execute processing related to printing based on the image data specified by the job control unit 303. As a result, the copy result of the original is output as a printed matter.

(Software Configuration of MFP Server 120)
Next, an example of the software configuration of the MFP server 120 will be described with reference to FIG.
Components related to realizing the functions of the MFP server 120 include, for example, a web browser communication unit 401, a data storage unit 402, a job control unit 403, an image processing unit 404, a cloud storage communication unit 405, an MFP communication unit 406, and an authentication control unit 407.
A series of software programs executed by the MFP server 120 may be held as distributed resources on a cloud, for example. Then, the programs may be loaded into a non-volatile memory allocated as the distributed resource, and then executed by a computing resource allocated as the distributed resource, thereby realizing the processing of each piece of software.

The web browser communication unit 401 executes processes related to presenting various information to the user and accepting instructions from the user in response to requests from a web browser operating on another device (e.g., the terminal device 113). The web browser communication unit 401 also controls the execution of processes linked to components such as buttons on a screen presented to the user via the web browser.

The data storage unit 402 executes processes related to storing various data in a storage area such as a distributed resource on the cloud and reading out various data from the storage area. As a specific example, let us consider a case where an instruction to change device settings related to the operation of the MFP server 120 is given via a web browser. In this case, the web browser communication unit 401 recognizes the content of the instruction from the user received by the web browser, and requests the data storage unit 402 to store the setting values of the above device settings according to the content of the instruction. Upon receiving the request from the web browser communication unit 401, the data storage unit 402 stores the setting values according to the instruction from the user in a specified storage area (e.g., a distributed resource on the cloud) as the setting values of the above device settings.

A job control unit 403 performs various controls related to the execution of a job.
The image processing unit 404 processes the image data into data of a desired format (for example, data in a format according to the intended purpose) in accordance with instructions from the job control unit 403 .
The cloud storage communication unit 405 controls communication with the cloud storage 121 by using the user's account information stored in a predetermined storage area by the data storage unit 402 .

The MFP communication unit 406 executes various processes related to communication with the MFP 100. As a specific example, the MFP communication unit 406 may execute processes related to accepting requests sent from the MFP 100, notifying the MFP 100 of various information, etc.

The authentication control unit 407 executes processing related to user authentication based on a predetermined authentication method. For example, the authentication control unit 407 may authenticate a user who uses the MFP server 120 by comparing the authentication information accepted by the web browser communication unit 401 with data related to the user stored in a predetermined storage area by the data storage unit 402.

<User Interface>
An example of a user interface of the information processing system according to this embodiment will be described with reference to FIGS.

(Menu screen)
First, Fig. 5 will be described. Fig. 5 is a diagram showing an example of a menu screen 501 displayed on the operation unit 102 of the MFP 100. The menu screen 501 is a screen for receiving instructions from a user relating to the execution of various functions possessed by the MFP 100. When the IC card reader 213 recognizes an IC card presented by a user, the authentication control unit 309 authenticates the target user based on information read from the IC card, and if the authentication is successful, the menu screen 501 is displayed.

The button 502 is a user interface for receiving an instruction from the user regarding the execution of the copy function.
The button 503 is a user interface for receiving instructions from the user regarding the execution of a print function (hereinafter also referred to as a “cloud print function”) that uses data (e.g., electronic documents) stored in the cloud storage 121 .
Button 504 is a user interface for receiving instructions from a user to execute a function (hereinafter also referred to as the “cloud scan function”) of reading a document and storing data corresponding to the results of the reading in cloud storage 121.

(Cloud print screen)
Next, Fig. 6 will be described. Fig. 6 is a diagram showing an example of a cloud print screen 601 displayed on the operation unit 102 of the MFP 100. The cloud print screen 601 is a screen for receiving instructions from a user regarding various settings of a print function (cloud print function) that uses data stored in the cloud storage 121. The cloud print screen 601 is displayed on the operation unit 102 when, for example, the button 503 is pressed on the menu screen 501.

The display area 602 is an area where a list of files available as targets for cloud printing (in other words, candidates for print targets) is displayed. The list of files displayed in the display area 602 may be displayed in a scrollable manner, for example, by a scroll bar. When a file presented in the list is selected, the file is set as a target for cloud printing.
It should be noted that the candidates for print targets displayed in the display area 602 may be limited to files linked to the account information of the user identified by authentication.
Furthermore, when multiple cloud storages are used as the cloud storage 121, for example, files stored in each of the multiple cloud storages may be displayed as a common list in the display area 602. In this case, information about the cloud storage in which each file is stored (for example, the cloud storage name, etc.) may be displayed in association with each file.

Check box 603 is a user interface for receiving an instruction from the user as to whether or not to associate information for identifying a path to a file selected to be printed (hereinafter also referred to as "URL information") with the file. By associating this URL information with a file, it becomes possible for another device, such as MFP 100, to identify the file (the file that is the source of a copy of the received file) managed by cloud storage 121 based on the URL information.

Button 604 is a user interface for receiving instructions from the user relating to the execution of the cloud print function in response to the user instructions received by display area 602 and check box 603. When button 604 is pressed, information corresponding to the user instructions received by display area 602 and check box 603 is sent to MFP server 120, and the MFP server 120 is instructed to execute various processes relating to the realization of the cloud print function.

The display area 605 is a display area where notification information (e.g., messages, etc.) for notifying the user of various situations is displayed. As a specific example, when an error occurs and the execution of a printing process is interrupted, information about the error may be displayed in the display area 605, thereby providing the user with feedback on the situation at that time.

(Cloud scan screen)
Next, Fig. 7 will be described. Fig. 7 is a diagram showing an example of a cloud scan screen 701 displayed on the operation unit 102 of the MFP 100. The cloud scan screen 701 is a screen for receiving instructions from a user regarding various settings of a function (cloud scan function) for reading an original and storing data corresponding to the results of the reading in the cloud storage 121. The cloud scan screen 701 is displayed on the operation unit 102 when, for example, the button 504 is pressed on the menu screen 501.

The display area 702 is an area where a list of cloud storages available as a destination for a file corresponding to the document reading result (e.g., a file of a scanned image) is displayed. The list of cloud storages displayed in the display area 702 may be displayed in a scrollable manner, for example, using a scroll bar. When a cloud storage presented in the list is selected, the cloud storage is set as a destination for a file corresponding to the document reading result.
A list of cloud storages available to a user identified by authentication may be displayed in the display area 702. In this case, for example, the MFP 100 may refer to information related to a connection to cloud storage for each user registered in the MFP server 120 to identify the list of cloud storages available to the user.

Check box 703 is a user interface for receiving an instruction from the user as to whether or not to use the URL information associated with the file selected as the print target when executing the cloud print described above for processing according to the scan results. By checking check box 703, for example, based on the URL information, cloud storage to which the file corresponding to the document reading results will be sent is specified, and the scan results are reflected in the file to be printed.

Button 704 is a user interface for receiving instructions from the user to execute the cloud scan function in response to the user instructions received by display area 702 and check box 703, respectively. When button 704 is pressed, the document is read under the control of the reading processing unit 306, and data (file) corresponding to the results of the reading is transmitted to the MFP server 120. At this time, information corresponding to the user instructions received by display area 702 and check box 703, respectively, is transmitted to the MFP server 120 in association with the data corresponding to the results of the reading of the document.

The display area 705 is a display area where notification information (e.g., messages, etc.) for notifying the user of various situations is displayed. As a specific example, when an error occurs and the execution of the process related to reading the document is interrupted, information related to the error may be displayed in the display area 705, thereby providing the user with feedback on the situation at that time.

(Login screen)
Next, Fig. 8 will be described. Fig. 8 is a diagram showing an example of a login screen 801 that the MFP server 120 causes to be displayed on a display unit (e.g., a browser screen, etc.) of another device (e.g., the terminal device 113) via a network. The login screen 801 is a screen for accepting input of information from a user that the MFP server 120 uses for user authentication. For example, when the MFP server 120 is accessed using an information processing device having a communication function such as the terminal device 113 in a state where user authentication has not been performed, the login screen 801 is displayed on the display unit of the information processing device.

Each of the input areas 802 and 803 is a user interface for accepting input of information used by the MFP server 120 to authenticate a user when the user logs in to the MFP server 120. Specifically, in the example shown in FIG. 8, the input area 802 is an input area for accepting input of identification information (so-called user ID) for identifying each user from the user. Furthermore, the input area 803 is an input area for accepting input of a password used to authenticate the user corresponding to the identification information entered in the input area 802 from the user.

Button 804 is a user interface for accepting instructions from the user regarding logging in to the MFP server 120. When button 804 is pressed, the information entered in each of input areas 802 and 803 is sent to the MFP server 120, and the MFP server 120 is requested to authenticate the user based on that information. In response to this request, the authentication control unit 407 of the MFP server 120 authenticates the user by comparing the information sent together with the request with the user's account information stored in a specified storage area by the data storage unit 402. If authentication is successful, the MFP server 120 permits access to the target user.

(Menu screen)
Next, Fig. 9 will be described. Fig. 9 is a diagram showing an example of a menu screen 901 that the MFP server 120 displays on a display unit (e.g., a browser screen, etc.) of another device (e.g., the terminal device 113) via a network. The menu screen 901 is a screen for receiving various instructions related to the use of the MFP server 120 from a user. The menu screen 901 is displayed on a display unit of an information processing device (e.g., the terminal device 113) when user authentication is performed based on information inputted on the login screen 801 shown in Fig. 8 and the authentication is successful.

Button 902 is a user interface for receiving an instruction from the user to display a screen for checking the history of processing involving the MFP server 120, such as the cloud scan described above. An example of this screen will be described separately later with reference to FIG. 10.

Button 903 is a user interface for receiving an instruction from the user to display a screen for receiving from the user the specification of various settings related to processing involving the MFP server 120, such as the cloud scan described above. An example of this screen will be described separately later with reference to FIG. 11.

(History confirmation screen)
Next, FIG. 10 will be described. FIG. 10 is a diagram showing an example of a history confirmation screen 1001 that the MFP server 120 displays on a display unit (e.g., a browser screen, etc.) of another device (e.g., the terminal device 113) via a network. The history confirmation screen 1001 is a screen for confirming the history of a process involving the MFP server 120, such as the above-mentioned cloud scan. In this embodiment, the history confirmation screen 1001 is used to confirm the history related to the execution of the cloud scan. The history confirmation screen 1001 is displayed on a display unit of an information processing device (e.g., the terminal device 113) when, for example, the button 902 is pressed on the menu screen 901 shown in FIG. 9.

Display area 1002 is a display area where the history of cloud scan execution is displayed. For each cloud scan process executed, the history may include, for example, information about the target file, information about the execution date and time, and information about whether the process of storing in cloud storage was successful.

As described above, when an error occurs in the MFP 100, information about the error is displayed in the display area 705 of the cloud scan screen 701 displayed on the operation unit 102 of the MFP 100. In contrast, when an error occurs in the MFP server 120, information about the error is displayed in the display area 1002 of the history confirmation screen 1001. With this configuration, there is no longer a need to occupy the MFP 100 after the processing executed in the MFP 100 is completed, and other users can then use the MFP 100 for other purposes.

Button 1003 is a user interface for receiving an instruction from the user to call up a screen (e.g., menu screen 901) from which history confirmation screen 1001 is called. When button 1003 is pressed, the screen from which history confirmation screen 1001 is called is displayed again.

(Settings screen)
Next, FIG. 11 will be described. FIG. 11 is a diagram showing an example of a setting screen 1101 that the MFP server 120 displays on a display unit (e.g., a browser screen, etc.) of another device (e.g., the terminal device 113) via a network. The setting screen 1101 is a screen for accepting, from a user, various settings related to processing involving the MFP server 120, such as the above-mentioned cloud scan. In this embodiment, the setting screen 1101 is used to specify settings related to cloud storage used to store a file corresponding to the result of reading a document when performing cloud scan. The setting screen 1101 is displayed on a display unit of an information processing device (e.g., the terminal device 113) when, for example, the button 903 is pressed on the menu screen 901 shown in FIG. 9.

The setting area 1102 is a user interface for accepting, from the user, specification of information related to the account used to access the cloud storage when performing a cloud scan. In the example shown in FIG. 11, the setting area 1102 is provided with an area for accepting specification of a user ID used to access the cloud storage for each cloud storage, and an area for accepting specification of a password used to authenticate the corresponding user.

Button 1003 is a user interface for receiving instructions from the user regarding the reflection of the information specified in setting area 1102. When button 1003 is pressed, the information specified in setting area 1102 is stored in a specified memory area by data storage unit 402. Note that the information stored in the memory area is used to authenticate the target user so that MFP server 120 can access cloud storage 121.

<Processing>
An example of processing of the information processing system according to this embodiment will be described with reference to FIGS.

(Cloud print function)
First, an example of a process related to realizing the cloud print function will be described with reference to Fig. 12. The series of processes shown in Fig. 12 is started, for example, when the button 604 on the cloud print screen 601 shown in Fig. 6 is pressed.

In S1201, the operation control unit 301 identifies the file designated by the user as the file to be printed, based on the selection state of the files displayed as a list in the display area 602 of the cloud print screen 601. The operation control unit 301 notifies the MFP server communication unit 310 of the file designated as the file to be printed and the input state of the check box 603, and requests the MFP server communication unit 310 to execute processing related to realizing the cloud print function.

In S1202, MFP server communication unit 310 transmits to MFP server 120 the file specified in S1201 and information corresponding to the input state of check box 603. In this manner, MFP server communication unit 310 requests MFP server 120 to acquire the specified file, and instructs MFP server 120 whether or not to associate URL information with the file, depending on the input state of check box 603.
Then, the MFP server communication unit 310 determines whether or not the transmission of the request to the MFP server 120 has been successful.
If the MFP server communication unit 310 determines in S1202 that the transmission of the request to the MFP server 120 has failed, the process proceeds to S1209. In S1209, the operation control unit 301 of the MFP 100 displays information about the error that has occurred in the display area 605 of the cloud print screen 601. Then, the series of processes shown in FIG. 12 ends.
On the other hand, if the MFP server communication unit 310 determines in S1202 that the transmission of the request to the MFP server 120 has been successful, the process proceeds to S1203.

In S1203, the MFP communication unit 406 receives the request transmitted from the MFP 100 in S1202, and registers the processing content corresponding to the request in the job control unit 403. The job control unit 403 requests the cloud storage communication unit 405 to perform processing in accordance with the registered processing content.
In response to this request, the cloud storage communication unit 405 attempts to acquire the file designated as the file to be acquired in S1202 from the cloud storage 121. At this time, the cloud storage communication unit 405 uses the account information corresponding to the target cloud storage stored in a specified storage area by the data storage unit 402 for authentication to access the cloud storage.
Then, the cloud storage communication unit 405 determines whether or not the acquisition of the target file has been successful.
If the cloud storage communication unit 405 determines in S1203 that acquisition of the target file has failed, the process proceeds to S1209. In S1209, the cloud storage communication unit 405 notifies the MFP 100 that acquisition of the target file has failed. In response to this notification, the operation control unit 301 of the MFP 100 displays information about the error that has occurred in the display area 605 of the cloud print screen 601. Then, the series of processes shown in FIG. 12 ends.
On the other hand, if the cloud storage communication unit 405 determines in step S1203 that the acquisition of the target file has been successful, the process proceeds to step S1204.

In S1204, the job control unit 403 determines whether or not to associate URL information with the target file, depending on the processing content registered in S1203.

If the job control unit 403 determines in S1204 that URL information is to be associated with the target file, the process proceeds to S1205. In S1205, the job control unit 403 requests the image processing unit 404 to create a rendering image to which the URL information has been added. The image processing unit 404 creates a rendering image by performing rendering on the file to be printed that was acquired in S1203. Examples of files to be printed include electronic documents in text format and electronic documents written based on a specified markup language.

Examples of markup languages used to write electronic documents include Extensible Markup Language (XML) and Hyper Text Markup Language (HTML). Of course, these are merely examples, and other markup languages may be used as long as they can be used to write electronic documents.
The file format of the rendering image may be, for example, a bitmap, but a file format in which an arbitrary print layout is specified, such as PDF (Portable Document Format), may also be applied.

The image processing unit 404 adds a cover page to the created rendering image, and assigns URL information indicating the location where the file to be printed is stored in cloud storage to the cover page in a specific description format. Examples of the description format include a character string indicating the URL or directory path to the target file, and a so-called barcode format such as a two-dimensional barcode.

On the other hand, if the job control unit 403 determines in S1204 that URL information is not to be associated with the target file, the process proceeds to S1208. In S1208, the job control unit 403 requests the image processing unit 404 to create a rendering image to which URL information has been added. The image processing unit 404 creates the rendering image by performing rendering on the file to be printed that was obtained in S1203. Note that the process related to creating the rendering image is the same as the process in S1205.

In S1206, the job control unit 403 transmits the rendering image created by the image processing unit 404 in S1205 or S1208 to the MFP 100 via the MFP communication unit 406.

In S1207, the MFP server communication unit 310 of the MFP 100 receives the rendering image sent by the MFP server 120 in S1206, and registers a print job for printing the rendering image in the job control unit 303. The job control unit 303 causes the print processing unit 305 to execute processing for printing the rendering image based on the registered print job.

(Cloud scan function)
Next, an example of a process related to realizing the cloud scan function will be described with reference to Fig. 13. The series of processes shown in Fig. 13 is started, for example, by pressing the button 704 on the cloud scan screen 701 shown in Fig. 7.

In S1301, the operation control unit 301 registers the processing content of the cloud scan in the job control unit 303 according to the selection state of the cloud storage in the display area 702 of the cloud scan screen 701 and the input state of the check box 703. Note that the selection state of the cloud storage in the display area 702 and the input state of the check box 703 may be registered as, for example, attributes of the processing content.
The job control unit 303 requests the reading processing unit 306 to execute processing related to reading the document placed on the scanner unit 104 in accordance with the registered processing content. In response to this request, the reading processing unit 306 controls the operation of the scanner unit 104 to execute processing related to reading the document placed on the reading surface of the scanner unit 104.
The scanned image obtained as a result of reading the document is output, for example, as bitmap data (file) in which information obtained by optically reading the document is arranged two-dimensionally.

In S1302, the MFP server communication unit 310 transmits data corresponding to the result of reading the document in S1301 (i.e., scanned image data) as a processing request to the MFP server 120. At this time, information regarding the cloud storage selected in the display area 702 and the input state of the check box 703 are also notified to the MFP server 120.
Then, the MFP server communication unit 310 determines whether or not the transmission of the data corresponding to the result of reading the document to the MFP server 120 has been successful.
If the MFP server communication unit 310 determines in S1302 that transmission of data corresponding to the document reading result to the MFP server 120 has failed, the process proceeds to S1319. In S1319, the operation control unit 301 of the MFP 100 displays information about the error that has occurred in the display area 705 of the cloud scan screen 701. Then, the series of processes shown in FIG. 13 ends.
On the other hand, if the MFP server communication unit 310 determines in S1302 that the transmission of the data corresponding to the result of reading the document to the MFP server 120 has been successful, the process proceeds to S1303.

In S1303, the MFP communication unit 406 receives the processing request sent from the MFP 100, and registers the processing content corresponding to the processing request in the job control unit 403. Based on the information registered as an attribute in the processing request, the job control unit 403 determines whether a cloud storage to which data corresponding to the document reading results is to be sent has been specified, or whether URL information has been associated with the data.

If the job control unit 403 determines in S1303 that the association of URL information with the data corresponding to the document reading result has not been specified and that a cloud storage destination has been specified, the process proceeds to S1321. In S1321, the job control unit 403 transmits the data corresponding to the document reading result (in other words, the data of the scanned image) to the cloud storage specified as the destination via the cloud storage communication unit 405.
In this case, since no URL information is associated with the data, no file on the cloud storage to be reflected by the data is specified. Therefore, the data may be placed in, for example, a directory set as a default in the cloud storage to which the data is to be sent.
The file name of the data may also be determined based on conditions set in the cloud storage, for example. As a specific example, the file name of the target data may be determined based on information regarding the date and time when the scan was performed.
In addition, the file format of the data may be determined based on conditions set in the cloud storage, for example. As a specific example, the target data may be stored as a PDF file.
Of course, the above is merely an example and may be modified as appropriate.

On the other hand, if the job control unit 403 determines in step S1303 that the association of URL information with data corresponding to the document reading result is specified, the process proceeds to step S1304.
In this embodiment, for convenience, the selection state of the cloud storage in the display area 702 on the cloud scan screen 701 and the input state in the check box 703 are mutually exclusive controlled by the operation control unit 301. Specifically, when the check box 703 is checked, the display area 702 is controlled to recognize that no cloud storage has been selected. In other words, the input state in the check box 703 takes priority over the selection state of the cloud storage in the display area 702.

In step S1304, the job control unit 403 has the image processing unit 404 analyze the target data, and based on the results of the analysis, attempts to identify a URL for accessing a file (electronic document) on cloud storage corresponding to the data.
For example, the image processing unit 404 may recognize URL information that is assigned in a predetermined description format (e.g., a two-dimensional barcode format) to the cover page that was added to the target data in S1205, and identify the URL based on the URL information.
As a specific example, image analysis may be performed on a partial image corresponding to a specified area of a scanned image obtained by reading a document (printed material) to recognize URL information presented in the partial image.
The scanned image resulting from the reading of the document corresponds to an example of a “first image.” The process of S1304 corresponds to an example of a “first specific process.”

When the URL is specified, the job control unit 403 accesses the cloud storage corresponding to the URL via the cloud storage communication unit 405 and attempts to obtain a file (electronic document) corresponding to the data from the cloud storage. At this time, the job control unit 403 may use, for example, account information corresponding to the target cloud storage stored in a predetermined storage area by the data storage unit 402 for authentication to access the cloud storage.
Then, the job control unit 403 determines whether or not a file on the cloud storage corresponding to the target data has been acquired.

If the job control unit 403 determines in S1304 that it has been unable to obtain a file on cloud storage that corresponds to the target data, it advances the process to S1320. As a specific example, if access to cloud storage fails, if authentication fails, or if the corresponding file does not exist on cloud storage, then obtaining the target file in S1304 fails. In this case, the job control unit 403 has been unable to obtain the file that is the target of the process relating to reflecting the proofreading results shown on the manuscript based on data corresponding to the results of reading the manuscript, which will be described in detail later as the process of S1305 to S1317.

In S1320, the job control unit 403 instructs the data storage unit 402 to store a history of the job processing in a specified storage area. In this case, information indicating a failure in the cloud scan processing is stored as the history of the job processing. Thereafter, the series of processes shown in FIG. 13 ends.

On the other hand, if the job control unit 403 determines in S1304 that a file on the cloud storage corresponding to the target data has been acquired, the process proceeds to S1305. In S1305, the job control unit 403 requests the image processing unit 404 to create a rendering image of the file (telegram document) acquired in S1304. In response to this request, the image processing unit 404 generates a rendering image by performing rendering on the file. Note that, for convenience, in this embodiment, the file is assumed to be an electronic document in text format or an electronic document described based on a predetermined markup language. Also, the data of the rendering image is assumed to be data in bitmap format.
The rendering image corresponds to an example of a "second image."

In S1306, the image processing unit 404 compares the scanned image corresponding to the result of reading the document in S1301 with the rendering image created in S1305, and obtains a difference image corresponding to the result of the comparison.
Note that the scanned image and the rendered image may differ in size and resolution. In such a case, for example, the image processing unit 404 may adjust the size and resolution of at least one of the scanned image and the rendered image. As a specific example, when creating a rendered image, the image processing unit 404 may control the size and resolution of the rendered image to be created in accordance with the size and resolution of the scanned image.
In addition, the image processing unit 404 may adjust the inclination of at least one of the scanned image and the rendered image, assuming a situation in which the scanned image is inclined relative to the rendered image. As a specific example, the image processing unit 404 may extract a difference between the scanned image and the rendered image while rotating the rendered image, and specify the rotation angle at which the difference is minimum as the rotation angle of the rendered image when comparing the scanned image and the rendered image.

In S1307, the image processing unit 404 recognizes the difference image acquired in S1306 as an image showing writing made to the printout of the electronic document from which the rendering image is generated (hereinafter also referred to as a "writing image") The image processing unit 404 breaks down the writing image into proofreading symbols and natural language written as a result of proofreading the printout of the electronic document.
Examples of proofreading symbols include symbols defined by the International Organization for Standardization as ISO 5776 and symbols defined by the Japanese Industrial Standards as JIS Z 8208.
The method of decomposing the written image into the review symbol and the natural language is not particularly limited. As a specific example, the written image may be decomposed into the review symbol and the natural language by pattern matching using a recognizer constructed by learning the characteristics of the review symbol and the natural language based on so-called machine learning.
The image processing unit 404 also performs various analytical processes (e.g., image analysis, etc.) on the review symbols and natural language extracted from the written image by the above-mentioned decomposition process to recognize correction instructions indicated by the review symbols and convert the natural language into text information. Converting the natural language into text information can be achieved by, for example, OCR (Optical Character Recognition) processing.
Note that the natural language written on the printed material during proofreading is used, for example, in combination with proofreading symbols, and plays a supplementary role in presenting correction candidates, etc.
The processes in S1306 and S1307 correspond to an example of a "second identification process."

In S1308, the image processing unit 404 identifies the coordinates on the rendering image of the proofreading symbol. In this disclosure, the coordinates on the rendering image refer to an area surrounded by four points represented by the X-axis and Y-axis on a bitmap described as a two-dimensional array. Since the above-mentioned written image (difference image) is obtained by comparing the scanned image and the rendering image in S1306, it goes without saying that it is possible to identify the positional correspondence between the rendering image and the written image.

In S1309, the image processing unit 404 extracts the area corresponding to the coordinates identified in S1308 from the rendering image. As described above, the rendering image is an image generated by rendering the electronic document to be reflected (in other words, the electronic document corresponding to the printed matter on which notes have been added during proofreading). Therefore, it is expected that the extracted area will present the character string that the user targeted for correction during proofreading.

In S1310, the image processing unit 404 identifies the character string presented in the area on the rendering image extracted in S1309. Note that the data of the area extracted in S1309 is image data in a format such as a bitmap. Therefore, the image processing unit 404 may identify the character string presented in the area by, for example, performing OCR processing on the area (i.e., the image in a format such as a bitmap). The image processing unit 404 may convert the character string identified from the area into text information and store it.

In S1311, the image processing unit 404 searches the character string identified in S1310 (i.e., the character string presented in the area on the rendering image) from the electronic document to be reflected. As a specific example, the image processing unit 404 may search the character string converted into text information in S1310 from the electronic document to be reflected based on a character string comparison with text information indicating the contents of the electronic document to be reflected.

In S1312, the image processing unit 404 identifies at least some of the candidates for character strings in the electronic document searched for in S1311 as reflection candidates. Then, the image processing unit 404 processes the electronic document to control the display mode so that the character strings that are reflection candidates are displayed in a mode different from other character strings.
In this embodiment, the image processing unit 404 processes the electronic document to be processed so that the character strings that are candidates for reflection are highlighted.

The processing for highlighting the character strings that are candidates for reflection may be appropriately changed according to the format of the electronic document to be processed.
As a specific example, when the target is an electronic document written based on a specified markup language, a process of assigning highlight attributes to the character strings that are candidates for reflection based on the format of the markup language may be performed on the electronic document.
As another example, when a text-format electronic document is the target, the appearance position of the character string that is the reflection candidate in the electronic document may be specified, and information about the appearance position may be stored in a predetermined storage area. Note that, for example, information indicating the appearance position in a text file may be information indicating the number of characters from the beginning of the file. Note that, the information indicating the appearance position is used to specify the position to be highlighted when rendering the electronic document.

As explained above, by carrying out processing related to control of the display mode, such as highlighting, it becomes possible to identify candidates for the character string to be corrected, for example, when a similar character string appears multiple times in the electronic document to be reflected. Details of this processing will be described separately later.

In step S1313, the image processing unit 404 generates a rendering image by performing rendering on the electronic document that has been processed in step S1312 to highlight the character strings that are candidates for reflection.
For example, when an electronic document written in a specified markup language is targeted, a rendering process is performed in the same manner as in normal cases, and the area corresponding to the character string that is a candidate for reflection is highlighted.
As another example, when a text-format electronic document is the target, a process for performing highlight processing is executed at the time of rendering. Specifically, when generating a rendering image, a process for changing the background color of an area corresponding to a character string that is a candidate for reflection is executed based on information regarding the appearance position of the character string.
Note that a rendering image generated by performing rendering on an electronic document that has been subjected to processing related to control of the display mode, such as highlight processing, as exemplified above, corresponds to an example of a "third image."

In S1314, the image processing unit 404 compares the rendering image generated in S1305 with the rendering image generated in S1313. Then, based on the result of the comparison, the image processing unit 404 identifies coordinates on the rendering image generated in S1305 that correspond to the area in the rendering image generated in S1313 that has been subjected to highlight processing.

In S1315, the image processing unit 404 determines whether the coordinates of the highlighted area identified in S1314 match the coordinates of the revision symbol identified in S1308. Note that the size of the highlighted area does not necessarily match the size of the revision symbol area. Therefore, for example, the image processing unit 404 may determine that the coordinates of these areas match when the size of a partial area that overlaps with the other area in at least one of these areas is equal to or greater than a predetermined percentage (e.g., equal to or greater than half).

If the image processing unit 404 determines in S1315 that the coordinates of the highlighted area do not match the coordinates of the proofreading mark, the process proceeds to S1312. In this case, the image processing unit 404 identifies another candidate and executes the process from S1312 onward for the other candidate.

On the other hand, if the image processing unit 404 determines in S1315 that the coordinates of the highlighted area match the coordinates of the revision mark, the process proceeds to S1316.
The processes relating to the specification of the reflection candidates shown as S1308 to S1315 correspond to an example of the "third specification process".
In S1316, the image processing unit 404 corrects the character string in the electronic document identified as the reflection candidate in S1312 according to the content indicated by the proofreading symbol. As a specific example, if the proofreading symbol is a strikethrough, the image processing unit 404 deletes the reflection candidate character string. As another example, if the proofreading symbol indicates a correction to a different character string, the image processing unit 404 replaces the reflection candidate character string with the different character string.

In S1317, the job control unit 403 transmits the electronic document corrected in S1316 to the cloud storage via the cloud storage communication unit 405, and instructs the cloud storage to update the electronic document to be reflected. As a result, the electronic document to be reflected that is stored in the cloud storage is overwritten with the electronic document transmitted from the job control unit 403 (i.e., the electronic document corrected in S1316).
In addition, if the cloud storage manages versions of the electronic document, the electronic document modified in the overwrite process is treated as a new version of the electronic document to be reflected. By performing such management, it becomes possible to control, for example, to return the electronic document stored in the cloud storage to the version before the overwrite process, or to present the modified part of the new version as a difference.

In S1318, the job control unit 403 causes the data storage unit 402 to store a history relating to the processing of the above-mentioned series of jobs in a predetermined storage area. The information stored as the history includes, for example, information on the file name of the electronic document to be reflected, the processing date and time, and the success or failure of the processing, which are displayed on the history confirmation screen 1001 shown in FIG. 10. In this case, information indicating the success of the cloud scan processing is recorded as the history relating to the job processing.

By applying the above-mentioned processing, even if the proofreading results are handwritten on a printed copy of an electronic document, it is possible to automatically make corrections to the electronic document stored in cloud storage according to the contents indicated by the proofreading results. This makes it possible to reflect the proofreading results in the target electronic document without the user having to perform the cumbersome task of electronically transcribing the handwritten contents on the printed copy of the electronic document using a keyboard or the like.

<Modification>
A modified example of the information processing system according to the present embodiment will be described below. In the above-described embodiment, when performing cloud scanning, a reflection candidate character string is identified by repeatedly searching for a series of character string candidates retrieved from an electronic document. In this modified example, a process related to control of the display mode is performed on the electronic document so that each of the series of character string candidates retrieved from the electronic document is displayed in a different manner, thereby making it possible to identify a reflection candidate character string without performing a repeated search.
Therefore, in this modified example, an example of a case in which each of a series of candidate character strings searched for from an electronic document is highlighted in a different color is described below, making it possible to identify candidate character strings to be reflected.
Moreover, in this modified example, the description will focus on the parts that are different from the previously described embodiment, and detailed descriptions of parts that are substantially the same as the previously described embodiment will be omitted.

Figure 14 is a flowchart showing an example of processing of the information processing system according to this modified example, and shows an example of processing related to realizing the cloud scan function. The series of processing shown in Figure 14 is started, for example, by pressing button 704 on the cloud scan screen 701 shown in Figure 7.

In step S<b>1401 , the operation control unit 301 registers the processing content of the cloud scan in the job control unit 303 according to the selection state of the cloud storage in the display area 702 of the cloud scan screen 701 and the input state of the check box 703 .
The job control unit 303 requests the reading processing unit 306 to execute processing related to reading the document placed on the scanner unit 104 in accordance with the registered processing content. In response to this request, the reading processing unit 306 controls the operation of the scanner unit 104 to execute processing related to reading the document placed on the reading surface of the scanner unit 104.

In S1402, the MFP server communication unit 310 transmits data corresponding to the result of reading the document in S1401 (i.e., data of the scanned image) as a processing request to the MFP server 120. At this time, information regarding the cloud storage selected in the display area 702 and the input state of the check box 703 are also notified to the MFP server 120.
Then, the MFP server communication unit 310 determines whether or not the transmission of the data corresponding to the result of reading the document to the MFP server 120 has been successful.
If the MFP server communication unit 310 determines in S1402 that the transmission of data corresponding to the document reading result to the MFP server 120 has failed, the process proceeds to S1419. In S1419, the operation control unit 301 of the MFP 100 displays information about the error that has occurred in the display area 705 of the cloud scan screen 701. Then, the series of processes shown in FIG. 14 ends.
On the other hand, if the MFP server communication unit 310 determines in S1402 that the transmission of the data corresponding to the result of reading the document to the MFP server 120 has been successful, the process proceeds to S1403.

In S1403, the MFP communication unit 406 receives the processing request sent from the MFP 100, and registers the processing content corresponding to the processing request in the job control unit 403. Based on the information registered as an attribute in the processing request, the job control unit 403 determines whether a cloud storage has been specified as a destination for data corresponding to the results of scanning the original, or whether URL information has been associated with the data.

If the job control unit 403 determines in S1403 that no URL information has been associated with the data corresponding to the results of scanning the original document, and that a cloud storage destination has been specified, the process proceeds to S1421. In S1421, the job control unit 403 transmits the data corresponding to the results of scanning the original document (in other words, the data of the scanned image) to the cloud storage specified as the destination, via the cloud storage communication unit 405.

On the other hand, if the job control unit 403 determines in S1403 that an association of URL information with data corresponding to the document reading results has been specified, the process proceeds to S1404.

In S1404, the job control unit 403 causes the image processing unit 404 to analyze the target data, and based on the results of the analysis, attempts to identify a URL for accessing a file (electronic document) on cloud storage corresponding to the data.
Then, the job control unit 403 determines whether or not a file on the cloud storage corresponding to the target data has been acquired.

If the job control unit 403 determines in S1404 that a file on cloud storage corresponding to the target data could not be acquired, the process proceeds to S1420. As a specific example, if access to cloud storage fails, if authentication fails, or if the corresponding file does not exist on cloud storage, acquisition of the target file fails in S1404.

In S1420, the job control unit 403 instructs the data storage unit 402 to store a history of the job processing in a specified storage area. In this case, information indicating a failure in the cloud scan processing is recorded as the history of the job processing. Thereafter, the series of processes shown in FIG. 14 ends.

On the other hand, if the job control unit 403 determines in S1404 that a file on the cloud storage corresponding to the target data has been acquired, the process proceeds to S1405. In S1405, the job control unit 403 requests the image processing unit 404 to create a rendering image of the file (telegram document) acquired in S1304. In response to this request, the image processing unit 404 generates a rendering image by rendering the file.

In S1406, the image processing unit 404 compares the scanned image resulting from reading the original in S1401 with the rendering image created in S1405, and obtains a difference image resulting from the comparison.

In S1407, the image processing unit 404 recognizes the difference image acquired in S1406 as a writing image that indicates writing applied to the printout of the electronic document from which the rendering image is generated.

In S1408, the image processing unit 404 identifies the coordinates of the revision symbol on the rendering image.

In S1409, the image processing unit 404 extracts the area corresponding to the coordinates identified in S1408 from the rendering image.

In S1410, the image processing unit 404 identifies the character string presented in the area on the rendering image extracted in S1409.

In S1411, the image processing unit 404 searches the electronic document to be reflected for the character string identified in S1410.

In S1412, the image processing unit 404 performs processing on the target electronic document to control the display mode so that each of the series of candidates for the character string in the electronic document searched for in S1411 is displayed in a different mode from other character strings. At this time, the image processing unit 404 controls the display mode of each candidate so that each of the series of candidates searched for is displayed in a different mode.
As described above, in this modified example, the image processing unit 404 processes the electronic document to highlight the candidate character strings. Specifically, the image processing unit 404 processes the electronic document to highlight each of the searched candidates in a different color. The conditions for color-coding the searched candidates may be set in advance.

In S1413, the image processing unit 404 generates a rendering image by performing rendering on the electronic document that has been processed to highlight the character strings corresponding to each of the series of candidates searched for in S1412.

In S1414, the image processing unit 404 compares the rendering image generated in S1405 with the rendering image generated in S1413. Then, based on the result of the comparison, the image processing unit 404 identifies coordinates on the rendering image generated in S1405 that correspond to the area in the rendering image generated in S1413 that has been subjected to highlight processing.

In S1415, the image processing unit 404 identifies an area that matches the coordinates of the proofreading symbol identified in S1408 from among the areas corresponding to each of the series of candidates that have been highlighted and identified in S1414. Then, based on the color of the highlighting applied to the identified area, the image processing unit 404 identifies the corresponding candidate from the series of candidates searched in S1412 as the target for reflection.

In S1416, the image processing unit 404 corrects the character string in the electronic document identified as a candidate for reflection in S1415 according to the content indicated by the proofreading symbol.

In S1417, the job control unit 403 transmits the electronic document modified in S1416 to the cloud storage via the cloud storage communication unit 405, and instructs the cloud storage to update the electronic document to be reflected. As a result, the electronic document to be reflected that is stored in the cloud storage is overwritten with the electronic document transmitted from the job control unit 403 (i.e., the electronic document modified in S1416).

In S1418, the job control unit 403 causes the data storage unit 402 to store a history relating to the processing of the above-mentioned series of jobs in a predetermined storage area. The information stored as the history includes, for example, information on the file name of the electronic document to be reflected, the processing date and time, the success or failure of the processing, etc., which are displayed on the history confirmation screen 1001 shown in FIG. 10. In this case, information indicating the success of the cloud scan processing is recorded as the history relating to the job processing.

By applying the above-mentioned processing, even if the proofreading results are handwritten on a printed copy of an electronic document, it is possible to automatically make corrections to the electronic document stored in cloud storage according to the contents indicated by the proofreading results. This makes it possible to reflect the proofreading results in the target electronic document without the user having to perform the cumbersome task of electronically transcribing the handwritten contents on the printed copy of the electronic document using a keyboard or the like.

<Other embodiments>
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a recording medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions.

100 MFP
120 MFP server 121 Cloud storage 403 Job control unit 404 Image processing unit

Claims (9)

a receiving means for receiving the first image, the first instruction, and the second instruction from a device including: a scanner unit for reading a printed matter to which information indicating the location on the cloud storage where the original electronic document is stored is attached as an image to generate a first image; and an operation unit for displaying a screen for receiving from the user a first instruction specifying whether or not to reflect a correction instruction written on the printed matter in the original electronic document stored in the location indicated by the information, and a second instruction specifying a cloud storage to which the first image is to be sent from a list of cloud storages available to the user;
a transmission means for transmitting the first image received by the receiving means to the cloud storage specified in the second instruction when the first instruction received by the receiving means specifies that the correction instruction should not be reflected in the original electronic document and the second instruction specifies a cloud storage to which the first image is to be transmitted;
a generating means for generating a second image by performing rendering on the original electronic document when the first instruction received by the receiving means specifies that the correction instruction should be reflected in the original electronic document, and when the original electronic document can be obtained from a location on the cloud storage based on information indicating a location on the cloud storage where the original electronic document is stored, the information being obtained by analyzing the first image received by the receiving means, and generating a difference image by extracting a difference between the first image received by the receiving means and the generated second image;
a specifying means for performing an analysis process and an OCR process on the difference image generated by the generating means, thereby recognizing the correction instruction indicated by the proofreading symbol included in the difference image and text information in natural language included in the difference image, and further specifying a character string in the electronic document that is a target for reflecting the correction instruction based on the coordinates of the proofreading symbol;
an execution unit that executes a correction process based on the correction instruction on the character string in the electronic document identified by the identification unit;
An information processing device comprising: The information processing device according to claim 1, further comprising a storage means for storing information indicating that reflecting the correction instructions in the original electronic document has failed when the first instruction received by the receiving means specifies that the correction instructions should be reflected in the original electronic document, and when the original electronic document cannot be obtained from the location on the cloud storage where the original electronic document is stored based on the information indicating the location on the cloud storage where the original electronic document is stored, which is obtained by analyzing the first image received by the receiving means. The information processing device according to claim 1, characterized in that the correction process executed by the execution means is a process of deleting a character string in the electronic document identified by the identification means when the correction instruction is deletion. The information processing device according to claim 1, characterized in that the correction process executed by the execution means is a process of correcting a character string in the electronic document identified by the identification means to another character string when the correction instruction is to correct the character string. The information processing device according to claim 1, characterized in that the electronic document is an electronic document written based on XML or HTML. The information processing device according to claim 1, characterized in that the location where the electronic document from which the printed matter is printed is stored is specified by a URL or a directory path written in the first image in a character string format or a barcode format. The information processing device according to claim 1, characterized in that the electronic document is an electronic document in text format. An information processing method executed by an information processing device,
a receiving step of receiving the first image, the first instruction, and the second instruction from a device including: a scanner unit that reads a printed matter to which information indicating the location on the cloud storage where the original electronic document is stored is attached as an image to generate a first image; and an operation unit that displays a screen for receiving from the user a first instruction specifying whether or not to reflect a correction instruction written on the printed matter in the original electronic document stored in the location indicated by the information, and a second instruction specifying a cloud storage to which the first image is to be sent from a list of cloud storages available to the user;
a transmission step of transmitting the first image received in the receiving step to the cloud storage specified in the second instruction when the first instruction received in the receiving step specifies that the modification instruction should not be reflected in the original electronic document and the second instruction specifies a cloud storage to which the first image is to be transmitted;
a generating step of generating a second image by performing rendering on the electronic document to be printed, if the first instruction received in the receiving step specifies that the correction instruction should be reflected in the electronic document to be printed, and if the electronic document to be printed can be obtained from a location on the cloud storage based on the information indicating a location on the cloud storage where the electronic document to be printed is stored, obtained by analyzing the first image received in the receiving step, and generating a difference image by extracting a difference between the first image received in the receiving step and the generated second image;
a step of performing an analysis process and an OCR process on the difference image generated in the generation step, thereby recognizing the correction instruction indicated by the proofreading symbol included in the difference image and text information in natural language included in the difference image, and further, identifying a character string in the electronic document that is a target for reflecting the correction instruction based on the coordinates of the proofreading symbol;
an execution step of executing a correction process based on the correction instruction on the character string in the electronic document identified by the identification step;
13. An information processing method comprising: A program for causing a computer to function as each of the means of an information processing device according to any one of claims 1 to 7.

Images