JP7703902B2 - DATA OUTPUT SYSTEM, INFORMATION PROCESSING SYSTEM, INFORMATION PROCESSING METHOD, AND PROGRAM - Google Patents

Description

The present invention relates to a data output system, an information processing system, an information processing method, and a program.

In the daily work of organizations such as companies and groups, the processing of documents such as expense reports, invoices, and purchase orders is often required. Personnel within the organization perform daily tasks such as extracting and tallying transaction dates, case numbers, and amounts from documents.

Technology has been devised for digitizing such forms (see, for example, Patent Document 1). Patent Document 1 discloses a technology that detects filled areas including areas filled with chromatic colors and enclosed areas including areas surrounded by chromatic colors in a scanned image of a document, and classifies the filled areas as areas for a first process and the enclosed areas as areas for a second process.

However, conventional technology has the problem that users cannot associate marks with processes. For example, if a user cannot register the correspondence between marks and processes by making marks on a designated piece of paper with a marker pen or the like, a person in charge will have to enter the correspondence between marks and processes in a file, etc.

In view of the above-mentioned problems, an object of the present invention is to provide a technique that enables identification information that a user adds to a specific paper sheet to be associated with a process.

In view of the above problems, the present invention provides an information processing system that is communicatively connected to one or more devices and executes a specified process, and includes a data storage unit that stores setting information that associates a mark that identifies the content of the process with the specified process based on a user's operation on a first device, a communication unit that receives second image data of a specified document including the mark added by the user from the first device, an area detection unit that detects an area including the mark from the second image data, a management unit that associates and stores position information of the detected area, a specified process corresponding to the mark included in the area, and an application selected by the user on the first device, and a processing unit that executes the specified process for third image data received from the second device based on the position information and the specified process associated with the application selected on the second device .

It is possible to provide an information processing system that can associate identification information that a user adds to a specific piece of paper with a process.

11A and 11B are diagrams illustrating the process or operation of the data output system for associating marks with processes. 11A and 11B are diagrams illustrating setting of area information in a form and acquisition of data from the form using the area information. FIG. 13 is a diagram for explaining the registration of the correspondence between colors and processes (task 1) using a registration form. FIG. 2 illustrates an example of a management site. FIG. 13 is a diagram for explaining the correspondence between area information and processing (task 2) using the original paper of a form. FIG. 13 is a diagram for explaining processing (task 3) on an area in a form using the form. 1 is a diagram showing a system configuration of an example of a data output system according to an embodiment of the present invention; FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing system and a terminal device. FIG. 2 is a diagram illustrating a hardware configuration of an example of an image forming apparatus. FIG. 2 is a diagram illustrating an example of a functional configuration of a device, a terminal device, and an information processing system. FIG. 4 is a diagram illustrating an example of application processing information stored in an application processing information storage unit. FIG. 2 is a diagram illustrating a series of processes performed by an application. 11A and 11B are diagrams illustrating a method for setting the correspondence between colors and processes using a registration form. FIG. 13 is a diagram showing an example of an administrator screen that is displayed when an administrator logs in to the device. 13 is a sequence diagram illustrating an example of a procedure in which an administrator uses a registration form to register the correspondence between colors and processes in a configuration file. FIG. 2 is a diagram for explaining detection of color information. FIG. 13 is a diagram showing an example of use of a registration form in which the process is applied to setting a file name. FIG. 2 illustrates an example of a management site. FIG. 13 is a diagram showing an example of a form in which an administrator designates two areas with a marker pen. FIG. 13 is a diagram showing a modified example of the registration form. FIG. 13 is a diagram illustrating a method for marking an area on a form. 13A and 13B are diagrams illustrating another example of a method for marking an area on a document. 13A and 13B are diagrams illustrating a comparative example of a method for specifying an area in a form. 13 is a diagram showing an example of a region registration screen displayed by the device when registering region information. FIG. 11 is an example of a sequence diagram illustrating a procedure in which an information processing system acquires area information from a form on which an administrator has marked areas, and creates a setting file. 1 is a diagram illustrating an example of detection of an area from image data; 11 is a sequence diagram showing an example of a procedure in which an administrator sets information in columns H to M of a setting file. 1 is a diagram illustrating an example of a correspondence between a setting file and a form. 13 is an example of a screen displayed by the terminal device regarding association between an application and a setting file. 11 is a sequence diagram illustrating an example of a process in which an administrator associates an application with a configuration file. FIG. 13 is a diagram showing a modified example of the association screen. 13 is a diagram illustrating an example of a selection screen for some applications. 13 is an example of a screen operated by an administrator when duplicating an application. 11 is a sequence diagram illustrating an example of a procedure in which the information processing system registers a copy of an application. 11A to 11C are diagrams illustrating an example of transition of screens displayed by a device. 1 is a diagram illustrating an example of a file stored in an external service system; 11 is an example of a sequence diagram in which the information processing system receives a PDF file including image data of a form from a device and acquires data from an area. 11 is an example of a sequence diagram in which the information processing system receives a PDF file including image data of a form from a device and acquires data from an area. 10A and 10B are diagrams illustrating an example of correction of an OCR processing result. FIG. 13 is a diagram showing an example of a file list screen of an external service system displayed on a terminal device. FIG. 13 is a diagram showing an example of an output data display screen displayed by the device. 13 is a diagram showing an example of a plurality of forms and an output file. 11A and 11B are diagrams for explaining a process or operation in which the data output system associates a mark with a process (Example 2). 11 is a diagram for explaining a process or operation in which the data output system creates a file using data acquired from a form (Example 2). FIG. FIG. 13 is a diagram for explaining the registration of the correspondence between colors and processes (task 1) using a registration form. FIG. 11 is a diagram illustrating an example of a management site (second embodiment); FIG. 13 is a diagram for explaining the correspondence between area information and processing (task 2) using a base paper of a form (Example 2). FIG. 13 is a diagram for explaining the correspondence between area information and processing (task 3) using a base paper of a form (Example 2). FIG. 11 is a diagram illustrating a functional configuration of an OCR processing unit (second embodiment). FIG. 11 is a diagram illustrating a series of processes performed by an application (second embodiment); 13A and 13B are diagrams for explaining a method for setting the correspondence between colors and processes using a registration form (Example 2). FIG. 11 is a diagram for explaining detection of color information (Example 2). FIG. 13 is a diagram showing an example of use of a registration form in which the process is applied to setting a file name (embodiment 2). FIG. 11 is a diagram illustrating an example of a management site (second embodiment); FIG. 13 is a diagram showing an example of a form in which an administrator designates two areas with a marker pen (Example 2). FIG. 11 is a diagram showing a modified example of the registration form (Example 2). 11A and 11B are diagrams for explaining a method for marking an area on a document (Example 2). FIG. 11 is a diagram illustrating an example of a region registration screen displayed by the device when registering region information (second embodiment); 13 is a diagram illustrating an example of a correspondence between a setting file and a form (Example 2); 13 is a sequence diagram illustrating an example of a process in which the information processing system receives a PDF file including image data of a form from a device and acquires data from an area (second embodiment); 13 is a sequence diagram illustrating an example of a process in which the information processing system receives a PDF file including image data of a form from a device and acquires data from an area (second embodiment);

Below, a data output system and a processing method performed by the data output system will be described as an example of a form for implementing the present invention.

<Outline of Processing or Operation>
Organizations such as companies and groups often process documents such as expense reports, invoices, and purchase orders. Personnel within the organization extract and tally up dates, case numbers, amounts, and other information from documents on a daily basis. Processed documents are filed and then stored, but the storage space also incurs costs. While the digitization of documents is gradually progressing, it is time-consuming for personnel to rename the file names that are automatically assigned to document files.

Although there are systems and services that automate billing and order processing, many of them are expensive, and not many small and medium-sized enterprises can afford to invest in them. Different organizations use different forms, and even if the same form is used, the items used for processing may differ from organization to organization. For this reason, the current situation is that these forms are inevitably processed manually.

In this embodiment, the information processing system receives image data from the device that associates a predetermined mark (such as the color or shape of a marker) with a process that corresponds to that mark. This allows the user to associate the mark with the process. Furthermore, the information processing system receives image data of a form on which the mark is formed from the device. The information processing system associates the mark, the process, and area information within the form with a setting file. When the form is scanned by the device, the information processing system executes the process that is associated with the area of the form identified by the area information.

This allows the information processing system to perform various processes specified by the user only on specific areas of the document.

The process or operation performed by the data output system will be outlined with reference to Figures 1 and 2. Figure 1 is a diagram explaining the process or operation performed by the data output system to associate marks with processes. Figure 2 is a diagram explaining the process or operation performed by the data output system 100 to create a file using data acquired from a form.

(1) The data output system 100 can link with the external service system 40. The data output system 100 links with the external service system 40 via an app. An app that runs on the data output system 100 may simply be called an "app," and an app that runs on the external service system 40 may be called a linked app. In this case, the app and the linked app are associated with each other by their app IDs, and the data output system 100 stores the corresponding relationship.

(2) The administrator 8 of the data output system 100 uses the device 20 to print out a registration form registered in the information processing system 10 and uses a marker pen to color the form with the color he or she wants to associate with the process. For example, the administrator 8 can associate any color with a process, such as green for OCR, blue for OMR, red for trimming, etc. The device 20 scans the registration form to generate image data (first image data) and transmits it to the information processing system 10.

(3) The information processing system 10 detects the correspondence between colors and processing from the image data, associates the colors with the processing, and registers the correspondence in a setting file. The information processing system 10 saves the setting file in the external service system 40.

Next, we will explain with reference to Figure 2. Figure 2(a) is a diagram explaining how to set area information in a form, and Figure 2(b) is a diagram explaining how to obtain data from a form using area information.

(4) The administrator 8 uses forms 7 that are used daily by users in the organization to specify with a marker pen which areas of the form 7 are to be processed (OCR: Optical Character Reader, OMR: Optical Mark Reader, trimming). For example, the administrator 8 may fill in or circle areas of the form 7 with a marker pen. The device 20 reads the form 7 with the marked areas and generates image data (second image data), which is then transmitted to the information processing system 10. Note that the form 7 to be marked may have text or the like written in the items, may only have the items, or may be blank.

(5) The information processing system 10 identifies areas in the image data of the form that are colored the same as the colors registered in the setting file obtained from the external service system 40, and stores the area information of the areas in the setting file in association with the colors and processing in the setting file. The area information is, for example, the coordinates of the diagonal vertices of a rectangular area in the image data. The information processing system 10 stores the setting file in the external service system 40.

(6)-1 The administrator 8 connects the terminal device 30 to the external service system 40, downloads the configuration file, and inputs the report language, date and time, etc. into the configuration file to complete the configuration file.

(6)-2 In addition, the administrator 8 connects the terminal device 30 to the information processing system 10 and performs an operation to associate the application ID of the application with the setting file. The administrator 8 can associate any number of application IDs with setting files. This associates an icon that accepts the selection of an application with area information in which image processing is performed.

(7) When a user 6 within an organization submits a form 7 to the organization for expense settlement or the like, the user presses the icon of the app that corresponds to the form 7 on device 20. The user can easily select an app from multiple form names, etc. The user has device 20 read a form on which the necessary information has been handwritten or a printed out form. Device 20 reads the app ID of the app and the form and generates image data (third image data, for example a PDF file) and transmits it to information processing system 10.

(8) The information processing system 10 identifies the setting file by the application ID and acquires it from the external service system 40. The information processing system 10 acquires (e.g., cuts out) the area of the area information contained in the acquired setting file from the image data for each process. Thus, the area of the form registered by the administrator 8 is acquired.

(9) The information processing system 10 performs processing associated with each area on the area. Therefore, for each area, character codes such as characters, whether or not a check mark is checked, image data, etc. are obtained. Hereinafter, these are simply referred to as acquired data.

(10) The information processing system 10 outputs the acquired data to an electronic file (e.g., a CSV (Comma Separated Value) file) and transmits the file and image data of the form to the external service system 40. Note that if the information processing system 10 specifies the application ID of the linked application, the file and image data can be transmitted by specifying the linked application.

(11) The external service system 40 registers the image data and the file in a folder (or directory) specified by the administrator 8. The file contains the information necessary for settlement, so the person in charge can process the settlement.

As described above, the data output system of this embodiment allows the administrator 8 to register the mark (color) and the content of the processing. The device 20 reads the form with the marked areas, and the information processing system 10 acquires the image data, so that processing can be associated and stored for each area of the form. The information processing system 10 can then acquire only the areas from the image data of the form sent from the device 20 and perform processing corresponding to the areas. This makes it possible to associate image processing areas and processing with each of any number of apps, and the user can perform the desired processing (e.g., OCR processing, etc.) on the desired area of the image data generated by scanning, simply by selecting an app according to the type of form, for example.

<Additional information regarding color and processing>
The series of processes explained in Figures 1 and 2 will be supplemented by dividing them into tasks 1 to 3. Figure 3 is a diagram for explaining the registration of the correspondence between colors and processes (task 1) using a registration form.

Figure 3 (a) shows the registration form before coloring.

Figure 3(b) shows the registration form after coloring. The registration form in Figure 3(b) is scanned by device 20.

Figure 3 (c) shows a part of a configuration file in which colors and processes are associated. There are two possible ways to associate colors and processes.

1: The information processing system has in advance information on how to link the coordinates (within the registration form) from which the color is obtained with the processing obtained from those coordinates.
Example of pre-defined parameters: The color extracted at coordinates (X coordinate: 100, 200 Y coordinate: 1, 50) is OCR
・The color extracted at coordinates (X coordinate: 100, 200 Y coordinate: 51, 100) is masked (OMR in Table 4).
- The color extracted at coordinates (X coordinates: 100, 200 Y coordinates: 101, 150) is trimmed. 2: The administrator links the color and processing on the management site.
4 shows an example of an administration site 150 for associating colors with processes. The administration site 150 is provided to the terminal device by the information processing system 10 as a Web page or a Web application. The administrator can register the color of the process using a radio button 151. In this case, the number of marker pen colors that the user can use is limited to three.

In either method, as shown in Figure 1(c), a number identifying the marker color (green) is associated with OCR. A number identifying the marker color (blue) is associated with OMR. A number identifying the marker color (red) is associated with trimming.

Figure 5 is a diagram explaining the correspondence between area information and processing (task 2) using the original form paper.

Figure 5 (a) is the original form.

Figure 5(b) shows the form after coloring. The form in Figure 5(b) is scanned by device 20.

Figure 5 (c) shows a portion of a configuration file in which area information and processing are associated.

As shown in FIG. 5(d), since the setting file associates colors with processes, the information processing system 10 detects the color of the registered paper (already registered in the setting file) from the form, and associates the coordinates of the area where this color is detected as area information with the processing of the setting file.

Therefore, in this embodiment, the administrator can change the processing for an area on a form by changing the color with which the area is colored.
Ex1: Color in blue → OCR, Color in red → Masking
Ex2: Colored in blue → beginning of file name, colored in red → end of file name

Figure 6 is a diagram explaining processing of an area within a form using the form (task 3).

Figure 6 (a) shows the original paper on which the user has filled out the necessary information.

Figure 6 (b) shows the processing performed on each area of the form. In the form title area 140, OCR and file naming are registered as processes, so the title becomes the file name. In the form pre-application area 141, masking is registered as a process, so the pre-application area is masked. In the form notes area 142, cropping is registered as a process, so the notes area is obtained as image data.

<Terminology>
A mark is a shape used as a means for distinguishing something from others. In this embodiment, a color is an example of a mark, but letters, alphabets, symbols, etc. may also be used as marks.

The first device is a device or an administrator's terminal device. The second device is, for example, a device.

"Based on user operation" includes cases where the user sets the settings using a registration form and cases where the user sets the settings using the administration screen.

The setting information is, for example, information that associates colors with the contents of processing (see Table 4, for example).

The administrator 8 is a system administrator on the client side of the data output system, who configures the data output system for general users. The users are general users on the client side. Since general users may also perform the work of the administrator 8, there is no need to distinguish between the two.

A form is a general term for ledgers, slips, etc. It is also called office paper with blank spaces for the user to fill in. In this embodiment, forms are mainly described, but the name of the document is not important as long as it is a document that can be read by device 20. For example, a form may be a document, paper, report, newspaper, magazine, etc. Note that there are cases where a form is not printed out on paper media (there are cases where device 20 does not need to read a form).

The type of image data to be processed can be any type, including PDF, JPEG, PNG, TIFF, BMP, etc.

In this embodiment, a "settings file" is a file in which processing and area information is set, and a "file" is a file to which acquired data is output. A file is a collection of data that is the basic unit of data managed by a computer.

Outputting data means allowing the user to visually check the data. In this embodiment, the data is output to a file, for example.

A web app is software or a mechanism that runs on a browser and operates through the cooperation of a program written in a programming language (e.g. JavaScript (registered trademark)) that runs on a browser and a program on the web server side.

<System configuration example>
First, the system configuration of the data output system 100 according to this embodiment will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of the system configuration of the data output system 100 according to this embodiment.

The data output system 100 shown in FIG. 7 includes an information processing system 10 and a device 20, which are communicatively connected via a wide area network N1 such as the Internet. The data output system 100 communicates with an external service system 40, which is an external system. In addition, any terminal device 30 can communicate with the information processing system 10 via the networks N1 and N2.

The information processing system 10 is realized by one or more information processing devices, and provides various services realized by a series of processes in cooperation with an external service system 40, such as a cloud service, via a network N1. Specific examples of services provided by the information processing system 10 according to this embodiment will be described later. The information processing system 10 may be realized by cloud computing, or may be realized by a single information processing device. Cloud computing refers to a form of use in which resources on a network are used without being aware of specific hardware resources. The information processing system 10 may exist on the Internet or on-premise. The series of processes is provided by one application, and the series of processes is also called a "processing flow" or "workflow".

The device 20 is various electronic devices used by the user. The device 20 is, for example, an image processing device such as an MFP (Multifunction Peripheral), an image forming device, a PC (Personal Computer), a projector, an electronic whiteboard, a digital camera, etc. The device 20 is connected to the network N2. The user can use the device 20 to utilize various services provided by at least one of the information processing system 10 or the external service system 40.

In the following description, when the multiple devices 20 are to be distinguished from one another, they will be referred to using subscripts such as "device 20 ₁ " and "device 20 ₂ ".

The terminal device 30 is, for example, a desktop PC (personal computer), notebook PC, smartphone, tablet terminal, etc. used by the administrator 8 or user. Alternatively, the terminal device 30 may be any device capable of running a web browser. The terminal device 30 is connected to the network N2. The administrator 8 or user can operate the terminal device 30 to use various services provided by the information processing system 10 or the external service system 40, and can further configure linked apps or apps.

For ease of explanation, the device 20 and the terminal device 30 will be described separately, but the device 20 and the terminal device 30 may be collectively referred to as the device.

In addition, hereinafter, when the multiple terminal devices 30 are to be distinguished from one another, they will be referred to using subscripts such as "terminal device 30 ₁ " and "terminal device 30 ₂ ".

The external service system 40 is one or more information processing devices that provide services by executing linked applications via the network N1. Data is managed or processed by executing linked applications, and the data itself or the results of the processing become the services.

Examples of the external service system 40 include, for example, a cloud service, an ASP (Application Service Provider), etc., and may include various external services provided via a network. In this embodiment, a storage service will be described as an example of a service. The external service system 40 may exist on the Internet or on-premise.

In the following description, when distinguishing between the multiple external service systems 40, the illustrated "external service system 40 ₁ " will be referred to as the external service system 40A, the "external service system 40 ₂ " will be referred to as the external service system 40B, etc.

<Hardware configuration example>
The hardware configuration of the information processing system 10 and the terminal device 30 included in the data output system 100 according to this embodiment will be described with reference to FIG.

<<Information Processing System and Terminal Device>>
Fig. 8 is a diagram showing an example of a hardware configuration of the information processing system 10 and the terminal device 30 according to this embodiment. As shown in Fig. 8, the information processing system 10 and the terminal device 30 are constructed by a computer, and include a CPU 501, a ROM 502, a RAM 503, a HD 504, a HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I/F (Interface) 508, a network I/F 509, a bus line 510, a keyboard 511, a pointing device 512, a DVD-RW (Digital Versatile Disk Rewritable) drive 514, and a media I/F 516.

Of these, the CPU 501 controls the overall operation of the information processing system 10 and the terminal device 30. The ROM 502 stores programs used to drive the CPU 501, such as IPL. The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls the reading or writing of various data from the HD 504 according to the control of the CPU 501. The display 506 displays various information such as a cursor, menu, window, character, or image. The external device connection I/F 508 is an interface for connecting various external devices. In this case, the external device is, for example, a USB (Universal Serial Bus) memory or a printer. The network I/F 509 is an interface for data communication using the network N2. The bus line 510 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 501 shown in FIG. 8.

The keyboard 511 is a type of input means equipped with multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving the cursor, etc. The DVD-RW drive 514 controls the reading and writing of various data from the DVD-RW 513, which is an example of a removable recording medium. The DVD-RW may be a DVD-R or the like. The media I/F 516 controls the reading and writing (storing) of data from the recording medium 515, such as a flash memory.

<<Equipment>>
9 is a hardware configuration diagram of an image forming apparatus, which is an example of the device 20. As shown in FIG. 9, the image forming apparatus includes a controller 910, a short-range communication circuit 920, an engine control unit 930, an operation panel 940, and a network I/F 950.

Of these, the controller 910 has a CPU 901, which is the main part of the computer, a system memory (MEM-P) 902, a north bridge (NB) 903, a south bridge (SB) 904, an ASIC (Application Specific Integrated Circuit) 906, a local memory (MEM-C) 907, which is a storage unit, a HDD controller 908, and a HD 909, which is also a storage unit, and is configured such that the NB 903 and the ASIC 906 are connected by an AGP (Accelerated Graphics Port) bus 921.

Of these, the CPU 901 is a control unit that performs overall control of the image forming apparatus. The NB 903 is a bridge that connects the CPU 901 with the MEM-P 902, the SB 904, and the AGP bus 921, and includes a memory controller that controls reading and writing to the MEM-P 902, a PCI (Peripheral Component Interconnect) master, and an AGP target.

The MEM-P 902 is composed of a ROM 902a, which is a memory for storing programs and data that realize the various functions of the controller 910, and a RAM 902b, which is used for expanding the programs and data, and as a drawing memory during memory printing. The programs stored in the RAM 902b may be provided by recording them in an installable or executable format on a computer-readable recording medium such as a CD-ROM, CD-R, or DVD.

SB904 is a bridge for connecting NB903 with PCI devices and peripheral devices. ASIC906 is an IC (Integrated Circuit) for image processing applications that has hardware elements for image processing, and acts as a bridge connecting AGP bus921, PCI bus922, HDD controller908, and MEM-C907. This ASIC906 is composed of a PCI target and AGP master, an arbiter (ARB) that is the core of ASIC906, a memory controller that controls MEM-C907, multiple DMACs (Direct Memory Access Controllers) that rotate image data using hardware logic, and a PCI unit that transfers data between scanner unit931, printer unit932, and facsimile unit via PCI bus922. In addition, ASIC 906 may have a USB (Universal Serial Bus) interface or an IEEE 1394 (Institute of Electrical and Electronics Engineers 1394) interface and may be connected to these.

MEM-C907 is a local memory used as an image buffer for copying and a code buffer. HD909 is a storage for storing image data, font data used during printing, and forms. HD909 controls the reading and writing of data from and to HD909 under the control of CPU901. AGP bus921 is a bus interface for a graphics accelerator card proposed to speed up graphic processing, and by directly accessing MEM-P902 with high throughput, the graphics accelerator card can be made faster.

The short-range communication circuit 920 also includes a short-range communication circuit antenna 920a. The short-range communication circuit 920 is a communication circuit such as NFC or Bluetooth (registered trademark).

The engine control unit 930 further includes a scanner unit 931, a printer unit 932, and a facsimile unit 933. The operation panel 940 includes a panel display unit 940a, such as a touch panel, that displays current settings and selection screens and receives input from the operator, and hard keys 940b, such as a numeric keypad that receives settings for image formation conditions such as density settings and a start key that receives a copy start command. The controller 910 controls the entire image forming apparatus, and controls, for example, drawing, communication, and input from the operation panel 940. The scanner unit 931 or the printer unit 932 includes an image processing unit such as error diffusion and gamma conversion.

The image forming device can sequentially switch between the document box function, copy function, printer function, and facsimile function using the application switching key on the operation panel 940. When the document box function is selected, the image forming device goes into document box mode; when the copy function is selected, the image forming device goes into copy mode; when the printer function is selected, the image forming device goes into printer mode; and when the facsimile mode is selected, the image forming device goes into facsimile mode.

The network I/F 950 is an interface for data communication using the network N2. The short-range communication circuit 920 and the network I/F 950 are electrically connected to the ASIC 906 via the PCI bus 922.

<About the function>
Next, the functional configuration of the data output system 100 according to the present embodiment will be described with reference to Fig. 10. Fig. 10 is a diagram showing an example of the functional configuration of the device 20, the terminal device 30, and the information processing system 10 according to the present embodiment.

<<Equipment>>
First, the device 20 has a second communication unit 21, a display control unit 22, an operation acceptance unit 23, an image data generation unit 24, a facsimile processing unit 25, an email processing unit 26, and an output unit 27. Each of these functional units is a function or means realized by the CPU 901 executing instructions included in one or more programs installed in the device 20. For example, the second communication unit 21, the display control unit 22, and the operation acceptance unit 23 are realized by a web browser, and the others are realized by individual applications (native apps).

The second communication unit 21 transmits and receives various information between the information processing system 10. In this embodiment, the second communication unit 21 receives screen information of the application list screen from the information processing system 10, and transmits image data of the form, application IDs, etc. to the information processing system 10.

The display control unit 22 interprets the screen information of various screens and displays it on the panel display unit 940a. The operation reception unit 23 accepts various operations by the administrator 8 or the user on the various screens displayed on the panel display unit 940a.

When the application selected by the operation receiving unit 23 generates image data, the image data generating unit 24 scans an original document with the scanner unit 931 to generate image data. The image data generating unit 24 has a scanning function. The image data generating unit 24 converts the image data into a PDF file. The image data generating unit 24 may not convert the image data and may leave the image data in its original format (JPEG, TIFF, PNG, etc.). The facsimile processing unit 25 performs processing related to the reception and transmission of facsimiles by the facsimile unit 933, and when a facsimile is received, requests the information processing system 10 to execute an application that has been associated with it in advance. The image data generating unit 24 may request an application that corresponds to the sender of the facsimile (fax number).

The email processing unit 26 performs processing related to sending and receiving emails, and when an email is received, it requests the information processing system 10 to execute an application that has been associated with the email in advance. The email processing unit 26 may also request an application that corresponds to the sender (email address) of the email.

The output unit 27 performs image formation processing to output paper on which an image has been formed. The image formation method may be an electrophotographic method or an inkjet method.

<<Terminal Device>>
The terminal device 30 has a first communication unit 31, a display control unit 32, and an operation reception unit 33. Each of these functional units is a function or means realized by the CPU 501 executing instructions included in one or more programs installed in the terminal device 30. Note that this program may be a web browser or may be dedicated software for communicating with the information processing system 10.

The first communication unit 31 transmits and receives various types of information between the information processing system 10 or the external service system 40. In this embodiment, it receives various types of screen information and the like from the information processing system 10 or the external service system 40, and transmits information set by the administrator 8 or the user to the information processing system 10 or the external service system 40.

The display control unit 32 interprets the screen information of various screens and displays it on the display 506. The operation reception unit 33 receives various operations by the administrator 8 or the user on the various screens displayed on the display 506.

<<Information Processing System>>
The information processing system 10 includes an application data service unit 11, a setting reception unit 12, an authentication service unit 14, a Web service processing unit 15, an input/output service unit 16, a component 18, and a communication unit 19. Each of these functional units is a function or means realized by the CPU 501 executing instructions included in one or more programs installed in the information processing system 10. Note that, for convenience of drawing in FIG. 10, one information processing system 10 includes each functional unit, but each functional unit may be distributed and arranged in a plurality of information processing devices.

The communication unit 19 transmits and receives various information between the terminal device 30 and the device 20. The setting reception unit 12 provides a setting site to the terminal device 30 and receives settings for the setting site. The site may refer to a web page or a web application, or may refer to a server-side system that provides the terminal device 30 with a web page or a web application. The setting reception unit 12 has a terminal screen providing unit 42, which provides an initial screen, an application list screen, a correspondence screen, and the like that allows the administrator 8 to set information required for executing the application. The terminal screen providing unit 42 generates screen information written in HTML, XML, CSS (Cascade Style Sheet), JavaScript (registered trademark), and the like in response to an HTTP request from the terminal device 30, and transmits the screen information to the terminal device 30 as an HTTP response.

The application data service unit 11 has an application data management unit 41. The application data management unit 41 manages application data 43 of applications purchased by customers such as companies for each customer. The application data 43, for example, associates the application with the above-mentioned setting file. The application data 43 is stored in the application data storage unit 13. Note that an organization such as a customer as viewed from the data output system 100 is called a tenant.

The web service processing unit 15 provides screen information to the device 20 and accepts application execution from the device 20. The web service processing unit 15 has an application execution unit 46 and a device screen providing unit 47. The application execution unit 46 accepts application execution from the device 20 and requests the input/output processing unit 48 to execute the application. The device screen providing unit 47 uses screen information 54 to provide screen information of an application list screen or an application screen to the device 20.

The input/output service unit 16 inputs and outputs data using the authentication service unit 14, the application processing information storage unit 17, and the component 18. The input/output service unit 16 has an input/output processing unit 48 and an application management unit 49. The input/output processing unit 48 executes an application requested by the application execution unit 46. During execution, the input/output processing unit 48 refers to the application processing information 53 and requests each process included in the application from the component 18.

The application management unit 49 manages the application processing information 53 and screen information 54 of the application stored in the application processing information storage unit 17. The application processing information 53 sets which processing the application will perform and in what order (for example, requesting processing from the external service system 40 after OCR). The screen information 54 is information on the screens displayed by the device 20, such as the icons of each application, the application list screen, and each application screen. The application processing information storage unit 17 stores the application processing information 53 and screen information 54 for each application.

The authentication service unit 14 performs processes related to authentication of users (including the administrator 8) and devices 20. The authentication service unit 14 has a user management unit 44 and a device management unit 45. The user management unit 44 manages information related to users. The authentication service unit 14 holds authentication information such as user names and passwords. The authentication information includes information about the information processing system 10 and information about the external service system 40. The device management unit 45 manages devices 20 associated with tenants. In other words, the identification information of the tenants and the devices 20 are managed in association with each other, and when the device 20 is identified, the tenant is also identified.

Component 18 is a collective term for the individual functions that actually perform each process. One block within component 18 corresponds to one process. Component 18 has, for example, an authentication processing unit 61, an external service processing unit 62, a judgment unit 63, a PDF processing unit 64, an area detection unit 65, a data storage unit 66, an area identification unit 67, an OCR processing unit 68, an OCR correction unit 69, an OMR processing unit 71, a trimming processing unit 72, and a color detection unit 73.

The authentication processing unit 61 is a component 18 that requests authentication-related processing from the authentication service unit 14.

The external service processing unit 62 is a component 18 that issues a processing request to the external service system 40. In other words, the external service processing unit 62 is a function for linking with the external service system 40. An external service processing unit 62 is provided for each type of external service system 40.

The judgment unit 63 performs various judgments and controls the branching of the processing flow. The PDF processing unit 64 performs processing related to PDF files, such as extracting image data from PDF files and creating PDF files with transparent text. This is because the device 20 transmits image data in the form of a PDF file.

The area detection unit 65 detects marks associated with a process from the image data of the registration form. The area detection unit 65 also detects rectangular areas marked with a marker pen or the like from the image data of the form, and creates the coordinates of the rectangular areas (area information).

The area identification unit 67 identifies the area to be processed from the image data of the form based on the area information. The area identification unit 67 acquires the identified area to be processed (for example, it may cut it out). The OCR processing unit 68 performs OCR processing on only the area and on the entire image data. The OCR correction unit 69 corrects the results of the OCR processing. The OMR processing unit 71 performs OMR processing on the area. The trimming processing unit 72 performs trimming processing on the area. Note that the component 18 may have components such as translation in addition to those shown in the figure.

The data storage unit 66 associates colors with processes, and further associates processes with area information and registers them in a setting file. The data storage unit 66 also outputs a file including the acquired data (data acquired from areas of the form) generated by the OCR processing unit 68, the OMR processing unit 71, and the trimming processing unit 72 through processing.

The color detection unit 73 detects the color that the administrator 8 has used with a marker pen on the registration form.

<External service system>
The external service system holds a registration form 5 and a setting file 9. Of these, the registration form 5 is registered in advance. The setting file 9 is registered by an administrator 8.

表１は、アプリ処理情報記憶部１７が保持するアプリ処理情報の一例を示す。アプリ処理情報記憶部１７はテナントごとに購入されている（ライセンスがある）アプリのリストを記憶する。表１に示す情報はアプリ処理情報記憶部が記憶するリストに含まれるアプリうち、本実施形態で説明する受信した画像データの一部に画像処理を行うアプリの一覧で、アプリＩＤ、アプリ名、画面情報、アプリ処理情報、共有アプリ、及び、外部サービスシステム４０の各項目を有している。アプリＩＤはアプリを識別する識別情報である。アプリ名は管理者８が付与したアプリの名称である。また、アプリのアイコンや設定画面を機器２０が表示する際に使用される画面情報５４（ＨＴＭＬ等で記述された）がアプリに対応付けられる。アプリには一連の処理の内容と順番（処理フロー）が定義されるので、各アプリにはアプリ処理情報５３が対応付けられる。後述するように、複製されたアプリはフロー情報が同じになる。共有アプリとは、一般のアプリがユーザーごとに利用権限が与えられるのに対し、共有アプリはテナントのユーザーが共通に使えるアプリであることを示す。表１では経費精算書アプリが共有アプリである。各ユーザーが共通に利用するためである。外部サービスシステム４０の項目は各アプリが連携する外部サービスシステム４０である。外部サービスシステム４０と連携しないアプリも存在する。

Table 1 shows an example of application processing information stored in the application processing information storage unit 17. The application processing information storage unit 17 stores a list of applications purchased (licensed) for each tenant. The information shown in Table 1 is a list of applications that perform image processing on a part of the received image data described in this embodiment among the applications included in the list stored in the application processing information storage unit, and has the following items: application ID, application name, screen information, application processing information, shared application, and external service system 40. The application ID is identification information that identifies the application. The application name is the name of the application given by the administrator 8. In addition, the application is associated with screen information 54 (described in HTML or the like) that is used when the device 20 displays the icon or setting screen of the application. Since the content and order of a series of processes (processing flow) are defined for the application, each application is associated with application processing information 53. As will be described later, the flow information is the same for the duplicated applications. The shared application indicates that, while a general application is given a usage right for each user, the shared application is an application that can be commonly used by users of the tenant. In Table 1, the expense report application is the shared application. This is because the applications are shared by all users. The item "external service system 40" is the external service system 40 with which each application is linked. There are also applications that do not link with the external service system 40.

When the administrator 8 or user logs in, the application management unit 49 obtains a list of applications for which the administrator 8 or user has been granted permission to use from the application processing information storage unit 17 and provides the list to the device 20. When an application is specified from the list of applications, the application management unit 49 identifies the screen information and application processing information associated with the application.

表２はユーザー管理部４４が管理するユーザーに関する情報の一例を示す。表２（ａ）はユーザーに関する情報として、情報処理システム１０におけるユーザーＩＤ、ユーザー名、メールアドレス、パスワード、利用できるアプリ、及び、外部サービスシステム情報の各項目を有している。このユーザーには管理者８も含まれる。ユーザーＩＤ、ユーザー名又はメールアドレスの少なくとも１つとパスワードは、ユーザーが情報処理システム１０にログインするための認証情報である。利用できるアプリはこのユーザーに利用権限が与えられているアプリである。利用権限は、利用人数などを指定してライセンスを購入したアプリを管理者８が各ユーザーに割り当てることで与えられる。利用できるアプリには個人アプリと共有アプリがある。個人アプリも共有アプリも複数のユーザーに使用されるが、個人アプリは個人のユーザーの外部サービスの認証情報を使用し、共有アプリは複数のユーザーが外部サービスに対し管理者８の認証情報を使用する。表２（ａ）では、U001～U003に共通のapp003が共有アプリである。その他は個人アプリである。個人アプリの場合、ユーザーごとに外部サービスシステムの認証情報の登録が必要である。外部サービスシステム情報はユーザーが連携設定済みの外部サービスシステム４０である。

Table 2 shows an example of information about a user managed by the user management unit 44. Table 2(a) has items of user ID, user name, email address, password, available apps, and external service system information in the information processing system 10 as information about the user. The administrator 8 is also included in this user. At least one of the user ID, user name, or email address, and the password are authentication information for the user to log in to the information processing system 10. Available apps are apps that the user has been given permission to use. The permission to use is given by the administrator 8 assigning apps for which licenses have been purchased by specifying the number of users, etc., to each user. Available apps include personal apps and shared apps. Both personal apps and shared apps are used by multiple users, but the personal app uses authentication information for an external service of an individual user, and the shared app uses authentication information of the administrator 8 for an external service by multiple users. In Table 2(a), app003, which is common to U001 to U003, is a shared app. The others are personal apps. In the case of a personal app, it is necessary to register authentication information for the external service system for each user. The external service system information is an external service system 40 for which the user has already set up a link.

Note that "linkage setup completed" means that the user wishes to link with the external service system 40 and has confirmed that they can log in using the authentication information for the external service system 40. The user inputs the authentication information for the external service system 40 from the setup site, and if they are able to log in, the identification information for the external service system is set in the external service system information in Table 2(a). Additionally, the authentication information for the external service system 40 is registered in Table 2(b).

Table 2(b) is information about the external service system for each user, and has the following fields: user ID, external service system 40, user ID, and password. Table 2(b) registers authentication information for the external service system 40 that the user has set up to link. The authentication information for the external service system 40 is authentication information for the user to log in to the external service system 40. As with user U002, authentication information may differ for each external service system 40. For this reason, authentication information for the external service system 40 is associated with each user of the data output system 100. The information processing system 10 identifies the authentication information for the linked external service system 40 according to authentication information such as the user ID and password for the information processing system 10 acquired from the device 20.

The authentication information may be authentication information of the external service system 40 or authentication information required for linking with the external service system 40, and may be a token obtained from the external service system 40 when linking with the external service system 40, in addition to a user ID and password.

Table 2 (c) shows authentication information for a shared app. The identification information of the shared app is associated with the external service system 40 and authentication information. Since the authentication information for the shared app is that of the administrator 8, the identification information of the administrator 8 is registered.

表３は、アプリデータ記憶部１３に記憶されているアプリデータ４３の一例を示す。アプリデータ４３は、アプリと設定ファイル（領域情報）を対応づける。対応付けは管理者８が行う。
・applicationIdは、連携元のアプリＩＤである。
・applicationNameは、アプリのアプリ名である。
・fileIdは、情報処理システム１０が作成した設定ファイルの識別情報である。
・fileNameは、情報処理システム１０が作成した設定ファイルのファイル名である。設定ファイルは外部サービスシステム４０に保存されるが、そのフォルダと共にファイル名が設定される。保存先は例えばＵＲＬで特定されてよい。
・アップロード先フォルダは、設定ファイルが保存される外部サービスシステム４０とそのフォルダである。アップロード先フォルダは、例えばＵＲＬでよい。管理者８がアップロード先フォルダを設定する。

Table 3 shows an example of the application data 43 stored in the application data storage unit 13. The application data 43 associates an application with a setting file (area information). The administrator 8 performs the association.
・applicationId is the application ID of the link source.
・applicationName is the app name of the app.
fileId is identification information of the setting file created by the information processing system 10.
fileName is the file name of the setting file created by the information processing system 10. The setting file is saved in the external service system 40, and the file name is set together with the folder. The save destination may be specified by, for example, a URL.
The upload destination folder is the external service system 40 in which the setting file is saved and the folder therefor. The upload destination folder may be, for example, a URL. The administrator 8 sets the upload destination folder.

表４は、外部サービスシステム４０に記憶されている設定ファイル９の一例を示す。設定ファイル９は、ＯＣＲ処理の対象となる領域情報を有するファイルである。設定ファイルは表形式（例えばＣＳＶ形式）になっており、各セルの位置が列番号と行番号で指定される。以下、Ａ列～Ｍ列の項目について説明する。なお、設定ファイルの１行目にある項目名は、設定ファイルの作成時に情報処理システム１０により自動的に設定される。

Table 4 shows an example of the setting file 9 stored in the external service system 40. The setting file 9 is a file that has area information that is the target of OCR processing. The setting file is in a table format (e.g., CSV format), and the position of each cell is specified by a column number and a row number. The items in columns A to M are explained below. Note that the item name in the first row of the setting file is automatically set by the information processing system 10 when the setting file is created.

- The ID is identification information for the area information. It is assigned automatically and the administrator 8 or the user does not need to be aware of it.

The marker color is the color of the cell (shown in Figure 13) read from the registration form (an example of identification information that identifies the contents of the process). The cell color is registered, for example, in the HSV color space. "H" (hue) in the HSV color space is called hue and indicates "color tone." The range of "H" is 0-360 degrees. "S" (saturation) is saturation and indicates "vividness." The range of "S" is 0-100%. "V" (Value) is luminosity and indicates "brightness." The range of "V" is 0-100%. The color space representing the marker color may be HSB, RGB, Lab, XYZ, etc.

The process is the content of the process that was associated with the cell on the registration form. Examples of processes are OCR, OMR, and trimming, but various other processes such as translation, masking (filling in), and encryption may also be included.

The start point X coordinate, start point Y coordinate, end point X coordinate, and end point Y coordinate are area information. These represent the diagonal vertices of the rectangular area of the marked document. The coordinates of one point, the width, and the height may be specified. The coordinate units may be, for example, cm or pixels.

The above information is automatically set by the information processing system 10. The following information is entered by the administrator 8.

- The format is the data format of the area. Examples include text, Y/N, image, date (YYYY/MM/DD), and numeric values (NUMERIC). By knowing the data format, the OCR processor 68 can perform OCR processing with higher accuracy. The format is a pull-down menu. In this way, the type of data is set for each area.

- The CSV column name is the name given to the area. In the case of a report, a rectangular frame title or the like is entered, but the administrator 8 can enter any value. The CSV column name (if a CSV position is specified) is output to the file along with the data.

- CSV position (column) specifies which column of the file the information processing system 10 will write the data extracted from the area to.

- The CSV position (start row) specifies which row of the file the information processing system 10 will write the data extracted from the area to. The row number increases each time a form is read, so the "start row" is set when no data is entered in the file. For example, data obtained from the first form is set to the second row, while data obtained from the second form is set to the third row. In contrast, the column position can be fixed.

- The OCR language is set to the language in which the area is written. Since the language is known, the OCR processing unit 68 can perform OCR processing with higher accuracy. The OCR language is set in a pull-down menu. In this way, the type of data is set for each area.

- In this embodiment, the order used in the file name is used in the file name of the file in which the data obtained from the area is output. The order used in the file name is the order in which the data is used in the file name of the file output by the information processing system 10. For example, if "Document name" is number 1, the file name will be "Expense report.csv" based on the data (expense report) obtained from the document name area.

<Application processing information>
Next, details of the application processing information will be described with reference to Fig. 11. Fig. 11 is a diagram showing an example of the application processing information 53 stored in the application processing information storage unit 17.

The application processing information 53 shown in FIG. 11 is information regarding a series of processes for realizing a service that requests "external service system B" to process image data generated by scanning with device 20.

The application processing information 53 includes a flow ID 1201A indicating the identification information of the application processing information 53, a flow name 1202A indicating the name of the application processing information 53, and flow details 1203A indicating the processing contents of a series of processes. The flow name 1202A also serves as the name of the application.

Furthermore, flow details 1203A includes processing content 1223A indicating that a processing request is to be made to "external service system B."

In the processing content 1223A, the component ID "ExService_B" of the component 18 that executes processing in cooperation with "external service system B", and the parameters "AppId" and "Folder" are specified. "ExService_B" corresponds to the external service processing unit 62 for external service system B. "AppId" stores the linked application ID (the correspondence between the application and the linked application is predetermined), and "Folder" stores the folder where the file will be saved (upload destination folder) (the upload destination folder set by the administrator 8 is set). This enables the information processing system 10 to save the scanned image data in "external service system B".

Note that for apps that do not link with a linked app, "AppId" is omitted. The parameters shown in the figure are only a partial list, and the app processing information may include many more parameters.

In FIG. 11, for simplicity, only one component is specified, but the application processing information 53 registers a series of processing contents 1223A performed by the application for each process (component).

Figure 12 is a diagram showing a schematic of a series of processes performed by an app. Figure 12(a) shows a series of processes of an app that registers the correspondence between marker colors and processes in a settings file (an app corresponding to color registration button 112, described later). One block corresponds to processing content 1223A. Details of each process will be explained in Figure 15.

Figure 12(b) shows a series of processes in an application that registers area information in the marker color and processing of the setting file (the application that corresponds to the form registration button 113 described later). Details of each process are explained in Figure 25.

Figure 12(c) shows a series of processes in an application that outputs data from a form area (e.g., an expense report application). Details of each process are explained in Figure 37.

<Registering the correspondence between colors and processing in the configuration file>
Next, the setting of the correspondence between colors and processes by the administrator 8 and the creation of a setting file will be described with reference to FIGS.

Figure 13 is a diagram explaining how to set the correspondence between colors and processes using a registration form. Figure 13 (a) shows the registration form before the administrator 8 colors it with a marker pen. The original paper of the registration form is registered in the external service system 40.

The first line of the registration form lists the items for each column (first column: marker color, second column: processing). In this embodiment, there are three types of "processing" (OCR, OMR, trimming), and the registration form is preset with the following processing in the order of second line: OCR, third line: OMR, fourth line: trimming. The marker color cell 130 of the original registration form is blank.

Figure 13(b) shows the administrator 8 coloring cells 130 with a marker pen for each process. The administrator 8 uses, for example, green, blue, and red marker pens from among the marker pens he owns. The administrator 8 assigns green to OCR processing, blue to OMR processing, and red to trimming processing. A predetermined gap is provided between the lines on the original paper of the registration form to prevent the colors from bleeding and mixing.

The administrator 8 colors the cells 130 on the registration form with a marker pen that he or she owns, according to the assignment of the desired marker color and each process that he or she has determined. The registration form is scanned by the device 20, and the correspondence between the marker color and the process is registered in the information processing system 10.

In addition, in FIG. 13, the cell 130 is filled in, but it is sufficient if some mark is set in the cell 130. For example, if symbols such as triangles or circles, alphabets, or enclosing lines are formed in the cell 130, the information processing system 10 registers these marks and processing in the setting file. In this case, the information processing system 10 may or may not take into account the color of the mark.

In addition, instead of filling in the cell 130 with a marker pen, the administrator 8 may specify the marker color using letters or symbols such as "green or G," "blue or B," or "red or R." In this case, the information processing system uses character recognition to associate the marker color with the content of the process.

In addition, the terminal device 30 may be connected to the information processing system 10, and the administrator 8 may input a color using text or the like into the web page displayed by the terminal device 30.

Furthermore, the administrator 8 may be able to register the registration form itself. The terminal device 30 is connected to the information processing system 10, and the administrator 8 registers the details of the process on the web page displayed by the terminal device 30, and creates a registration form as shown in FIG. 13(a).

<Device screen example>
14 is an example of an administrator screen 110 that is displayed when the administrator 8 logs in to the device 20. The administrator screen 110 has an administrator menu 118 and a general user menu 119. The administrator menu 118 is a menu that allows the administrator 8 to register a setting file. The general menu is a list of apps that the administrator 8 can use as a user.

The registration paper button 111 on the administrator menu 118 is a button that allows the administrator 8 to print a registration paper. The color registration button 112 is a button that allows the administrator 8 to have the device 20 scan a registration paper and register the correspondence between the marker color and the process in the settings file. The form registration button 113 is a button that allows the administrator 8 to have the device 20 scan a marked form and register the correspondence between the process and the area information in the settings file.

Note that the registration paper button 111, color registration button 112, and form registration button 113 are each a web application, and the administrator 8 can register the setting file by working according to the web application.

When the registration form button 111 on the administrator menu 118 is pressed, the device 20 displays a registration form. The administrator 8 prints out the registration form and colors it with a marker pen. Next, the administrator 8 presses the color registration button 112 on the administrator menu 118 and scans the registration form colored with the marker pen, and the image data of the registration form is sent to the information processing system 10. The information processing system 10 registers the correspondence between the marker color and the process in a settings file.

Next, the administrator 8 presses the form registration button 113 on the administrator menu 118, and scans the original form colored with a marker pen with the device 20. The image data of the form is sent to the information processing system 10, which registers the correspondence between the process and the area information in the settings file.

When the report registration button 113 is pressed, the device 20 may display the correspondence between the marker color and the process received from the information processing system 10. For example, the device 20 may display information such as that shown in FIG. 13(b) on the operation panel. If the administrator 8 wishes to change the correspondence between the color and the process, the administrator 8 presses the up and down buttons 114 on the operation panel. The device 20 shifts the process column by one row each time the up and down button 114 is pressed. In this way, the administrator 8 can change the correspondence between the color and the process.

When the administrator 8 as a user presses the app in the general user menu 119, the app is launched. When the user scans a completed form with the device 20, the information processing system executes the process associated with the area specified by the area information in the setting file for each area.

Figure 15 is an example of a sequence diagram showing the procedure for an administrator 8 to use a registration form to register the correspondence between colors and processes in a configuration file.

S1: The administrator 8 operates the device 20 to log in to the information processing system 10 and presses the registration form button 111 from the administrator menu 118.

S2: The second communication unit 21 of the device 20 sends an execution request for the app corresponding to the registration paper button 111 to the information processing system 10.

S3: The communication unit 19 of the information processing system 10 receives a request to execute an app. The app execution unit 46 specifies the app ID of the app corresponding to the registration paper button 111 and requests the input/output processing unit 48 to execute the app. The input/output processing unit 48 executes the app by referring to the app processing information 53 associated with the app ID of the app corresponding to the registration paper button 111. First, the input/output processing unit 48 requests a list of registration papers from the external service processing unit 62.

S4: The external service processing unit 62 communicates with the external service system 40 using the administrator 8's authentication information and requests a list of registration forms that the administrator 8 can use.

S5: The second communication unit 21 of the device 20 receives the list of registered papers, and the display control unit 22 displays the list of registered papers. The administrator 8 selects the registered paper for which the process to be executed is registered. The operation reception unit 23 accepts the selection of the registered paper. If there is only one type of registered paper, the user need only instruct printing rather than selection.

S6: The output unit 27 of the device 20 prints the registration form selected by the administrator 8. Note that in the sequence diagram, the process in which the device 20 obtains the registration form from the external service system 40 has been omitted. This registration form may include identification information for the registration form or position information for the cell 130 to be colored, formed by a barcode or the like. Details will be described in FIG. 16.

S7: The administrator 8 uses a marker pen to color the cells 130 associated with the processing of the printed registration form. The administrator 8 then presses the color registration button 112 from the administrator menu 118.

S8: The operation acceptance unit 23 of the device 20 accepts execution of the color registration button 112, and the image data generation unit 24 scans the registration form and generates a PDF file.

S9: The second communication unit 21 of the device 20 sends the PDF file and a request to execute the app corresponding to the color registration button 112 to the information processing system 10.

S10: The communication unit 19 of the information processing system 10 receives a PDF file. The application execution unit 46 specifies the application ID of the application corresponding to the color registration button 112 and requests the input/output processing unit 48 to execute the application. The input/output processing unit 48 executes the application by referring to the application processing information 53 associated with the application ID of the application corresponding to the color registration button 112. First, the input/output processing unit 48 specifies the PDF file and requests the PDF processing unit 64 to extract image data. The PDF processing unit 64 extracts image data of the registration form from the PDF file.

S11: Next, the input/output processing unit 48 sends the image data to the color detection unit 73 and requests detection of the correspondence between color and processing.

S12: The color detection unit 73 detects color information from a predetermined cell 130 of the image data.

Figure 16 shows a schematic diagram of the detection of color information. A barcode 101 indicating orientation is formed in a corner, such as the upper left corner, on the registration sheet, and the color detection unit 73 rotates the image data so that the barcode 101 is at the upper left. The color detection unit 73 then detects color information from a cell 130 (position is fixed) that is predetermined for each process.

As shown in FIG. 16(a), OCR processing is determined by the colors within a rectangle at two diagonal position coordinates AB (an example of position information), OMR processing is determined by the colors within a rectangle at position coordinate CD, and trimming processing is determined by the colors within a rectangle at position coordinate EF. The correspondence between such position coordinates AB, CD, and EF and processing can be stored in the barcode 101 or held by the application data management unit 41 (in this case, it is preferable that the barcode 101 also contains at least identification information of the registered paper, but if there is only one type of registered paper, identification information is not necessary).

As shown in FIG. 16B, the barcode 101 or the application data management unit 41 stores the process and the position coordinates of the cell 130 to be colored in association with each other. The color detection unit 73 obtains the HSV value of H from each cell 130 based on the position coordinates in FIG. 16B. If a color corresponding to red (H=(0-60°, 301-360°)) is detected from the cell 130 associated with trimming, the color detection unit 73 associates H=(0-60°, 301-360°) in HSV with trimming and registers it in the setting file. If a color corresponding to green (H=(61-180°)) is detected from the cell 130 associated with OCR, the color detection unit 73 associates H=(61-180°) with OCR and registers it in the setting file. When a color corresponding to blue (H = (181 to 300°)) is detected from cell 130 associated with OMR, color detection unit 73 associates H = (181 to 300°) with OMR and registers it in the setting file. Rather than registering the entire range of H values for red, blue, and green in this way, color detection unit 73 may register in the setting file a color (H value) with a margin of ±10% or the like set around the detected H value.

Associating colors with processes also involves an administrator using the management site 150 described in FIG. 4. The setting sequence using the management site 150 is as follows. The communication unit 19 transmits screen information of the management site 150 to the terminal device 30, and receives the correspondence between marks and processes accepted by the terminal device 30. The data storage unit 66 associates the processes with the marks and stores them in a setting file.

S13: Returning to FIG. 15, the explanation will be given. The application execution unit 46 obtains the correspondence between colors and processes from the color detection unit 73. The application execution unit 46 sends the correspondence between colors and processes to the data storage unit 66 and requests the generation of a setting file. The data storage unit 66 assigns an ID and creates a setting file in which the marker color and the process are associated. The data storage unit 66 sends the setting file to the application execution unit 46. At this point, information is entered in columns A to C of Table 4. Note that the correspondence between marker color and process may be saved in a separate independent file rather than in the setting file.

S14, S15: The application execution unit 46 sends the setting file to the external service system 40 via the external service processing unit 62. The external service processing unit 62 acquires the authentication information of the administrator 8 in the external service system 40 from the user management unit 44. As a result, the external service processing unit 62 logs into the external service system 40 and saves the setting file in the top folder of the administrator 8 (or a folder designated by the administrator 8). If the administrator 8 specifies a file name for the setting file, the external service processing unit 62 assigns this file name to the setting file. If a file name is not set, the current date and time, etc., becomes the file name.

With this, the configuration file in which the marker color and the process are associated has been registered in the external service system 40.

<<Specifying file name by process>>
In this embodiment, the file name of the CSV file in which the acquired data is stored is set by the administrator 8. The administrator 8 can set the file name of the CSV file by coloring the registration form.

Figure 17 shows an example of the use of a registration form where processing is applied to setting file names. In Figure 17, processing is limited to OCR, making it possible for the data storage unit 66 to use acquired data from multiple locations for the file name. The administrator 8 colors the registration form with the marker pen color specified for the file name. The registration form in Figure 17(a) is set with OCR as processing, and the order in which the file names will be used, "File name 1" and "File name 2", are set.

As shown in FIG. 17(b), the administrator 8 uses a marker pen to specify the marker pen color (e.g., green) to be used for file name 1 and the marker pen color (e.g., blue) to be used for file name 2.

As shown in Figure 17 (c), a numerical value indicating a color and a process (used in the file name) are associated with a configuration file. There are two possible ways to associate a color with a process.

1: The information processing system has in advance information on how to link the coordinates (within the registration form) from which the color is obtained with the processing obtained from those coordinates.
・The color extracted at coordinates (X coordinates: 100, 200 Y coordinates: 1, 50) is filename 1
・The color extracted at coordinates (X coordinates: 100, 200 Y coordinates: 51, 100) is filename 2

2: The administrator links colors and processes on the management site.
18 shows an example of an administration site 152 for associating colors with processes. The administrator can register the colors used to specify file names using radio buttons 153. In this case, the number of marker pen colors that a user can use is limited to three.

In either method, as shown in FIG. 17(c), the numeric value identifying the marker color (green) is associated with file name 1. The numeric value identifying the marker color (blue) is associated with file name 2. Note that file names 1 and 2 are used in the order in which they are used in file names.

Figure 19 shows an example of a form on which the administrator 8 has designated two areas with a marker pen. Figure 19(a) shows the original form, and Figure 19(b) shows the form colored with a marker pen. When the device 20 scans the form in Figure 19(b), the information processing system 10 identifies the colored areas on the form and registers the area information in the settings file.

In Figure 19 (b), the area for the report title, "Expense Report," is marked with a green marker pen, and the area for the application date is marked with a blue marker pen. For example, if the application date is "2020/10/10," the file name of the CSV file when the device scans the report in Figure 19 (b) will be "Expense Report 20201010.csv." Note that three or more marker pen colors may be specified in the file name.

Also, it is not necessary to register the order of the areas used in the file names (file 1, 2). In this case, the data storage unit 66 can set the file names in the order in which the colors are registered in the setting file. Therefore, it is not necessary to change the color of the marker pen for each cell on the registration form.

<<Modifications of the registration form>>
The CSV location may be registered in advance in the registration form as shown in Fig. 20. Fig. 20 shows a modified example of the registration form. Fig. 20(a) shows the registration form before coloring. This registration form has columns for CSV column name 154 and CSV location 155, so the administrator does not need to manually set the CSV location in the setting file.

Figure 20(b) shows the registration form after coloring. The registration form in Figure 20(b) is scanned by device 20.

Figure 20 (c) shows a portion of a settings file in which color, processing, and area information are associated.

<Registration of area information>
Next, the registration of area information will be described with reference to FIG. 21 and the like. FIG. 21 is a diagram for explaining a method of marking an area on a form. First, FIG. 21(a) shows a form (original paper) before an area is designated with a marker pen. On the original paper of the form, the manager 8 designates an area with a marker pen of the color registered in the area where the process registered on the registration form is to be executed. For example, the manager 8 wants to set the title area 140 to OCR, the advance application area 141 to OMR, and the remarks area 142 to trimming.

Figure 21 (b) shows a form with areas colored with a marker pen. Area 140 of the form's title, "Expense Report," is colored with a green marker pen. Area 141 of "Advance Application" is colored with a blue marker pen. Area 142 of "Notes" at the bottom of the form is colored with a red marker pen. When device 20 scans this form, the association between the areas and the processes for the form is registered in the settings file.

According to the configuration file in Table 4, the OCR processing unit 68 performs OCR processing on the area 140 of the document title, "Expense report," to generate acquired data. The OMR processing unit 71 also performs OMR processing on the area 141 of "Advance application," to generate acquired data. The trimming processing unit 72 performs trimming on the remarks area 142, to generate acquired data (for example, "Sorry for the delay in submitting the application."). The trimmed acquired data is image data (JPEG, GIF, PNG, TIFF, BMP, etc.).

Figure 21 (c) is an example of a form on which the user has filled in the necessary information. From the colored state in Figure 21 (b), the acquired data are "Expense report," "Please apply in advance," and "Sorry for the late application."

Also, the area does not have to be filled in as shown in FIG. 22(a). In FIG. 22(a), a frame line 602 surrounding the area is formed with a marker pen. The area detection unit 65 detects pixels other than black and white, and detects a circumscribing rectangle for the pixels other than black and white (frame line 602).

The method of marking an area is not limited to the method shown in FIG. 21 or FIG. 22. For example, an area may be designated by a double line, or an area may be designated by a range surrounded by a predetermined symbol (%, $, #, △, etc.).

Also, as shown in FIG. 22(b), a single form may have multiple areas colored with the same color. For example, as shown in FIG. 22(b), when the administrator 8 designates an area 143 of data on the form with a green marker pen, all areas 143 are subject to OCR processing.

Next, it is assumed that the administrator 8 presses the form registration button 113 in FIG. 14. The administrator 8 logs into the information processing system 10 from the device 20 and selects the form registration button 113.

The effect of an administrator specifying an area by filling it in will be described with reference to Figure 23. Figure 23 shows a comparative example of a method for specifying an area on a form.

Figure 23(a) shows a method of designating an area in which the administrator colors the area with a marker pen and then handwrites a △ mark 161 in the same area. However, this method requires double work.

Figure 23 (b) shows a method of designation in which the administrator handwrites a triangle mark 162 with a marker pen to surround the area. However, with this method, the area is not completely defined, and the information processing system 10 must recognize the triangle mark 162 and then recognize the rectangle of the area, making the area recognition process complicated.

In contrast, by coloring (filling in) an area with a marking pen, the administrator can easily "designate the area" and "designate the process" at the same time.
・Area specification → Shape... Mainly rectangle ・Processing specification → Other than shape... Color

<Area registration screen>
24 is an example of an area registration screen 610 displayed by the device 20 when registering area information. The area registration screen 610 has a message 611 saying "Please set the form with the area you want to process colored and press 'Start'", a read setting field 612, and a start button 613. The user sets the original (form) on the contact glass and presses the start button 613. The user may set the read conditions (color/black and white, single-sided/double-sided, orientation, resolution, etc.) in the read setting field 612 as necessary. Note that it is preferable that the color/black and white setting is fixed to color so that the user does not mistakenly set it to black and white.

It is preferable that the administrator 8 can specify the file name of the configuration file from the area registration screen 610 in FIG. 24. This is to make it easier to search for the configuration file from the external service system 40. It is also preferable that the administrator 8 can set the folder in which the configuration file is to be saved.

Figure 25 is an example of a sequence diagram showing the procedure in which the information processing system 10 obtains area information from a form on which an administrator 8 has marked areas, and creates a configuration file.

S101: The administrator 8 operates the device 20 to log in to the information processing system 10 and presses the form registration button 113 from the administrator menu 118. This causes the display control unit 22 of the device 20 to display the area registration screen 610. The administrator 8 places the form with the area marked on the contact glass and presses the start button 613. The image data generation unit 24 generates image data of the form and converts it into a PDF file.

S102: The second communication unit 21 of the device 20 sends the PDF file to the information processing system 10 along with a request to execute the app corresponding to the report registration button 113.

S103: The communication unit 19 of the information processing system 10 receives the PDF file. The application execution unit 46 specifies the application ID of the application corresponding to the form registration button 113 and requests the input/output processing unit 48 to execute the application. The input/output processing unit 48 executes the application by referring to the application processing information 53 associated with the application ID of the application corresponding to the form registration button 113. First, the input/output processing unit 48 specifies the PDF file and requests the PDF processing unit 64 to extract image data. The PDF processing unit 64 extracts the image data of the form from the PDF file.

S104: The application execution unit 46 requests a configuration file from the external service processing unit 62 to determine the colors to be extracted from the image data.

S105: The external service processing unit 62 obtains from the external service system 40 a configuration file that can be obtained using the authentication information of the administrator 8. Note that the administrator 8 may select a desired configuration file from a list of configuration files.

S106: Next, the input/output processing unit 48 requests the area detection unit 65 to acquire an area by specifying the image data and the marker color registered in the setting file. The area detection unit 65 acquires an area from the image data based on the color of the image data.

Figure 26 is a diagram explaining the detection of an area from image data. As shown in Figure 26 (a), a color-applied portion 601 specified with a marker pen has a color corresponding to the marker pen. For example, in the HSV color space, the range of red is H = (0 to 60°, 301 to 360°), the range of green is H = (61 to 180°), and the range of blue is H = (181 to 300°).

The area detection unit 65 searches the image data for colors that are included in the range of the marker color H in the setting file, for example. Note that the area detection unit 65 only detects areas where the colored area is continuous and has a certain area or more. This prevents the detection of parts that have been colored by the user by mistake.

Figure 26(b) shows the detected color attachment portion 601. The area detection unit 65 sets a circumscribing rectangle 620 for the detected color attachment portion 601. The area detection unit 65 then sets margins on the top, bottom, left, and right of the circumscribing rectangle 620, and determines an area 621 including the margins. In Figure 26(b), a [mm] is set on the left and right, and b [mm] on the top and bottom. The margins are set to prevent positional deviations and the like from occurring when the device 20 reads the document. The coordinates of the upper left corner 622 in Figure 26(b) are the start X and Y coordinates of the area information, and the coordinates of the lower right corner 623 are the end X and Y coordinates of the area information (area information).

Also, as shown in FIG. 26(c), the area detection unit 65 may detect the outer frame 625 of the color attachment portion 601 detected based on the H of HSV. The area detection unit 65 searches for black pixels vertically outward from the color attachment portion 601 (circumscribed rectangle 620), and if a black pixel is found, traces the black pixel, and when it returns to the original position, determines that the outer frame 625 has been detected. In this way, the maximum area in which data is written can be detected, making it easier for the area identification unit 67 to identify the data even if the device 20 is misaligned when reading the document.

The administrator 8 can set how much margin the area detection unit 65 should provide for the area specified by the marker pen. Details will be explained in the association screen 640-2 of FIG. 31. The area detection unit 65 sets area information based on the settings in the association screen 640-2.

S107: Let's go back to Figure 25. Next, the input/output processing unit 48 specifies the image data in which the area is designated with the marker pen, and requests the external service processing unit 62 to upload the image data to the external service system 40.

S108: The external service processing unit 62 obtains authentication information for the administrator 8 in the external service system 40 from the user management unit 44. As a result, the external service processing unit 62 logs into the external service system 40 and saves the image data in the administrator 8's top folder (or a folder specified by the administrator 8). If the administrator 8 specifies a file name for the configuration file on the area registration screen 610, the external service processing unit 62 assigns the same file name to the image data file. If a file name is not set, the current date and time, etc., becomes the file name.

S109: Next, the input/output processing unit 48 specifies the area information detected by the area detection unit 65, and requests the data storage unit 66 to create a setting file. The data storage unit 66 places the area information in a specific column of the CSV file and generates the setting file. At this point, information is entered in columns A to G of Table 4.

S110: Next, the input/output processing unit 48 specifies the configuration file and requests the external service processing unit 62 to upload it to the external service system 40.

S111: The external service processing unit 62 has already logged in to the external service system 40, so it saves (overwrites) the setting file in the top folder of the administrator 8 (or a folder designated by the administrator 8). Since the file names of the image data and the setting file are the same, the administrator 8 can easily determine which image data the setting file was generated from.

<Registering other information in the configuration file>
Setting of columns H to M in the setting file will be described with reference to Figures 27 and 28. Figure 27 is an example of a sequence diagram showing the procedure in which the administrator 8 sets information in columns H to M of the setting file.

S31: After registering the setting file in the external service system 40, the administrator 8 operates the terminal device 30 to connect the terminal device 30 to the external service system 40. The administrator 8 inputs his/her own authentication information and logs in to the external service system 40.

S32: Since the external service system 40 is a storage service, it sends a list of files registered by the administrator 8 to the terminal device 30. For example, a list of files in the top folder is sent. If the files are not in the top folder, the administrator 8 specifies a hierarchy. The list of files also includes setting files and image data files registered by the administrator 8 from the device 20.

S33: The administrator 8 selects a setting file to check the contents of the setting file and set information in columns H to M. The operation reception unit 33 of the terminal device 30 receives the selection, and the first communication unit 31 specifies the file ID of the setting file and requests the setting file from the external service system 40.

S34: The communication unit 19 of the external service system 40 receives the request for the configuration file and transmits the configuration file to the terminal device 30.

S35: The administrator 8 checks the contents of the configuration file and sets information for columns H to M. The operation reception unit 33 of the terminal device 30 accepts the settings.

S36: When the administrator 8 performs an operation to upload a configuration file, for example, the first communication unit 31 transmits the configuration file in which information is set in columns H to M to the information processing system 10.

In this way, the configuration file is registered in the external service system 40.

Figure 28 is a diagram explaining the correspondence between the configuration file and the report. As explained in Table 4, the administrator 8 sets the format, CSV column name, CSV position (column), CSV position (row), OCR language, and the order to be used in the file name. Note that the administrator 8 does not necessarily have to set all columns. Also, depending on the processing content of column C, the specifications of H to M may be ignored.

In the example of Figure 28, the data obtained from the area information with ID=1 is set to correspond to the CSV column name "Document Name". "Document Name" is used as the first character in the file name. The data obtained from the area information with ID=2 is set to correspond to the CSV column name "Advance Application". In addition, since the CSV position of Advance Application is set to 3 columns, row 2, the data obtained from the area information with ID=2 is written out to the 3rd column, row 2 of the file. The data obtained from the area information with ID=3 is set to correspond to the CSV column name "Notes". In addition, since the CSV position of Notes is set to 4 columns, row 2, the data obtained from the area information with ID=3 is written out to the 4th column, row 2 of the file.

<Associating apps with configuration files>
Next, the association between an application and a setting file will be described with reference to Fig. 29 and Fig. 30. Fig. 29 is an example of a screen displayed by the terminal device 30 regarding the association between an application and a setting file. The administrator 8 operates the terminal device 30 to cause the terminal device 30 to communicate with the information processing system 10.

First, FIG. 29(a) shows an initial screen 630 that is displayed when the administrator 8 logs in to the information processing system 10. A menu of operations that are permitted to the administrator 8 is displayed. To associate an app with a configuration file, the administrator 8 selects an icon 631 that displays a list of apps.

As shown in Table 2, the user management unit 44 registers the apps and external service system information that can be used by each user, so the information processing system 10 can provide the terminal device 30 with a list of apps that the administrator 8 can use for each external service system 40.

Figure 29 (b) shows an example of an app list screen 632. The app list screen 632 displays a list of apps that the administrator 8 can use. From the app list screen 632, the administrator 8 selects an app 633 (e.g., an expense report app) to associate with the setting file registered in the external service system 40. When the operation reception unit 33 accepts the selection of the app 633, the association screen 640 shown in Figure 29 (c) is displayed. The association screen 640 has an external service system selection field 641, an upload destination folder field 642, and a setting file field 643.

- The external service system selection field 641 is a field for the administrator 8 to set the external service system 40 in which the file to which the data obtained from the area is written is saved. Therefore, the administrator 8 can change the external service system 40 that is pre-associated with the app, or set a new one.

- The upload destination folder field 642 is a field for the administrator 8 to set a destination folder in the external service system 40 for the file to which the data obtained from the area is written. When the administrator 8 presses the folder selection button 642a, the information processing system 10 logs into the external service system 40 using the authentication information of the administrator 8 in the external service system 40, and the top folder in the external service system 40 that the administrator 8 can access is displayed on the terminal device 30. The administrator 8 can select any folder.

- The setting file column 643 is a column for the administrator 8 to select a setting file to be associated with the app. When the administrator 8 presses the select file button 644, the information processing system 10 logs into the external service system 40 using the authentication information of the administrator 8 in the external service system 40, and the top folder in the external service system 40 that the administrator 8 can access is displayed on the terminal device 30. The top folder (or a folder set by the administrator 8) contains setting files registered by the information processing system 10 and updated by the administrator 8. If there is no setting file in the top folder, the administrator 8 can search for the setting file in the folders below the top folder.

Figure 30 is an example of a sequence diagram explaining the process in which the administrator 8 associates an app with a configuration file via the screen in Figure 29. The process in Figure 30 starts with the screen in Figure 29(a) displayed.

S41: The administrator 8 logs in to the information processing system 10 and inputs an operation to display the app list from the initial screen 630 to the terminal device 30. The operation reception unit 33 of the terminal device 30 receives the operation, and the first communication unit 31 requests the app list from the information processing system 10.

S42: The communication unit 19 of the information processing system 10 receives the request for the app list, and the terminal screen providing unit 42 obtains the apps associated with the user ID of the administrator 8 from the user management unit 44. The terminal screen providing unit 42 creates screen information of an app list screen including apps that the administrator 8 can use for each external service system 40. The communication unit 19 transmits the screen information of the app list screen to the terminal device 30.

S43: The administrator 8 selects an app to which the configuration file is to be associated. The operation acceptance unit 33 of the terminal device 30 accepts the selection. After this, the terminal device 30 displays the association screen 640 as described above.

S44: The administrator 8 presses the Select File button 644 on the association screen 640. The operation reception unit 33 receives the press, and the first communication unit 31 sends to the information processing system 10 a request for a list of files stored in the external service system 40 set on the association screen 640.

S45: The communication unit 19 of the information processing system 10 receives the request for the file list, and the application execution unit 46 requests the file list from the input/output processing unit 48 by specifying the external service system 40. The input/output processing unit 48 requests the file list from the external service processing unit 62 corresponding to the external service system 40. The external service processing unit 62 obtains authentication information for the administrator 8's external service system 40 (which is associated with the application or set by the user on the association screen) from the user management unit 44, and transmits a request for the file list to the external service system 40.

S46: The external service system 40 permits the administrator 8 to log in based on the authentication information of the administrator 8, and obtains a list of files associated with the administrator 8. The list of files is sent to the terminal device 30. Note that in FIG. 30, it is assumed that the external service system 40 in which the setting file is saved is the same as the external service system 40 in which the data file acquired from the area is saved, but the two may be different. In this case, the administrator 8 selects the setting file and then changes the external service system 40 on the association screen 640.

S47: The display control unit 32 of the terminal device 30 displays a list of files. The administrator 8 selects a configuration file to be associated with the app. The operation acceptance unit 33 of the terminal device 30 accepts the selection.

S48: The first communication unit 31 of the terminal device 30 specifies the identification information of the setting file, the identification information of the app, the external service system 40, and the upload destination folder, and sends a request to associate the setting file with the app to the information processing system 10.

S49: The communication unit 19 of the information processing system 10 receives a request to associate the setting file with the application, and the application data management unit 41 registers the setting file and the application in the application data 43 in association with each other. That is, the application data management unit 41 registers the applicationId, applicationName, fileId, fileName, and upload destination folder in association with each other (see Table 3). The external service system 40 is registered in the application processing information 53.

The app and the settings file are now associated. From then on, a general user can output the data in the form area to a file simply by pressing the icon on the device 20.

Note that in FIG. 30, a method in which the administrator 8 associates an app with area information on the association screen 640 is described, but the app and area information may be associated when the area information is registered.

In this case, for example, a field for inputting the app name is displayed on the area registration screen 610 in FIG. 24. The administrator 8 inputs the app name of the app that is to be associated with the setting file. After creating the setting file, the information processing system 10 can associate the setting file with the app with the input app name. In other words, if the administrator 8 sets the app name and has the form read, a setting file is generated and the link source file and the setting file are associated.

In addition, in the association screen 640 shown in FIG. 29(c), the administrator 8 can set the expansion width of the area. FIG. 31 shows a modified example of the association screen 640. Note that the explanation of FIG. 31 will mainly focus on the differences from FIG. 29(c).

The matching screen 640-2 in FIG. 31 has an area expansion width setting field 645. In the area expansion width setting field 645, the administrator 8 can set how much the area information (OCR area) should be expanded beyond the area specified with the marker pen. As a result, the area information in Table 4 becomes wider than the area marked with the marker pen, and even if the area specified by the administrator 8 with the marker pen is narrower than the area that should be marked, or if the user writes outside the area of the form, the OCR processing unit 68 can perform OCR processing on the characters in the area.

However, this is not a limitation, and the administrator 8 may set on the association screen 640-2 whether to include only the area surrounded by the marker (for example, if the OCR area is surrounded by the marker, the marker portion is not included in the area) or the area includes the marker portion.

In addition, the correspondence screen 640-2 may accept information such as color, pen type, and shape regarding the format in which the administrator 8 specifies the area. For example, the administrator 8 can set a rectangle with a yellow marker, a circle with a red pen, or fill in the area with a green marker.

<App duplication>
Since forms have various shapes, the location of the area where the user writes data also varies depending on the form. Therefore, in this embodiment, the administrator 8 can register area information according to the shape of each form. As one method of registering area information for each form, a method in which the administrator 8 duplicates an application will be described.

Figure 32 shows a selection screen 650 for several apps. As an example, if there is an itemized statement app, a payment slip app, and an OCR designated area app, the itemized statement area information is associated with the itemized statement app, and the payment slip area information is associated with the payment slip app in the app data 43. When a user wants to obtain data from the itemized statement area, the user can launch the itemized statement app and scan the itemized statement to output data in the desired area. When a user wants to obtain data from the payment slip area, the user can launch the payment slip app and scan the payment slip to output data in the desired area.

The processing performed by each app is described in app processing information 53, and although the file save destination, upload destination folder, and setting file are different for the processing performed by the statement app and the payment slip app, the components executed are the same. Therefore, if the administrator 8 duplicates the statement app or the payment slip app, the information processing system 10 can create an app corresponding to any form by specifying an area, acquiring an area, creating a setting file, and associating the setting file with the duplicated app.

The duplication of an app will be described with reference to Figure 33. Figure 33 is an example of a screen operated by the administrator 8 when duplication of an app. Figure 33 (a) shows an initial screen 630 displayed by the terminal device 30 when the administrator 8 logs in to the information processing system 10. The initial screen 630 displays a setting button 651 that allows the administrator 8 to make various settings.

Figure 33 (b) shows an app management screen 652 that is displayed after pressing the settings button 651. The app management screen 652 displays a group name 653 into which several apps are grouped. Note that the screen transitions are an example for the purpose of explanation, and the screens shown in the figure may be omitted, or new screens may be added.

A list of apps belonging to this group is displayed to the right of the group name 653. The apps displayed in the list of apps are those for which the administrator 8 has been granted permission to use. Some apps are for the administrator only, and some are used by other users. The administrator 8 presses the settings button 654 for the app to be copied. Pressing the settings button 654 displays a menu that contains a copy button 655.

When the administrator 8 presses the duplicate button 655, a duplicate app setting screen (pop-up screen) 660 shown in FIG. 33(c) is displayed. The duplicate app setting screen 660 has an app name setting field 661, an icon list field 662, and a duplication request button 663. In the app name setting field 661, the administrator 8 inputs an arbitrary app name for the new app created by duplication. The app name can be arbitrary, but duplicate app names are eliminated or a sub-number such as "-(2)" is added. The app name may be called an icon name or a setting name in addition to the app name.

The icon list field 662 displays a list of icons. The administrator 8 can select a desired icon. It is also possible to display another icon by pressing the reference button 664. When the administrator 8 presses the copy request button 663, the application ID, application name, and icon of the application to be copied are transmitted to the information processing system 10. The information processing system 10 receives these, and the application management unit 49 registers a new application using the application processing information 53 and screen information 54 of the application to be copied. Among the application processing information 53, the application ID is numbered so that it does not overlap, and the application name entered by the administrator 8 is set as the application name. The screen information is the same as the application to be copied, but among these, the icon is selected by the administrator 8. The application processing information, shared application, and external service system 40 items are the same as the application to be copied. For example, if the user can identify the application by the application name, etc., it is not necessary to set a different icon for every application.

In addition, in this embodiment, a new image processing area setting is added when the app is duplicated, but instead of duplicating the app, the administrator 8 may add a setting or add an icon to associate the addition of a new image processing area with identification information that accepts selection on the device.

Figure 34 is an example sequence diagram showing the procedure by which the information processing system 10 registers a copy of an app.

S51: The administrator 8 inputs authentication information for the information processing system 10 into the terminal device 30. The operation reception unit 33 of the terminal device 30 accepts the input of the authentication information, and the first communication unit 31 transmits the authentication information to the information processing system 10, thereby logging in the administrator 8 to the information processing system 10.

S52: By logging in, the terminal screen providing unit 42 of the information processing system 10 transmits screen information of the initial screen to the terminal device 30 via the communication unit 19. After that, in response to an operation by the administrator 8, the terminal screen providing unit 42 transmits screen information of the screen shown in FIG. 33 to the terminal device 30.

S53: The administrator 8 inputs the app name, etc. into the terminal device 30, and the operation reception unit 33 accepts the input.

S54: The first communication unit 31 of the terminal device 30 transmits the app ID, app name, and icon of the app to be copied to the information processing system 10.

S55: The communication unit 19 of the information processing system 10 receives these, and the application management unit 49 uses the application processing information 53 and screen information 54 of the original application to register a new application.

In this way, the administrator 8 can duplicate an app. The administrator 8 can associate a configuration file with the new app created by duplication, and obtain data from a specified area of the desired form.

The administrator 8 can create multiple new apps from a single original app, and the information processing system 10 can accept the selection of a different setting file for each of the multiple new apps. Since it is sufficient to assign some kind of identification information to the same processing flow, duplicating an app is just one example, and identification information may also be assigned to the same processing flow in the form of adding an icon or adding settings.

<Acquisition of data from reports using association between apps and configuration files>
After the application and the configuration file have been associated with each other, the user can have the device 20 read the form, so that the information processing system 10 can obtain data from the form.

The acquisition of data from a form will be described with reference to Figures 35 and 36. Figure 35 shows an example of the transition of screens displayed by the device 20.

(1) As shown in FIG. 35(a), the user causes the device 20 to display an app list screen. For example, when the user wants to settle expenses, the user selects the icon of an expense report app. The user knows which report corresponds to which app, but it is preferable that the app name displayed on the app icon 690 and the name of the report (original) match. This app name may be called the icon name or setting name. In other words, it is preferable that the icon 690 includes the name of the report. Furthermore, it is preferable that the icon 690 includes a thumbnail of the report. This makes it easier to prevent the user from selecting an icon that does not correspond to the report. Alternatively, when the user presses the icon 690, a thumbnail of the report may be displayed. The user can check the icon 690 by enlarging or partially displaying the thumbnail.

In addition, a link may be embedded in the icon 690. The link has, for example, a URL including the application ID, or a URL that is associated with the application ID on the information processing system side. When the icon 690 is pressed, the device 20 connects to the URL, thereby notifying the information processing system 10 of the application ID.

In addition, if the user selects a form that does not correspond to the icon 690, the device 20 may warn the user. For example, the device 20 compares the image data of the scanned form with the image data associated with the icon 690, and if the two are different, asks the user by displaying a message asking whether to continue processing. The comparison of image data may be performed by the information processing system 10. Note that, as a comparison method, for example, the device 20 or the information processing system 10 may perform pattern matching on the image data of the form scanned with the image data associated with the icon 690. In addition, if an application ID is printed on the form as a barcode or the like, the device 20 or the information processing system 10 may compare this application ID with the application ID associated with the icon 690.

Also, what the user selects may be an icon, and what the device 20 transmits to the information processing system 10 may be a character string such as an app ID or a setting number. In other words, what is selected does not have to be an app ID (identification information).

(2) When the device 20 scans an expense report, the user prepares the expense report (Figure 35 (b)). The expense report may be prepared by the user by hand, or may be printed out by the device 20. It does not have to be something that the user printed out themselves (such as an officially issued document or a document sent from another company).

(3) The app selected by the user displays the screen in FIG. 35(c) based on the screen information 54. The screen in FIG. 35(c) is the same as that in FIG. 24. The user places the printed expense statement on the contact glass and presses the start button 613 to have the device 20 read it.

The image data of the form is sent to the information processing system 10, and the file is output by the process described below.

Figure 36 is a diagram explaining files stored in the external service system 40. Figure 36 (a) shows an example of an upload destination folder in the external service system 40. That is, the file output by the information processing system 10 is saved in the upload destination folder set by the administrator 8 in Figure 29. The file hierarchy of "My Drive > Expense Report Storage" in Figure 36 (a) is one example, and the administrator 8 can specify a folder to which he or she has access rights.

Figure 36(b) shows two files saved in the upload destination folder. Two files are generated by one read operation.
A. A CSV file 681 in which data obtained from the form area is saved.
B. A PDF file 682 containing image data of the form.

CSV files created by the same app will be separate with each read operation. A serial number and date are added to the file name, so files will not be overwritten. On the other hand, there may be cases where data obtained with the same app will be added to the same CSV file even if the read operation is performed at a different time. This method will be described later. In this way, general affairs staff and other personnel can perform settlements and other procedures for each employee's data simply by accessing a single CSV file.

<<If the device does not scan documents>>
35, the expense report is scanned by the device 20, but the information processing system 10 can receive image data of the form from the terminal device 30 or the like and obtain the data from the image data area. In this case, the terminal device 30 transmits image data of the form to the information processing system 10 as follows.

- The user displays the expense report in a word processor running on the terminal device 30 and enters the necessary information. Therefore, the necessary information is in character code. The user connects the terminal device 30 to the information processing system 10, logs in, and displays a list of apps. The screen of the terminal device 30 will be similar to the screen in Figure 35 (a), and the user selects the desired app. The terminal screen providing unit 42 of the information processing system 10 sends screen information capable of accepting the expense report to the terminal device 30. When the user selects a PDF file or the like containing image data of the expense report on this screen, the terminal device 30 can send the PDF file to the information processing system 10.

Therefore, the information processing system 10 can obtain data from the form without the user having to print out the form. In this case, the processing of the device 20 in the processing of FIG. 37 described later is performed by the terminal device 30.

<Operation procedure>
FIG. 37 is an example of a sequence diagram in which the information processing system 10 receives a PDF file including image data of a form from the device 20 and acquires acquisition data from an area.

S61: The user who printed out the form operates the device 20 to input his/her own authentication information for the information processing system 10. The operation acceptance unit 23 of the device 20 accepts the input of the authentication information, and the second communication unit 21 transmits the authentication information to the information processing system 10. This allows the user to log in to the information processing system 10. Upon logging in, the display control unit 22 of the device 20 displays a list of apps that the user has permission to use (Figure 35 (a)). The user selects, for example, an app with a name that matches the name of the printed out form. The operation acceptance unit 23 accepts the selection.

S62: The second communication unit 21 of the device 20 specifies the application ID and sends an application screen acquisition request to the information processing system 10.

S63: The communication unit 19 of the information processing system 10 receives the application screen acquisition request, and the device screen providing unit 47 acquires screen information 54 of the application identified by the application ID from the application management unit 49. The screen information of the application screen of the application is transmitted to the device 20. As a result, the display control unit 22 of the device 20 displays the application screen (Figure 35 (c)).

S64: The user places the printed form on the contact glass and presses the start button 613. The operation reception unit 23 of the device 20 receives the press, and the image data generation unit 24 generates image data of the form and converts this image data into a PDF file. The form may contain multiple pages.

S65: The second communication unit 21 of the device 20 specifies the app ID of the app and the PDF file, and requests the information processing system 10 to execute the app.

S66: The communication unit 19 of the information processing system 10 receives a request to execute the app, and the app execution unit 46 starts executing the app. The app execution unit 46 specifies the app ID and requests the input/output service unit 16 to execute the app. The input/output service unit 16 obtains the app processing information 53 via the app management unit 49. This results in the processing flow of the app. In addition, the input/output processing unit 48 obtains the app data 43 associated with the app ID from the app data storage unit 13. This allows the input/output processing unit 48 to identify the setting file associated with the app.

The input/output processing unit 48 requests the external service processing unit 62 to obtain a configuration file based on the application data 43 and the application processing information 53.

S67: The external service processing unit 62 connects to the URL of the fileName item registered in the application data 43 and obtains the setting file from the external service system 40. If the application requested to be executed is not a shared application, the external service processing unit 62 logs in to the external service system 40 using the user's authentication information in the external service system 40. If the application requested to be executed is a shared application, the external service processing unit 62 logs in to the external service system 40 using the administrator 8's authentication information in the external service system 40.

S68: Next, the input/output processing unit 48 requests the data storage unit 66 to obtain area information from the setting file based on the application processing information 53.

S69: The data storage unit 66 acquires all area information from predetermined rows and columns of the configuration file and returns it to the input/output processing unit 48. Since area information is registered for each process, area information is acquired for each process.

S70: Next, the input/output processing unit 48 specifies the image data and area information based on the application processing information 53, and requests the area identification unit 67 to obtain (cut out) the area from the image data.

S71: The area identification unit 67 identifies and acquires the processing target area specified by the area information from the image data, and returns images of the processing target area to the input/output processing unit 48 for the number of processes. Note that the area identification unit 67 does not need to acquire (cut out) the processing target area for OCR processing and OMR processing from the image data. In this case, the OCR processing unit 68 and the OMR processing unit 71 only need to perform OCR processing or OMR processing on the processing target area of the image data identified by the area information.

If there are multiple areas to be processed by OCR processing, steps S72 and S73 are repeated as many times as there are areas to be processed.

S72: Based on the application processing information 53, the input/output processing unit 48 specifies an image of the processing target area and requests the OCR processing unit 68 to perform OCR processing of the processing target area. At this time, the input/output processing unit 48 may specify the format and OCR language obtained from the configuration file in association with the processing target area.

S73: The OCR processing unit 68 performs OCR processing on the processing target area using a recognition engine that corresponds to the format and OCR language. The OCR processing unit 68 returns the data converted into character code (OCR processing result) to the input/output processing unit 48.

The recognition engine is an identifier that uses training data consisting of a set of character images and character codes to learn the correspondence between the two, and outputs OCR processing results (character codes) from the images. The simpler the training data is, such as Japanese only, English only, numbers only, or dates only, the higher the recognition accuracy of the identifier. On the other hand, if the training data contains a mixture of Japanese, English, numbers, and dates, it becomes possible to recognize a variety of data, but the recognition accuracy tends to decrease. Therefore, the OCR processing unit 68 can improve recognition accuracy by using a recognition engine that is compatible with the format and OCR language.

Here, machine learning is a technology that allows a computer to acquire human-like learning capabilities, and refers to a technology in which a computer autonomously generates algorithms necessary for judgments such as data identification from training data that is previously loaded, and applies these to new data to make predictions. The learning method for machine learning may be any of supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and deep learning, or may be a combination of these learning methods; any learning method for machine learning is acceptable. In addition, the recognition engine does not have to be one generated by the learning effect of machine learning.

Next, if there are multiple areas to be processed in the OMR process, steps S74 and S75 are repeated the same number of times as there are areas to be processed.

S74: Based on the application processing information 53, the input/output processing unit 48 specifies an image of the processing target area and requests the OMR processing unit 71 to perform OMR processing of the processing target area.

S75: The OMR processing unit 71 performs OMR processing on the processing target area. The OMR processing unit 71 returns the acquired data (OMR processing result) converted to indicate the presence or absence of a mark to the input/output processing unit 48.

Next, if there are multiple areas to be processed for the trimming process, steps S76 and S77 are repeated the same number of times as there are areas to be processed.

S76: Based on the application processing information 53, the input/output processing unit 48 specifies an image of the processing target area and requests the trimming processing unit 72 to perform trimming processing of the processing target area.

S77: The trimming processing unit 72 performs trimming processing on the processing target area. The trimming processing unit 72 returns image data (trimming processing result) to the input/output processing unit 48 as acquired data acquired from the processing target area.

S78: Next, based on the application processing information 53, the input/output processing unit 48 specifies the acquired data acquired from each processing target area, the CSV column name, the CSV position, and the file name (acquired from the configuration file), and requests the data storage unit 66 to output a file containing the acquired data.

S79: The data storage unit 66 writes the CSV column name to the beginning of the column, writes the acquired data to the cell of the CSV file specified by the CSV position, and returns the CSV file to the input/output processing unit 48. The CSV file is given the specified file name. Note that the format of the output file is not limited to a CSV file, and may be any table format file, a text file, or a file of another application.

S80: Next, the input/output processing unit 48 requests the OCR correction unit 69 to correct the OCR processing result by specifying the OCR processing result (CSV file), format, and OCR language based on the application processing information 53. This format is obtained from the configuration file shown in Table 4 and is set by the administrator 8.

Figure 38 is a diagram explaining the correction of OCR processing results. If a format is set in the configuration file, the OCR correction unit 69 can correct the OCR processing results. The OCR correction unit 69 defines the rules for specified character strings and conversion characters as shown in Figure 38. Conversion rule 2201 in Figure 38 is the rule when the format is NUMERIC. For example, since I, B, and Z are not included in numerical values, they are corrected to 1, 8, and 2, respectively. Conversion rule 2202 in Figure 38 is the rule when the language is English. For example, since 1, 2, and 8 are not included in English, they are corrected to I, B, and Z, respectively.

S81: Returning to FIG. 37B, the OCR correction unit 69 returns the corrected OCR processing result (CSV file) to the input/output processing unit 48.

Steps S82 to S85 are executed as many times as there are CSV files. The number of CSV files created in one read operation may be one regardless of the number of forms, or the same as the number of form pages. Which is used is set in the app.

S82: Next, the input/output processing unit 48 requests the external service processing unit 62 to upload the OCR processing result (CSV file) based on the application processing information 53. The upload destination folder is set in the application data 43.

S83: The external service processing unit 62 saves the CSV file in the upload destination folder of the external service system 40.

S84: If it is possible to set metadata, step S84 is executed. If the external service system 40 supports the setting of metadata (items set by the user on the external service system, such as character strings and dates), the user can set the metadata to be linked to the CSV file to be uploaded. The user can input the metadata from a screen displayed by the device 20. The input/output processing unit 48 requests the external service processing unit 62 to upload the metadata. The upload destination folder is the same as the CSV file.

S85: The external service processing unit 62 saves the metadata in the upload destination folder of the external service system 40.

S86: When multiple report data are loaded, the user may wish to save them in individual files. In this case, the input/output processing unit 48 requests the PDF processing unit 64 to split the pages along with the image data of multiple pages based on the application processing information 53. This process may be omitted, in which case the input/output processing unit 48 may save the multiple data as a single file.

S87: The PDF processing unit 64 returns the PDF file broken down into pages to the input/output processing unit 48.

S88: Based on the application processing information 53, the input/output processing unit 48 specifies a PDF file and requests the OCR processing unit 68 to perform OCR processing.

S89: The OCR processing unit 68 performs OCR processing on the PDF file and sets the character code generated by the OCR processing as transparent text in the PDF file.

Steps S90 to S93 are executed as many times as there are PDF files.

S90: The input/output processing unit 48 requests the external service processing unit 62 to upload a PDF file with text based on the application processing information 53. The upload destination folder is set in the application data 43.

S91: The external service processing unit 62 saves the PDF file with text in the upload destination folder of the external service system 40.

S92: If the external service system 40 supports the above metadata (items set by the user on the external service system, such as character strings and dates), the user can input the metadata from the screen displayed by the device 20. The input/output processing unit 48 requests the external service processing unit 62 to upload the metadata. The upload destination folder is the same as the PDF file.

S93: The external service processing unit 62 saves the metadata in the upload destination folder of the external service system 40.

As a result, the CSV file and a separate PDF file from the CSV file have been registered in the upload destination folder of the external service system 40.

Note that in Figure 37, the image data to be processed is a PDF file, but the image data to be processed can be image data in any format, such as JPEG, PNG, TIFF, BMP, etc.

In this embodiment, it is assumed that the device 20 will scan and process a large number of forms and the like all at once. On the other hand, there are cases where a user will only use forms of the same format once, and in cases where the processing area changes each time, registering area information in the setting file will only be a hassle for the administrator. For this reason, the task of registering area information can be omitted, and the areas of the forms filled in by the user can be directly read by the device 20. The information processing system 10 performs processing for each area according to the correspondence between the color and processing registered in the setting file.

<Displaying CSV and PDF files registered in an external service system>
When a company employee calculates expenses for employees, the employee operates the terminal device 30 to connect to the external service system 40. Since the employee has been notified of the upload destination folder, the employee can obtain the CSV file and the PDF file on the terminal device 30.

Figure 39 is an example of a file list screen 670 of the external service system 40 displayed by the terminal device 30. As shown in Figure 39(a), a PDF file 671 and a CSV file 672 are saved in the upload destination folder. Figure 39(b) shows the contents of the CSV file 672 displayed by the terminal device 30. That is, Figure 39(b) is data output from a form. In Figure 39(b), data obtained from two areas is saved. The CSV column name set in the settings file is also set for each column, allowing the user to determine what data each column represents.

As shown in FIG. 40, it is also preferable that the device 20 also displays the processing results. FIG. 40 is an example of an output data display screen 691 displayed by the device 20. The output data display screen 691 displays the name of the executed application 692 and a list 693 of files output by the application. These are sent from the information processing system 10 to the device 20. The CSV file contains the contents of the CSV file, and when the user presses the CSV file, the device 20 can display data 694 that has been obtained from the form area and saved in the CSV file.

By checking the file name and data, users can determine whether the report was processed correctly.

<When multiple documents are scanned>
A case where multiple forms are read in one reading operation will be described with reference to Fig. 41. Fig. 41 is a diagram showing multiple forms and output files. As shown in Fig. 41(a), multiple forms are read in one reading operation when the application is executed. One reading operation refers to a case where the start button 613 is pressed only once. The device 20 reads multiple forms in sequence, and the information processing system 10 acquires multiple data from the same area of the image data of each form.

Figure 41 (b) shows a CSV file when multiple forms are read in a single reading operation. The first line is the data obtained from the first form, and the second line is the data obtained from the second form. In this way, multiple data from multiple forms are compiled into a single CSV file, making it easier for the person in charge to tally and graph the data.

When multiple forms are read in a single read operation, the data storage unit 66 increments the CSV position (start row) of the configuration file by one. In other words, the data storage unit 66 writes data in the row below for each form.

When multiple forms are read in a single reading operation, the information processing system 10 can also output CSV files for the number of forms.

In addition, when multiple documents are read in multiple reading operations (when read at different times), it is also possible for the information processing system 10 to output the data to one and the same CSV file. There are cases where the external service system 40 merges the data and cases where the information processing system 10 merges the data.

- When the external service system 40 performs merging, the external service processing unit 62 requests the external service system 40 to merge the CSV files. If the application is the same, the file names of the CSV files contain the same character string (e.g., expense report), so the external service processing unit 62 can request merging of the CSV files by specifying the file name. The external service system 40 adds data of the same column number of the CSV file requested to be merged to the same column number of the already saved CSV file.

- When the information processing system 10 performs merging, the external service processing unit 62 specifies the file name and downloads the CSV file from the external service system 40. The external service processing unit 62 adds data of the same column number of the newly created CSV file to the same column number of the acquired CSV file. The external service processing unit 62 uploads the merged CSV file to the external service system 40.

In this way, the information processing system 10 obtains the same area from the multiple image data it receives, the OCR processing unit 68 performs OCR processing on the same area in the multiple image data, and the data storage unit 66 can output the multiple data generated from the same area in the multiple image data to a single file.

<Major Effects>
As described above, the data output system 100 of this embodiment allows the administrator 8 to associate marks on registration forms with processes using a marker pen. Furthermore, the device 20 reads the forms on which areas have been marked, and the information processing system 10 stores the process and area information for each form. The area is then obtained from the image data of the form sent from the device 20, and the information processing system 10 performs the process set in the area. This makes it possible to easily create a file that extracts only specific areas from multiple forms of the same type.

In this embodiment, we will explain an information processing system 10 that uses OCR to recognize different recognition targets depending on the color of the area.

The process or operation performed by the data output system 100 will be outlined with reference to Figures 42 and 43. Note that the explanation of Figures 42 and 43 will mainly focus on the differences from Figures 1 and 2.

In the process of Figure 42, process (2) is different from that of Figure 1.

(2) The administrator 8 of the data output system 100 uses the device 20 to print out a registration form registered in the information processing system 10 and uses a marker pen to color it with the color he or she wants to associate with the process. For example, the administrator 8 can associate any color with a process, such as green for "OCR kanji," blue for "OCR dates," and red for "OCR alphanumeric characters." The device 20 scans the registration form to generate image data (first image data) and sends it to the information processing system 10.

Next, we will refer to Figure 43 for further explanation. Figure 43(a) is a diagram for explaining how to set area information in a form, and Figure 43(b) is a diagram for explaining how to obtain data from a form using area information. The process in Figure 43 is similar to that in Figure 2, so we will explain the main differences.

(4) The administrator 8 uses a form 7 that is used daily by users in the organization and uses a marker pen to designate which area of the form 7 is to be processed (OCR kanji, OCR dates, OCR alphanumeric characters).

The subsequent steps (5) to (11) can be the same as in Figure 2.

In addition to the effects of the first embodiment, the data output system 100 of this embodiment allows the administrator 8 to register more detailed processing contents and marks (colors). It is known that in OCR, the recognition rate varies greatly depending on whether the recognition target is limited. For example, when the OCR processing unit 68 performs OCR for the postal code "〒", if it is not known in advance that the recognition target is a postal code, it will often erroneously recognize it as "T". In this embodiment, the OCR recognition target is set for each area, which improves the OCR recognition accuracy.

Another option is for the administrator to use a website to set what is to be written in each area of the form, but in this embodiment, the administrator can set the OCR recognition target for each area by simply marking the registration form.

<Additional information regarding color and processing>
The series of processes explained in Figures 42 and 43 will be supplemented by dividing them into tasks 1 to 3. Figure 44 is a diagram for explaining the registration of the correspondence between colors and processes (task 1) using a registration form.

Figure 44 (a) shows the registration form before coloring.

Figure 44(b) shows the registration form after coloring. The registration form in Figure 44(b) is scanned by device 20.

Figure 44 (c) shows a part of a configuration file in which colors and processes are associated. There are two possible ways to associate colors and processes.

1: The information processing system has in advance information on how to link the coordinates (within the registration form) from which the color is obtained with the processing obtained from those coordinates.
Example of pre-defined parameters: The color extracted at coordinates (X coordinate: 100, 200 Y coordinate: 1, 50) is used to OCR kanji characters.
・The color extracted at coordinates (X coordinate: 100, 200 Y coordinate: 51, 100) is the date and time OCR
・The color extracted at coordinates (X coordinates: 100, 200 Y coordinates: 101, 150) is OCR alphanumeric characters
2: The administrator links colors and processes on the management site.
45 shows an example of the management site 150 for associating colors with processes. The management site 150 is provided to the terminal device by the information processing system 10 as a Web page or a Web application. The administrator can register the color of the process using a radio button 171. In this case, the number of colors of the marker pen that the user can use is limited to three.

In either method, as shown in FIG. 44(c), a number identifying the marker color (green) is associated with the OCR of kanji characters. A number identifying the marker color (blue) is associated with the OCR of dates. A number identifying the marker color (red) is associated with the OCR of alphanumeric characters.

Figure 46 is a diagram explaining the correspondence between area information and processing (task 2) using a form base. Figure 46 (a) shows the form base.

Figure 46(b) shows the form after coloring. The form in Figure 46(b) is scanned by device 20. Figure 46(c) shows a portion of a setting file in which area information and processing are associated.

As shown in FIG. 46(d), since the setting file associates colors with processes, the information processing system 10 detects the color of the registered paper (already registered in the setting file) from the form, and associates the coordinates of the area where this color is detected as area information with the processing of the setting file.

Therefore, in this embodiment, the administrator can change the content of the OCR processing for an area by changing the color with which the area is colored on the form.
Ex1: Color in blue → OCR kanji, color in red → OCR date
Ex2: Color in blue → OCR the postal code, color in red → OCR the URL
47A and 47B are diagrams for explaining processing of an area in a form (operation 3) using the form. Fig. 47A shows an original sheet on which the user has filled in the necessary information.

Figure 47 (b) shows the processing performed in each area of the form. In the form's name area 172, the process of OCR in kanji is registered as OCR1, so the name is treated as kanji (including katakana and hiragana) and OCRed. In the form's application date area 173, the process of OCR date and time is registered as OCR2, so the application date is treated as date and time and OCRed. In the form's employee number area 174, the process of OCR alphanumeric characters is registered as OCR3, so the employee number is treated as alphanumeric characters and OCRed.

<About the function>
In this embodiment, the hardware configuration diagrams of FIG. 8 and FIG. 9 and the functional configuration diagram shown in FIG. 10 described in the above embodiment can be used.

Figure 48 is a diagram showing the functional configuration of the OCR processing unit 68 in this embodiment. The OCR processing unit 68 has a first OCR processing unit 68a, a second OCR processing unit 68b, and a third OCR processing unit 68c.

The first OCR processing unit 68a performs character recognition of kanji characters. The second OCR processing unit 68b performs character recognition of dates. The third OCR processing unit 68c performs character recognition of alphanumeric characters. The first OCR processing unit 68a to the third OCR processing unit 68c are each an identification device that has learned the correspondence between images of kanji characters, dates, or alphanumeric characters and character codes through machine learning.

Machine learning is a technology that allows computers to acquire human-like learning capabilities, in which the computer autonomously generates algorithms necessary for judgments such as data identification from training data that is previously loaded, and applies these to new data to make predictions. The learning method for machine learning may be any of supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and deep learning, or a combination of these learning methods. Any learning method for machine learning is acceptable.

Also, as shown in Table 5, in this embodiment, the configuration file associates area colors with OCR languages.

表５は、本実施例において、外部サービスシステム４０に記憶されている設定ファイル９の一例を示す。なお、表５の説明では、主に表４との相違を説明する。表５の設定ファイルでは、処理の項目にＯＣＲ１～３が設定されている。また、ＯＣＲ言語の項目には領域の色に応じた認識対象が設定されている。表５では、例えば、緑が漢字、青が日付、赤が英数字である。

Table 5 shows an example of the setting file 9 stored in the external service system 40 in this embodiment. Note that in the explanation of Table 5, differences from Table 4 will be mainly explained. In the setting file of Table 5, OCR1 to 3 are set in the processing item. Also, in the OCR language item, a recognition target according to the color of the area is set. In Table 5, for example, green is kanji, blue is date, and red is alphanumeric.

The first OCR processor 68a performs character recognition of kanji characters when the area is green. The second OCR processor 68b performs character recognition of dates when the area is blue. The third OCR processor 68c performs character recognition of alphanumeric characters when the area is red.

Therefore, in this embodiment, the recognition targets are limited according to the kanji characters, dates, and alphanumeric characters written in the area, improving recognition accuracy.

Note that the types of OCR (recognition targets) are not limited to these, and may include postal codes, email addresses, half-width alphanumeric characters, numbers (telephone numbers, postal codes, etc., in a format without hyphens between numbers), numbers (telephone numbers, postal codes, etc., in a format with hyphens between numbers), or URLs. In this embodiment, kanji includes strings of hiragana and katakana. Various languages (English, French, German, Chinese, Arabic, etc.) may also be recognized.

<Application processing information>
Next, the application processing information will be described in detail. The same applies to the present embodiment with respect to FIG.

Figure 49 is a diagram that shows a schematic of a series of processes performed by an app. In explaining Figure 49, differences from Figure 12 will be mainly explained. Figure 49 (c) shows a series of processes of an app that outputs data from an area of a form (e.g., an expense report app). In this embodiment, OCR of kanji, date and time, and alphanumeric characters is performed according to the OCR language (recognition target) set in the configuration file.

<Registering the correspondence between colors and processing in the configuration file>
Next, the setting of the correspondence between colors and processes by the administrator 8 and the creation of a setting file will be described with reference to FIG.

Figure 50 is a diagram explaining how to set the correspondence between colors and processes using a registration form. The explanation of Figure 50 will mainly focus on the differences from Figure 13. Figure 50 (a) shows the registration form before the administrator 8 has colored it with a marker pen. The original paper of the registration form is registered in the external service system 40.

The first line of the registration form lists the items for each column (first column: marker color, second column: process). In this embodiment, there are three types of "processing" (OCR kanji, OCR date and time, and OCR alphanumeric characters), and the registration form has the following pre-set processes: second line: OCR kanji, third line: OCR date and time, and fourth line: OCR alphanumeric characters. The marker color cell 130 on the registration form in its original state is blank.

Figure 50 (b) shows the administrator 8 coloring cells 130 with a marker pen for each process. The administrator 8 uses, for example, green, blue, and red marker pens from among the marker pens he owns. The administrator 8 assigns green to OCR kanji, blue to OCR date and time, and red to OCR alphanumeric characters. A predetermined gap is provided between the lines on the base paper of the registration form to prevent the colors from bleeding and mixing.

The administrator 8 colors the cells 130 on the registration form with a marker pen that he or she owns, according to the assignment of the desired marker color and each process that he or she has determined. The registration form is scanned by the device 20, and the correspondence between the marker color and the process is registered in the information processing system 10.

<Device screen example>
An example of the screen of the device 20 may be the same as that shown in Fig. 14. A sequence diagram showing the procedure in which the administrator 8 uses a registration form to register the correspondence between colors and processes in the setting file may be the same as that shown in Fig. 15. However, the process in step S12 is different.

S12: The color detection unit 73 detects color information from a predetermined cell 130 of the image data.

Figure 51 is a diagram that shows a schematic diagram of detection of color information. The explanation of Figure 51 will mainly focus on the differences from Figure 16. As shown in Figure 51 (a), the OCR processing of kanji characters is determined by the color within a rectangle at two diagonal position coordinates AB (an example of position information), the OCR processing of date and time is similarly determined by the color within a rectangle at position coordinates CD, and the OCR processing of alphanumeric characters is determined by the color within a rectangle at position coordinates EF.

As shown in FIG. 51(b), the barcode 101 or the application data management unit 41 stores the process and the position coordinates of the cell 130 to be colored in association with each other. The color detection unit 73 obtains the HSV H value from each cell 130 based on the position coordinates in FIG. 51(b). When a color corresponding to red (H=(0-60°, 301-360°)) is detected from a cell 130 in which alphanumeric characters are associated with OCR, the color detection unit 73 registers H=(0-60°, 301-360°) in HSV in the setting file, associating the alphanumeric characters with OCR. When a color corresponding to green (H=(61-180°)) is detected from a cell 130 in which kanji characters are associated with OCR, the color detection unit 73 registers H=(61-180°) in the setting file, associating the kanji characters with OCR. If a color corresponding to blue (H=(181-300°)) is detected from cell 130 where the date and time are associated with OCR, the color detection unit 73 registers H=(181-300°) in the configuration file, associating the date and time with OCR.

Associating colors with processes can also be done by an administrator using the management site 150 described in Figure 45.

<<Specifying file name by process>>
In this embodiment as well, the administrator 8 can set the file name of the CSV file by coloring the registration form.

Figure 52 shows an example of the use of a registration form where processing is applied to setting file names. The explanation of Figure 52 will mainly focus on the differences from Figure 17. As with Figure 17, processing is limited to OCR. The registration form in Figure 52(a) has OCR1 and OCR2 set as processing, and the order to be used for file names, "File name 1" and "File name 2", set.

As shown in FIG. 52(b), the administrator 8 uses a marker pen to specify the marker pen color (e.g., green) to be used for file name 1 and the marker pen color (e.g., blue) to be used for file name 2.

As shown in FIG. 52(c), a numerical value indicating a color and a process (used in the file name) are associated with a setting file. The two methods for associating a color with a process can be the same as those shown in FIG. 17.

Figure 53 is an example of a management site 152 for associating colors with processes. The administrator 8 can register the color to be used to specify the file name using radio buttons 153. In this case, the number of marker pen colors that the user can use is limited to three.

As shown in FIG. 52(c), the numeric value identifying the marker color (green) is associated with file name 1. The numeric value identifying the marker color (blue) is associated with file name 2. Note that file names 1 and 2 are used in the order in which they are named.

Figure 54 is an example of a form on which the administrator 8 has designated two areas with a marker pen. Figure 54(a) shows the original form, and Figure 54(b) shows the form colored with a marker pen. When the device 20 scans the form in Figure 54(b), the information processing system 10 identifies the colored areas on the form and registers the area information in the settings file.

In FIG. 54(b), the "Name" area 172 is designated with a green marker pen, and the application date area 173 is designated with a blue marker pen. For example, if the name is "Riko Taro" and the application date is "2020/10/10", the file name of the CSV file when the document in FIG. 54(b) is scanned by device 20 will be "Riko Taro 20201010.csv". Note that three or more marker pen colors may be specified in the file name.

Also, it is not necessary to register the order of the areas used in the file names (file 1, 2). In this case, the data storage unit 66 can set the file names in the order in which the colors are registered in the setting file. Therefore, it is not necessary to change the color of the marker pen for each cell on the registration form.

<<Modifications of the registration form>>
As shown in Figure 55, the CSV location may be registered in advance on the registration form. In the explanation of Figure 55, the differences from Figure 20 will be mainly explained. Figure 55(a) shows the registration form before coloring. This registration form has columns for CSV column name 154 and CSV location 155, so the administrator does not need to manually set the CSV location in the setting file.

Figure 55(b) shows the registration form after coloring. The registration form in Figure 55(b) is scanned by device 20. Figure 55(c) shows a portion of a setting file in which colors, processing, and area information are associated with each other.

<Registration of area information>
Next, the registration of area information will be described with reference to FIG. 56 and the like. FIG. 56 is a diagram for explaining a method of marking an area on a form. First, FIG. 56(a) shows a form (original paper) before the area is designated with a marker pen. On the original paper of the form, the manager 8 designates an area with a marker pen of the color registered in the area where the process registered on the registration form is to be executed. For example, the manager 8 wants to set the name area 172 as an area for OCR of kanji, the application date area 173 as an area for OCR of date and time, and the employee number area 174 as an area for OCR of alphanumeric characters.

Figure 56 (b) shows a form with areas colored with a marker pen. The employee area 172 is colored with a green marker pen. The application date area 173 is colored with a blue marker pen. The employee number area 174 is colored with a red marker pen. When the device 20 scans this form, the association between the areas and the processes for the form is registered in the configuration file.

According to the configuration file in Table 5, the first OCR processing unit 68a performs OCR processing on the name area 172, with kanji characters as the recognition target, to generate acquired data. The second OCR processing unit 68b performs OCR processing on the alphanumeric area 173, with date and time as the recognition target, to generate acquired data. The third OCR processing unit 68c performs OCR processing on the employee number area 174, with alphanumeric characters as the recognition target, to generate acquired data.

Figure 56 (c) is an example of a form on which the user has filled in the necessary information. From the colored state in Figure 56 (b), "Riko Taro", "2020/10/10", and "50412" are the acquired data.

<Area registration screen>
Fig. 57 is an example of an area registration screen 610 displayed by the device 20 when registering area information. In the explanation of Fig. 57, the differences from Fig. 24 will be mainly explained. In Fig. 57, the areas marked on the form are different from those in Fig. 24.

The procedure in which the information processing system 10 acquires area information from a form on which the administrator 8 has marked areas and creates a configuration file may be the same as that shown in FIG. 25. The procedures in FIGS. 26 and 27 may also be the same as those in the first embodiment.

Figure 58 is a diagram explaining the correspondence between configuration files and reports. In the example of Figure 58, the data obtained from the area information with ID=1 is set to correspond to the CSV column name "Name". "Name" is used as the first character in the file name. The data obtained from the area information with ID=2 is set to correspond to the CSV column name "Application Date". In addition, since the CSV position of the application date is set to column 3, row 2, the data obtained from the area information with ID=2 is written to column 3, row 2 of the file. The data obtained from the area information with ID=3 is set to correspond to the CSV column name "Employee Number". In addition, since the CSV position of notes is set to column 4, row 2, the data obtained from the area information with ID=3 is written to column 4, row 2 of the file.

After this, Figures 29 to 36 can be the same as in Example 1.

<Operation procedure>
Fig. 59 is an example of a sequence diagram in which the information processing system 10 receives a PDF file including image data of a form from the device 20 and acquires acquisition data from an area. In the explanation of Fig. 59, differences from Fig. 37 will be mainly explained.

First, the processing of steps S61 to S71 can be the same as that shown in FIG. 37.

If there are multiple areas to be processed by OCR processing, steps S72 and S73 are repeated as many times as there are areas to be processed.

S72: Based on the application processing information 53, the input/output processing unit 48 specifies an image of the processing target area and requests the OCR processing unit 68 to perform OCR processing of the processing target area. In this embodiment, the input/output processing unit 48 specifies the format and OCR language (kanji, date and time, alphanumeric characters) obtained from the configuration file in association with the processing target area.

S73: The OCR processing unit 68 performs OCR processing on the processing target area using a recognition engine that corresponds to the format and OCR language. That is, the first OCR processing unit 68a recognizes kanji, the second OCR processing unit 68b recognizes dates, and the third OCR processing unit 68c recognizes alphanumeric characters. The OCR processing unit 68 returns the data converted into character codes (OCR processing results) to the input/output processing unit 48.

S78, S79: Next, the input/output processing unit 48, based on the application processing information 53, specifies the acquired data acquired from each processing target area, the CSV column name, the CSV position, and the file name (acquired from the configuration file), and requests the data storage unit 66 to output a file containing the acquired data.

The process from step S80 to S93 can be the same as in Figure 37.

This makes it possible to display CSV files and PDF files registered in the external service system, but the method for displaying these can be the same as in Example 1.

<Major Effects>
As described above, in addition to the effects of the first embodiment, the data output system 100 of this embodiment allows the administrator 8 to register more detailed processing contents and marks (colors). Since the OCR recognition target is set for each area, the OCR recognition accuracy can be improved. Also, in this embodiment, the administrator can set the OCR recognition target for each area by the simple method of marking the registration form.

<Other application examples>
The above describes the best mode for carrying out the present invention using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

For example, in this embodiment, the information processing system 10 outputs the data acquired from the form area to a file, but instead of creating a file, the data may be sent by email or displayed on the device 20.

The configuration examples in FIG. 10 and the like are divided according to main functions to facilitate understanding of the processing by the terminal device 30, the device 20, and the information processing system 10. The present invention is not limited by the manner in which the processing units are divided or the names of the processing units. The processing by the terminal device 30, the device 20, and the information processing system 10 can also be divided into even more processing units depending on the processing content. Also, it can be divided so that one processing unit includes even more processes.

Furthermore, the devices described in the examples are merely illustrative of one of several computing environments for implementing the embodiments disclosed herein. In one embodiment, the information processing system 10 includes a plurality of computing devices, such as a server cluster. The plurality of computing devices are configured to communicate with each other via any type of communication link, including a network, shared memory, etc., and perform the processes disclosed herein.

Furthermore, the information processing system 10 can be configured to share the processing steps disclosed in this embodiment, such as FIG. 37, in various combinations. For example, a process executed by a specific unit can be executed by multiple information processing devices possessed by the information processing system 10. Furthermore, the information processing system 10 may be integrated into one server device, or may be divided into multiple devices.

Each function of the embodiments described above can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and devices such as an ASIC (Application Specific Integrated Circuit), DSP (Digital Signal Processor), FPGA (Field Programmable Gate Array), and conventional circuit modules designed to execute each function described above.

10 Information processing system 20 Device 30 Terminal device 100 Data output system

JP 2019-096178 A

Claims (17)

An information processing system that is communicatively connected to one or more devices and executes a predetermined process,
a data storage unit that stores setting information in which a mark for identifying the content of the process is associated with the predetermined process based on an operation by a user of a first device;
a communication unit that receives second image data of a predetermined document including the mark added by a user from the first device;
an area detection unit that detects an area including the mark from the second image data;
a management unit that stores location information of the detected area, a predetermined process corresponding to the mark included in the area, and an application selected by a user of the first device in association with each other;
a processing unit that executes a predetermined process on the third image data received from the second device based on location information and a predetermined process associated with an application selected on the second device;
An information processing system comprising: 2. The information processing system according to claim 1, wherein the mark is color information. 3. The information processing system according to claim 1, further comprising a communication unit for transmitting screen information of a screen for accepting the association between the mark and the process to the first device. The information processing system according to any one of claims 1 to 3, characterized in that the data storage unit stores the process in association with the mark based on the mark contained in the first image data received from the first device. The information processing system according to claim 4 , wherein the data storage unit stores the process performed by the information processing system in association with the mark based on position information of the mark formed in the first image data. the first image data includes position information of an area corresponding to the content of the processing;
An information processing system as described in claim 4 or 5, wherein the data storage unit stores the mark formed in the area of the location information and the content of the processing corresponding to the area of the location information in association with the processing . the information processing system holds position information of an area corresponding to the content of the processing in association with identification information of the first image data;
An information processing system as described in any one of claims 4 to 6, wherein the data storage unit stores the mark formed in the area of the location information and the content of the processing corresponding to the area of the location information in association with the processing. The information processing system according to any one of claims 1 to 7, characterized in that the processing is OCR, OMR, trimming, translation, masking, or encryption. The information processing system according to claim 2, characterized in that the processing is OCR, and the data storage unit stores information indicating that different recognition targets are recognized depending on the color information. The information processing system according to claim 9, characterized in that the recognition target is at least one of a character string, a postal code, an email address, half-width alphanumeric characters, numbers (without hyphens between numbers), numbers (with hyphens between numbers), or a URL. a communication unit that receives second image data of a predetermined document from the first device;
an area detection unit that detects an area in which the mark associated with the process is formed from the second image data;
11. The information processing system according to claim 1, wherein the data storage unit stores location information of the area in association with the process. the communication unit receives the third image data of the predetermined document from the second device;
a region specifying unit that specifies a processing target region from the third image data based on position information of the region;
The processing unit executes the processing associated with position information of the processing target area,
The information processing system according to claim 11 , wherein the data storage unit outputs acquired data acquired by the processing unit processing the processing target region. The data storage unit outputs a file including the acquired data;
13. The information processing system according to claim 12, further comprising an external service processing unit that stores the file in an external service system. The process specifies one or more file names;
The information processing system according to claim 13 , wherein the data storage unit sets acquired data acquired by the processing unit processing the processing target region as a file name of the file. The information processing system according to any one of claims 1 to 14, characterized in that the first device and the second device are the same device. An information processing method performed by an information processing system that is communicatively connected to one or more devices and executes a predetermined process, comprising:
a data storage unit storing setting information in which a mark for identifying the content of the process is associated with the predetermined process based on an operation by a user of the first device;
receiving, from a first device, second image data of a predetermined document including the mark added by a user, by a communication unit;
A step of detecting an area including the mark from the second image data by an area detection unit;
a step of storing, by a management unit, position information of the detected area, a predetermined process corresponding to the mark included in the area, and an application selected by a user of the first device in association with each other;
A step of executing a predetermined process on the third image data received from the second device, based on location information and a predetermined process associated with the application selected on the second device, by a processing unit;
13. An information processing method comprising: An information processing system that is communicatively connected to one or more devices and executes a predetermined process,
a data storage unit that stores setting information in which a mark for identifying the content of the process is associated with the predetermined process based on an operation by a user of a first device;
a communication unit that receives second image data of a predetermined document including the mark added by a user from the first device;
an area detection unit that detects an area including the mark from the second image data;
a management unit that stores location information of the detected area, a predetermined process corresponding to the mark included in the area, and an application selected by a user of the first device in association with each other;
a processing unit that executes a predetermined process on the third image data received from the second device based on location information and a predetermined process associated with the application selected on the second device;
A program to function as a

Images