JP7765256B2 - Information processing device, information processing method, and information processing program - Google Patents

Description

The present invention relates to an information processing device, an information processing method, and an information processing program.

Recently, there has been an increasing demand for automated application operations known as RPA (Robotic Process Automation). There are automation tools available for automating RPA operations.

In RPA, identifying the cause of an error when an automated operation fails is a key issue. Previously, only a screenshot was recorded immediately after the error occurred. This made it difficult to identify the cause of the error if the original screen was not captured or if the screen itself disappeared.

A similar effect could also be achieved by taking screenshots at every RPA processing step, but this was not a practical method because the binary data for the images would increase disk space.

Furthermore, even if it was possible to capture screenshots before and after the error occurred, there was no viewer to intuitively compare them.

Conventional RPA systems include, for example, the system described in Patent Document 1.

Japanese Patent Application Laid-Open No. 2020-115246

However, Patent Document 1 does not disclose any method for identifying the cause of an error if one occurs during automatic operation.

The present invention has been made in view of the above, and when an error occurs during automatic operation,
An object of the present invention is to provide an information processing device, an information processing method, and an information processing program that enable the cause of an error to be visually and intuitively understood.

To solve the above-mentioned problems and achieve the objectives, the present invention provides an information processing device that automatically operates an application in accordance with a definition that prescribes a series of operations, and is characterized by comprising: a screenshot means that records screenshots before and after an error that occurs during the automatic operation of the application in a memory area; and a screen display means that displays the screenshots before and after the error recorded in the memory area on a confirmation screen.

Furthermore, according to one aspect of the present invention, the screenshot means may take a screenshot each time a screen item is automatically operated, and may store in the storage area only the screenshots taken before an error that correspond to a set number of steps recorded immediately before the error.

Furthermore, according to one aspect of the present invention, the screenshot means may store the screenshot before the error in the storage area using a FI/FO method.

Furthermore, according to one aspect of the present invention, the confirmation screen may display a list of thumbnails of screenshots before and after the error occurred.

Furthermore, according to one aspect of the present invention, the confirmation screen may display a comparison image that highlights the difference between screenshots before and after the error.

Furthermore, according to one aspect of the present invention, when the confirmation screen displays comparison images of three or more screenshots, the differences between the different images may be highlighted differently.

Furthermore, according to one aspect of the present invention, the highlighting of the differences may include highlighting using a highlight color.

Furthermore, according to one aspect of the present invention, when the confirmation screen displays comparison images of three or more screenshots, the differences between the different images may be highlighted in different colors to visualize the gradual differences in the screen changes.

Furthermore, according to one aspect of the present invention, the automated operation may be automated operation based on RPA (Robotic Process Automation).

Furthermore, according to one aspect of the present invention, the definition is composed of a first layer that defines the flow, a second layer that defines the robot execution, and a third layer that is the internal implementation of the screen operation processing, and the definition of the screenshot means may be implemented in the third layer.

Furthermore, according to one aspect of the present invention, the application may include an accounting application.

Furthermore, in order to solve the above-mentioned problems and achieve the objectives, the present invention provides an information processing method executed by an information processing device that automatically operates an application in accordance with a definition that prescribes a series of operations, and may include a screenshot step of recording screenshots of the screen before and after an error that occurs during the automatic operation of the application in a storage area, and a screen display step of displaying the screenshots of the screen before and after the error recorded in the storage area on a confirmation screen.

Furthermore, in order to solve the above-mentioned problems and achieve the objectives, the present invention provides an information processing program to be executed by an information processing device that automatically operates an application in accordance with a definition that prescribes a series of operations, and is characterized by the information processing program executing a screenshot process that records screenshots before and after an error that occurs during the automatic operation of the application in a memory area, and a screen display process that displays the screenshots before and after the error recorded in the memory area on a confirmation screen.

This invention has the advantage that, if an error occurs during automatic operation, the cause of the error can be visually and intuitively understood.

FIG. 1 is a diagram for explaining how to take a screen shot when a robot is automatically executing an application to be operated. FIG. 2 is a diagram showing an example of a screen shot. FIG. 3 is a diagram showing an example of a screen shot. FIG. 4 is a diagram for explaining a case where a screen shot is taken immediately before each operation of a screen item. FIG. 5 is a block diagram showing an example of the configuration of an information processing device according to this embodiment. FIG. 6 is a diagram for explaining a specific example of processing by the control unit of the information processing device according to the present embodiment. FIG. 7 is a diagram for explaining a specific example of processing by the control unit of the information processing device according to the present embodiment. FIG. 8 is a diagram for explaining a specific example of processing by the control unit of the information processing device according to the present embodiment. FIG. 9 is a diagram for explaining a specific example of processing by the control unit of the information processing device according to the present embodiment. FIG. 10 is a diagram for explaining a specific example of processing by the control unit of the information processing device according to the present embodiment. FIG. 11 is a diagram for explaining a specific example of processing by the control unit of the information processing device according to the present embodiment. FIG. 12 is a diagram for explaining a specific example of processing by the control unit of the information processing device according to the present embodiment. FIG. 13 is a diagram for explaining a specific example of processing by the control unit of the information processing device according to the present embodiment. FIG. 14 is a diagram for explaining a specific example of processing by the control unit of the information processing device according to the present embodiment. FIG. 15 is a diagram for explaining a specific example of processing by the control unit of the information processing device according to the present embodiment. FIG. 16 is a diagram for explaining a specific example of processing by the control unit of the information processing device according to the present embodiment. FIG. 17 is a diagram for explaining a specific example of processing by the control unit of the information processing device according to the present embodiment. FIG. 18 is a diagram for explaining a specific example of processing by the control unit of the information processing device according to the present embodiment. FIG. 19 is a diagram for explaining a specific example of processing by the control unit of the information processing device according to the present embodiment. FIG. 20 is a diagram for explaining a specific example of processing by the control unit of the information processing device according to the present embodiment.

Embodiments of an information processing device, information processing method, and information processing program according to the present invention will be described in detail below with reference to the accompanying drawings. Note that the present invention is not limited to these embodiments.

[1. Overview]
Recently, there has been an increasing demand for automated application operations known as RPA (Robotic Process Automation).

This RPA has automation tools for automating operations. Specifically, multiple operations performed by the user are registered (defined) and then executed, allowing the standardized operations to be easily executed from the second time onwards.

Responding when automated operations fail is an important issue, and screenshot logs are useful when investigating the cause of a failure. Screenshot logs (1) allow for a more intuitive understanding of the situation at the time of failure than text logs of processing details, and (2) contain more information, making them useful for identifying the cause. Specifically, they can record error messages displayed on the screen and input values for items that have not yet been operated.

Figure 1 is a diagram illustrating how to take a screenshot when a robot is automatically executing an application to be operated. In Figure 1, let's assume that operations 1 and 2 performed by the robot on the application to be operated are successful, but operation 3 fails. An example of a failure would be when an invalid value is entered into a text box in "operation 2," causing an error message to be displayed, making it impossible to operate the item to be entered in "operation 3." A screenshot is taken immediately after the error occurs, and the cause of the failure is investigated using the screenshot.

Figures 2 and 3 show examples of screenshots. Most systems take a screenshot immediately after an error occurs and notify the user. This system is technically easy to implement.

Figure 2 shows Screenshot Example 1. Screenshot Example 1 shows a case where an error occurred because an unexpected dialog box appeared, preventing the robot from continuing processing. The dialog box "Error: Some fields have not been entered" is displayed, and this screenshot is taken.

Figure 3 shows Screenshot Example 2. Screenshot Example 2 shows a case where the screen transition expected by the robot did not occur, the item to be operated on could not be found, and an error occurred. The message "500 Internal Server Error Please contact your system administrator" is displayed, and this screen shot is taken.

Examples of logic by which the robot determines an operation has failed include: (1) when the screen item to be operated does not exist on the screen; (2) when a dialog box (such as an error) that the robot did not anticipate appears; or (3) when a screen other than the one to be operated is displayed.

Next, we will explain the issues with screenshots. First, as a business issue (1), there are cases where the cause of an error cannot be determined from just the screenshot taken immediately after the error occurs. Examples of cases where the items entered using RPA cannot be confirmed include when the input screen is partially hidden by an error dialog, the user is redirected to an error page, or the items entered in the application are cleared.

Operational issue (2) is that there are cases where it is not possible to take a screenshot immediately after an error occurs. Irregular cases where the screen disappears include when the RPA execution times out and is forced to close, when a person accidentally closes a screen that is being automatically operated, or when the computer running the robot is forced to shut down.

For this reason, it is desirable to have a screenshot of the screen immediately before an error occurs. However, there are implementation challenges (1) and (2) to implementing this feature:

(Implementation Issue (1))
Since the process of taking a screenshot for each operation must be set in the definition (L2), the creation of the definition becomes cumbersome and the definition itself becomes complicated. L (layer) will be described later.

(Implementation Issues (2))
It is necessary to take a screenshot every time just before operating a screen item, but this would result in a large screenshot log (and of course, it is not practical to take a video). Specifically, because images are binary data, a large amount of disk space is required to store them, and due to the characteristics of RPA, a large number of images are taken during a series of execution processes.

Figure 4 is a diagram illustrating the case where a screen shot is taken each time a screen item is operated. In Figure 4, for example, operations 1 through 98 are successful, operation 99 fails, and a screen shot (image data No. 100) is taken when an error occurs. Screen data Nos. 1 through 100 are stored in the image data storage area, and No. 99 is the screen shot taken immediately before the error occurs and is necessary to identify the cause of the error, while Nos. 1 through 98 are not necessary to identify the cause of the error.

(Solution to implementation problem (1))
This can be solved by incorporating the screen shot capture process into the internal process (L3) of the screen operation process. The definition is simplified because the process can be incorporated without the user creating the RPA definition (L2) being aware of it.

(Solution to implementation problem (2))
A screenshot is taken every time a screen item is operated, but the image data from the previous capture is discarded and only the most recent one is saved. This allows you to take screenshots before and after an error occurs without taking up space in the image data storage area.

The solutions to implementation issues (1) and (2) will be discussed in detail below.

2. Configuration
The configuration of information processing device 100 according to this embodiment will be described with reference to Fig. 5. Fig. 5 is a block diagram showing an example of the configuration of information processing device 100 according to this embodiment.

The information processing device 100 is, for example, a commercially available desktop personal computer. Note that the information processing device 100 is not limited to a stationary information processing device such as a desktop personal computer, but may also be a portable information processing device such as a commercially available notebook personal computer, PDA (Personal Digital Assistant), smartphone, or tablet personal computer.

As shown in FIG. 5, the information processing device 100 includes a control unit 102, a communication interface unit 104, a memory unit 106, and an input/output interface unit 108. The units included in the information processing device 100 are connected to each other so that they can communicate with each other via any communication path.

The communication interface unit 104 communicatively connects the information processing device 100 to the network 300 via a communication device such as a router and a wired or wireless communication line such as a dedicated line. The communication interface unit 104 has the function of communicating data with other devices via the communication line. Here, the network 300 has the function of connecting the information processing device 100 and the server 200 so that they can communicate with each other, and is, for example, the Internet or a LAN (Local Area Network). Note that the data stored in the memory unit 106, which will be described later, may also be stored on the server.

An input device 112 and an output device 114 are connected to the input/output interface unit 108. The output device 114 may be a monitor (including a home television), a speaker, or a printer. The input device 112 may be a keyboard, a mouse, a microphone, or a monitor that functions as a pointing device in conjunction with a mouse. Note that, below, the output device 114 may be referred to as the monitor 114, and the input device 112 may be referred to as the keyboard 112 or the mouse 112. Furthermore, the operation performed by a user using the input device 112 on the screen (GUI, etc.) of the output device (monitor) 114 may be simply referred to as a "user operation."

The storage unit 106 stores various databases, tables, files, and the like. The storage unit 106 stores computer programs that work in conjunction with the OS (Operating System) to issue commands to the CPU (Central Processing Unit) to perform various processes. The storage unit 106 can be, for example, a memory device such as RAM (Random Access Memory) or ROM (Read Only Memory), a fixed disk device such as a hard disk, a flexible disk, or an optical disk.

The memory unit 106 includes a definition file 106a, a parameter master 106b, a log file 106c, an image file 106d, etc.

The definition file 106a is a file in which the automatic operation unit (RPA system) 102a stores definitions (also called "RPA robot execution definitions") that stipulate a series of operation commands for automatically operating an application. A "robot or RPA robot" is a unit that defines a series of operation commands required for one processing execution (one loop), defined for each application to be operated. Details of the "RPA robot execution definition" will be provided later.

The log file 106c is a file for storing execution logs created by the automatic operation unit (RPA system) 102a. Execution logs include, for example, three levels of logs: summary log, detailed log, and detailed log (see Figures 8 to 11).

The parameter master 106b is a master for registering various parameters, for example,
The number of steps recorded immediately before an error is registered in the parameter master 106b, which defines the number of steps recorded immediately before an error in the image file 106d of the screen shot taken by the screen shot unit 102a1. The number of steps recorded immediately before an error can be set by the operator.

The image file (storage area) 106d is a file for storing screenshots taken by the screenshot unit 102a1.

The control unit 102 is a CPU or the like that provides overall control of the information processing device 100. The control unit 102 has internal memory for storing control programs such as an OS, programs that define various processing procedures, and required data, and executes various information processes based on these stored programs.

The control unit 102 is configured to be able to access the definition file 106a, parameter master 106b, log file 106c, image file 106d, etc. stored in the memory unit 106. Note that these master files may also be stored in other locations (e.g., server 200) as long as they are accessible by the control unit 102.

Functionally, the control unit 102 comprises an automatic operation unit (RPA system) 102a and an application 102b.

The automatic operation unit (RPA system) 102a executes automatic operations on application 102b in accordance with a definition that specifies a series of operations on application 102b, which is registered in definition file 106a.

The automatic operation unit (RPA system) 102a conceptually comprises a screenshot unit 102a1 and a screen display control unit 102a2. The automatic operation unit 102a is incorporated into the RPA system described above. In addition to being incorporated into RPA, it may also be incorporated into other automatic operation tools such as RDA (Robotic Desktop Automation).

The screenshot unit 102a1 records screenshots before and after an error occurs during automatic operation of an application in the image file 106d.

The screenshot unit 102a1 may take a screenshot every time a screen item is automatically operated, and may store in the image file 106d only the screenshots taken before an error that correspond to the number of steps recorded immediately before the error, as set in the parameter master 106b.

The screenshot unit 102a1 may store the pre-error screenshots in the image file 106d using a FI/FO (First In/First Out) method, for the number of recording steps immediately before the error.

The screen display control unit 102a2 displays on the monitor 114 a confirmation screen for displaying the screenshots before and after the error that are recorded in the image file 106d. The confirmation screen may display a list of thumbnails of the screenshots before and after the error. The confirmation screen may also display a comparison image that highlights the differences between the screenshots before and after the error.

When displaying comparison images of three or more screenshots on the confirmation screen, the differences between the different images may be highlighted differently. Highlighting of differences may include highlighting using highlight colors. Note that highlighting is not limited to highlight colors; various displays are possible, such as highlighting by changing the color, font, size, brightness, etc. of text, or highlighting with arrows. Any display format that can distinguish and display the differences is acceptable.

When displaying comparison images of three or more screenshots on the confirmation screen, the differences between the different images may be highlighted in different colors to visualize the gradual differences in the screen changes.

Application 102b includes, for example, accounting applications for inputting various slips such as order entry, purchase entry, order entry, and sales entry, and for printing out various slips. The present invention can also be applied to other applications. Applications are stored in the memory unit 106, but the applications stored in the memory unit 106 that are executed by the control unit 102 are conceptually referred to as application 102b of the control unit 102 in terms of function.

3. Specific examples of processing
Specific examples of the processing performed by the control unit 102 of the information processing device 100 in this embodiment will be described with reference to Figures 6 to 20. Figures 6 to 20 are diagrams for explaining specific examples of the processing performed by the control unit 102 of the information processing device 100 in this embodiment.

The automatic operation unit (RPA system) 102a of the control unit 102 executes the following process (flow) in accordance with the RPA robot execution definition (program and data) stored in the definition file 106a. The process (flow) shown below shows the process executed by the automatic operation unit 102a in accordance with the RPA execution definition, but for ease of explanation, the RPA execution definition may be described as the subject of the operations.

(3-1. Common (basic) definition)
The common definitions of the RPA robot execution definitions will be described with reference to Figures 6 to 11. First, the basic common definitions will be described.

(RPA flow definition)
6 is a diagram showing the execution flow including the processes before and after the execution of the RPA robot. It is assumed that the data to be input to the robot is prepared in Excel or the like and then inputted repeatedly.

In Figure 6, when the RPA flow is started (step S1), the RPA data source is read (step S2). Here, for example, a data source prepared in Excel or similar is read.

Execute the RPA data source loop (step S3). Here, loop processing is performed for each item of data read.

When the loop starts (step S4), the RPA robot is executed (step S7). Detailed settings for how the robot will move are made separately. The execution result is determined to be success or failure. If the execution result is success (step S8), the next loop is executed (step S10). If the execution result is failure (step S9), a handling pattern is selected depending on the operation. (1) In the case of a continue-on-error type (processing continues even after an error occurs) (step S11), the next loop is executed (step S10). (2) In the case of a stop-on-error type (processing stops when an error occurs) (step S12), the RPA scenario is terminated (step S13).

When the loop ends (step S5), the RPA flow ends (step S6).

(RPA robot execution definition)
This section explains the detailed definition of "RPA robot execution." The robot's behavior is defined by combining processing components. The RPA robot execution definition consists of (1) Layer 1 (L1): RPA flow definition, (2) Layer 2 (L2): robot execution definition, and (3) Layer 3 (L3): internal implementation of screen operation processing.

(1) Layer 1 (L1): RPA flow definition This is the "RPA flow definition" mentioned above. This is the layer set by the RPA execution setter.

(2) Layer 2 (L2): Robot Execution Definition This is the detailed execution definition of the robot. This is the layer set by the RPA execution configuration person. Operation processes corresponding to screen items are prepared as components, and these are combined to define the behavior of the robot. In general RPA tools, various processing components are prepared in addition to screen operation processes.

For example, the screenshot capture process is a process that can be incorporated when you want to take a screenshot at any time. The standby process is a process that waits for time-consuming processes such as screen transitions and registration processes. It is common to set the standby time during setup. The screen item value acquisition process is a process that acquires the values entered on the screen. The branching process is a process that branches the robot's behavior based on conditions such as the value of a screen item. The loop process is a process used when repeated input is required, such as for detail items.

(3) Layer 3 (L3): Internal implementation of screen operation processing This is the internal implementation of the "screen operation processing component" incorporated into the robot execution definition. This is a layer that the RPA execution configuration person is not aware of. By incorporating common processing into this layer, it is possible to simplify the RPA configuration while making it highly functional.

Figure 7 shows an example of the processing flow for (1) Layer 1 (L1): RPA flow definition, (2) Layer 2 (L2): robot execution definition, and (3) Layer 3 (L3): internal implementation of screen operation processing.

In Figure 7, in the L1:RPA flow definition, when the PRA flow is started (step S21), a series of pre-processing steps are performed (step S22), the RPA robot is executed (step S23), a series of post-processing steps are performed (step S24), and the RPA flow is terminated (step S25).

L2: In the robot execution definition, when RPA robot execution is started (step S31), for example, text boxes are filled in (step S32), check boxes are filled in (step S33), radio buttons are filled in (step S34), a series of processes are executed (step S35), the registration button is pressed (step S36), and RPA robot execution is terminated (step S37).

L3: In the internal implementation of the screen operation process, when the operation process starts (step S41), the item to be operated is identified (step S42). For example, the item to be operated is identified based on its ID or coordinates. If the identification of the item to be operated fails (step S44), i.e., if the item to be operated does not exist, an error is generated and "RPA robot execution" is immediately interrupted.

If the target item to be operated is successfully identified (step S43), input data is obtained (step S45). Specifically, data to be input into the target item is obtained from the data read in by "Read RPA data source."

If the input data acquisition fails (step S47), an error is detected and "RPA robot execution" is immediately interrupted.

If the input data is successfully acquired (step S46), a series of pre-processing steps are performed (step S48), operation processing is performed (step S49), a series of post-processing steps are performed (step S50), an execution log is output and stored in log file 106c (step S51), and operation processing is terminated (step S52).

(Execution log)
The execution log is assumed to have three levels: summary log, detailed log, and detailed log.
A summary log is a log created for each execution instruction of an RPA flow. A detailed log is a log output for each processing unit (L1: each step unit of the RPA flow definition) in one execution of an RPA flow. A detailed log is a log output for each processing unit (L2: each step unit of the RPA robot execution definition) in the RPA robot execution definition. Note that the execution log may also include data on success and error (failure) results of the execution.

Figure 8 shows an example of a summary log. The summary log may include the following data: SEQ, log level, execution definition name, execution start date and time, execution end date and time, execution ID, and execution type CD.

Figure 9 shows an example of a detailed log. The detailed log may include data such as the execution ID, SEQ, log level, execution content, executing robot name, target application name, loop SEQ, execution start date and time, and execution end date and time.

Figure 10 shows an example of a detailed log. The detailed log may include data such as the execution ID, detailed SEQ, SEQ, log level, execution content, name of the item to be operated on, input information, and execution date and time.

Figure 11 shows an example of a screenshot log. The screenshot log may include an execution ID, a detailed SEQ, a detailed SEQ, and an example of screen data.

(3-2. Implementation and Processing of Screen Shot Unit 102a1)
The screen shot unit 102a1 records screen shots taken before and after an error occurs during automatic operation of an application in an image file (storage area) 106d. Alternatively, the screen shot unit 102a1 may take a screen shot immediately before each automatic operation of a screen item, and store only the screen shots taken before the error, the number of steps recorded immediately before the error set in the parameter master 106b, in the image file 106d. Alternatively, the screen shot unit 102a1 may store the screen shots taken before the error in the image file 106d in a FI/FO format.

The implementation and processing of the screenshot section 102a1 will be explained in relation to solutions to the above implementation issues (1) and (2).

(Solution to implementation problem (1))
If a screen shot capture process is set in the definition (L2) for each operation, the creation of the definition becomes cumbersome, and the definition itself also becomes complicated. To address this issue, this embodiment incorporates the screen shot capture process into the internal process (L3) of the screen operation process. This simplifies the definition because the process can be incorporated without the user creating the RPA definition (L2) being aware of it.

The reason for implementing the screen shot capture process in the internal processing (L3) of the screen operation process will be explained in detail with reference to Figures 12 and 13. Figure 12 is a diagram for explaining the case where the screen shot capture process is implemented in the RPA definition (L2). In Figure 12, steps equivalent to those in Figure 7 are assigned the same step numbers.

In Figure 12, when implementing the screenshot capture process in the RPA definition (L2), the screenshot creation process must be set in the RPA definition (L2) for each operation. In the example shown in Figure 12, the screenshot capture process must be set for each text box input, check box input, radio button input, and registration button operation. This makes creating the RPA definition (L2) cumbersome, and the definition itself becomes complicated.

Figure 13 is a diagram for explaining in detail the case where the screenshot capture process is implemented as an internal process (L3) of the screen operation process. In Figure 13, steps that are equivalent to those in Figure 7 are assigned the same step numbers.

In Figure 13, the definition can be simplified by incorporating the screenshot capture process (step T1) before the identification of the operation target item (step S41) in the internal process (L3) rather than in the RPA definition (L2). As a result, a screenshot can be captured when the "immediate process in the RPA definition (L2)" has completed successfully. For example, if the step in question is "(2) Checkbox input (step S33)," a screenshot will be captured when "(1) Text box input (step S32)" has completed successfully.

Errors are detected at each processing step of the internal processing (L3), and if an error occurs (step S44), a screenshot is taken.

(Solution to implementation problem (2))
To address the issue of the large data volume of the screenshot log if a screenshot is taken every time a screen item is operated (identifying the item to be operated (step S42)), a screenshot is taken every time a screen item is operated (identifying the item to be operated (step S42)), but the image data from the previous shot is discarded and only the most recent one is kept. This makes it possible to take screenshots before and after an error occurs without putting pressure on the image file 106d (storage area).

Figure 14 is a diagram illustrating a case where, for example, a screen shot is taken every time a screen item is operated, but the image data from the previous shot is discarded and only the most recent one is kept.

In Figure 14, for example, if operations 1 through 98 are successful, operation 99 fails, and a screenshot (image data No. 100) is taken at the time the error occurred, image file 106d (image data storage area) temporarily stores image data for each capture, but the image data from the previous capture is discarded and only the most recent data (image data No. 99) is retained.

The number of steps recorded immediately before an error in parameter master 106b may be set to the number of steps recorded going back immediately before the error. For example, if the number of steps recorded immediately before an error is 3, image data for three screen shots immediately before the error occurred will be recorded. In the example shown in Figure 14, image data for three screen shots No. 97 to 99 will be recorded.

Figure 15 is a diagram explaining how screenshots are stored in memory. In FIFO format, when a new screenshot is taken, the data for the oldest screenshot is discarded. The number of screenshots to be recorded can be changed depending on the number of steps recorded immediately before an error in parameter master 106b. In the example shown in Figure 15, the number of steps recorded immediately before an error is set to 3.

If step 3 is successful, screenshots (1), (2), and (3) are saved as shown in Figure 15. If step 4 is successful, the oldest screenshot (1) is discarded. If an error occurs in step 5, the screenshot immediately after the error occurred, as well as screenshots (2), (3), and (4), are saved.

(3-3. Processing of Screen Display Control Unit 102a2)
The display processing of the screen display control unit 102a2 will be described in detail with reference to Figures 16 to 20. Figures 16 to 20 show images of reference screens. The screen display control unit 102a2 displays an RPA log list screen based on the logs recorded in the log file 106c on the monitor 114, and displays the screen shots before and after the error recorded in the image file 106d on a screen shot confirmation screen in response to an operator's operation on the RPA log list screen. The screen display control unit 102a2 displays a list of thumbnails of the screen shots before and after the error on the screen shot confirmation screen, and may also highlight the differences between the screen shots before and after the error.

Figure 16 (A) is a diagram showing an example of the RPA log list screen. The RPA log list screen displays the SEQ, the result of the automatic operation (success or failure), the target item name, the processing content of the automatic operation, and a screenshot confirmation button linked to the error log when the result of the automatic operation is "failure." Pressing the screenshot confirmation button launches the screenshot confirmation screen shown in Figure 16 (B).

Figure 16 (B) shows an example of the screenshot confirmation screen. The screenshot confirmation screen includes a thumbnail list area where a list of captured screenshots is displayed as thumbnails, and an image display area where captured screenshots are displayed.

The thumbnail list area allows you to scroll through thumbnails. The image display area displays the screenshot selected in the thumbnail list area, and you can switch between screenshots using the buttons to switch between previous and next images.

Figure 17 is a diagram illustrating an example of how to operate the screenshot confirmation screen. Figure 18 is a diagram showing an example of comparison image 1 (image No. 99: immediately before the error). Figure 19 is a diagram showing an example of comparison image 2 (image No. 100: immediately after the error). Comparison image 2 in Figure 19 (image No. 100: immediately after the error) shows an error when an invalid code was entered when entering the XX code.

In Figure 17, when multiple images (thumbnails) are selected in the thumbnail list area and the image comparison button is pressed, a comparison image of the selected images is displayed in the image display area. In the image display area, multiple images are compared pixel by pixel, and an enhanced comparison image is displayed with several pixels surrounding points of difference highlighted. This makes it possible to provide information that allows for more intuitive cause identification.

You can select two or more images as comparison images. For example, if you select two images, Image 1 and Image 2, a comparison image will be displayed in which the differences between Image 1 and Image 2 are highlighted. If you select three images, Image 1, Image 2, and Image 3, a comparison image will be displayed in which the differences between adjacent images are highlighted, and a comparison image will be displayed in which the differences between Image 1 and Image 2 and the differences between Image 2 and Image 3 are highlighted. If you select four images, Image 1, Image 2, Image 3, and Image 4, a comparison image will be displayed in which the differences between Image 1 and Image 2, the differences between Image 2 and Image 3, and the differences between Image 3 and Image 4 are highlighted. Each difference may be displayed in a different highlight color.

In the example shown in Figure 17, image No. 99 (just before the error) and image No. 100 (just after the error) are selected in the thumbnail list area and the image comparison button is pressed, and a comparison image is displayed in the image display area, with the few pixels surrounding the points of difference between image No. 99 (just before the error) and image No. 100 (just after the error) highlighted in highlight color. In the example shown in Figure 17, the difference between the two images, "XXX This code cannot be used because it is classified as B", is highlighted in highlight color.

Figure 20 shows an example of the display when three images are selected on the screenshot confirmation screen. When three or more screenshots are selected from the thumbnail list display area and compared, the gradual differences in the images are visualized by using multiple highlight colors to indicate the differences.

In the example shown in Figure 20, three images - image No. 98 (before the error), image No. 99 (just before the error), and image No. 100 (just after the error) - are selected in the thumbnail list area and the image comparison button is pressed. Comparison images are then displayed in the image display area, highlighting in different highlight colors the differences between image No. 98 (before the error) and image No. 99 (just before the error) and the differences between image No. 99 (just before the error) and image No. 100 (just after the error).

As described above, this embodiment includes a screenshot unit 102a1 that records screenshots before and after an error that occurs during automatic operation of an application in a storage area, and a screen display control unit 102a2 that displays the screenshots before and after the error recorded in the storage area on a confirmation screen. Therefore, if an error occurs during automatic operation, it becomes possible to visually and intuitively understand the cause of the error.

[4. Contribution to the United Nations-led Sustainable Development Goals (SDGs)]
This embodiment can contribute to improving business efficiency and promoting appropriate management decisions by companies, thereby contributing to the achievement of SDGs Goals 8 and 9.

Furthermore, this embodiment can contribute to reducing waste and promoting paperless and electronic systems, thereby contributing to SDGs goals 12, 13, and 15.

Furthermore, this embodiment can contribute to strengthening control and governance, thereby contributing to Goal 16 of the SDGs.

5. Other Embodiments
The present invention may be implemented in various different embodiments other than those described above within the scope of the technical concept set forth in the claims.

For example, among the processes described in the embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods.

In addition, the processing procedures, control procedures, specific names, registered data for each process, information including parameters such as search conditions, screen examples, and database configurations shown in this specification and drawings may be changed as desired unless otherwise specified.

Furthermore, with regard to the information processing device 100, the components shown in the figure are functional concepts and do not necessarily have to be physically configured as shown.

For example, all or any part of the processing functions of the information processing device 100, particularly the processing functions performed by the control unit 102, may be implemented by a CPU and a program interpreted and executed by the CPU, or may be implemented as hardware using wired logic. The program is recorded on a non-transitory, computer-readable recording medium containing programmed instructions for causing the information processing device to execute the processes described in this embodiment, and is mechanically read by the information processing device 100 as needed. That is, a computer program is recorded on the storage unit 106, such as a ROM or HDD (Hard Disk Drive), for working with the OS to issue commands to the CPU and perform various processes. This computer program is executed by being loaded into RAM, and works with the CPU to configure the control unit 102.

In addition, this computer program may be stored on an application program server connected to the information processing device 100 via any network, and all or part of it may be downloaded as needed.

In addition, the program for executing the processing described in this embodiment may be stored on a non-temporary computer-readable recording medium, or may be configured as a program product. Here, this "recording medium" includes memory cards, USB (Universal Serial Bus) memories, SD (Secure Digital) cards, flexible disks, magneto-optical disks, ROMs, EPROMs (Erasable Programmable Read Only Memory), EEPROMs (registered trademark) (Electrically Erasable and Programmable Read Only Memory), CD-ROMs (Compact Disk Read Only Memory), MOs (Magneto-Optical disks), DVDs (Digital Versatile Disks), and more. This includes any "portable physical media" such as a Blu-ray Disc (registered trademark) and a Blu-ray (registered trademark) Disc.

A "program" is a data processing method written in any language or description method, and may be in any format, such as source code or binary code. It should be noted that a "program" is not necessarily limited to a single structure, but also includes a distributed structure consisting of multiple modules or libraries, or a structure that achieves its function by working in conjunction with a separate program, such as an OS. The specific structure and reading procedure for reading a recording medium in each device shown in the embodiments, as well as the installation procedure after reading, can use well-known structures and procedures.

The various databases stored in the memory unit 106 are storage devices such as memory devices such as RAM and ROM, fixed disk devices such as hard disks, flexible disks, and optical disks, and store various programs, tables, databases, and web page files used for various processes and website provision.

In addition, the information processing device 100 may be configured as an information processing device such as a known personal computer or workstation, or may be configured as an information processing device to which any peripheral device is connected. In addition, the information processing device 100 may be realized by implementing software (including programs, data, etc.) that causes the information processing device to perform the processing described in this embodiment.

Furthermore, the specific form of distribution and integration of the devices is not limited to that shown in the figures, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various additions or functional additions. In other words, the above-mentioned embodiments may be implemented in any combination, or embodiments may be implemented selectively.

100 Information processing device 102 Control unit 102a Automatic operation unit (RPA system)
102a1 Screen shot unit 102a2 Screen display control unit 102b Application 104 Communication interface unit 106 Storage unit 106a Definition file 106b Parameter master 106c Log file 106d Image file 108 Input/output interface unit 112 Input device 114 Output device 200 Server 300 Network

Claims (12)

An information processing device that automatically operates an application in accordance with a definition that prescribes a series of operations,
a screen capture means for recording screen shots taken before and after an error occurs during the automatic operation of the application in a storage area;
a screen display means for displaying a confirmation screen for confirming the screen shots before and after the error recorded in the storage area;
Equipped with
The information processing device is characterized in that the screenshot means takes a screenshot every time just before automatically operating a screen item, and for screenshots before an error, only the screenshots of the number of steps recorded just before the error that are set are stored in the memory area . 2. The information processing apparatus according to claim 1 , wherein the screen shot means stores the screen shot before the error in the storage area in a FIFO manner. 3. The information processing apparatus according to claim 1, wherein the confirmation screen displays a list of thumbnails of screen shots before and after the error. 4. The information processing apparatus according to claim 1, wherein the confirmation screen displays a comparison image in which a difference between screen shots before and after the error is highlighted. 5. The information processing apparatus according to claim 4 , wherein, when three or more comparative images of the screen shots are displayed on the confirmation screen, differences between the different images are highlighted differently. 6. The information processing apparatus according to claim 4 , wherein the highlighting of the difference includes highlighting in a highlight color. The information processing device according to claim 6, characterized in that when the confirmation screen displays comparison images of three or more screen shots, the differences between the different images are highlighted in different colors to visualize the gradual differences in screen changes. 8. The information processing device according to claim 1, wherein the automatic operation is an automatic operation based on RPA (Robotic Process Automation). The information processing device described in claim 8, characterized in that the definition is composed of a first layer that defines a flow, a second layer that defines robot execution, and a third layer that is an internal implementation of screen operation processing, and the definition of the screenshot means is implemented in the third layer. 10. The information processing apparatus according to claim 1, wherein the application includes an accounting application. An information processing method executed by an information processing device that automatically operates an application in accordance with a definition that prescribes a series of operations, comprising:
a screenshot process for recording screenshots of the application before and after an error occurs during the automatic operation of the application in a storage area;
a screen display step of displaying the screen shots before and after the error recorded in the storage area on a confirmation screen;
Including,
In the screen shot process, a screen shot is taken every time just before an automatic operation of a screen item is performed, and for screen shots before an error, only the screen shots of the set number of recording steps just before the error are retained in the memory area . An information processing program for causing an information processing device to automatically operate an application in accordance with a definition that prescribes a series of operations, the information processing program comprising :
The control unit
a screenshot process for recording screenshots of the application before and after an error occurs during the automatic operation of the application in a storage area;
a screen display step of displaying the screen shots before and after the error recorded in the storage area on a confirmation screen;
An information processing program for executing
In the screen shot process, a screen shot is taken every time just before an automatic operation of a screen item is performed, and for screen shots before an error, only the screen shots of the set number of steps recorded just before the error are stored in the memory area.