JP7662857B2 - Screenshot capture device, screenshot capture method and program - Google Patents

Description

The present invention relates to a screen shot capture device, a screen shot capture method, and a program.

As companies are increasingly adopting the automated operation of business systems (application programs) using RPA (Robotic Process Automation) tools, there is a growing need to understand the execution status of automated operations on screens provided by business systems (in other words, various scenes during the execution of the RPA tool) for the purpose of collecting evidence. However, because the execution speed of automated operations is fast, it is difficult for a person to visually check the execution status, and if a person who checks the execution status must be present when the automated operation is being executed, there is a risk that the significance of adopting an RPA tool in the first place will be lost. In addition, while it is possible for a person to understand the execution status by looking at and checking a text log that describes the execution status, it is difficult for a person to intuitively understand the execution status from a text log alone. Therefore, in order to support a person's intuitive understanding of the execution status, screenshots of the screen that is the target of the automated operation are taken.

Patent document 1 describes a screenshot method for a terminal in which a screenshot is taken once based on the image change status in the screenshot reference area each time the screen scrolls a specified distance, and a final screenshot image is obtained based on the images screenshotted multiple times and the initial screenshot image.

Special table 2019-533251 publication

However, up until now, screenshots of the screen that is the target of automatic operation (the operation target screen) have been taken without considering the state of the operation target screen. As a result, when multiple application programs are running, the screenshots are taken with the operation target screens overlapping each other, resulting in a problem in that it is not possible to check the hidden screen items.

The present invention has been made in consideration of the above, and aims to provide a screen shot taking device, a screen shot taking method, and a program that can take screen shots including areas where the screens to be operated overlap and are hidden.

In order to solve the above-mentioned problems and achieve the object, the screen shot photographing device according to the present invention is a screen shot photographing device including a control unit that photographs screen shots of all of a plurality of screens provided by a target program that is an application program to be operated, the control unit including an execution identifier acquisition means for acquiring an execution identifier of the target program being operated, a screen identifier acquisition means for acquiring screen identifiers of all of the screens provided by the target program identified by the execution identifier acquired by the execution identifier acquisition means, and a screen identifier acquisition means for acquiring screen shots of the screens identified by each of all the screen identifiers acquired by the screen identifier acquisition means. the capture means captures an image of an error occurring state of an object program and a storage means stores all the screen shots captured by the capture means in a storage unit, the storage means storing the screen shots by assigning numbers indicating the order in which the screen shots were taken so that a person can later check the screen in a state in which the error occurrence state can be understood from the context of the screen and the results of the automatic operation can be kept as a record, the execution identifier is a process of the target program and the screen identifier is a window handle of the screen, and the capture means specifies the window handle of the screen to be captured and captures screen shots of each layer that do not overlap with other screens, starting from the bottom layer.

The screen shot capture device according to the present invention is also characterized in that the screens provided by the target program are multiple.

In addition, the screen shot capturing device of the present invention is capable of accessing master data in which a screen identifier is linked to one or more control identifiers for vertical or horizontal scrolling and/or a control identifier for page movement and in the screen shot capturing device of the present invention, the capturing means executes the following processes (1) to (3) for all screen identifiers acquired by the screen identifier acquisition means.
(1) Obtain a control identifier associated with the screen identifier from the master data.
(2) If the control identifier cannot be obtained, a screenshot of the screen identified by the screen identifier is taken.
(3) If a control identifier is obtained, (31) obtains scrolling information or page movement information from each control identifier, (32) sets the scrolling and/or page movement state of the screen identified by the screen identifier to an initial state according to each piece of obtained information, and takes screen shots, and (33) performs scrolling and/or page movement processing according to each piece of obtained information on the screen identified by the screen identifier, and takes screen shots until the processing can no longer be performed.

In addition, in the screen shot taking device according to the present invention, the taking means takes a screen shot while performing the processes of scrolling and page moving in the process (33) in that order.

Furthermore, in the screen shot photographing device according to the present invention, the photographing means is characterized in that in the process (33), the scrolling process is performed in the order of vertical scrolling and horizontal scrolling.

The screen shot capture device according to the present invention is also characterized in that the scroll process uses the size of the scroll box as the unit of movement of the scroll box.

The screen shot capture device according to the present invention is also characterized in that the save execution means saves the screen shots arranged on one sheet based on the scrolling information and/or page navigation information saved when the screen shots were captured.

The screen shot taking device according to the present invention is also characterized in that the taking means takes screen shots in order starting from the bottom screen.

In the screen shot taking device according to the present invention, the save execution means saves the screen shots by assigning them numbers indicating the order in which they were taken.

The screen shot photographing device according to the present invention is also characterized in that the application program is a business application program.

The screen shot capture device according to the present invention is also characterized in that the operation is an automatic operation performed by an RPA (Robotic Process Automation) tool or an operation performed by a user to capture a screen shot.

In addition, in the screen shot capture device according to the present invention, the execution identifier is the process of the target program, the screen identifier is the window handle of the screen, and the capture means specifies the window handle of the screen to be captured, and captures the screen shots in order from the bottom screen.

Furthermore, a screen shot taking method according to the present invention is a screen shot taking method for taking screen shots of all of a plurality of screens provided by a target program which is an application program to be operated and which is executed by an information processing device having a control unit, the method comprising the steps of: acquiring an execution identifier of the target program being operated and executed by the control unit; acquiring screen identifiers of all screens provided by the target program identified by the acquired execution identifier; taking screen shots of the screens identified by each of the acquired screen identifiers by an image taking means; and saving all the screen shots taken by the image taking means in a memory unit to leave a record of the results of the automatic operation so that a person can later check the error occurrence situation in a state where the error occurrence situation can be understood in relation to the context of the screens, wherein in the saving step, the screen shots are saved by assigning numbers indicating the order in which they were taken, the execution identifier is a process of the target program and the screen identifier is a window handle of the screen, and in the taking step, the window handle of the screen to be photographed is specified, and screen shots of each layer that do not overlap with other screens are taken, starting from the bottommost layer screen.

Furthermore, a program according to the present invention is a program for taking screen shots of all of a plurality of screens provided by a target program, which is an application program to be operated, to be executed by an information processing device having a control unit, the program comprising: a step of acquiring an execution identifier of the target program being operated to be executed by the control unit; a step of acquiring screen identifiers of all screens provided by the target program identified by the acquired execution identifier; a step of taking screen shots of the screens identified by each of all the acquired screen identifiers by an image capturing means ; and a saving step of saving all the screen shots taken by the image capturing means in a memory unit to leave a record of the results of the automatic operation so that a person can later check the error occurrence situation in a state where the error occurrence situation can be understood in relation to the context of the screens, wherein in the saving step, the screen shots are saved by assigning numbers indicating the order in which they were taken, the execution identifier is a process of the target program, and the screen identifier is a window handle of the screen, and in the capturing step, the window handle of the screen to be captured is specified, and screen shots of each layer that do not overlap with other screens are taken, starting from the bottommost layer screen.

The present invention has the advantage that it is possible to take a screen shot that includes areas where the target screens overlap and are hidden.

FIG. 1 is a diagram showing an example of a problem that occurs when taking a screen shot of a screen to be operated. FIG. 2 is a diagram showing a mechanism for dividing and photographing only the screen related to the automatic operation target. FIG. 3 is a diagram showing a mechanism for capturing an image of the entire screen to be operated, taking into account vertical and horizontal scrolling. FIG. 4 is a diagram showing a mechanism for capturing images of the entire screen to be operated, taking into consideration scrolling and page movement. FIG. 5 is a diagram showing an example of the configuration of the screen shot photographing device 100. As shown in FIG. FIG. 6 is a diagram showing an example of a flowchart relating to the screen shot photographing process executed by the screen shot photographing device 100 when only the screen related to the automatic operation target is divided and photographed in its entirety. FIG. 7 is a diagram showing an example of a flowchart relating to a screen shot capturing process executed by a screen shot capturing device 100 whose OS is Windows 10 (registered trademark) when only the screen related to the automatic operation target is divided and captured in its entirety. FIG. 8 is a diagram showing an example of a flowchart relating to the screen shot photographing process executed by the screen shot photographing device 100 when the screen is divided and photographed in its entirety, taking into consideration screen scrolling and page movement. FIG. 9 is a diagram showing an example of a screen shot storage image when the screen is divided and all of the screen is photographed, taking into consideration screen scrolling and page movement.

The following describes in detail an embodiment of a screen shot shooting device, a screen shot shooting method, and a program according to the present invention with reference to the drawings. Note that the present invention is not limited to the present embodiment.

[1. Overview]
Here, an overview of this embodiment will be described with reference to FIG. 1 to FIG. 4 and the like.

During RPA execution, there are various situations where you may want to take a screen shot. Examples of these situations include "after inputting important items in the screen to be operated" (Scene Example 1), "immediately before confirmation processing after inputting all items in the screen to be operated" (Scene Example 2), "immediately after confirmation processing after inputting all items in the screen to be operated" (Scene Example 3), "after the screen to be operated changes" (Scene Example 4), and "immediately before automatic operation stops due to an error" (Scene Example 5). Note that the purpose of taking a screen shot in Scene Example 1 is, for example, to check whether values have been input into important items. Also, the purpose of taking a screen shot in Scene Example 2 is, for example, to check whether all input information is correct. Also, the purpose of taking a screen shot in Scene Example 3 is, for example, to check whether confirmation processing has been performed appropriately. Also, the purpose of taking a screen shot in Scene Example 4 is, for example, to check whether the screen has switched appropriately. Also, the purpose of taking a screen shot in Scene Example 5 is, for example, to identify the cause of the operation stopping.

However, up until now, screenshots of the target screen have been taken without considering the state of the target screen. As a result, as shown in Figure 1, for example, issues have arisen such as "message dialogs or other screens overlap, making it impossible to confirm the input item you want to check from the screenshot" (Example Problem 1) and "screen items are sometimes hidden by scrolling or moving pages within the screen, making it impossible to confirm the input item you want to check from the screenshot" (Example Problem 2). In other words, it was sometimes impossible to confirm the screen item you wanted to check from the screenshot. Figure 1 shows an example of an issue that arises when taking a screenshot of the target screen.

Therefore, as shown in Figure 2, we developed a system that identifies only the screen to be operated and its related screens, splits them up one by one, and takes screenshots of them. With this system, screens that are not related to the screen to be operated are not captured, so no unnecessary screenshots are taken. Also, with this system, screenshots are taken in order starting from the bottommost overlapping screen, so the chronological relationship of the screens can be understood. Figure 2 shows the system that splits up and captures only the screens related to the automatic operation target.

As shown in Figure 3, we have also developed a mechanism based on the above mechanism that recognizes the vertical and horizontal scroll bars of the target screen and scrolls the screen while dividing it into individual shots and taking them all at once. With this mechanism, the screen shots are rearranged and displayed taking into account the vertical and horizontal scrolling, so the relative positions of each screen shot can be understood at a glance. Figure 3 shows the mechanism that divides the target screen and takes all the shots while taking into account the vertical and horizontal scrolling.

Furthermore, as shown in Figure 4, based on the above mechanism, we have also developed a mechanism that recognizes the scroll bars and page navigation buttons on the target screen, automatically changes the screen, and divides it into individual shots to capture them. With this mechanism, the screen shots are rearranged and displayed taking into account scrolling and page navigation, so item information can be understood at a glance. Figure 4 shows the mechanism that divides the target screen and captures all of it, taking into account scrolling and page navigation.

2. Configuration
Here, an example of the configuration of the screen shot photographing device 100 according to this embodiment will be described with reference to FIG.

Figure 5 is a block diagram showing an example of the configuration of the screen shot capture device 100. The screen shot capture device 100 captures screen shots of all screens (the screens provided by the target program may be single or multiple) provided by a target program, which is an application program (e.g., a business application program) that is the target of an operation (e.g., an automatic operation by an RPA tool or an operation by a user related to taking a screen shot), and is, for example, a program in which a business application program and an RPA tool that are the target of an automatic operation by an RPA tool have been introduced.

The screen shot photographing device 100 is a commercially available desktop personal computer. Note that the screen shot photographing device 100 is not limited to a stationary information processing device such as a desktop personal computer, but may be a portable information processing device such as a commercially available notebook personal computer, a PDA (Personal Digital Assistant), a smartphone, or a tablet personal computer.

The screen shot photographing device 100 includes a control unit 102, a communication interface unit 104, a storage unit 106, and an input/output interface unit 108. Each unit included in the screen shot photographing device 100 is connected to each other so as to be able to communicate with each other via any communication path.

The communication interface unit 104 communicatively connects the information processing device 100 to the network 300 via a communication device such as a router and a wired or wireless communication line such as a dedicated line. The communication interface unit 104 has a function of communicating data with other devices via the communication line. Here, the network 300 has a function of connecting the screen shot photographing device 100 and the server 200 so that they can communicate with each other, and is, for example, the Internet or a LAN (Local Area Network). The data stored in the memory unit 106 may be stored in the server device 200, for example.

An input device 112 and an output device 114 are connected to the input/output interface unit 108. The output device 114 may be a monitor (including a home television), a speaker, or a printer. The input device 112 may be a keyboard, a mouse, a microphone, or a monitor that works with a mouse to provide a pointing device function. In the following, the output device 114 may be referred to as the monitor 114, and the input device 112 may be referred to as the keyboard 112 or the mouse 112.

The storage unit 106 stores various databases, tables, files, and the like. The storage unit 106 records computer programs that work in cooperation with the OS (Operating System) to give instructions to the CPU (Central Processing Unit) to perform various processes. The storage unit 106 can be, for example, a memory device such as a RAM (Random Access Memory) or a ROM (Read Only Memory), a fixed disk device such as a hard disk, a flexible disk, an optical disk, or the like.

The storage unit 106 includes, for example, master data 106a, and a screen shot storage unit 106b in which captured screen shots are stored. The master data 106a stores a screen identifier (e.g., a screen ID) and one or more control identifiers (e.g., control IDs) for vertical or horizontal scrolling and/or a control identifier (e.g., control ID) for page movement, in association with each other.

Returning to FIG. 5, the control unit 102 is a CPU or the like that performs overall control of the screen shot photographing device 100. The control unit 102 has an internal memory for storing control programs such as an OS (e.g., Windows (registered trademark), macOS (registered trademark), UNIX (registered trademark) (including Linux (registered trademark), etc.), Android (registered trademark), iOS (registered trademark), etc.), programs that define various processing procedures, etc., required data, etc., and executes various information processing based on these stored programs.

The control unit 102 conceptually includes, functionally, an execution identifier acquisition unit 102a, a screen identifier acquisition unit 102b, a photographing unit 102c, and a save execution unit 102d.

The execution identifier acquisition unit 102a acquires the execution identifier (e.g., execution ID) of the target program during operation execution. For example, when the OS is Windows 10 (registered trademark), the execution identifier acquisition unit 102a acquires the process (ID that can uniquely identify the execution of the program) of the target program during operation execution.

The screen identifier acquisition unit 102b acquires screen identifiers (e.g., screen IDs) of all screens provided by the target program identified by the execution identifier acquired by the execution identifier acquisition unit 102a. For example, if the OS is Windows 10 (registered trademark), the screen identifier acquisition unit 102b acquires window handles (IDs that can uniquely identify a screen in order to recognize the screen or control) of all screens provided by the target program identified by the process acquired by the execution identifier acquisition unit 102a.

The image capturing unit 102c captures screen shots of the screens identified by all the screen identifiers acquired by the screen identifier acquisition unit 102b. The image capturing unit 102c may capture screen shots in order from the bottom screen. For example, when the OS is Windows 10 (registered trademark), the image capturing unit 102c specifies the window handle of the screen to be captured, and captures screen shots in order from the bottom screen.

Furthermore, the photographing unit 102c may execute the following processes (1) to (3) for all screen identifiers acquired by the screen identifier acquiring unit 102b.
(1) A control identifier (e.g., control ID) associated with the screen identifier is obtained from the master data 106a.
(2) If the control identifier cannot be obtained, a screenshot of the screen identified by the screen identifier is taken.
(3) If a control identifier is obtained, (31) obtains scrolling information or page movement information from each control identifier, (32) sets the scrolling and/or page movement state of the screen identified by the screen identifier to an initial state according to each piece of obtained information, and takes screen shots, and (33) performs scrolling and/or page movement processing according to each piece of obtained information on the screen identified by the screen identifier, and takes screen shots until the processing can no longer be performed.

The image capturing unit 102c may capture screen shots while performing the processes of scrolling and page movement in the process of (33). The image capturing unit 102c may perform the scroll processes of vertical scrolling and horizontal scrolling in the process of (33). The scroll process may use the size of the scroll box as the unit of movement of the scroll box.

The save execution unit 102d saves all the screen shots taken by the image capture unit 102c in the screen shot storage unit 106b.

The save execution unit 102d may also save the screen shots by assigning them numbers indicating the order in which they were taken.

The save execution unit 102d may also arrange and save the screen shots on one sheet based on the scrolling information and/or page navigation information saved when the screen shots were taken.

3. Specific examples of processing
Here, a specific example of the process executed by the screen shot photographing device 100 will be described with reference to Figs. 6 to 9 and the like.

[3-1. Specific example 1]
FIG. 6 is a diagram showing an example of a flowchart relating to the screen shot photographing process executed by the screen shot photographing device 100 when only the screen related to the automatic operation target is divided and photographed in its entirety.

First, the execution identifier acquisition unit 102a acquires the execution ID of the program to be operated (step SA1).

Next, the screen identifier acquisition unit 102b acquires all screen IDs of the program identified by the execution ID acquired in step SA1 (step SA2).

Next, the image capture unit 102c captures screen shots of the screens identified by the screen IDs in ascending order of the screen IDs based on all the screen IDs acquired in step SA2 (step SA3).

Next, the save execution unit 102d assigns the order of capture to the file names of all the screen shots taken in step SA3, and saves the screen shots after assignment in the screen shot storage unit 106b (step SA4).

[3-2. Specific example 2]
FIG. 7 is a diagram showing an example of a flowchart relating to a screen shot capturing process executed by a screen shot capturing device 100 whose OS is Windows 10 (registered trademark) when dividing and capturing only the screen related to the automatic operation target.

First, the execution identifier acquisition unit 102a acquires the process of the program to be operated (step SB1).

Next, the screen identifier acquisition unit 102b acquires all window handles of the screens to be operated by the program identified by the process acquired in step SB1 (step SB2).

Next, the image capture unit 102c specifies the window handle of the screen to be captured from among the window handles obtained in step SB2 (step SB3).

Next, the image capture unit 102c takes screen shots of the screen to be captured, starting from the bottom layer, based on the window handle specified in step SB3 (step SB4).

Next, the save execution unit 102d assigns the order of capture to the file names of all the screen shots taken in step SB4, and saves the screen shots after assignment in the screen shot storage unit 106b (step SB5).

[3-3. Specific example 3]
Fig. 8 is a diagram showing an example of a flowchart relating to a screen shot photographing process executed by the screen shot photographing device 100 when a screen is divided and photographed in its entirety, taking into consideration screen scrolling and page movement. Note that this flowchart is a specific example of step SA3 in the flowchart shown in Fig. 6, and therefore a description of the processes corresponding to steps SA1 and SA2 shown in Fig. 6 will be omitted.

First, the image capture unit 102c obtains the control ID associated with the obtained screen ID from the master data 106a (step SC1).

Next, the image capture unit 102c acquires information on scrolling and page movement of the screen to be operated from the control ID acquired in step SC1 (step SC2).

Next, the image capture unit 102c moves the scroll and page movement to the initial position and captures a screen shot after the movement (step SC3).

Next, the image capture unit 102c determines whether page navigation is possible for the screen to be operated (step SC4).

Next, if page movement is possible, the image capture unit 102c determines whether horizontal scrolling is possible on the operation target screen (step SC5). Specifically, the image capture unit 102c determines whether horizontal scrolling is possible based on the size of the scroll area and the size of the scroll box.

Next, if horizontal scrolling is possible, the image capture unit 102c determines whether vertical scrolling is possible for the operation target screen (step SC6). Specifically, the image capture unit 102c determines whether vertical scrolling is possible based on the size of the scroll area and the size of the scroll box.

Next, if vertical scrolling is possible, the image capturing unit 102c executes vertical scrolling processing and captures a screen shot of the screen to be operated (step SC7). Specifically, the image capturing unit 102c moves the scroll position by the size of the vertical scroll box and captures a screen shot of the screen to be operated. After capturing the image, the process returns to step SC6.

Steps SC6 and SC7 are repeated until vertical scrolling is no longer possible. Specifically, they are repeated until the scroll position of the vertical scroll box moves to the final position. This results in screen shots taken while fixing the page and horizontal scrolling and only performing vertical scrolling (specifically, each screen shot (z = 0, 1, ...) corresponding to (0, 0, z) shown in Figure 9).

Next, the image capturing unit 102c executes a horizontal scroll process and captures a screen shot of the screen to be operated (step SC8). Specifically, the image capturing unit 102c moves the scroll position by the size of the horizontal scroll box and captures a screen shot of the screen to be operated. After capturing the screen shot, the process returns to step SC5.

Steps SC5 to SC8 are repeated until horizontal scrolling is no longer possible. Specifically, they are repeated until the scroll position of the horizontal scroll box moves to the final position. This results in screen shots taken while the page is fixed and horizontal scrolling is further performed (specifically, each screen shot corresponding to (0, y, z) shown in Figure 9 (y = 0, 1, 2, ...; z = 0, 1, ...). Steps SC5 to SC8 are also performed the same number of times as there are scroll bars.

Next, the image capturing unit 102c executes a page moving process and captures a screen shot of the screen to be operated (step SC9). Specifically, the image capturing unit 102c moves the page by one page and captures a screen shot of the screen to be operated. After capturing the screen shot, the process returns to step SC4.

Steps SC4 to SC9 are repeated until page navigation is no longer possible. Specifically, they are repeated until the page moves to the final page. This allows screen shots taken while page navigation is still being performed (specifically, each screen shot corresponding to (x, y, z) shown in FIG. 9 (x=0, 1, ...; y=0, 1, 2, ...; z=0, 1, ...) to be obtained.

Then, the save execution unit 102d arranges the screen shots on one sheet as shown in FIG. 9 based on the information on scrolling and page movement saved when the screen shots were taken. When page movement and horizontal scrolling are applied, the save execution unit 102d arranges the screen shots in order horizontally as shown in FIG. 9. When vertical scrolling is applied, the save execution unit 102d arranges the screen shots in order vertically as shown in FIG. 9. The save execution unit 102d then saves this sheet in the screen shot storage unit 106b. FIG. 9 is a diagram showing an example of a screen shot saving image when the screen is divided and all of the screen shots are captured, taking into account screen scrolling and page movement.

[4. Summary of the present embodiment]
As described above, according to this embodiment, when taking a screen shot, it is possible to take a picture including an area that would normally be hidden. Specifically, it is possible to take a picture of an item that is hidden by overlapping other screens, or to take a picture of an item outside the scroll display area. This makes it possible to grasp all items on the screen, and makes the screen shot more meaningful as a record of the results of automatic operations. Furthermore, the screen shot makes it possible to more efficiently investigate the cause of an error.

Until now, it has been desirable to understand the status of automated operations by taking screenshots of the screen during RPA operations, but in normal screen shots it was sometimes not possible to capture all items, making it impossible to confirm the items one wanted to check. However, according to this embodiment, it is possible to capture all items related to the target of automated operation without omission, so it is now possible to reliably confirm the items one wants to check from the screen shot. Furthermore, according to this embodiment, it can be used not only during RPA operations, but also when a user takes a screenshot of an application screen, so it is possible to capture all screen items without omission.

5. Other embodiments
The present invention may be embodied in various different embodiments other than those described above within the scope of the technical concept set forth in the claims.

For example, among the processes described in the embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method.

In addition, the processing procedures, control procedures, specific names, registered data for each process, information including parameters such as search conditions, screen examples, and database configurations shown in this specification and drawings may be changed as desired unless otherwise specified.

Furthermore, with regard to the screen shot capture device 100, each component shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure.

For example, the processing functions of the screen shot photographing device 100, particularly the processing functions performed by the control unit, may be realized in whole or in part by a CPU and a program interpreted and executed by the CPU, or may be realized as hardware using wired logic. The program is recorded on a non-transient computer-readable recording medium that includes programmed instructions for causing the information processing device to execute the processes described in this embodiment, and is mechanically read by the screen shot photographing device 100 as necessary. That is, a computer program for giving instructions to the CPU in cooperation with the OS and performing various processes is recorded in a storage unit such as a ROM or HDD (Hard Disk Drive). This computer program is executed by being loaded into RAM, and cooperates with the CPU to form the control unit.

This computer program may also be stored in an application program server connected to the screen shot capture device 100 via any network, and all or part of it may be downloaded as needed.

In addition, the program for executing the processing described in this embodiment may be stored on a non-transitory computer-readable recording medium, and may also be configured as a program product. Here, the term "recording medium" refers to a memory card, a USB (Universal Serial Bus) memory, a SD (Secure Digital) card, a flexible disk, a magneto-optical disk, a ROM, an EPROM (Erasable Programmable Read Only Memory), an EEPROM (registered trademark) (Electrically Erasable and Programmable Read Only Memory), a CD-ROM (Compact Disk Read Only Memory), an MO (Magneto-Optical disk), a DVD (Digital Versatile This includes any "portable physical media" such as a Blu-ray (registered trademark) Disc, and a Blu-ray (registered trademark) Disc.

A "program" is a data processing method written in any language or description method, and may be in any format, such as source code or binary code. Note that a "program" is not necessarily limited to a single configuration, but also includes a distributed configuration consisting of multiple modules or libraries, and a program that works in conjunction with a separate program, such as an OS, to achieve its function. Note that the specific configuration and reading procedure for reading a recording medium in each device shown in the embodiments, as well as the installation procedure after reading, can use well-known configurations and procedures.

The various databases stored in the memory unit are storage devices such as RAM, ROM, etc., fixed disk devices such as hard disks, flexible disks, and optical disks, and store various programs, tables, databases, and web page files used for various processes and providing websites.

The screen shot photographing device 100 may be configured as an information processing device such as a known personal computer or workstation, or may be configured as the information processing device to which any peripheral device is connected. The screen shot photographing device 100 may be realized by implementing software (including programs or data, etc.) that causes the device to realize the processing described in this embodiment.

Furthermore, the specific form of distribution and integration of the devices is not limited to that shown in the figures, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various additions or functional loads. In other words, the above-mentioned embodiments can be implemented in any combination, or the embodiments can be implemented selectively.

The present invention is particularly useful for companies that have adopted automated operation of their business systems using RPA tools.

100 Screen shot photographing device 102 Control unit
102a Execution identifier acquisition unit
102b Screen identifier acquisition unit
102c Photography section
102d storage execution unit 104 communication interface unit 106 storage unit
106a Master Data
106b Screen shot storage unit 108 Input/output interface unit 112 Input device 114 Output device 200 Server 300 Network

Claims (10)

A screen shot photographing device including a control unit that photographs all of a plurality of screens provided by a target program that is an application program to be operated, the screen shot photographing device comprising:
The control unit is
an execution identifier acquisition means for acquiring an execution identifier of a target program being operated;
a screen identifier acquiring means for acquiring screen identifiers of all screens provided by a target program identified by the execution identifier acquired by the execution identifier acquiring means;
a photographing means for photographing a screen shot of a screen identified by each of all the screen identifiers acquired by the screen identifier acquiring means;
a storage and execution means for storing all screen shots taken by the photographing means in a storage unit so that a person can later confirm the error occurrence state in a state where the error occurrence state can be understood in the context of the screen , and for leaving a record of the automatic operation result;
The storage execution means stores the screen shots by assigning numbers to the screen shots indicating the order in which the screen shots were taken;
the execution identifier is a process of the target program,
the screen identifier is a window handle for the screen;
said photographing means designates a window handle of a screen to be photographed, and photographs screen shots of each layer, starting from the bottom screen, without overlapping with other screens;
A screen shot taking device comprising: Master data in which a screen identifier and one or more control identifiers for vertical or horizontal scrolling and/or a control identifier for page movement are associated with each other and stored is accessible;
the imaging means executes the following processes (1) to (3) for all screen identifiers acquired by the screen identifier acquisition means;
(1) From the master data, obtain the control identifier associated with the screen identifier:
(2) If the control identifier cannot be obtained, take a screenshot of the screen identified by the screen identifier:
(3) If the control identifiers are obtained, (31) obtain scroll information or page movement information from each control identifier, (32) set the scroll and/or page movement state of the screen identified by the screen identifier to an initial state according to each piece of obtained information, and take screen shots, and (33) take screen shots while performing a scroll and/or page movement process according to each piece of obtained information on the screen identified by the screen identifier, until the process can no longer be performed.
2. The screen shot photographing device according to claim 1, The photographing means photographs a screen shot while performing each process in the process of (33) in the order of a scroll process and a page move process;
3. The screen shot photographing device according to claim 2, The photographing means, in the process of (33), executes the scroll process in the order of vertical scroll and horizontal scroll;
4. The screen shot photographing device according to claim 3, The scroll process uses the size of the scroll box as a unit of movement of the scroll box;
5. The screen shot photographing device according to claim 2, wherein: the save execution means saves the screen shots arranged on one sheet based on scroll information and/or page movement information saved when the screen shots were taken;
6. The screen shot photographing device according to claim 2, wherein: The application program is a business application program;
7. The screen shot photographing device according to claim 1 , The operation is an automatic operation by a Robotic Process Automation (RPA) tool or an operation by a user to take a screen shot;
8. The screen shot photographing device according to claim 1, wherein : A screen shot capturing method for capturing screen shots of all of a plurality of screens provided by a target program, which is an application program to be operated and executed in an information processing device having a control unit, comprising:
Executed by the control unit,
obtaining an execution identifier of a target program during which an operation is being performed;
acquiring screen identifiers of all screens provided by the target program identified by the acquired execution identifier;
taking, by an imaging means , screen shots of the screens identified by each of all the acquired screen identifiers;
a storage step of storing all the screen shots taken by the photographing means in a storage unit so that a person can later confirm the error occurrence situation in a state where the error occurrence situation can be understood in the context of the screen , and keeping a record of the automatic operation result;
In the saving step, the screen shots are saved by assigning numbers to the screen shots indicating the order in which they were taken;
the execution identifier is a process of the target program,
the screen identifier is a window handle for the screen;
In the step of taking the images, a window handle of a screen to be photographed is designated, and a screen shot of each layer that does not overlap with other screens is taken in order from the bottom screen;
A method for taking a screenshot, comprising: A program for taking screenshots of all of a plurality of screens provided by a target program, which is an application program to be operated, to be executed by an information processing device having a control unit,
To cause the control unit to execute
obtaining an execution identifier of a target program during which an operation is being performed;
acquiring screen identifiers of all screens provided by the target program identified by the acquired execution identifier;
taking, by an imaging means , screen shots of the screens identified by each of all the acquired screen identifiers;
a storage step of storing all the screen shots taken by the photographing means in a storage unit so that a person can later confirm the error occurrence situation in a state where the error occurrence situation can be understood in the context of the screen , and keeping a record of the automatic operation result;
In the saving step, the screen shots are saved by assigning numbers to the screen shots indicating the order in which they were taken;
the execution identifier is a process of the target program,
the screen identifier is a window handle for the screen;
In the step of taking the images, a window handle of a screen to be photographed is designated, and a screen shot of each layer that does not overlap with other screens is taken in order from the bottom screen;
A program characterized by.

Images