JP7433144B2 - Screen shot shooting device, screen shot shooting method and program - Google Patents

Description

The present invention relates to a screen shot photographing device, a screen shot photographing method, and a program.

As more and more companies are adopting automatic operation of business systems (application programs) using RPA (Robotic Process Automation) tools, the execution status of automatic operations on screens provided by business systems (in other words, RPA tools There is a growing need to understand various situations during execution for the purpose of collecting evidence. However, because the execution speed of automated operations is fast, it is difficult for humans to visually check the execution status, and moreover, it is difficult for people to check the execution status when the automated operations are executed. There is a risk that the significance of this matter will be reduced to mere mere tokens. Furthermore, although it is possible for a person to understand the execution status by checking a text log that describes the execution status, it is difficult for a person to intuitively understand the execution status from the text log alone. Therefore, in order to help people intuitively understand the execution status, screen shots of the screens targeted for automatic operation are taken.

Patent Document 1 discloses that each time the screen scrolls a predetermined distance, a screenshot is taken once based on the image change status of the screenshot reference area, and a final screenshot image is taken based on the image that has been screenshotted multiple times and the initial screenshot image. It describes how to take a screenshot of the device.

Special Publication No. 2019-533251

However, until now, a screen shot of a screen to be automatically operated (operation target screen) has been taken without considering the state of the operation target screen. Therefore, when multiple application programs are running, a screen shot is taken with the operation target screens overlapping each other, resulting in a problem in that hidden screen items cannot be confirmed.

The present invention has been made in view of the above, and the present invention is a screen shot photographing device, a screen shot photographing method, and a program capable of photographing screen shots including areas where operation target screens overlap and are hidden. The purpose is to provide

In order to solve the above-mentioned problems and achieve the purpose, a screen shot photographing device according to the present invention takes screen shots of all screens provided by a target program that is an application program to be operated. A screen shot photographing device comprising a control unit, wherein the control unit includes an execution identifier acquisition unit that acquires an execution identifier of a target program that is being operated, and a target that is specified by the execution identifier acquired by the execution identifier acquisition unit. a screen identifier acquisition means for acquiring screen identifiers of all screens provided by the program; a photographing means for photographing a screen shot of a screen specified by each of all the screen identifiers acquired by the screen identifier acquisition means; a storage execution means for storing all screen shots taken by the photographing means in a storage unit ; a screen identifier; one or more control identifiers for vertical scrolling or horizontal scrolling; and/or a control identifier for page movement. is able to access master data stored in association with the image capturing means, and the photographing means executes the following processes (1) to (3) for all the screen identifiers acquired by the screen identifier acquisition means. It is characterized by:
(1) Obtain the control identifier linked to the screen identifier from the master data.
(2) If the control identifier cannot be obtained, take a screen shot of the screen specified by the screen identifier.
(3) If the control identifier can be obtained, (31) obtain scroll information or page movement information from each control identifier, and (32) display the screen specified by the screen identifier according to each obtained information. (33) Set the scrolling and/or page movement state to the initial state, take a screen shot, and (33) scroll and/or page move the screen specified by the screen identifier according to the acquired information. While executing the process, take screen shots until the process can no longer be executed.

Furthermore, the screen shot photographing device according to the present invention is characterized in that a plurality of screens are provided by the target program.

Further, in the screen shot photographing device according to the present invention, the photographing means may photograph the screen shot while executing each process in the order of scrolling process and page movement process in the process (33). Features.

Further, in the screen shot photographing device according to the present invention, the photographing means is characterized in that, in the process (33), the scrolling process is performed in the order of vertical scrolling and horizontal scrolling.

Further, in the screen shot photographing device according to the present invention, the scrolling process uses the size of the scroll box as a unit of movement of the scroll box.

Further, in the screen shot photographing device according to the present invention, the storage execution means stores the screen shots arranged in one sheet based on scroll information and/or page movement information saved at the time of screen shot photographing; It is characterized by

Further, in the screen shot photographing device according to the present invention, the photographing means sequentially photographs screen shots starting from the lowest layer of the screen.

Further, in the screen shot photographing device according to the present invention, the storage execution means stores the screen shots by assigning a number indicating the photographing order to the screen shots.

Further, in the screen shot photographing device according to the present invention, the application program is a business application program.

Further, in the screen shot photographing device according to the present invention, the operation is an automatic operation using an RPA (Robotic Process Automation) tool or an operation related to taking a screen shot by a user.

Further, in the screen shot photographing device according to the present invention, the execution identifier is a process of the target program, the screen identifier is a window handle of the screen, and the photographing means is configured to detect a window handle of the target screen. The feature is that the screen shots are taken in order from the bottom screen when specified.

Further, the screen shot photographing method according to the present invention provides a method for taking screen shots of all screens provided by a target program that is an application program to be operated, which is executed by an information processing device including a control unit. A shot photographing method, which includes a step of acquiring an execution identifier of a target program being operated by the control unit, and all screens provided by the target program specified by the acquired execution identifier. The method includes a step of acquiring an identifier, a step of taking a screen shot of a screen specified by each of all the acquired screen identifiers, and a step of saving all the taken screen shots in a storage unit . , it is possible to access master data in which a screen identifier and one or more control identifiers for vertical scrolling or horizontal scrolling and/or control identifiers for page movement are stored in association with each other; The method is characterized in that the following processes (1) to (3) are executed for all screen identifiers acquired in the step of acquiring screen identifiers.
(1) Obtain the control identifier linked to the screen identifier from the master data.
(2) If the control identifier cannot be obtained, take a screen shot of the screen specified by the screen identifier.
(3) If the control identifier can be obtained, (31) obtain scroll information or page movement information from each control identifier, and (32) display the screen specified by the screen identifier according to each obtained information. (33) Set the scrolling and/or page movement state to the initial state, take a screen shot, and (33) scroll and/or page move the screen specified by the screen identifier according to the acquired information. While executing the process, take screen shots until the process can no longer be executed.

Further, the program according to the present invention is a program that takes screen shots of all screens provided by a target program, which is an application program to be operated, to be executed by an information processing apparatus including a control unit. and acquiring an execution identifier of the target program that is being operated to be executed by the control unit, and acquiring screen identifiers of all screens provided by the target program specified by the acquired execution identifier. the step of taking a screen shot of the screen specified by each of all the acquired screen identifiers, and the step of saving all the taken screen shots in a storage unit. , one or more control identifiers for vertical scrolling or horizontal scrolling and/or one or more control identifiers for page movement are stored in a linked manner, and the photographing step acquires the screen identifier. The method is characterized in that the following processes (1) to (3) are executed for all the screen identifiers acquired in the step of performing the following steps.
(1) Obtain the control identifier linked to the screen identifier from the master data.
(2) If the control identifier cannot be obtained, take a screen shot of the screen specified by the screen identifier.
(3) If the control identifier can be obtained, (31) obtain scroll information or page movement information from each control identifier, and (32) display the screen specified by the screen identifier according to each obtained information. (33) Set the scrolling and/or page movement state to the initial state, take a screen shot, and (33) scroll and/or page move the screen specified by the screen identifier according to the acquired information. While executing the process, take screen shots until the process can no longer be executed.

According to the present invention, it is possible to take a screen shot including a region where operation target screens overlap and are hidden.

FIG. 1 is a diagram illustrating an example of a problem that occurs when taking a screen shot of an operation target screen. FIG. 2 is a diagram showing a mechanism for dividing only the screen related to the automatic operation target and photographing the entire screen. FIG. 3 is a diagram showing a mechanism for photographing the entire operation target screen in consideration of vertical and horizontal scrolling. FIG. 4 is a diagram showing a mechanism for photographing the entire operation target screen in consideration of scrolling and page movement. FIG. 5 is a diagram showing an example of the configuration of the screen shot photographing device 100. FIG. 6 is a diagram illustrating an example of a flowchart regarding a screen shot photographing process executed by the screen shot photographing device 100 when dividing only the screen related to the automatic operation target and photographing the entire screen. FIG. 7 is a diagram illustrating an example of a flowchart regarding a screen shot photographing process executed by the screen shot photographing device 100 whose OS is Windows 10 (registered trademark) when dividing only the screen related to the automatic operation target and photographing the entire screen. It is. FIG. 8 is a diagram illustrating an example of a flowchart regarding a screen shot photographing process executed by the screen shot photographing device 100 when dividing the screen and photographing the entire screen in consideration of screen scrolling and page movement. FIG. 9 is a diagram illustrating an example of a saved screen shot image when the screen is divided and all images are taken in consideration of screen scrolling and page movement.

DESCRIPTION OF EMBODIMENTS Below, embodiments of a screen shot photographing device, a screen shot photographing method, and a program according to the present invention will be described in detail based on the drawings. Note that the present invention is not limited to this embodiment.

[1. overview]
Here, an overview of this embodiment will be explained with reference to FIGS. 1 to 4 and the like.

During RPA execution, there are various situations in which you may want to take screen shots. Examples of this situation include ``After inputting important items on the operation target screen'' (Example scene 1), ``Immediately before confirmation processing after inputting to all items on the operation target screen'' (Example scene 2), ``Operation "Immediately after confirmation processing after inputting all items on the target screen" (Scene example 3), "After transition of the operation target screen" (Scene example 4), and "Just before automatic operation stops due to an error" (Scene example 5) Examples include. Note that the purpose of taking a screen shot in Scene Example 1 is, for example, to check whether values have been entered incorrectly in important items. Further, the purpose of taking a screen shot in scene example 2 is, for example, to check whether all input information is correct. Further, the purpose of taking a screen shot in scene example 3 is, for example, to confirm whether the confirmation process has been appropriately performed. Further, the purpose of taking a screen shot in scene example 4 is, for example, to confirm whether the screen has been appropriately switched. Further, the purpose of taking a screen shot in scene example 5 is, for example, to identify the cause of the stoppage of the operation.

However, until now, screen shots of the operation target screen have been taken without considering the state of the operation target screen. As a result, as shown in Figure 1, for example, issues such as ``message dialogs and other screens overlap, making it impossible to confirm the input items you want to confirm from the screen shot'' (Example Issue 1), and ``scrolling and Screen items may be hidden when the page is moved, and the input item you want to confirm cannot be confirmed from the screen shot.'' (Example issue 2). In other words, sometimes it was not possible to check the screen item you wanted to check from the screen shot. FIG. 1 is a diagram illustrating an example of a problem that occurs when taking a screen shot of an operation target screen.

Therefore, as shown in Figure 2, we developed a system that specifies only the screen to be operated and its related screens, and separates them one by one to take screen shots. With this mechanism, screens that are not related to the screen to be operated are not photographed, so unnecessary screen shots are not taken. Additionally, with this system, screen shots are taken in order from the lowest overlapped screen, so you can understand the context of the screens. FIG. 2 is a diagram showing a mechanism for dividing only the screen related to the automatic operation target and photographing the entire screen.

Additionally, as shown in Figure 3, we have also developed a system that recognizes the vertical and horizontal scroll bars on the target screen and takes screenshots by dividing the screen one by one while scrolling the screen, as shown in Figure 3. . With this system, the screen shots are rearranged and displayed with vertical and horizontal scrolling in mind, so you can understand the positional relationship of each screen shot at a glance. FIG. 3 is a diagram showing a mechanism for dividing the operation target screen and photographing the entire screen in consideration of vertical and horizontal scrolling.

Furthermore, as shown in Figure 4, a system based on the above mechanism recognizes the scroll bars and page movement buttons on the screen to be operated, and automatically changes the screen while taking screen shots one by one. ” was also developed. This system sorts and displays screen shots taking into account scrolling and page movement, so item information can be understood at a glance. FIG. 4 is a diagram showing a mechanism for dividing the operation target screen and photographing the entire screen in consideration of scrolling and page movement.

[2. composition]
Here, an example of the configuration of the screen shot photographing device 100 according to the present embodiment will be described with reference to FIG. 5 and the like.

FIG. 5 is a block diagram showing an example of the configuration of the screen shot photographing device 100. The screen shot capturing device 100 captures all information provided by a target program that is an application program (for example, a business application program) that is to be operated (for example, an automatic operation by an RPA tool or an operation related to capturing a screen shot by a user). This is to take a screen shot of the screen (the screen provided by the target program may be single or multiple screens). It is something that

Screen shot photographing device 100 is a commercially available desktop personal computer. Note that the screen shot capturing device 100 is not limited to a stationary information processing device such as a desktop personal computer, but also a portable computer such as a commercially available notebook personal computer, a PDA (Personal Digital Assistant), a smartphone, or a tablet personal computer. It may be a type information processing device.

The screen shot photographing device 100 includes a control section 102, a communication interface section 104, a storage section 106, and an input/output interface section 108. Each part included in the screen shot photographing device 100 is communicably connected via an arbitrary communication path.

The communication interface unit 104 communicatively connects the information processing device 100 to the network 300 via a communication device such as a router and a wired or wireless communication line such as a dedicated line. The communication interface unit 104 has a function of communicating data with other devices via a communication line. Here, the network 300 has a function of connecting the screen shot photographing device 100 and the server 200 so that they can communicate with each other, and is, for example, the Internet or a LAN (Local Area Network). Note that the data stored in the storage unit 106 may be stored in the server device 200, for example.

An input device 112 and an output device 114 are connected to the input/output interface unit 108 . As the output device 114, in addition to a monitor (including a home television), a speaker or a printer can be used. As the input device 112, in addition to a keyboard, a mouse, and a microphone, a monitor that cooperates with the mouse to realize a pointing device function can be used. Note that in the following description, the output device 114 may be referred to as a monitor 114, and the input device 112 may be referred to as a keyboard 112 or a mouse 112.

The storage unit 106 stores various databases, tables, files, and the like. The storage unit 106 stores a computer program for giving instructions to a CPU (Central Processing Unit) to perform various processes in cooperation with an OS (Operating System). As the storage unit 106, for example, a memory device such as a RAM (Random Access Memory) or a ROM (Read Only Memory), a fixed disk device such as a hard disk, a flexible disk, an optical disk, etc. can be used.

The storage unit 106 includes, for example, master data 106a and a screen shot storage unit 106b in which captured screen shots are stored. The master data 106a includes a screen identifier (e.g. screen ID), one or more control identifiers for vertical scrolling or horizontal scrolling (e.g. control ID), and/or control identifiers for page movement (e.g. control ID). are stored in a linked manner.

Returning to FIG. 5, the control unit 102 is a CPU or the like that centrally controls the screen shot photographing device 100. The control unit 102 uses an OS (for example, Windows (registered trademark), macOS (registered trademark), UNIX (registered trademark) (including Linux (registered trademark), etc.), Android (registered trademark), iOS (registered trademark), etc.), etc. It has an internal memory for storing control programs, programs defining various processing procedures, required data, etc., and executes various information processing based on these stored programs.

Functionally, the control unit 102 includes an execution identifier acquisition unit 102a, a screen identifier acquisition unit 102b, a photographing unit 102c, a storage execution unit 102d, and the like.

The execution identifier acquisition unit 102a acquires an execution identifier (eg, execution ID) of a target program that is currently being operated. For example, when the OS is Windows 10 (registered trademark), the execution identifier acquisition unit 102a acquires the process (ID that can uniquely identify the execution of the program) of the target program that is currently being operated.

The screen identifier acquisition unit 102b acquires screen identifiers (eg, screen IDs) of all screens provided by the target program specified by the execution identifier acquired by the execution identifier acquisition unit 102a. For example, when the OS is Windows 10 (registered trademark), the screen identifier acquisition unit 102b acquires window handles (recognizes screens and controls) of all screens provided by the target program specified by the process acquired by the execution identifier acquisition unit 102a. (ID) that can uniquely identify the screen.

The photographing unit 102c photographs a screen shot of the screen specified by each of all the screen identifiers acquired by the screen identifier acquiring unit 102b. Note that the photographing unit 102c may photograph screen shots in order from the bottom screen. For example, if the OS is Windows 10 (registered trademark), the window handle of the screen to be photographed is specified, and screen shots are taken in order from the bottom screen.

Furthermore, the imaging unit 102c may perform the following processes (1) to (3) on all screen identifiers acquired by the screen identifier acquisition unit 102b.
(1) Obtain a control identifier (eg, control ID) linked to the screen identifier from the master data 106a.
(2) If the control identifier cannot be obtained, take a screen shot of the screen specified by the screen identifier.
(3) If the control identifier can be obtained, (31) obtain scroll information or page movement information from each control identifier, and (32) display the screen specified by the screen identifier according to each obtained information. (33) Set the scrolling and/or page movement state to the initial state, take a screen shot, and (33) scroll and/or page move the screen specified by the screen identifier according to the acquired information. While executing the process, take screen shots until the process can no longer be executed.

Further, in the process (33), the photographing unit 102c may take a screen shot while executing each process in the order of scrolling process and page movement process. Further, in the process (33), the imaging unit 102c may perform the scroll process in the order of vertical scroll and horizontal scroll. Further, the scrolling process may be performed using the size of the scroll box as a unit of movement of the scroll box.

The storage execution unit 102d stores all screen shots taken by the imaging unit 102c in the screen shot storage unit 106b.

Note that the storage execution unit 102d may save the screen shot by assigning a number indicating the shooting order to the screen shot.

Further, the storage execution unit 102d may arrange and save the screen shots on one sheet based on the scroll information and/or page movement information saved when the screen shots were taken.

[3. Specific example of processing]
Here, a specific example of processing executed by the screen shot photographing device 100 will be described with reference to FIGS. 6 to 9 and the like.

[3-1. Specific example 1]
FIG. 6 is a diagram illustrating an example of a flowchart regarding a screen shot photographing process executed by the screen shot photographing device 100 when dividing only the screen related to the automatic operation target and photographing the entire screen.

First, the execution identifier acquisition unit 102a acquires the execution ID of the operation target program (step SA1).

Next, the screen identifier acquisition unit 102b acquires all the screen IDs of the programs specified by the execution ID acquired in step SA1 (step SA2).

Next, the photographing unit 102c photographs screen shots of the screens specified by the screen IDs in ascending order of the screen IDs based on all the screen IDs acquired in step SA2 (step SA3).

Next, the storage execution unit 102d adds the shooting order to the file names of all the screen shots taken in step SA3, and saves the screen shots after the addition in the screen shot storage unit 106b (step SA4).

[3-2. Specific example 2]
FIG. 7 is a diagram illustrating an example of a flowchart regarding a screen shot photographing process executed by the screen shot photographing device 100 whose OS is Windows 10 (registered trademark) when dividing only the screen related to the automatic operation target and photographing the entire screen. It is.

First, the execution identifier acquisition unit 102a acquires the process of the operation target program (step SB1).

Next, the screen identifier acquisition unit 102b acquires all the window handles of the operation target screen of the program specified by the process acquired in step SB1 (step SB2).

Next, the photographing unit 102c specifies the window handle of the screen to be photographed from among the window handles acquired in step SB2 (step SB3).

Next, the photographing unit 102c sequentially photographs screen shots of the screen to be photographed starting from the lowest layer based on the window handle specified in step SB3 (step SB4).

Next, the storage execution unit 102d adds the shooting order to the file names of all the screen shots taken in step SB4, and saves the screen shots after the addition in the screen shot storage unit 106b (step SB5).

[3-3. Specific example 3]
FIG. 8 is a diagram illustrating an example of a flowchart regarding a screen shot photographing process executed by the screen shot photographing device 100 when dividing the screen and photographing the entire screen in consideration of screen scrolling and page movement. Note that since this flowchart is a specific example of step SA3 in the flowchart shown in FIG. 6, explanations of processes corresponding to step SA1 and step SA2 shown in FIG. 6 will be omitted.

First, the photographing unit 102c acquires a control ID linked to the acquired screen ID from the master data 106a (step SC1).

Next, the photographing unit 102c acquires information on scrolling and page movement of the operation target screen from the control ID acquired in step SC1 (step SC2).

Next, the photographing unit 102c moves the scroll and page movement to the initial position, and photographs a screen shot after the movement (step SC3).

Next, the photographing unit 102c determines whether page movement is possible with respect to the operation target screen (step SC4).

Next, if page movement is possible, the photographing unit 102c determines whether horizontal scrolling is possible with respect to the operation target screen (step SC5). Specifically, the imaging unit 102c determines whether horizontal scrolling is possible based on the size of the scroll area and the size of the scroll box.

Next, if horizontal scrolling is possible, the photographing unit 102c determines whether vertical scrolling is possible for the operation target screen (step SC6). Specifically, the imaging unit 102c determines whether vertical scrolling is possible based on the size of the scroll area and the size of the scroll box.

Next, if vertical scrolling is possible, the photographing unit 102c executes vertical scrolling processing and photographs a screen shot of the operation target screen (step SC7). Specifically, the photographing unit 102c moves the scroll position by the size of the vertical scroll box and photographs a screen shot of the operation target screen. After photographing, the process returns to step SC6.

Steps SC6 and SC7 are repeated until vertical scrolling becomes impossible. Specifically, this is repeated until the scroll position of the vertical scroll box moves to the final position. As a result, screen shots taken while fixing the page and horizontal scrolling while only performing vertical scrolling (specifically, each screen shot corresponding to (0, 0, z) shown in FIG. ,...)) is obtained.

Next, the photographing unit 102c executes horizontal scroll processing and photographs a screen shot of the operation target screen (step SC8). Specifically, the photographing unit 102c moves the scroll position by the size of the horizontal scroll box and photographs a screen shot of the operation target screen. After photographing, the process returns to step SC5.

Steps SC5 to SC8 are repeated until horizontal scrolling becomes impossible. Specifically, this is repeated until the scroll position of the horizontal scroll box moves to the final position. As a result, screen shots taken while the page is fixed and horizontal scrolling is also performed can be obtained (specifically, each screen shot corresponding to (0, y, z) shown in FIG. 9 (y=0, 1,2,...;z=0,1,...)). Furthermore, steps SC5 to SC8 are performed as many times as there are scroll bars.

Next, the photographing unit 102c executes page movement processing and photographs a screen shot of the operation target screen (step SC9). Specifically, the photographing unit 102c moves the page by one page and photographs a screen shot of the operation target screen. After photographing, the process returns to step SC4.

Steps SC4 to SC9 are repeated until page movement becomes impossible. Specifically, this is repeated until the page moves to the final page. As a result, screen shots taken while moving the page (specifically, each screen shot corresponding to (x, y, z) shown in FIG. 9 (x = 0, 1, ...; y = 0,1,2,...;z=0,1,...) are obtained.

Then, the saving execution unit 102d arranges the screen shots on one sheet as shown in FIG. 9 based on the scroll and page movement information saved at the time of taking the screen shots. Note that when page movement and horizontal scrolling are applied, the storage execution unit 102d arranges the screen shots in order in the horizontal direction, as shown in FIG. Furthermore, when vertical scrolling is applied, the storage execution unit 102d arranges the screen shots in order in the vertical direction, as shown in FIG. Then, the storage execution unit 102d stores this sheet in the screen shot storage unit 106b. FIG. 9 is a diagram illustrating an example of a saved screen shot image when the screen is divided and all images are taken in consideration of screen scrolling and page movement.

[4. Summary of this embodiment]
As described above, according to the present embodiment, when taking a screen shot, it is possible to take a picture including an area that is originally hidden. Specifically, it is possible to take pictures of items that are hidden by overlapping other screens, and it is also possible to take pictures of items that are outside the scroll display area. This makes it possible to keep track of all the items on the screen, making screen shots more meaningful as records of automated operation results. Screen shots also make investigating the cause of errors more efficient.

Until now, users wanted to understand the automatic operation status by taking screen shots of RPA operations, but sometimes it was not possible to capture all the items with normal screen shots, so it was sometimes not possible to check the items that they wanted to check. However, according to the present embodiment, all items related to the automatic operation target can be taken without exception, so it is now possible to reliably check the items that the user wants to check from the screen shot. Further, according to the present embodiment, it can be used not only when operating the RPA but also when the user takes a screen shot of an application, so that screen items can be taken without omission.

[5. Other embodiments]
In addition to the embodiments described above, the present invention may be implemented in various different embodiments within the scope of the technical idea described in the claims.

For example, among the processes described in the embodiments, all or part of the processes described as being performed automatically can be performed manually, or all of the processes described as being performed manually can be performed manually. Alternatively, some of the steps can be performed automatically using known methods.

In addition, unless otherwise specified, information including processing procedures, control procedures, specific names, parameters such as registered data and search conditions for each process, screen examples, and database configurations shown in this specification and drawings are included. It can be changed arbitrarily.

Further, regarding the screen shot photographing device 100, each illustrated component is functionally conceptual, and does not necessarily need to be physically configured as illustrated.

For example, all or any part of the processing functions provided by the screen shot photographing device 100, particularly each processing function performed by the control unit, may be realized by a CPU and a program interpreted and executed by the CPU. Alternatively, it may be realized as hardware using wired logic. Note that the program is recorded on a non-temporary computer-readable recording medium that includes programmed instructions for causing the information processing device to execute the processing described in this embodiment. Mechanically read by device 100. That is, a storage unit such as a ROM or an HDD (Hard Disk Drive) stores a computer program that cooperates with an OS to give instructions to a CPU and perform various processes. This computer program is executed by being loaded into the RAM, and constitutes a control unit in cooperation with the CPU.

Further, this computer program may be stored in an application program server connected to the screen shot photographing device 100 via an arbitrary network, and it is also possible to download all or part of it as necessary. It is.

Further, a program for executing the processing described in this embodiment may be stored in a non-temporary computer-readable recording medium, or may be configured as a program product. Here, this "recording medium" includes a memory card, a USB (Universal Serial Bus) memory, an SD (Secure Digital) card, a flexible disk, a magneto-optical disk, a ROM, an EPROM (Erasable Programmable Read Only Memory), and an EEPR. OM (registration) Trademark) (Electrically Erasable and Programmable Read Only Memory), CD-ROM (Compact Disk Read Only Memory), MO (Magneto-Optical disk), DVDs (Digital Versatile Discs), Blu-ray (registered trademark) Discs, etc. shall include any “portable physical medium”.

Further, a "program" is a data processing method written in any language or writing method, and does not matter in the form of source code or binary code. Note that a "program" is not necessarily limited to a unitary structure, but may be distributed as multiple modules or libraries, or may work together with separate programs such as an OS to achieve its functions. Including things. Note that well-known configurations and procedures can be used for the specific configuration and reading procedure for reading the recording medium in each device shown in the embodiments, and the installation procedure after reading.

The various databases stored in the storage unit are storage means such as memory devices such as RAM and ROM, fixed disk devices such as hard disks, flexible disks, and optical disks, and are used for various processing and website provision. Stores programs, tables, databases, web page files, etc.

Further, the screen shot photographing device 100 may be configured as an information processing device such as a known personal computer or workstation, or may be configured as the information processing device to which any peripheral device is connected. Further, the screen shot photographing device 100 may be realized by installing software (including a program, data, etc.) that causes the device to realize the processing described in this embodiment.

Furthermore, the specific form of dispersion and integration of devices is not limited to what is shown in the diagram, and all or part of them can be functionally or physically divided into arbitrary units according to various additions or functional loads. It can be configured in a distributed/integrated manner. That is, the embodiments described above may be implemented in any combination, or the embodiments may be implemented selectively.

The present invention is particularly useful in companies that have adopted automatic operation of business systems using RPA tools.

100 Screen shot photographing device 102 Control unit
102a Execution identifier acquisition unit
102b Screen identifier acquisition unit
102c Photography Department
102d Storage execution unit 104 Communication interface unit 106 Storage unit
106a Master data
106b Screen shot storage unit 108 Input/output interface unit 112 Input device 114 Output device 200 Server 300 Network

Claims (13)

A screen shot capturing device that includes a control unit that captures screen shots of all screens provided by a target program that is an application program to be operated,
The control unit includes:
Execution identifier acquisition means for acquiring an execution identifier of a target program during operation execution;
screen identifier acquisition means for acquiring screen identifiers of all screens provided by the target program specified by the execution identifier acquired by the execution identifier acquisition means;
a photographing means for photographing a screen shot of a screen specified by each of all the screen identifiers acquired by the screen identifier acquisition means;
and storage execution means for storing all screen shots taken by the photographing means in a storage unit ,
It is possible to access master data in which a screen identifier and one or more control identifiers for vertical scrolling or horizontal scrolling and/or control identifiers for page movement are stored in association with each other,
The photographing means executes the following processes (1) to (3) on all screen identifiers acquired by the screen identifier acquisition means;
(1) Obtain the control identifier linked to the screen identifier from the master data:
(2) If the control identifier cannot be obtained, take a screen shot of the screen specified by the screen identifier:
(3) If the control identifier can be obtained, (31) obtain scroll information or page movement information from each control identifier, and (32) display the screen specified by the screen identifier according to each obtained information. (33) Set the scrolling and/or page movement state to the initial state, take a screen shot, and (33) scroll and/or page move the screen specified by the screen identifier according to the acquired information. Take screen shots while performing the process until the process can no longer be performed:
A screen shot shooting device featuring: The number of screens provided by the target program is plural;
The screen shot photographing device according to claim 1, characterized in that: In the process (33), the photographing means photographs a screen shot while executing each process in the order of a scroll process and a page movement process;
The screen shot photographing device according to claim 1 or claim 2, characterized in that: In the process (33), the photographing means executes the scrolling process in the order of vertical scrolling and horizontal scrolling;
The screen shot photographing device according to claim 1 or claim 2, characterized in that: The scrolling process uses the size of the scroll box as a unit of movement of the scroll box;
The screen shot photographing device according to any one of claims 1 to 4, characterized in that: The storage execution means arranges and stores the screen shots on one sheet based on scroll information and/or page movement information saved at the time of taking the screen shots;
The screen shot photographing device according to any one of claims 1 to 5, characterized in that: The photographing means sequentially photographs screen shots starting from the lowest screen;
The screen shot photographing device according to any one of claims 1 to 6, characterized in that: The storage execution means saves the screen shots by assigning a number indicating the shooting order to the screen shots;
The screen shot photographing device according to any one of claims 1 to 7, characterized in that: the application program is a business application program;
The screen shot photographing device according to any one of claims 1 to 8, characterized in that: The operation is an automatic operation using an RPA (Robotic Process Automation) tool or an operation related to taking a screen shot by the user;
The screen shot photographing device according to any one of claims 1 to 9, characterized in that: The execution identifier is a process of the target program,
the screen identifier is a window handle of the screen;
The photographing means specifies a window handle of a screen to be photographed and sequentially photographs screen shots starting from the lowest screen;
The screen shot photographing device according to any one of claims 1 to 10 . A screen shot shooting method for taking screen shots of all screens provided by a target program that is an application program to be operated, which is executed on an information processing device including a control unit, the method comprising:
executed by the control unit,
obtaining an execution identifier of a target program during execution of the operation;
acquiring screen identifiers of all screens provided by the target program identified by the acquired execution identifier;
taking a screen shot of the screen identified by each of all the acquired screen identifiers;
a step of saving all the captured screen shots in a storage unit,
It is possible to access master data in which a screen identifier and one or more control identifiers for vertical scrolling or horizontal scrolling and/or control identifiers for page movement are stored in association with each other,
The step of photographing includes performing the following processes (1) to (3) on all the screen identifiers obtained in the step of obtaining the screen identifiers;
(1) Obtain the control identifier linked to the screen identifier from the master data:
(2) If the control identifier cannot be obtained, take a screen shot of the screen specified by the screen identifier:
(3) If the control identifier can be obtained, (31) obtain scroll information or page movement information from each control identifier, and (32) display the screen specified by the screen identifier according to each obtained information. (33) Set the scrolling and/or page movement state to the initial state, take a screen shot, and (33) scroll and/or page move the screen specified by the screen identifier according to the acquired information. Take screen shots while performing the process until the process can no longer be performed:
A screen shot shooting method featuring: A program for taking screen shots of all screens provided by a target program, which is an application program to be operated, to be executed by an information processing device including a control unit,
for the control unit to execute,
obtaining an execution identifier of a target program during execution of the operation;
acquiring screen identifiers of all screens provided by the target program identified by the acquired execution identifier;
taking a screen shot of the screen identified by each of all the acquired screen identifiers;
a step of saving all the captured screen shots in a storage unit,
It is possible to access master data in which a screen identifier and one or more control identifiers for vertical scrolling or horizontal scrolling and/or control identifiers for page movement are stored in association with each other,
The step of photographing includes performing the following processes (1) to (3) on all the screen identifiers obtained in the step of obtaining the screen identifiers;
(1) Obtain the control identifier linked to the screen identifier from the master data:
(2) If the control identifier cannot be obtained, take a screen shot of the screen specified by the screen identifier:
(3) If the control identifier can be obtained, (31) obtain scroll information or page movement information from each control identifier, and (32) display the screen specified by the screen identifier according to each obtained information. (33) Set the scrolling and/or page movement state to the initial state, take a screen shot, and (33) scroll and/or page move the screen specified by the screen identifier according to the acquired information. Take screen shots while performing the process until the process can no longer be performed:
A program featuring.

Images