JP7845465B2 - Specific device, specific method, and specific program - Google Patents

Description

This invention relates to a specific device, a specific method, and a specific program.

Conventionally, operation logs, which record the actions taken by workers in terminal-based tasks, have been used for business analysis to understand actual work practices and for business support using machine learning (see, for example, Non-Patent Documents 1 and 2).

Furthermore, information in the operation log regarding the scope of operations performed by the operator on GUI components, etc., is useful for visualizing business workflows and designing machine learning systems.

For example, a known method for obtaining operation logs for the operation screen of a GUI application involves obtaining attribute values of GUI components that make up the operation screen when an event occurs, and then identifying changes before and after the event occurred (see, for example, Patent Document 1).

Japanese Patent Publication No. 2015-153210

Yokose et al., "Business Visualization Technology Contributing to DX Promotion," NTT Technical Journal, 2020 vol.32 No.2. M. Fukai, M. Tadokoro, and H. Oishi, "Study on Automating Decision-Making by Learning Optimal Processes from PC Work," Proc. The 22nd Asia-Pacific Network Operations and Management Symposium (APNOMS 2021), Sept. 2021.

However, conventional technologies have a problem in that it can sometimes be difficult to easily determine the area of operation on the screen.

For example, when performing terminal-based tasks using a variety of applications, it may be difficult to determine the scope of operation if GUI component information cannot be obtained for some applications.

Furthermore, in environments where only screen capture and mouse/keyboard operation information can be obtained, such as thin client environments, it is difficult to obtain information about GUI components and determine the scope of operation.

Another approach is to infer the location of the operation from the operation event information and captured images, and manually set the operation range. However, there are cases where it is difficult to infer the intended purpose of the operation, and manually setting the range for each operation would be extremely time-consuming.

The present invention has been made in view of the above, and aims to easily specify the operating range on the screen.

To solve the above-mentioned problems and achieve the objective, the specific device is characterized by having an acquisition unit that acquires operation events generated in response to operations on the screen of a terminal device and a captured image of the screen, and an identification unit that identifies the range targeted by the operation based on information regarding the color of the position corresponding to the operation event in the captured image.

According to the present invention, the operating range on the screen can be easily identified.

Figure 1 shows an example of the configuration of a specific device according to the first embodiment. Figure 2 shows an example of a method for determining the operating range. Figure 3 shows an example of a method for determining the operating range. Figure 4 shows an example of a method for determining the operating range. Figure 5 shows an example of a method for determining the operating range. Figure 6 is a flowchart showing an example of the data acquisition process. Figure 7 is a flowchart showing an example of the generation process flow. Figure 8 is a flowchart showing an example of the flow of a specific process. Figure 9 shows an example of a computer running a specific program.

The embodiments of the specific apparatus, method, and program related to this application will be described in detail below with reference to the drawings. However, the present invention is not limited to the embodiments described below.

For example, the tasks in the embodiment include inputting data into a terminal device such as a PC (Personal Computer). Furthermore, tasks using a terminal device are performed by operating input devices such as a mouse, keyboard, or touch panel.

The identifying device in this embodiment identifies the operating range on the screen of the terminal device. For example, the identifying device can output the identified operating range as an image.

Furthermore, the operating range identified by a specific device can be used, for example, in business analysis procedures such as process visualization and timeline visualization, as described in Non-Patent Document 1.

Furthermore, the operating range identified by a specific device may be used as training data for a model (e.g., a neural network) that automatically performs operations on a terminal device that mimic human input.

[Configuration of the first embodiment]
First, the configuration of the specific device according to the first embodiment will be described using Figure 1. Figure 1 is a diagram showing an example of the configuration of the specific device according to the first embodiment.

As shown in Figure 1, the specific device 10 is connected to the terminal device 20. The terminal device 20 is a device such as a PC used by the worker to perform their duties. Furthermore, the specific device 10 does not necessarily have to be directly connected to the terminal device 20; it is sufficient if it can obtain information regarding operations on the terminal device 20 by any means.

Here, we will describe the various parts of the specific device 10. As shown in Figure 1, the specific device 10 has a communication unit 11, a storage unit 12, and a control unit 13.

The communication unit 11 is an interface for data input and output. For example, the communication unit 11 is a NIC (Network Interface Card). The communication unit 11 can send and receive data with other devices.

The storage unit 12 is a storage device such as an HDD (Hard Disk Drive), SSD (Solid State Drive), or optical disc. Alternatively, the storage unit 12 may be a rewritable semiconductor memory such as RAM (Random Access Memory), flash memory, or NVSRAM (Non-Volatile Static Random Access Memory).

The memory unit 12 stores the OS (Operating System) and various programs executed on the specific device 10. The memory unit 12 also stores operation log information 121.

Operation log information 121 includes an operation log. The operation log is information that associates screen capture images with operation events. The method for generating the operation log will be described later.

The control unit 13 controls the entire specific device 10. The control unit 13 is, for example, an electronic circuit such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), or GPU (Graphics Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

Furthermore, the control unit 13 has an internal memory for storing programs and control data that define various processing procedures, and executes each process using the internal memory. The control unit 13 also functions as various processing units through the operation of various programs. For example, the control unit 13 includes an acquisition unit 131, a generation unit 132, and a specification unit 133.

The acquisition unit 131 acquires operation events generated in response to operations on the screen of the terminal device 20, as well as captured images of the screen. The acquisition unit 131 can acquire operation events and captured images via the communication unit 11.

The acquisition unit 131 acquires the operation event at the time it occurs. For example, the operation event is information related to mouse operations and keyboard operations. Mouse operations include mouse button operations and mouse wheel operations. Keyboard operations include key inputs.

Mouse operation events include information identifying the button or wheel that was operated, and the type of operation (click, press down, push up, scroll, etc.). Keyboard operation events include the type of key that was pressed.

Furthermore, the acquisition unit 131 acquires, along with the operation event, the time (timestamp) when the operation event occurred, and the position of the mouse cursor at that time.

Furthermore, regardless of the occurrence of an operation event, the acquisition unit 131 acquires a captured image of the screen, the time (timestamp) at which the captured image was acquired, and the position of the mouse cursor at that time from the terminal device 20 at regular time intervals.

Furthermore, if possible, the acquisition unit 131 may acquire detailed information such as window identification information along with the captured image.

The generation unit 132 generates an operation log based on the information acquired by the acquisition unit 131. The generation unit 132 compares the timestamp of the operation event with the timestamp of the captured image and associates the captured image acquired at the time closest to the time indicated by the operation event's timestamp with that operation event.

The generation unit 132 adds the associated operation event and captured image, along with the mouse cursor position corresponding to the operation event or captured image, to the operation log information 121 as an operation log.

The identification unit 133 identifies the range to be operated on (operation range) based on color information at the position corresponding to the operation event in the captured image. The color information is, for example, the pixel value of each pixel in the image. For example, the pixel value is expressed in the RGB color space or the HLS color space. Hereafter, unless otherwise specified, the pixel value will be expressed in the RGB color space.

Here, using Figures 2, 3, 4, and 5, the method for specifying the operating range by the specific unit 133 will be explained with concrete examples. Figures 2, 3, 4, and 5 are diagrams illustrating examples of methods for specifying the operating range. Note that the method for specifying the operating range is not limited to those illustrated here.

The rectangular areas in Figures 2, 3, 4, and 5 represent all or part of the captured image. Furthermore, while symbols, patterns, line styles, fonts, and enlarged views are added to each figure as appropriate, these are for illustrative purposes only and do not imply actual editing or processing of the captured image.

(First identification method: Identification method based on the mouse cursor position)
As shown in Figure 2, the identification unit 133 can identify the operating range based on the position of the mouse cursor (hereinafter referred to as the first identification method).

First, if the operation event of the specified target included in the operation log information 121 relates to mouse operation, the identification unit 133 records the pixel value of the position on the captured image corresponding to the position of the mouse cursor at the time the operation event occurred (step S11). For example, the identification unit 133 records the pixel value of the pixel at the tip of the mouse cursor.

Next, the identification unit 133, starting from the position of the mouse cursor where the pixel value was recorded, acquires an adjacent area (the area with the pattern in step S12) that is the same color as the recorded color (equal pixel value), and identifies that area as the operating range (within the thick-lined frame in step S13).

For example, the interior of a text box may be a single color (e.g., white (pixel value: (255,255,255)) while the border is a different color (e.g., white (pixel value: (0,0,0))). In such a case, if the tip of the mouse cursor is inside the text box, the identification unit 133 identifies the entire interior of the text box as the operation range.

In other words, in image editing software, the area to be filled when a fill operation is performed on a pixel that has recorded a pixel value is identified as the operation range by the identification unit 133.

In this case, if the specified unit 133 contains areas of different colors within the area, it includes those areas in the operation range. In the example in Figure 2, the specified unit 133 includes the area containing the string "Enter station name" inside the text box labeled "Departure station" in the operation range.

Furthermore, the identification unit 133 may identify an area of color similar to the recorded color as the operating range, rather than an area of the same color as the recorded color. For example, the identification unit 133 may consider an area to be within the operating range if its similarity to the recorded pixel value (e.g., the distance between vectors) is below a threshold.

In this way, the identification unit 133 can identify the range of the operation based on the color of the mouse cursor position when an operation event occurs. This makes it easy to identify the range of mouse operations.

(Second identification method: Identification method based on the mouse cursor position and adjacent coordinates)
As shown in Figure 3, the identification unit 133 can identify the operating range based on the position of the mouse cursor and adjacent coordinates (hereinafter referred to as the second identification method).

For example, if the manipulated GUI component is a button containing text, and the text is clicked, it may be difficult to accurately determine the exact range of the operation.

Therefore, if the operation range identified by executing the first identification method satisfies either the first or second condition below, the identification unit 133 records not only the coordinates of the clicked position but also the color of the coordinates surrounding those coordinates.
First condition: The width and height of the specified operating range are below the threshold.
The second condition is that the total number of pixels in the specified operating range is less than or equal to the threshold.

In the example shown in Figure 3, first, the identification unit 133 records the pixel value (117, 117, 117) of the position on the character "力" (step S21). Then, the identification unit 133 identifies the area with the pattern of the character "力" as the operating range using the first identification method (step S22). Note that the pixel value (117, 117, 117) is a color between white and black, and may be called gray.

The control area identified here is narrower in both height and width compared to the entire text box, and has a smaller total number of pixels. Therefore, the control area satisfies at least one of the first or second conditions.

Therefore, the identification unit 133 further performs the first identification method on pixels adjacent to the operation range identified in step S22, i.e., the parts of the text box where no characters exist, thereby identifying the entire interior of the text box as the operation range (step S23).

Furthermore, the vertical and horizontal thresholds in the first condition may be based on the thickness of the lines that make up the characters of the font used. Also, the total number of pixels in the second condition may be based on the average number of pixels occupied by the characters of the font used.

(Third identification method: Identification method based on pixel values around the changed area)
As shown in Figure 4, the identification unit 133 can identify the operating range based on the pixel values surrounding the change portion (hereinafter referred to as the third identification method).

Let's consider a case where the operation event relates to keyboard input, and the screen changes due to text input. For example, suppose that pressing the "y" key on the keyboard causes the screen to change from the state in step S31 of Figure 4 to the state in step S32.

At this point, the identification unit 133 records the pixel values for the coordinates around the area where the change occurred, and performs the first identification method on these pixel values. As a result, the identification unit 133 identifies the area inside the text box displaying the letter "y" as the operation range (step S33).

The area surrounding the changed region is the outer edge of the letter "y," and corresponds to the area where the text in the text box is not displayed.

Furthermore, the operator may move the focus to a text box by clicking and then enter text into that text box. Therefore, the specific unit 133 may set the operation range while the operator continues keyboard operation to be the same as the operation range acquired during the previous click operation.

In this way, the acquisition unit 131 acquires captured images at regular time intervals. Then, the identification unit 133 identifies the range to be operated on based on the parts of the captured image that have changed over time. In particular, the identification unit 133 identifies the range to be operated on based on the color surrounding the parts of the captured image that have changed over time. This makes it easy to identify the range of operations related to keyboard character input.

(4. Method for identifying the changeable portion as the operating range)
Using the third identification method, if the area is acquired based on the color of the coordinates around the changed part, it may not be possible to obtain an accurate range of operation.

For example, if the keyboard operation is a Tab key press, acquiring the coordinates around the change point will include areas outside the text box.

For example, if the Tab key is pressed in the state shown in step S41 of Figure 5, the cursor may appear inside the text box as shown in step S42. In this case, the coordinates around the cursor may be either within the border of the text box or outside the text box.

Therefore, the identification unit 133 defines the area on the screen where a change occurs when a key is pressed as the operation range (step S43). When the Tab key is pressed in the state of step S41, a cursor appears in the text box as shown in step S42, and the border of the text box where the cursor appears is highlighted as shown in step S43. At this time, the identification unit 133 identifies the area surrounded by the highlighted border as the area where a change has occurred, i.e., the operation range.

For example, the identification unit 133 may execute the fourth identification method when a specific type of key, such as the Tab key or the Enter key, is operated, and execute the third identification method when any other key is operated.

The identification unit 133 identifies a closed region that has changed over time in the captured image as the range to be operated on. This makes it possible to identify the operation range even when it is difficult to identify the operation range based solely on color information.

[Processing according to the first embodiment]
The processing flow of the specific device 10 will be explained using Figures 6, 7, and 8.

Figure 6 is a flowchart showing an example of the acquisition process flow. For example, the acquisition process is performed by the acquisition unit 131.

As shown in Figure 6, first, if the operator using the terminal device 20 has not stopped processing and the operator has not shut down the terminal device 20 (step S101, No.), the specific device 10 repeats steps S102 to S104. Note that shutting down the terminal device 20 means shutting down the terminal device 20 and turning off the power to the terminal device 20.

Furthermore, the specific device 10 terminates processing if the operator stops processing or if the operator drops the terminal device 20 (step S101, Yes).

In step S102, the specific device 10 temporarily saves captured images of the terminal device 20's screen at regular intervals.

Then, if an operation event occurs (step S103, Yes), the specific device 10 proceeds to step S104. If no operation event occurs (step S103, No), the specific device 10 terminates the process.

In step S104, the specific device 10 acquires an operation event from the terminal device 20. The operation event includes the event type (mouse click, keyboard key press, etc.), the time of occurrence, the mouse cursor position at the time of occurrence, etc.

Figure 7 is a flowchart showing an example of the generation process flow. For example, the generation process is performed by the generation unit 132.

As shown in Figure 7, the specific device 10 repeats steps S202 and S203 until it targets all operation events acquired in the acquisition process (step S201, No). When the specific device 10 has targeted all operation events (step S201, Yes), it terminates the process.

In step S202, the identification device 10 determines the target operation event. Then, the identification device 10 associates the determined operation event with the capture image acquired at the time closest to the time the operation event occurred (step S203). The identification device 10 stores the associated data as operation log information 121 in the storage unit 12.

Figure 8 is a flowchart showing an example of the flow of a specific process. For example, the specific process is performed by the specific unit 133.

As shown in Figure 8, the specific device 10 repeats the processing from step S302 onward until it targets all operation events acquired in the acquisition process (step S301, No). When the specific device 10 has targeted all operation events (step S301, Yes), it terminates the process.

In step S302, the specific device 10 determines the target operation event. If the determined operation event is a mouse operation (step S303, Yes), the specific device 10 proceeds to step S304; otherwise, it proceeds to step S310.

In step S304, the identification device 10 records the pixel value of the mouse cursor position at the time the operation event occurs. Then, the identification device 10 obtains a region with the same pixel value from the mouse cursor position at the time the operation event occurred (step S305). Step S305 corresponds to the first identification method.

Here, the specific device 10 determines whether the vertical and horizontal dimensions of the acquired area are greater than or equal to the standard (step S306). Furthermore, if the vertical and horizontal dimensions of the acquired area are greater than or equal to the standard (step S306, Yes), the specific device 10 determines whether the total number of pixels in the acquired area is greater than or equal to the standard (step S307).

In step S306, the first condition is checked. Then, in step S307, the second condition is checked.

If the vertical and horizontal dimensions of the acquired area are not equal to or greater than the standard (step S306, No.), or if the total number of pixels in the acquired area is not equal to or greater than the standard (step S307, No.), the specific device 10 proceeds to step S308.

In step S308, the identification device 10 obtains pixel values for coordinates adjacent to the area acquired in step S305, and acquires areas with the same pixel values. Steps S306, S307, and S308 correspond to the second identification method.

Furthermore, if the vertical and horizontal dimensions of the acquired area are greater than or equal to the standard (step S306, Yes), and the total number of pixels in the acquired area is greater than or equal to the standard (step S307, Yes), the specific device 10 identifies the area acquired in step S305 as the operating range (step S309).

In step S310, the identification device 10 acquires the region that has changed due to the operation event (step S310). If the operation is not text input (step S311, No), the identification device 10 proceeds to step S309. In this case, steps S310 and S309 correspond to the fourth identification method.

On the other hand, if the operation is text input (step S311, Yes), the identification device 10 acquires an area with the same pixel value based on the coordinates adjacent to the change (step S312), and proceeds to step S306. In this case, steps S310 and S312 correspond to the third identification method.

[Effects of the First Embodiment]
As explained above, the identification device 10 has an acquisition unit 131 and an identification unit 133. The acquisition unit 131 acquires operation events that occur in response to operations on the screen of the terminal device 20, and a captured image of the screen. The identification unit 133 identifies the range targeted by the operation based on information about the color of the position corresponding to the operation event in the captured image.

As a result, the specific device 10 does not need attribute information of GUI components, and can identify the target range of operation from limited information such as operation events and captured images. Thus, according to this embodiment, the operation range on the screen can be easily identified.

[System Configuration, etc.]
Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed or integrated in any unit according to various loads and usage conditions. Moreover, each processing function performed by each device can be implemented, in whole or in any part, by a CPU (Central Processing Unit) and a program that is analyzed and executed by the CPU, or by hardware using wired logic. Note that the program may be executed not only by the CPU but also by other processors such as a GPU.

Furthermore, among the processes described in this embodiment, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified.

[program]
In one embodiment, the specific device 10 can be implemented by installing a specific program that performs the above-mentioned specific processing as packaged software or online software on a desired computer. For example, by having the above-mentioned specific program run on an information processing device, the information processing device can be made to function as the specific device 10. The information processing device referred to here includes desktop or notebook personal computers. In addition, the category of information processing device also includes mobile communication terminals such as smartphones, mobile phones and PHS (Personal Handyphone System), and slate terminals such as PDA (Personal Digital Assistant).

Furthermore, the specific device 10 can also be implemented as a specific server device that uses a user's terminal device as a client and provides services related to the specific processing described above to that client. For example, the specific server device is implemented as a server device that takes operation events and captured images from the terminal device as input and provides a specific service that outputs a specified range of operations. In this case, the specific server device may be implemented as a web server, or it may be implemented as a cloud service that provides services related to the specific processing described above through outsourcing.

Figure 9 shows an example of a computer running a specific program. Computer 1000 has, for example, memory 1010 and a CPU 1020. Computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

Memory 1010 includes ROM (Read Only Memory) 1011 and RAM (Random Access Memory) 1012. ROM 1011 stores, for example, a boot program such as BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. For example, a removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.

The hard disk drive 1090 stores, for example, the OS 1091, application programs 1092, program modules 1093, and program data 1094. That is, the programs defining each process of the specific device 10 are implemented as program modules 1093 containing code executable by a computer. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, a program module 1093 for performing processes similar to those in the functional configuration of the specific device 10 is stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

Furthermore, the configuration data used in the processing of the above-described embodiment is stored as program data 1094 in, for example, memory 1010 or hard disk drive 1090. The CPU 1020 then reads the program module 1093 and program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as needed and executes the processing of the above-described embodiment.

Furthermore, the program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090; for example, they may be stored in a removable storage medium and read by the CPU 1020 via a disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (LAN (Local Area Network), WAN (Wide Area Network), etc.). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via a network interface 1070.

10 Identification device 11 Communication unit 12 Storage unit 13 Control unit 20 Terminal device 121 Operation log information 131 Acquisition unit 132 Generation unit 133 Identification unit

Claims (6)

An acquisition unit that acquires operation events generated in response to operations on the screen of a terminal device, and a captured image of the screen.
A identifying unit that identifies the range targeted by the operation based on information regarding the color of the position corresponding to the operation event in the captured image,
It has,
The acquisition unit acquires the captured images at regular time intervals.
The aforementioned identification unit identifies the range to be targeted by the operation based on the portion of the captured image that has changed over time.
A specific device characterized by the following features. The identification device according to claim 1, characterized in that the identification unit identifies the range targeted by the operation based on the color of the mouse cursor position when the operation event occurs. The identification device according to claim 1 , characterized in that the identification unit identifies a closed region that has changed over time in the captured image as the range targeted by the operation. The identification device according to claim 1 , characterized in that the identification unit identifies the range to be targeted by the operation based on the color surrounding the portion of the captured image that has changed over time. A specific method performed by a specific device,
An acquisition step of acquiring operation events that occur in response to operations on the screen of a terminal device, and a captured image of the said screen,
A process of identifying the range targeted by the operation based on information regarding the color of the position corresponding to the operation event in the captured image,
Includes,
The acquisition step involves acquiring the captured images at regular time intervals.
The identification step is characterized by identifying the range to be targeted by the operation based on the portion of the captured image that has changed over time . An acquisition step to acquire operation events that occur in response to operations on the screen of a terminal device, and a captured image of the screen,
A selection step to identify the range targeted by the operation based on information regarding the color of the position corresponding to the operation event in the captured image,
Have the computer run it ,
The acquisition step involves acquiring the captured images at regular time intervals.
The aforementioned specific step is a specific program characterized by identifying the range to be targeted by the operation based on the portion of the captured image that has changed over time .