JP7520143B2 - Screen capture device and screen capture method - Google Patents

Description

The present disclosure relates to a screen capture device and a screen capture method.

Conventionally, automatic test equipment that performs operational tests has been proposed. The automatic test equipment displays various information related to the test, such as the results of the operational test, on a display. The automatic test equipment obtains copy data of the image displayed on the display as evidence of the test, and records the copy data.

On this display, some of the information may be displayed in a window with a scroll bar. In this case, the remainder of the information will not be visible on the display unless the scroll bar is operated. Therefore, the remainder of the information cannot be obtained as evidence by simply obtaining a copy of the image shown on the display once.

One technique that can be used to solve such problems is that described in, for example, Patent Document 1. Patent Document 1 describes a mobile phone that includes a touch panel. This mobile phone displays a setting image related to printing on the touch panel, and obtains screenshot data (i.e., copy data) of the displayed image on the touch panel. When the size of the setting image is large, the mobile phone displays a part of the setting image in the display area of the touch panel with a scroll bar.

The mobile phone acquires the size of the setting image, and if that size is larger than the size of the display area, it determines that only a portion of the setting image is being displayed, and sequentially scrolls the setting image to display it on the touch panel, acquiring copy data of the displayed image each time. This allows the entire setting image to be acquired.

JP 2018-73143 A

However, with the technology of Patent Document 1, it is necessary to obtain the size of the setting image. In other words, it is necessary to obtain size information of the source data to be displayed on the display (i.e., the setting image). In order to obtain this size information, it is necessary to support the data format. Since there are many types of data formats, it is time-consuming to support all of the data formats.

Therefore, the present disclosure aims to provide a technology that can obtain multiple display screen data including the source data using a method other than the method for obtaining the size of the source data.

A screen capture device according to the present disclosure is a screen capture device that is connected to a computer device and acquires display image data displayed on a display of the computer device, the screen capture device including an image acquisition unit that acquires the display image data including a part of original data to be displayed on the display and a scroll bar, an identification unit that identifies a state of the scroll bar based on the display image data acquired by the image acquisition unit and preset scroll image data indicating the scroll bar, an operation processing unit that instructs the computer device to operate the scroll bar to scroll the original data to be displayed on the display, and a screen capture unit that controls the operation processing unit based on the state of the scroll bar identified by the identification unit to scroll the original data to be displayed on the display. and a processing control unit that causes an undisplayed image area of the original data to be displayed on the display and causes the image acquisition unit to acquire the display image data including the image area displayed on the display , wherein a scroll box of the scroll bar moves between a first end and a second end of the scroll bar, and first scroll image data showing the scroll bar in a state in which the scroll box is not positioned at the second end of the scroll bar is set in advance as the scroll image data, and when a first image similar to or matching the first scroll image data is detected from the display image data, the processing control unit controls the operation processing unit to move the scroll box toward the second end by one page or less, and causes the undisplayed image area to be displayed on the display .
A screen capture device according to the present disclosure includes an image acquisition unit that acquires display image data including a part of original data and a scroll bar displayed on a display, an identification unit that identifies a state of the scroll bar based on the display image data acquired by the image acquisition unit and preset scroll image data indicating the scroll bar, an operation processing unit that operates the scroll bar to scroll the original data to be displayed on the display, and a screen capture device that controls the operation processing unit based on the state of the scroll bar identified by the identification unit to display an image area of the original data that has not been displayed on the display, and and a processing control unit that causes the image acquisition unit to acquire the display image data including the image area, wherein a scroll box of the scroll bar moves between a first end and a second end of the scroll bar, and first scroll image data showing the scroll bar in a state in which the scroll box is not positioned at the second end of the scroll bar is set in advance as the scroll image data, and when a first image similar to or matching the first scroll image data is detected from the display image data, the processing control unit controls the operation processing unit to move the scroll box toward the second end by one page or less, thereby displaying the undisplayed image area on the display.

A screen capture method according to the present disclosure is a screen capture method using a screen capture device connected to a computer device and acquiring display image data displayed on a display of the computer device, the method comprising: acquiring the display image data displayed on the display; identifying a state of the scroll bar based on the acquired display image data and scroll image data indicating a preset scroll bar displayed on the display, the scroll bar including a scroll box that moves between a first end and a second end of the scroll bar; issuing an instruction to the computer device to operate the scroll bar based on the identified state of the scroll bar; displaying an image area of the original data that has not been displayed on the display; acquiring the display image data including the image area displayed on the display ; first scroll image data indicating the scroll bar in a state in which the scroll box is not located at the second end of the scroll bar is preset as the scroll image data; and when displaying the image area on the display, when a first image similar to or matching the first scroll image data is detected from the display image data, moving the scroll box toward the second end by one page or less to display the image area that has not been displayed on the display .
A screen capture method according to the present disclosure includes a first step of acquiring display image data displayed on a display; a second step of identifying a state of the scroll bar based on the acquired display image data and preset scroll image data indicating a scroll bar displayed on the display, and operating the scroll bar based on the identified state of the scroll bar to cause an undisplayed image area of the original data to be displayed on the display; and a third step of acquiring the display image data including the image area displayed on the display, wherein a scroll box of the scroll bar moves between a first end and a second end of the scroll bar, and first scroll image data indicating the scroll bar in a state in which the scroll box is not located at the second end of the scroll bar is preset as the scroll image data, and in the second step, when a first image similar to or matching the first scroll image data is detected from the display image data, the scroll box is moved toward the second end by one page or less, to cause the undisplayed image area to be displayed on the display.

According to the present disclosure, multiple display screen data including the source data can be obtained using a method other than the method for obtaining the size of the source data.

The objects, features, aspects, and advantages of the present disclosure will become more apparent from the following detailed description and accompanying drawings.

1 is a block diagram illustrating an example of a configuration of a screen capture device according to an embodiment. FIG. 10 is a diagram illustrating an example of procedure data showing a procedure of an operation test. FIG. 2 is a diagram illustrating an example of a display image displayed on a display. 10 is a diagram illustrating an example of a relationship between the position of a scroll box and the position of an image area displayed on a display in source data. FIG. 10 is a diagram illustrating an example of a relationship between the position of a scroll box and the position of an image area displayed on a display in source data. FIG. 10 is a diagram illustrating an example of a relationship between the position of a scroll box and the position of an image area displayed on a display in source data. FIG. FIG. 4 is a diagram illustrating an example of first scroll image data. FIG. 4 is a diagram illustrating an example of first scroll image data. FIG. 10 is a diagram illustrating an example of second scroll image data. 10 is a flowchart showing an example of the operation of the screen capture device. FIG. 2 is a diagram illustrating an example of a hardware configuration of a screen capture device. FIG. 2 is a diagram illustrating an example of a hardware configuration of a screen capture device.

Hereinafter, the embodiments will be described with reference to the attached drawings. In the following embodiments, detailed features are shown to explain the technology, but they are merely examples and are not necessarily all essential features for the embodiments to be feasible.

Note that the drawings are schematic, and for ease of explanation, configurations may be omitted or simplified as appropriate in the drawings. Furthermore, the relative sizes and positions of components shown in different drawings are not necessarily described accurately, and may be changed as appropriate. Furthermore, hatching may be used in drawings such as plan views that are not cross-sectional views, in order to make it easier to understand the contents of the embodiments.

In addition, in the following description, similar components are illustrated with the same reference numerals, and their names and functions are also similar. Therefore, detailed descriptions of them may be omitted to avoid duplication.

In addition, in the description below, when a certain component is described as "comprising," "including," or "having," unless otherwise specified, this is not an exclusive expression that excludes the presence of other components.

In addition, in the description below, even if ordinal numbers such as "first" or "second" are used, these terms are used for convenience to facilitate understanding of the contents of the embodiments, and are not limited to the ordering that may result from these ordinal numbers.

In addition, in the following description, expressions indicating an equal state, such as "same", "equal", "uniform" or "homogeneous", unless otherwise specified, are intended to indicate a strictly equal state as well as cases in which differences occur within the tolerance or range in which the same level of functionality is obtained.

Figure 1 is a block diagram showing an example of the configuration of a screen capture device 20 according to an embodiment. The screen capture device 20 is a device for capturing a display image displayed on the display screen 31a of the display 31. In other words, the screen capture device 20 is a device for obtaining copy data of display image data D1 showing the display image displayed on the display screen 31a. This copy data can also be said to be screenshot data. Note that since the copy data of the display image data D1 is substantially the same as the display image data D1, hereinafter the copy data may also be simply referred to as the display image data D1.

In the example of FIG. 1, the screen capture device 20 is electrically connected to a computer device 30 such as a personal computer. For example, the computer device 30 includes a display 31 such as a liquid crystal display, and a computer main body 32. The computer main body 32 outputs display image data D1 to the display 31, and the display 31 displays the display image data D1 on the display screen 31a. The display image data D1 is, for example, data including pixel values (e.g., brightness values) for each pixel that constitutes the display screen 31a of the display 31. By the display 31 displaying the display image data D1 on the display screen 31a, various information such as characters, symbols, and images included in the display image data D1 is displayed on the display screen 31a of the display 31.

The computer main body 32 has the function of performing various arithmetic processing. For example, the computer main body 32 performs processing in response to input by a user to an input device (e.g., a mouse and a keyboard). The computer main body 32 can display the results of the processing on the display 31. For example, the computer main body 32 generates display image data D1 that reflects the results of the processing, and outputs the display image data D1 to the display 31. As a result, the display screen 31a of the display 31 displays the display image data D1 that reflects the processing. The computer main body 32 also has the function of storing copy data of the display image data D1 displayed on the display screen 31a.

In the example of FIG. 1, the screen capture device 20 is also electrically connected to an automatic test equipment 40. The automatic test equipment 40 is a device for automatically performing operational tests. The test referred to here is, for example, a test for determining whether or not a specific part operates properly. The specific part may include software. The software may be installed in, for example, the computer main body 32 and executed by the computer main body 32.

Procedure data 50 indicating the test procedure is input to the automatic test equipment 40. For example, the procedure data 50 is a script that specifies the test procedure, and may be specified in a specific programming language. The automatic test equipment 40 automatically performs the test based on the input procedure data 50.

2 is a diagram showing an example of the contents of procedure data 50. In the example of FIG. 2, procedure data 50 specifies test procedures 51 and 52. As an example, the software executed by computer main body 32 is the subject of the test. Procedure 51 specifies clicking button A, clicking button B, and screen capture in this order from top to bottom. Buttons A and B are software keys displayed on display screen 31a by computer main body 32 executing software. Clicking button A changes the display content of display screen 31a in response to button A, and clicking button B changes the display content of display screen 31a in response to button B.

The automatic test equipment 40 executes each step defined in the procedure data 50 in order from the top. Specifically, the automatic test equipment 40 instructs the computer main body 32 to click buttons A and B in that order. The automatic test equipment 40 may output the instructions to the computer main body 32 via the screen capture device 20, or may output the instructions directly to the computer main body 32.

In response to this instruction, the computer main body 32 clicks button A and then button B. That is, the computer main body 32 executes processing corresponding to button A and processing corresponding to button B in that order, and sequentially generates display image data D1 reflecting the results. The computer main body 32 sequentially displays these display image data D1 on the display 31. As the latest display image data D1 includes the results of both clicking buttons A and B, the display screen 31a of the display 31 displays the processing results for clicking buttons A and B.

After receiving an instruction to click buttons A and B, the automatic test equipment 40 instructs the screen capture device 20 to perform a screen capture. In response to the instruction, the screen capture device 20 performs processing to capture the display image on the display screen 31a. For example, the screen capture device 20 instructs the computer main body 32 to perform a screen capture. In response to the instruction, the computer main body 32 stores a copy of the latest display image data D1 after buttons A and B are clicked. Alternatively, the screen capture device 20 may obtain a copy of the latest display image data D1 from the computer main body 32 and store the copy.

By testing using procedure 51, display image data D1 including the processing results in response to clicking buttons A and B can be stored as evidence.

In step 52, the keyboard input of "test", the click of the "OK" button, and the screen capture are specified in this order from top to bottom. The "OK" button is a software key displayed on the display screen 31a by the computer main body 32 executing software. When the "OK" button is clicked, the display content of the display screen 31a changes according to the "OK" button.

When the automatic test equipment 40 executes procedure 52 defined in procedure data 50, the computer main body 32 performs processing in response to the input of the characters "test", and then performs processing in response to the clicking of the "OK" button. The computer main body 32 sequentially generates display image data D1 reflecting these processing results, and outputs them sequentially to the display 31. The display 31 sequentially displays the display image data D1 on the display screen 31a.

After receiving the instruction to input "test" from the keyboard and click the "OK" button, the automatic test equipment 40 instructs the screen capture device 20 to perform a screen capture. In response to the instruction, the screen capture device 20 performs processing to capture the display image of the display screen 31a. For example, the screen capture device 20 instructs the computer main body 32 to perform a screen capture. In response to the instruction, the computer main body 32 stores a copy of the latest display image data D1 after the keyboard input of "test" and the click of the "OK" button. Alternatively, the screen capture device 20 obtains a copy of the latest display image data D1 from the computer main body 32 and stores the copy.

According to such a test using procedure 52, the latest display image data D1 includes the processing results for the keyboard input of "test" and the click of the "OK" button. Therefore, the display image data D1 including the processing results for the keyboard input of "test" and the click of the "OK" button can be stored as evidence.

Note that a window with a scroll bar may be displayed on the display screen 31a of the display 31, as described below. In this case, only a portion of the original information is displayed in the window, and the remaining portion of the information is displayed by scrolling the image in the window. Therefore, in this case, in order to properly obtain the required information, it is necessary to capture the displayed image while scrolling the image.

In the example of Figure 1, the screen capture device 20 includes an image acquisition unit 1, an identification unit 2, an image position acquisition unit 3, an operation processing unit 4, and a processing control unit 5.

The image acquisition unit 1 acquires display image data D1 from the computer main body 32. The image acquisition unit 1 may store the display image data D1 in a non-transient non-volatile storage unit. It can also be said that the image acquisition unit 1 captures the display image displayed on the display screen 31a of the display 31.

Figure 3 is a diagram showing an example of a display image displayed on the display screen 31a of the display 31. In the example of Figure 3, the display image includes a first window 34 and a second window 35. In the example of Figure 3, the first window 34 does not have a scroll bar, and the second window 35 has a scroll bar 33. The following description will focus mainly on the second window 35.

Scroll bar 33 includes an element 331 indicating its top end (corresponding to the first end), an element 332 indicating its bottom end (corresponding to the second end), and a scroll box 333. Scroll box 333 is vertically movable between the top and bottom ends of scroll bar 33. Scroll box 333 may also be called a knob, valve, thumb, scroller, or scroll thumb.

The display 31 displays a portion of the source data D2 on the display screen 31a together with the scroll bar 33. Here, the source data D2 and the position of the scroll box 333 in the scroll bar 33 will be described. Figures 4 to 6 are diagrams that show an example of the relationship between the position of the scroll box 333 and the position of the image area (here, the second window 35) of the source data D2 displayed on the display 31.

In the example of Figure 4, the scroll box 333 is located at the top end of the scroll bar 33. At this time, the uppermost area of the original data D2 is displayed in the second window 35. In the example of Figure 5, the scroll box 333 is located at the midpoint between the top and bottom ends of the scroll bar 33. Therefore, at this time, the middle area of the original data D2 is displayed in the second window 35. In the example of Figure 6, the scroll box 333 is located at the bottom end of the scroll bar 33. Therefore, at this time, the lowermost area of the original data D2 is displayed in the second window 35.

As described above, the position of the scroll box 333 in the scroll bar 33 indicates the image area of the source data D2 displayed in the second window 35. Conversely, if the state of the scroll bar 33 can be identified, it is possible to identify the image area of the source data D2 displayed in the second window 35. Here, the state of the scroll bar 33 may include the position and size of the scroll box 333 in the scroll bar 33.

Therefore, in order to specify the state of the scroll bar 33 displayed on the display screen 31a, scroll image data showing the scroll bar 33 is stored in advance in the storage unit 10. The storage unit 10 is, for example, a non-volatile storage unit, and as a more specific example, it may be a memory (storage medium) including a hard disk drive (i.e., HDD), a read only memory (i.e., ROM), a flash memory, an erasable programmable read only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a non-volatile semiconductor memory, a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, or a DVD, or any storage medium to be used in the future.

As the scroll image data, for example, first scroll image data D31 showing the scroll bar 33 in a state where the scroll box 333 is not located at the bottom end of the scroll bar 33, and second scroll image data D32 showing the scroll bar 33 in a state where the scroll box 333 is located at the bottom end of the scroll bar 33 are set in advance. In other words, the first scroll image data D31 and the second scroll image data D32 are stored in advance in the storage unit 10.

7 and 8 are diagrams showing an example of the first scroll image data D31, and FIG. 9 is a diagram showing an example of the second scroll image data D32. The first scroll image data D31 and the second scroll image data D32 are, for example, data obtained by cutting out a part of the display image data D1, and are image data showing the lower end area of the scroll bar 33 (corresponding to a part of the area on the second end side). Both the first scroll image data D31 and the second scroll image data D32 include the element 332 and a part above the element 332. The first scroll image data D31 in FIG. 7 does not include the scroll box 333. Although the first scroll image data D31 in FIG. 8 includes the lower end of the scroll box 333, there is a gap between the scroll box 333 and the element 332, and the scroll box 333 is located above the lower end of the scroll bar 33. Note that more image data with different positions of the scroll box 333 may be adopted as the first scroll image data D13. The second scroll image data D32 in FIG. 9 includes the scroll box 333 located at the lower end of the scroll bar 33.

The identification unit 2 identifies the state of the scroll bar 33 in the display image data D1 acquired by the image acquisition unit 1 based on the scroll image data stored in the storage unit 10 and the display image data D1. Specifically, the identification unit 2 acquires the first scroll image data D31 and the second scroll image data D32 from the storage unit 10. Then, the identification unit 2 determines whether the first scroll image data D31 or the second scroll image data D32 can be detected from the display image data D1.

For example, the identification unit 2 performs template matching between the display image data D1 and the first scroll image data D31. As illustrated in Fig. 7 and Fig. 8, when there are multiple types of first scroll image data D31, it is preferable to perform template matching for each of the first scroll image data D31. When a first image similar to or matching each of the first scroll image data D31 is included in the display image data D1, the first image is detected by the template matching. In this case, in the scroll bar 33 displayed on the display screen 31a, the scroll box 333 is located above the bottom end position.

The identification unit 2 also performs template matching between the display image data D1 and the second scroll image data D32. When the display image data D1 contains a second image similar to or matching the second scroll image data D32, the second image is detected by the template matching. In this case, the scroll box 333 is located at the bottom end position in the scroll bar 33 displayed on the display screen 31a.

The image position acquisition unit 3 outputs position information of the scroll box 333 of the scroll bar 33 to the processing control unit 5. For example, when the identification unit 2 detects a first image similar to or matching the first scroll image data D31, the image position acquisition unit 3 notifies the processing control unit 5 that the scroll box 333 is at a position other than the bottom end position. When the identification unit 2 detects a second image similar to or matching the second scroll image data D32, the image position acquisition unit 3 notifies the processing control unit 5 that the scroll box 333 is at the bottom end position. When neither the first image nor the second image is detected from the display image data D1, the image position acquisition unit 3 may notify the processing control unit 5 of that effect.

The operation processing unit 4 performs processing related to the operation of the scroll bar 33. The operation processing unit 4 operates the scroll bar 33 to scroll the source data D2 to be displayed on the display 31. For example, the operation processing unit 4 instructs the computer main body 32 to operate the scroll bar 33. Specifically, the operation processing unit 4 instructs the computer main body 32 to move the scroll box 333 downward so as to display an image area of the source data D2 that is one page or less below the image area currently being displayed. The operation of moving the image area downward by one page is also called a page down operation.

The operation processing unit 4 may further perform processing related to a screen capture operation. For example, the operation processing unit 4 may instruct the computer main body 32 to perform a screen capture operation. In response to the instruction, the computer main body 32 stores copy data of the display image data D1 displayed on the display 31 in, for example, a non-transient non-volatile storage unit.

The processing control unit 5 performs overall processing of the entire screen capture device 20. For example, the processing control unit 5 outputs various instructions to the image acquisition unit 1, the identification unit 2, the image position acquisition unit 3, and the operation processing unit 4, and also receives various information from the image acquisition unit 1, the identification unit 2, the image position acquisition unit 3, and the operation processing unit 4.

Figure 10 is a flowchart showing an example of the operation of the screen capture device 20. In step S1, the process control unit 5 determines whether or not a screen capture instruction has been received from the automatic test equipment 40. If the instruction has not been received, the process control unit 5 executes step S1 again.

When an instruction for screen capture is received, in step S2, the process control unit 5 causes the image acquisition unit 1 to acquire display image data D1. The image acquisition unit 1 acquires the display image data D1 from the computer main body 32. When causing the computer main body 32 to perform screen capture, the process control unit 5 causes the operation processing unit 4 to instruct the computer main body 32 to perform a screen capture operation. In response to the instruction, the computer main body 32 stores copy data of the display image data D1 as evidence in a non-transient non-volatile storage unit. Note that when the screen capture device 20 performs screen capture, it is preferable for the image acquisition unit 1 to store the display image data D1 in the storage unit. As a result, in step S2, the currently displayed display image data D1 is stored in the storage unit as evidence.

Next, in step S3, the process control unit 5 causes the identification unit 2 to determine whether a first image similar to or matching the first scroll image data D31 has been detected from the display image data D1. Specifically, the identification unit 2 performs template matching between the display image data D1 acquired by the image acquisition unit 1 and the first scroll image data D31 stored in the storage unit 10. When multiple types of first scroll image data D31 are stored in the storage unit 10, the identification unit 2 performs template matching for each first scroll image data D31.

When any of the first images is detected, in step S3, the process control unit 5 causes the operation processing unit 4 to operate the scroll bar 33 to display the undisplayed image area of the original data D2 in the second window 35 of the display 31. For example, the operation processing unit 4 moves the scroll box 333 to the bottom end by one page or less to display the undisplayed image area in the second window 35 of the display 31. As a more specific example, the operation processing unit 4 instructs the computer main body 32 to perform a page down operation. The computer main body 32 performs the page down operation based on the instruction. As a result, an image area of the original data D2 that is one page lower than the image area displayed in the second window 35 is displayed in the second window 35.

The page down operation can be realized, for example, by clicking on a part of the area of the scroll bar 33 between the scroll box 333 and the element 332 (see, for example, position P1 in FIG. 7). The click position P1 is, for example, a position above the element 332 in the first scroll image data D31 (first image). The image position acquisition unit 3 may identify the click position P1 based on the position of the first image detected by the identification unit 2 (see FIGS. 7 and 8), and the operation processing unit 4 may specify the position P1 identified by the image position acquisition unit 3 and instruct the computer main body 32 to click. In response to the instruction, the computer main body 32 performs a page down operation by clicking on position P1.

Next, the process control unit 5 executes step S2 again. That is, an image area of the original data D2 that was not displayed is displayed in the second window 35 by the scroll operation in step S4, and therefore in order to acquire the display content, the image acquisition unit 1 acquires the display image data D1 displayed on the display screen 31a in step S2. Note that the display image data D1 being displayed in step S2 is stored in the non-volatile storage unit as evidence by the computer main body 32 or the image acquisition unit 1.

A set of steps S4 and S2 is repeatedly executed as long as the first image is detected from the display image data D1 in step S3. In other words, when the first image is detected, the original data D2 contains an undisplayed image area, so the operation processing unit 4 performs a scroll operation to display the undisplayed image area on the display screen 31a of the display 31 (step S4). Then, the display image data D1 including the image area displayed on the display screen 31a is acquired and stored (step S2). This allows image areas of the original data D2 that have not yet been acquired to be acquired sequentially as display image data D1 and stored in the storage unit.

On the other hand, when the first image is not detected from the display image data D1 in step S3, the process control unit 5 causes the identification unit 2 to determine whether or not the second image is detected from the display image data D1 in step S5. Specifically, the identification unit 2 performs template matching between the display image data D1 acquired by the image acquisition unit 1 and the second scroll image data D32 stored in the storage unit 10.

When the second image is detected from the display image data D1, the scroll box 333 is located at the bottom end position, and the process is terminated. In other words, when the scroll box 333 is located at the bottom end position, the image area of the data D2 is displayed sequentially on the display screen 31a by the previous scroll operation (step S4), and the display image data D1 is acquired and stored in the non-volatile memory unit for each scroll operation. As a result, the entire image area of the original data D2 is divided and acquired into multiple display image data D1. In other words, sufficient evidence has been acquired, and the process is terminated.

When the second image is not detected from the display image data D1 in step S5, the process control unit 5 determines that an abnormality has occurred in step S6 and performs error processing. In other words, when neither the first image nor the second image can be detected from the display image data D1, the process control unit 5 determines that an abnormality has occurred. As error processing, the process control unit 5 may, for example, cause the display 31 to display that the state of the scroll bar 33 could not be identified.

As described above, in this embodiment, the state of the scroll bar 33 is identified based on the display image data D1 and the preset scroll image data, and a scroll operation and a capture of the display screen are performed based on the state of the scroll bar 33. With this type of processing, there is no need to acquire size information of the source data D2. In other words, the necessity for a scroll operation can be determined by a method other than the method of acquiring size information of the source data D2, and the entire image area of the source data D2 can be divided and acquired into multiple display image data D1. Therefore, the screen capture device 20 does not need to perform processing corresponding to the data format of the data D2, making it easier to introduce the screen capture device 20.

Also, in the above example, each time it is determined that scrolling is necessary, the scroll box 333 is moved from the top to the bottom by one page or less (step S4), and the display image is captured each time. Therefore, according to this embodiment, each image area of the original display data D2 can be captured in order from the top. In other words, each image area can be acquired as display image data D1 in order from the top.

With this scroll order, when the scroll box 333 is not yet positioned at the bottom end, the source data D2 contains image areas that have not yet been acquired.

Therefore, the screen capture device 20 performs template matching between the first scroll image data D31, which indicates the bottom end area of the scroll bar 33 that does not include the scroll box 333, and the display image data D1. This allows the screen capture device 20 to easily identify a state in which the scroll box 333 is not located at the bottom end position. In other words, the screen capture device 20 can easily identify that an image area that has not yet been acquired as evidence is not included in the original display data D2, and a scroll operation is required.

Also, according to the above-mentioned scrolling order, when the scroll box 333 is positioned at the bottom end position, all of the image areas of the source data D2 have been captured.

Therefore, the screen capture device 20 performs template matching between the second scroll image data D32, which shows the scroll bar 33 when the scroll box 333 is located at the bottom end position, and the display image data D1. This allows the screen capture device 20 to easily identify the state in which all of the image areas of the original data D2 have been acquired as evidence.

In the above example, the process control unit 5 determines that an abnormality has occurred when both the first image and the second image cannot be detected from the display image data D1 by template matching. Therefore, the process control unit 5 can perform error processing at an appropriate timing. As the error processing, the process control unit 5, for example, displays on the display 31 that an abnormality has occurred.

Note that in the above example, scroll bar 33 is a vertical scroll bar, but it may also be a horizontal scroll bar.

Next, an example of the hardware configuration of the screen capture device 20 will be described. Figures 11 and 12 are diagrams that show an example of the hardware configuration of the screen capture device 20 illustrated in Figure 1.

Note that the hardware configurations illustrated in Figures 11 and 12 may not match the numbers, etc., of the configuration illustrated in Figure 1, but this is because the configuration illustrated in Figure 1 represents conceptual units.

Figure 11 shows a processing circuit 21A that performs calculations and a memory device 22 as the hardware configuration for realizing the image acquisition unit 1, identification unit 2, image position acquisition unit 3, operation processing unit 4, and processing control unit 5 in Figure 1.

The storage device 22 may be, for example, a hard disk drive (HDD), random access memory (RAM), read only memory (ROM), flash memory, erasable programmable read only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM), or other volatile or non-volatile semiconductor memory, a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, or a DVD, or any other storage medium to be used in the future.

The processing circuit 21A may execute a program stored in the storage device 22, an external CD-ROM, an external DVD-ROM, or an external flash memory. That is, the processing circuit 21A may be, for example, a central processing unit (CPU), a microprocessor, a microcomputer, or a digital signal processor (DSP).

When the processing circuit 21A executes a program stored in the storage device 22, an external CD-ROM, an external DVD-ROM, or an external flash memory, the image acquisition unit 1, the identification unit 2, the image position acquisition unit 3, the operation processing unit 4, and the processing control unit 5 are realized by software, firmware, or a combination of software and firmware in which the program stored in the storage device 22 is executed by the processing circuit 21A. Note that the functions of the image acquisition unit 1, the identification unit 2, the image position acquisition unit 3, the operation processing unit 4, and the processing control unit 5 may be realized, for example, by multiple processing circuits working together.

The software and firmware may be written as a program and stored in the storage device 22. In this case, the processing circuit 21A realizes the above-mentioned functions by reading and executing the program stored in the storage device 22. In other words, the storage device 22 may store a program that, when executed by the processing circuit 21A, results in the above-mentioned functions being realized.

Processing circuitry 21B may also be dedicated hardware. That is, processing circuitry 21B may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a combination thereof.

When the processing circuit 21B is a dedicated hardware, the image acquisition unit 1, the identification unit 2, the image position acquisition unit 3, the operation processing unit 4, and the processing control unit 5 are realized by the operation of the processing circuit 21B. Note that the functions of the image acquisition unit 1, the identification unit 2, the image position acquisition unit 3, the operation processing unit 4, and the processing control unit 5 may be realized by separate circuits or by a single circuit.

The functions of the image acquisition unit 1, the identification unit 2, the image position acquisition unit 3, the operation processing unit 4 and the processing control unit 5 may be partly realized in a processing circuit 21A which executes a program stored in a memory device 22, and partly realized in a processing circuit 21B which is dedicated hardware.

In addition, the functions of the computer main body 32 and the functions of the automatic test equipment 40 may be realized by the processing circuit 21A and the memory device 22, similar to the screen capture device 20, or may be realized by the processing circuit 21B.

As mentioned above, the present disclosure has been described in detail, but the above description is illustrative in all respects and is not limiting. It is understood that countless variations not illustrated can be envisioned.

Furthermore, unless a contradiction arises, any component described in the above embodiments as being provided in "one" quantity may be provided in "one or more" quantities.

Furthermore, each component in the embodiments described above is a conceptual unit, and the scope of the technology disclosed in this specification includes cases where one component is made up of multiple structures, where one component corresponds to part of a structure, and even where multiple components are provided in one structure.

In addition, each component in the embodiments described above is intended to include structures having other structures or shapes so long as they perform the same function.

Furthermore, the descriptions in this specification are incorporated by reference for all purposes related to the present technology, and none of them are admitted to be prior art.

Furthermore, each of the components described in the embodiments described above is envisioned as either software or firmware, or as corresponding hardware, and in both concepts, each component is referred to as a "part" or "processing circuit" (circuitry), etc.

In addition, the technology disclosed in this specification may take the form of a system in which each component is distributed across multiple devices.

1 Image acquisition unit, 2 Identification unit, 31 Display, 33 Scroll bar, 333 Scroll box, 4 Operation processing unit, 5 Processing control unit, D1 Display image data, D31 First scroll image data, D32 Second scroll image data.

Claims (6)

A screen capture device that is connected to a computer device and captures display image data displayed on a display of the computer device,
an image acquisition unit that acquires the display image data including a part of the original data and a scroll bar, the display image data being displayed on the display;
a determination unit that determines a state of the scroll bar based on the display image data acquired by the image acquisition unit and preset scroll image data indicating the scroll bar;
an operation processing unit that operates the scroll bar to instruct the computer device to scroll the source data to be displayed on the display;
a processing control unit that controls the operation processing unit based on the state of the scroll bar identified by the identification unit to cause an undisplayed image area of the original data to be displayed on the display, and causes the image acquisition unit to acquire the display image data including the image area displayed on the display ;
a scroll box of the scroll bar moving between a first end and a second end of the scroll bar;
As the scroll image data, first scroll image data showing the scroll bar in a state where the scroll box is not positioned at the second end of the scroll bar is set in advance,
The processing control unit
A screen capture device that, when a first image similar to or matching the first scroll image data is detected from the display image data, controls the operation processing unit to move the scroll box toward the second end by one page or less, thereby displaying the undisplayed image area on the display . an image acquisition unit that acquires display image data including a part of the original data and a scroll bar, the display image data being displayed on the display;
a determination unit that determines a state of the scroll bar based on the display image data acquired by the image acquisition unit and preset scroll image data indicating the scroll bar;
an operation processing unit that operates the scroll bar to scroll the source data to be displayed on the display;
a processing control unit that controls the operation processing unit based on the state of the scroll bar identified by the identification unit to cause an undisplayed image area of the original data to be displayed on the display, and causes the image acquisition unit to acquire the display image data including the image area displayed on the display;
a scroll box of the scroll bar moving between a first end and a second end of the scroll bar;
As the scroll image data, first scroll image data showing the scroll bar in a state where the scroll box is not positioned at the second end of the scroll bar is set in advance,
The processing control unit
A screen capture device that, when a first image similar to or matching the first scroll image data is detected from the display image data, controls the operation processing unit to move the scroll box toward the second end by one page or less, thereby displaying the undisplayed image area on the display. second scroll image data showing the scroll bar in a state where the scroll box is located at the second end of the scroll bar is preset as the scroll image data;
The screen capture device according to claim 1 or 2 , wherein the processing control unit terminates the processing when a second image similar to or matching the second scroll image data is detected from the display image data. The screen capture device according to claim 3 , wherein the processing control unit determines that an abnormality has occurred when neither the first image nor the second image is detected from the display image data. 1. A screen capture method using a screen capture device that is connected to a computer device and acquires display image data displayed on a display of the computer device, comprising:
Acquire the display image data displayed on the display;
determining a state of the scroll bar based on the acquired display image data and preset scroll image data showing a scroll bar displayed on the display, the scroll bar including a scroll box moving between a first end and a second end of the scroll bar ;
giving an instruction to the computer device to operate the scroll bar based on the specified state of the scroll bar, and displaying an undisplayed image area of the source data on the display;
acquiring the display image data including the image area displayed on the display ;
As the scroll image data, first scroll image data showing the scroll bar in a state where the scroll box is not positioned at the second end of the scroll bar is set in advance,
A screen capture method in which, when the image area is displayed on the display, when a first image similar to or matching the first scroll image data is detected from the display image data, the scroll box is moved toward the second end by one page or less to display the undisplayed image area on the display. A first step of acquiring display image data displayed on a display;
a second step of specifying a state of the scroll bar based on the acquired display image data and preset scroll image data indicating a scroll bar to be displayed on the display, and manipulating the scroll bar based on the specified state of the scroll bar to display an image area of the original data that has not been displayed on the display;
and a third step of acquiring the display image data including the image area displayed on the display,
a scroll box of the scroll bar moving between a first end and a second end of the scroll bar;
As the scroll image data, first scroll image data showing the scroll bar in a state where the scroll box is not positioned at the second end of the scroll bar is set in advance,
In the second step,
A screen capture method, when a first image similar to or matching the first scroll image data is detected from the display image data, moving the scroll box toward the second end by one page or less to display the undisplayed image area on the display.

Images