JP5916136B2 - Screen change detection apparatus, method and program - Google Patents

Description

  The present invention periodically captures (image data acquisition) a screen (display screen) displayed on a display device of a computer at short time intervals, and there is a difference in content between the latest image data and previous image data. The present invention relates to a technique for detecting a screen change depending on whether or not there is.

  Generally, it is possible to immediately detect a change in the screen by setting the time interval for capturing the display screen to 50 msec or the like. In addition, by capturing the entire area of the display screen, it is guaranteed that changes can be detected without omission. However, the process of capturing an image in a short period and in the entire area is not light enough to ignore the load even with a current computer (PC) with improved processing capability. When the load of the original calculation processing is heavy, it is necessary for the user (operator) to change the capture setting to lower the detection accuracy.

  On the other hand, depending on the purpose of detection, there is a case where it is desired to lower the detection accuracy.

  For example, if the entire area of the display screen is captured, there are many areas that change at any time regardless of the user's operation, such as blinking display of the cursor indicating the text editing position, current time display, and network connection status display.

  In addition, for example, when the difference between the latest image data and the image data immediately before it is captured in a short cycle and the difference is compared, the display of the GUI element is changed by hovering the mouse (the pointer only moves on the button, the high Even when the mouse passes over the GUI, it reacts as a screen change.

  In this way, the detection is frequently performed including those that are not related to the change of the screen to be detected. Here, these changes in the screen state that are not desired for practical purposes are called “background noise-like screen changes”.

  As a prior art for this, in order to exclude small area and short-time changes from the target of change detection, a countermeasure is applied by applying a threshold by a predetermined setting. Specifically, for the area, once the entire area of the screen is captured and the change area is detected, the area and vertical and horizontal lengths are calculated. Also, with respect to time, the duration of the difference between the comparison images is calculated and ignored if the difference is less than or equal to the threshold value, which necessitates the need for filtering.

  In addition, as a countermeasure that combines the suppression of the load and the above-mentioned filter, the environment for detecting the screen change (the content of the background noise-like screen change that occurs on the PC performance and the window system) and the purpose Depending on (content of the screen change to be detected), a means for the user to set the range of the area to be captured and the length of the time interval is also provided. Using this means, the user changes the set value within a predetermined width, and determines the set value while observing the actual screen and the detection result.

  As another detection method, a method of setting sampling pixels in a random or predetermined pattern on the screen and comparing changes in the pixel values can be considered.

  There is software that acquires image data of a display screen of a PC through an operating system (OS) and applies the data. For example, in Prior Art 1 (see Patent Document 1), an agent (software) that automatically executes a PC operation acquires image data between automatic execution operations, and the position of a window to be operated or Use as a clue to know the state.

  In this case, it is required to acquire image data in real time in parallel with the user operation on the PC and the execution of the process of the application program (AP). Then, it may be acquired at a timing based on an operation event issued by a user through an input device (mouse, keyboard, etc.) or an internal event issued by the AP itself in software. The problem shown in 2 occurs. It is useful if the start and completion of changes in the image content on the display screen can be monitored and the data acquisition timing can be detected.

  In addition, when monitoring the display screen of a PC as a function of these agents, the hardware configuration change of the PC, low-level modifications / changes in software configuration such as OS and device drivers, specifically, the PC Replace the graphics board with a special one for monitoring, apply a special patch to the graphics processing of the OS, install a special graphics device driver, replace the runtime library provided by the OS to the AP, etc. Whether it is necessary is an important point.

  If these modifications and changes are necessary, not only the introduction cost and preparation work are required, but also the resources necessary for executing any AP on the PC are changed. , Problems such as being unable to receive support (upgrading and troubleshooting support) of providers such as APs, and technically difficult to completely return to the state before the introduction of the agent, may leave some influence Arise.

  As described above, when executing the detection of the screen change, it is not necessary to modify or change the PC environment in terms of introducing it into a PC owned by an unspecified general user or a terminal that is in operation. This is an essential requirement.

  There is a method for setting a sample point to detect a time-series change of an image (refer to Prior Art 2: Patent Document 2). Therefore, on the basis of this method, a device for improving detection accuracy has been devised by setting sample points at a high density for an image region where a detection target exists.

The main change of the display screen assumed by the present invention corresponds to a high-frequency change that occurs on the desktop of the window system, and there are many cases where detection is not possible even if sample points are arranged at high density. . The main examples of changes that occur on the desktop are:
-Appearance, movement, transformation, and disappearance of the rectangular area corresponding to the frame window-Display of highlight color when the frame window is active-Appearance and disappearance of the rectangular area corresponding to the pop-up window-Up, down, left, and right of the image inside the scroll view Scroll movement ・ Changes in text drawing contents ・ Highlight color display by text selection ・ Blink of caret waiting for text input ・ Changes in appearance such as button press during GUI element action ・ Highlight color when focus of GUI element・ Highlight color display during mouse hover of GUI elements.

  These are only possible examples and are not limited to these. For example, an AP in which the appearance and disappearance of GUI elements (buttons, text fields, tab panels, scroll panels, etc.) arranged on the frame window are dynamically considered is also conceivable. Although it is not common, there are rare APs that are moved or deformed. These are considered to be image changes similar to the appearance, disappearance, movement, and deformation of the frame window. A different feature is that the GUI elements do not overlap each other like the frame window, which corresponds to a situation in which drawing elements do not overlap on a uniform background color. As described above, it is not possible to enumerate all individual display update events, but the feature of image change is considered to be included in the above case.

  In the above, the pop-up window is a generic term for a menu window or submenu window displayed when a menu is selected, a list window displayed when a combo box is selected, a context menu window displayed when the right mouse button is pressed, or the like. . The GUI element is a generic term including buttons, text fields, tab panels, scroll panels, menu items, and windows themselves. A window is a generic term for a frame window or a pop-up window.

  Examples of these changes are listed below when they cannot be detected by the method using sample points.

  For example, it cannot be detected when the frame width is small (the number of pixels occupied in the width direction of the band-like image drawn as the boundary portion of the frame window is small) and does not overlap the sample point. Especially in cases where the image inside the window does not change and only the frame part changes (such as when the window is highlighted when active), even if the occupied area of the entire window overlaps the sample point In some cases, a change cannot be detected.

  For example, when characters are sparsely present in the text area in the scroll panel, there may be a case where all the sample points are located in the background before and after the scroll movement and cannot be detected by chance.

  • For example, if the horizontal character string is small and does not overlap the sample point, it cannot be detected.

  For example, when the rectangular area of the GUI element is small and does not overlap the sample point, it cannot be detected.

  ・ For example, if the caret is small and does not overlap the sample point, it cannot be detected.

  Even if the interval between the sampling points is reduced, the probability of the worst case in which these cases occur cannot be suppressed. This is because an extremely vertically or horizontally long rectangle may not overlap the sample point depending on the position of the object, and whether it can be detected depends on the situation at the time of execution.

  Since the scan line of the present invention is a lattice-like pattern composed of vertical and horizontal combinations, the dependency on the position of the detection target is small, and the detection sensitivity can be effectively improved by narrowing the interval. In addition, since the vertical and horizontal lengths of one square of the lattice correspond to the minimum size of the change rectangle that can be detected reliably, the conditions for the object that can guarantee detection are given clearly and concisely. This is because the present invention uses image features derived from the image of the GUI element on the desktop, not the image of the natural object as in the prior art 2.

  Further, in the means for monitoring the camera image and detecting the difference, “after comparing the sampling image with the reference image and detecting that the sampling image and the reference image are different from each other, the sampling image is “Abnormality detection means for determining that the imaging means is in an abnormal operation state when the same state continues” and “partial image extraction means for extracting a part of the monitored image captured by the imaging means as a partial image. In addition, the reference image recording means records the partial image extracted by the partial image extraction means as a reference image in a state where there is no abnormality in the monitoring target, and the image acquisition means also uses the partial image extraction means for the monitoring target. There is a technique for acquiring an extracted partial image as a sampling image (see Prior Art 3: Patent Document 3).

  In the present invention, when a difference is found by comparing the latest image in the time series with the previous image, it is determined that the screen change has started, and when the state where the images remain the same for a predetermined time or longer, the image change There are two types of statuses to be notified to the upper module as a result of the determination, such as determining to end. In addition, the pattern used when the monitoring target is divided into partial images and sampled is configured by a grid of scanline-like band-like regions, not tile-like rectangular regions. This makes it possible to detect a change in a vertically long / horizontal region. Furthermore, a superposition of patterns with different resolutions is dynamically generated. This can cover the entire screen while reducing the calculation load.

  “By performing a predetermined calculation between images separated by n × Δt (n ≧ 2, n is an integer) with respect to each successively captured image, where Δt is an image capture time interval. By detecting an intruding object (extracting a change area), there is a method in which the detecting performance of an intruding object is not different from the conventional one, while an intruding object detection result can be obtained at a fast time interval (prior art 4). : Patent Document 4). The entire image area is targeted for image capture and computation between images at all timings. Here, a situation in which detection results are obtained at fast time intervals is referred to as a high detection reaction speed.

  In the present invention, the target at one timing is not the entire image area, but a predetermined portion thereof, and the processing load can be kept low. In addition, the detection result cannot be obtained at a uniform fast time interval, but the detection result can be adaptively obtained at a fast time interval depending on conditions. This feature is important in practice. Ideally, it is desirable that the detection reaction speed be a fast time interval. However, if the load of the detection process increases for this reason, the ability to allocate to the AP process in the PC is reduced. It will not serve its original purpose. For this reason, it is practically important that the detection can be performed with as little delay as possible in the large area within the load range that does not affect the performance, and the overall load can be reduced in the small area at the expense of the detection reaction speed. is there.

  Specifically, when a scan line pattern at a certain resolution is taken up and considered, the following can be said. When attention is paid to one scan line in the pattern, there is an interval of one cycle (16 × Δt in the case of 16 divisions) between the capture and calculation between images. Therefore, considering only one pixel in the region, the change is detected after the worst case ((16-1) +16) × Δt after the actual change. This is the same as the conventional method in which the prior art 4 is not applied.

  Instead, in the present invention, at each capture timing (Δt) within one period, the captured image area corresponds to each scan line of the capture pattern, and the position is slightly shifted. The change in the detection target actually changes not only one pixel but a large number of pixels in a practical situation. As the vertical and horizontal sizes of these change areas are larger, the number of lines that simultaneously overlap a plurality of scan lines increases, so that the detection result is actually obtained at a time interval faster than one cycle.

  It should be noted here that in the present invention, the worst value can be strictly guaranteed according to the capture pattern for each resolution, instead of simply changing the resolution and the detection speed in a probabilistic manner according to the detection target. For example, the scan line interval of a 4-division capture pattern is 256 pixels, and each scan line is subjected to processing such as image capture and comparison between the previous and next images every 40 msec, and the scan line interval of the 16-division capture pattern. Similarly, when processing is performed every 160 msec with 64 pixels, a rectangle whose long side exceeds 256 pixels can be detected with a delay of less than 80 msec after the start of actual change at worst, and a rectangle with more than 64 pixels has a delay of less than 320 msec at worst Can be detected.

  The smaller the region, the longer the detection delay time, but the worst case time is guaranteed according to the range of the long side of the region. This requires that the present invention be attached to an unspecified target AP from another process, such as automatic GUI operation or user operation monitoring, and operate in real time while sharing CPU resources. When implemented as a general-purpose agent-type product, it is a useful approach compared to the conventional method in which the time required for detection depends on the contingency such as the timing at the time of execution and the display contents and does not give a clear guarantee. From the viewpoint of clearly presenting the performance of automatic operation and operation monitoring to the user, and agreeing on the restrictions at the time of execution at the time of user support, it is important in the characteristics of the agent type product.

  In addition, there is a method of “characterizing that the screen is monitored by setting a region to be monitored by thinning the screen with a horizontal line, and sequentially shifting the region” (Prior Art 5: Patent Document 5). reference).

  In the present invention, the scan line is set not only in the horizontal direction but also in the vertical direction. In addition, a plurality of patterns are superimposed and set so that determination can be made simultaneously with a plurality of resolutions. A fine resolution pattern is captured with a long period, and a coarse resolution pattern is captured with a short period. Thereby, the resolution according to the size of the region to be detected can be realized simultaneously.

  This can be implemented in consideration of two points. One is that the number of captures can be halved by utilizing the fact that fine resolution patterns include patterns with coarse resolution. For one, it is possible to eliminate variations in the number of scan lines at each capture timing and level the load.

  In addition, it is possible to preferentially determine a changing area by dynamically controlling in which order the scan lines are captured in one cycle.

  In addition, by dynamically controlling the width of the scan-line-shaped rectangular area that is actually captured, the load when the screen is changing or when the CPU usage rate is high is reduced, while the screen is stationary or the CPU usage rate is reduced. Can be adaptively improved when the resolution is low. In the normal method, it is an implementation that sacrifices either one.

  In addition, a natural determination operation can be realized by continuously adjusting the resolution according to the degree to which the screen change subsides or the CPU usage rate decreases. In the normal method, since the set value is a coarse step value, the determination operation changes unnaturally by switching dynamic control.

  Further, since the setting values such as the threshold value and weight for dynamic control of the resolution can be selected from fine step values, the performance of the PC to which the present invention is applied, the number of pixels of the screen, the change tendency of the GUI image, etc. Adjustment according to the environmental conditions of the application destination is possible. In the normal method, when the environmental condition of the application destination changes, the capture period, capture pattern (region shape and area), and judgment logic (balance) while balancing the required resolution and allowable CPU load accordingly. It is necessary to re-implement the judgment criteria / conditions and judgment position / cycle in the comparison algorithm.

  In general, a background noise-like screen change is made so as not to be noticeable to the user so that the display content desired to be transmitted to the user and human recognition of the change are not disturbed. Various display formats provided by the window system and UIs designed by application developers are considered to be consistent with this. Therefore, it is assumed that the smaller the area is, and the shorter the duration from when the display changes to another content or the original content is shorter, the lower the degree of detection is desired. it can. A detection method capable of setting a filter based on this concept (stage detection model) is desired.

  As a problem of the conventional method, it is not possible to handle a stepwise and complex filter threshold based on this assumption. For example, only one combination of area and time thresholds can be set, so if the area is large, it will be notified even if it changes for a short time, and if it is small, it will be notified that it will not change for a long time. You cannot make settings such as

  In the method of setting sampling pixels at random, detection according to the above assumption can be performed probabilistically, but the worst value of detection omission can be guaranteed only probabilistically. This is because when the operation setting is required for the set value of the agent, for example, when the screen history is used as evidence for the test by the developer or the important operation by the user, or when the agent executes the mission critical automatic operation, For example, it becomes a practical problem (in terms of contract provisions, on the demarcation of responsibility).

  In the method of setting sampling pixels in a predetermined pattern, there are cases where a screen change in an extremely vertically or horizontally long rectangular area is missed. Also, no pattern is provided that provides a stepwise and complex filter threshold. Furthermore, no means for changing these patterns according to the set values is presented. Furthermore, no detection model that can intuitively correspond to the set value and the area and change time that can be detected is not presented.

  Further, in handling such a stepwise filter threshold, a measure for suppressing the load by roughening the filter generally does not work effectively. In other words, it is necessary to fix the capture area (or regional granularity) and time (or periodic granularity) based on the finer setting of the stepwise filter threshold. The data obtained by this is over-spec, and a large amount of changes that do not satisfy the conditions are ignored. In other words, it means that wasteful capture processing and image data area reservation frequently occur. As a result, the computer load is dominated by the heaviest combination of filter settings and will not be as low as expected.

In the present invention, in order to solve the above problem,
A device for detecting changes in the display screen of a computer,
A scan line pattern generation unit that generates a pattern by a combination of scan lines specified with a predetermined interval in the X direction or the Y direction or both of the display screen;
A capture processing unit that captures a rectangular image on the screen for each scan line;
A capture result storage unit for storing captured image data;
A scanline difference comparison unit that compares the difference between the results accumulated in the previous cycle and the latest results,
A screen change notification unit for notifying the start and end of the screen change continuation state to the outside ,
The scan line pattern generation unit generates a scan line pattern having a predetermined change size sensitivity and change timing resolution,
The scan line difference comparison unit refers to a time stamp of the latest change time associated with each scan line .

In the present invention, in order to solve the above-mentioned problem,
A method for detecting changes in a display screen of a computer,
A step of generating a pattern by a combination of scan lines specified with a predetermined interval in the X direction or the Y direction or both of the display screen by the scan line pattern generation unit;
A step of capturing a band-shaped rectangular image on the screen for each scan line by the capture processing unit;
A step of storing captured image data by a capture result storage unit;
A step of comparing the difference between the result accumulated in the previous cycle and the latest result by the scan line difference comparison unit,
The screen change notification unit, at least viewed including the step of notifying towards the point beginning and end of the continuous state of the screen changes to the outside,
In the scan line pattern generation step, a scan line pattern having a predetermined change size sensitivity and change timing resolution is generated,
In the scan line difference step, the time stamp of the latest change time associated with each scan line is referred to.

  According to the present invention, it is possible to provide the minimum height and the minimum width of the change region in which it is ensured that there is no omission of detection by acquiring the screen state with the pattern of the scan line instead of the sample pixel.

  In addition, by preparing a plurality of patterns for each resolution and applying them by overlapping them at a predetermined cycle / frequency, detection based on the above-described step detection model can be performed.

  Further, the load of the capture process can be smoothed by decomposing the high-resolution pattern and applying the pattern in a plurality of times within one period.

  Further, by dynamically changing the pattern according to the state of screen detection, the operation can be adapted according to the allowable load of the capture process and the required detection accuracy without changing the processing framework.

  Furthermore, by increasing the width of the scan line statically or dynamically, it is possible to increase the ratio that the change area is applied to the scan line and to improve the effective sensitivity.

Explanatory drawing showing how the display is updated when the window is brought to the front Sequence diagram of general display update processing when bringing a window to the front Basic flowchart of screen change detection processing Explanatory diagram of the detection omission area depending on the type of sampling pattern Illustration of multiple scan line patterns with different resolutions Explanatory drawing when multiple scan line patterns with different resolutions are implemented at appropriate intervals Explanatory drawing when the scan line in the scan line pattern is divided into multiple times Explanatory diagram of the scan line with the overlap between multiple scan line patterns with different resolutions removed Explanatory diagram when multiple scan line patterns with different resolutions are removed at appropriate intervals and divided into multiple times Configuration diagram of a PC environment including a screen change detection device of the present invention Functional configuration diagram showing an example of an embodiment of a screen change detection device of the present invention Flowchart of processing in screen change detection device of the present invention Flowchart of processing of detection mode control unit and scan line pattern generation unit Flowchart of processing of the scan line difference comparison unit Flowchart of processing of capture result storage unit Illustration of data structure of internal processing Flowchart of other processing of detection mode control unit and scan line pattern generation unit Explanatory drawing of the judgment cycle of the difference between the prior art and the present invention Explanatory diagram when the determination cycle of differences and the resolution of the pattern are dynamically changed Explanatory drawing showing pseudo code of scanline pattern generation function Explanatory drawing of index of scan line display position Illustration of the coordinates of the display position of the scan line

  FIG. 1 shows an example of a problem in a conventional screen capture process, in which an example of bringing one window to the front of another window. In conventional screen capture, image data in the middle of display update may be acquired. On the other hand, a method of capturing with a standby time set in advance is common, but the optimal standby time differs depending on the operating environment, the target application, and the window. In addition, the time required for display update may vary depending on the situation at the time of execution, and there is always a possibility that a state in the middle will be captured with a fixed time setting. An object of the present invention is to monitor a change in image content, detect the start of the change, observe whether the change continues under a predetermined condition, and detect the end of the change (background art). In this section, detection of the start of change is described in mind, but in reality, the purpose is to recognize the continuation of change and detect the end of change.)

  FIG. 2 shows a general display update processing sequence when the window is brought to the front. As can be seen from this figure, even if an operation event can be detected, it is theoretically impossible to synchronize the display update, that is, the timing of the end of the drawing process, even by devising the processing flow.

  The biggest reason for this is that, as indicated by (1) in the figure, the OS specification may continue asynchronously without waiting for the hook return, and is largely dependent on the situation at the time of execution. Specifically, in the OS of Microsoft (registered trademark) such as Windows (registered trademark) XP, Vista, 7, etc., in order to detect an operation event related to the GUI by another process, a Windows (registered trademark) message is displayed. It is common to use the global hook mechanism. As a specification of this mechanism, in order not to freeze the process that generated the event, when the process does not return from the hook procedure for a certain period of time, the next process is continued without waiting for the return. Therefore, if a process that takes time such as a capture process is performed, there is a possibility that the drawing process has already started at the time of capture completion. Even if you try to acquire the image immediately before the start of operation by detecting the event “Start window activation from now on” (just before the start of operation) in the Windows® message, the drawing of the image immediately after the operation has already started. The case where the state that has been done occurs cannot be avoided.

  On the other hand, even if you try to acquire the image immediately after the completion of the operation by detecting the event “Windows activation is completed and completed” (immediately after the operation is completed) in the Windows (registered trademark) message, the drawing is still in progress It is not possible to avoid the case where this occurs. This is because, as shown in (2) to (5) in the figure, the capture may be earlier than the drawing completion timing, and greatly depends on the situation at the time of execution. This is the second reason. Specifically, the processing trigger is registered in the queue in (2) and (4), this is taken out from another thread and the subsequent processing is executed, and each corresponding in (3) and (5) This is because the execution time required to obtain the actual effect of the process is different, or the priority order of other processing requests within the processing unit is different.

  The actual processing contents of (2) to (5) are as follows. (2) is based on the repaint algorithm on the GUI platform of the detection target AP, but generally only the flag is set to the GUI element that needs to be redrawn in the thread that requested the repaint, and the GUI platform window When control is passed to the event loop thread in the manager, the actual drawing process is executed. For (3), depending on the specifications of the GUI platform and the OS's graphics environment, when hardware acceleration is enabled, graphics drawing commands generated by drawing processing are sent to a drawing engine such as Open GL. A series of processes until the data is stored and executed on the graphic memory is performed using a predetermined execution time.

  For (4), in order to shorten the processing time in the hook procedure as described above, the event is registered in the original event queue inside the agent and the processing is returned promptly, and the registration is performed by the agent's own polling event. By extracting the contents, processing corresponding to the event contents is performed. In addition to screen capture, the agent uses the same event hook mechanism to acquire and store user operation logs, and to perform other processes such as executing automatic operations based on scenarios. For this reason, the execution time of the agent for processing corresponding to the event content extracted from the event queue is different, and control such as changing the priority of the processing order is required depending on the event content. As a result, the timing from when an event requiring screen capture is registered in the queue to when the capture processing request is actually transmitted to the OS depends on the execution situation such as the internal implementation of the agent and the contents of events before and after.

  In addition, (2) and (3) depend on the internal implementation of the AP to be detected and the GUI platform, and are provided by an external development vendor different from the agent. Of course, it is impossible to see the source code, and (4) and (5) must be implemented universally for multiple types of detection targets. It should be noted that special implementations are not allowed.

  It should be noted that, as described separately, it is not permissible as a premise of the present invention to handle the OS and hardware. In addition, the above has been described on the assumption that the environment of the Windows (registered trademark) OS and the GUI platform thereon is the same, but the same problem occurs in other environments such as the Linux (registered trademark) OS and the GUI platform on the same. It is thought to occur. However, if it tries to cope with it as a device on the general processing flow of the display update described above, it becomes an individual implementation and the development is costly. Furthermore, in the case of display updates other than window fronting (such as button presses), the processing flow is the same as the above, but the necessary devices differ individually, and it is realistic to consider the corresponding method for each. Have difficulty. For this reason, a method of directly monitoring and comparing and detecting changes in image content through screen capture is an important method from the viewpoint of practicality.

  FIG. 3 shows a basic flow of a method for detecting a change in the screen by capturing the entire screen or a specified predetermined partial area. In this method, if the rectangular area to be captured is too large, the processing load becomes heavy. In addition, if the capture cycle is short, the change detection sensitivity increases, but if there is a momentary pause in the middle of the display update, it is detected one by one, and in the flow of granularity that humans can recognize the change Cannot determine whether the overall change has subsided. Furthermore, obtaining image data from a rectangular area using a bitmap is advantageous in that it compares without leaking even one pixel, but the sensitivity is too high because all the changes are detected for changes in a minute area. It is not possible to judge whether the overall change within the spread of

  Although not shown in the figure, pixel sampling with randomly specified positions can detect the above requirements stochastically, but cannot guarantee the minimum detection sensitivity.

  FIG. 4A shows an example of pixel sampling that is regularly positioned. The detection sensitivity can be guaranteed by the pixel interval. However, in comparison in units of pixels, detection cannot be guaranteed if it falls below either the minimum height or the minimum width of the change area that can be detected, such as the area indicated by the broken line. This is contrary to human intuition with respect to image changes that occur on the display screen, and it is difficult to determine the appropriateness of the application environment and settings in practical use. For example, it can be seen that there is a high possibility of omission of detection for the display update of the window frame that is first performed when the window is brought into front as shown in FIG.

  Therefore, a scan rectangular shape having a capture rectangular area of 1 pixel width (setting a plurality of pixel widths is within the scope of the invention) according to the concept shown in FIGS. 4B, 4C, and 4D. This makes it possible to solve the above problems while reducing the processing load. 4B is a vertical scan line pattern (fixed in the X direction), FIG. 4C is a horizontal scan line pattern (fixed in the Y direction), and FIG. 4D is the same figure. Examples in which the scan line patterns of (b) and (c) are superimposed are shown.

  Furthermore, a plurality of scan line patterns in which the minimum height and minimum width (resolution) of the change area that can be detected are changed are generated based on the concept shown in FIG. Further, according to the concept shown in FIG. 6, at each timing on the capture cycle, a capture process is performed based on a pattern in which scan line patterns for each resolution are superimposed at appropriate intervals. It is possible to realize an operation such as notifying even if there is a change in the number, or not informing if the change is small for a long time. Furthermore, with the concept shown in FIG. 7, if a large number of scan lines on the pattern are distributed and arranged at each timing on the capture cycle and it is guaranteed that one scan line is recaptured and compared in one cycle, Leveling can be realized.

  Further, as shown in FIG. 8, since the fine resolution pattern has a scan line that overlaps the coarse resolution pattern, the number of scan lines to be processed when distributed and arranged at a plurality of times in FIG. 7 can be halved. The effect is obtained. In addition, as shown in FIG. 9, when the capture timings are distributed within one period while overlapping a plurality of patterns having different resolutions, the number of lines captured at each timing can be leveled. It is done.

  FIG. 10 is a configuration diagram of a PC environment including a screen change detection device (screen change detection agent) according to the present invention, in which 1 is a computer, 2 is a display (display device), 3 is a keyboard (input device), Reference numeral 4 denotes a mouse (input device). The computer 1 includes a hard disk (storage device) 5, an internal memory 6, a system bus 7, and a CPU 8, and on the internal memory 6, a screen change detection device of the present invention together with a Windows (registered trademark) OS (operating system) 10. 20 is implemented.

  FIG. 11 is a functional configuration diagram showing an example of an embodiment of the screen change detection device 20 of the present invention, in which a pattern is formed by a combination of scan lines specified with a predetermined interval in the X and Y directions of the display screen. A scan line pattern generation unit 21 to generate, a capture processing unit 22 to capture a strip-shaped rectangular image on the screen for each scan line, a capture result storage unit 23 to store captured image data, and a previous cycle A scan line difference comparison unit 24 that compares the difference between the accumulated result and the latest result, a screen change notification unit 25 that notifies the start or continuation or end of the screen change to the outside of the apparatus, and a screen change notification unit 25 Memorize the notification contents, change the detection mode according to the combination of the current state on the display screen and / or the state of the external environment. And a detection mode control unit 26.

  FIG. 12 shows the flow of processing in the screen change detection apparatus 20 of the present invention. In s1, the scan line generation unit 21 generates a scan line pattern. The scan line generation unit 21 holds the pattern referred to by the capture processing unit 22 as a current pattern. If it is the same as the previous pattern, the current pattern for subsequent processing is not updated. Update if it is different. Details of the processing are shown in FIG. In s2, an image is captured for each scan line. Details of the processing are shown in FIG. In s3, the acquired captured content (image) is compared with the previous captured content. This can be realized by comparing the capture area on the screen buffer, which will be described later, with the captured content, but it is also equivalent to other implementation methods, such as preparing an array that holds the acquired content immediately before the captured content. Can be realized.

  In s4, the previous captured content is compared with the current captured content, and a conditional branch is made as to whether or not there is any difference. If the branch is Yes (at least one difference occurs), if the current updating flag is OFF in s5, it is set to ON and the start of the screen change is notified to the higher function. The notification is executed via the screen change notification unit 25. If the branch is No (no difference has occurred), s6 and subsequent steps are executed.

  In s6, the time stamp of the latest change time held for each scan line is referred to and compared with the current time. If even one of them has a time difference exceeding a time difference of one cycle, a conditional branch is performed. This can be determined by referring to the change start time lastTime [] that is the result of s4, and all are null, that is, not set. When the above is false, if the current updating flag is ON in s7, it is set to OFF, and the end of the screen change is notified to the higher function. When the above is true, the updating flag is not changed and no notification is given to the upper function. This means that it is in the middle of change.

  Here, the flow branched conditionally by the determination of s4 and s6 joins in s8 through the above process. Here, the current capture content is saved for later comparison. The specific method can be considered variously as described in the explanation of s3. Here, an image object is generated in advance as a screen buffer having the same screen size, and the currently captured content is copied to a predetermined position. This position is defined as x, y, w, h in the rectangular area for each scan line (x, y are the coordinate positions of the upper left corner of the area starting from the upper left corner of the screen, w is the width of the area, h is The scan line area rect [] array holding the area height) is prepared in advance and determined by referring to this.

  FIG. 13 shows details of the operation flow of the detection mode control unit 26 and the scan line pattern generation unit 21. Here, the value of the update flag and the value of the CPU load are acquired as the current screen change detection state by the function of the detection mode control unit 26. Then, conditional branching is performed according to the range of the value, an appropriate pattern is generated by the function of the scan line pattern generation unit 21 (s11, s12), and comparison with the current pattern used so far in the loop processing is performed. If necessary, it is updated (s13). In this embodiment, in addition to the state of image content change, the CPU load of the PC is measured, and the pattern is adapted depending on whether it is above or below the threshold.

  That is, when the load is high, there is a high possibility that the display update is continued even if the image change is momentarily stopped. Therefore, the resolution is lowered so that stillness is not detected unnecessarily. Specifically, when the load is low, a 2-64 divided pattern is generated to detect stillness as soon as possible (s11). When the load is high, only the 16 divided patterns are validated to detect only the divided patterns having a specific change size resolution and change timing resolution, and a pattern in which the others are paused is generated (s12).

  FIG. 14 shows the details of the operation flow of the scan line difference comparison unit. In s21, loopIndex is incremented, and if it exceeds the maximum value Tmax of one cycle, it is reinitialized to zero. In s22, the function calcIndex () is called, and the index number corresponding to the current loopIndex and patternIndex is assigned to the scanIndex variable. In s23, a rectangular area to be actually captured on the screen is acquired from the scanRect [] array, and the area is captured. In s24, the previous captured content is re-acquired from the screen buffer screenBuffer image and compared with the captured content. In s25, if there is a difference as a result of the image comparison, the time stamp of the change start is stored in the latest change time lastTime []. Specifically, refer to the change start time lastTime [] for each scan line captured at that time, and if there is no difference in the image comparison result and it exceeds the time difference of one cycle from the current time, it is overwritten with null. Not set. If there is a difference between the image comparison results, the image is newly overwritten with the current time. In s26, for each of the 2, 4, 8, 16, 32, and 64 divided patterns, conditional branching is performed depending on whether the processing related to the corresponding pattern in the patternIndex loop is completed.

  FIG. 15 shows the details of the operation flow of the capture result storage unit. In s31, the captured content is stored in the memory for comparison in the next loop for each scan line. Specifically, the captured content is overwritten in the area on the image object in the screen buffer.

  Here, there are points to be noted in understanding the contents of the flow. First, the detailed flow of FIG. 14 corresponds to two of s2 and s3 of FIG. The detailed flow in FIG. 15 corresponds to s8 in FIG. 12, but the actual processing may be performed between s24 and s26 in FIG. Further, in the conditional branching of s4 and s6 in FIG. 12, the substantial comparison processing is performed in s24 in FIG. 14, and the result is only stored and referred to in lastTime []. These points occur because the structure of FIG. 12 is left in correspondence with the flow of FIG. 3, which is a conventional method, in order to facilitate understanding of the flow of FIG. As detailed processing contents, there are different parts as described above, but the flow is substantially the same.

  FIG. 16 shows work variables, fixed variable values, and functions that are necessary for the internal processing described above. Pattern class, Rectangle class, and Image class mean that the implementation programming language is an object-oriented language, respectively, and the library class if it already exists in the execution environment, and the class that is individually defined if it does not exist. ing. Pattern class is a class for converting work variables and fixed variable values into objects for each scan line pattern of 2 to 64 divisions. Instances are generated one by one in internal processing, that is, as singletons. Rectangle class is a class for converting the geometric information of a rectangular figure on the coordinates of the display screen into an object. The x coordinate and y coordinate of the upper left corner, and the width w (abbreviation of width) and height h (abbreviation of height) of the rectangle are held as internal variables. The Image class is a class for converting the image data in the implementation into an object.

  In another embodiment, instead of coarsening the change size sensitivity (minimum size that guarantees detection at a given time) in detecting the start of change, change timing resolution (detection reaction speed, ie, detection at a fast time interval) Explain the case where the ability to obtain results) is fine.

  FIG. 17 shows an adaptive control method based on the runtime situation and user settings based on s11 and s12 of FIG.

  Here, s41 is a step added to reflect the above-mentioned idea. This is a method in which, in the state where the updating flag is OFF and the display is not updated, it is considered that the update is started only when a certain region or more changes at a stroke from the viewpoint of avoiding unnecessary load. In order to realize this, a pattern having a high change size resolution is paused so as not to detect the start of change for a change in a small region. Specifically, the processing load is reduced to 1/5 by operating only the 8-division pattern and pausing the remaining 2, 4, 16, 32 division patterns. Therefore, the change timing resolution is increased for the change size resolution using the available processing capacity. Specifically, the same change timing resolution as that of the conventional two-division pattern is realized by capturing the eight-division pattern for four scan lines at one timing. In the figure, this is called a quadruple speed pattern.

  This assumes a display update for fronting the window. As shown in FIG. 1, when the window is brought to the front, display update is started by displaying a window frame (ie, the height and width of the change area is large), and the internal area is changed as the display update progresses. It is based on the characteristic of changing, and it is intended to perform control so that the start of the update is detected with as little time lag as possible, and the end of the update is suppressed as much as possible.

  s42 is a step added to guarantee the minimum change size resolution although the change timing resolution is poor. This is effective when the setting for ignoring the change in the small area is turned off in advance by the user setting, and an instruction is given to maintain the detection sensitivity at the worst level at the conventional level. Specifically, by operating a 32 divided pattern in parallel with the 8 divided pattern quadruple speed type and capturing 5 scan lines at one timing, the processing load is the same as the conventional load level corresponding to s11. However, it is possible to realize a detection process in which the detection sensitivity of the change size and change timing in the worst case is comparable.

  Since the pattern generated in this way is in a lattice shape, a small area detection is inherently inevitable. In the present invention, if the change size resolution can be set as finely as possible, the detection omission in a small region can be reduced to the theoretical limit. However, in order to cover a general display screen size of 1024 pixels or 2048 pixels, it is necessary to handle 10 or 11 scan lines at one time. Depending on the characteristics and performance of the graphics engine, capturing a rectangular area with a relatively large area at a time may have a smaller load in terms of processing time and memory consumption than capturing many elongated rectangular areas. . Also, in practice, when the rectangular area size that is guaranteed to be more than a certain level is reduced, it is considered that there is almost no utility for setting the guarantee of the detection delay time in fine steps.

  Therefore, it is conceivable to cover the entire display screen area without a gap by making the scan line thicker for a certain division pattern or more. s43 is a step added to reflect the idea. Specifically, only when the CPU load is less than 50%, the scan line width should be 16 pixels (the direction of the display screen is 1024 or less) or 32 pixels (the direction of the display screen is 2048 or less). And this is realized. Here, this means a pattern that masks the entire screen, and this is referred to as a mask version of a scan line pattern.

  As described above, according to the state definition of the detection mode control unit 26 and the generation logic of the scan line pattern generation unit 21, these adjustments can be easily changed according to the application target. In particular, since dynamic change is possible, adaptive control can be easily implemented and customized in the configuration of the present invention. For example, it is possible to add a scan line of a pattern with high resolution such as 64 divisions and 256 divisions, or a double-speed type that processes a scan line of 256 divisions two at a time. The combination can be selected according to the performance of the graphics engine, and since there is no logical dependency between the resolutions set in multiple stages, there is an effect that the handling is easy.

  FIG. 18 shows how the time difference from when an actual screen change occurs until a detection result is obtained can be suppressed to be shorter than in the conventional method. Although the effect of detecting the start of change is clear, there is a one-cycle time difference as in the conventional case for detecting change completion. However, the existence of the possibility of completion of the change can be detected with a shorter time difference than in the past. This is apparent from the fact that the solid arrows at the bottom of the drawing appear with a short time difference. For this reason, when there is a possibility of completion of the change, it is possible to detect with a short time difference by adaptively increasing the resolution.

  FIG. 19 shows a state in which a pattern with high resolution is enabled and set to 2 × speed or 4 × speed in the above state. Furthermore, by increasing the width of the scan line of a pattern with high resolution so that there is no gap between the lines, detection determination covering the entire display screen becomes possible. On the other hand, when the CPU load is large, it is possible to prevent the screen change detection agent from taking away the CPU resources of other processes on the terminal by suspending the pattern with high resolution adaptively.

  In the above description, scan lines in one direction on the display screen are described. However, in reality, it is assumed that the same method and implementation are performed in the vertical direction and the horizontal direction, and both are operated simultaneously. It is said. This is a problem area to which the present invention is applied, because it corresponds to both extremely elongated rectangles in the horizontal direction and elongated rectangles in the vertical direction. When the detection target only needs to correspond to one of the other, Even if omitted, the effect of the present invention is basically effective.

  A specific example of an algorithm for generating a pattern in which scan lines are arranged in a grid pattern necessary for screen capture will be described.

  Here, only the horizontal direction is taken into consideration, and the concept when the number of scan lines captured at one time is 5 and the screen size is 1024 pixels is shown. It is assumed that the resolution of the pattern is detailed in half, such as when the size of the change region in which detection is guaranteed is 1/2 of 1024, and when the size is 1/4. Further, it is assumed that the position of the scan line at each timing in one resolution pattern simply moves from left to right. By changing these assumptions, as already described elsewhere, the order is changed so that the entire screen is examined at the timing dispersed within the capture cycle. It may be possible to control the order so that it is investigated by extending the length or to change it arbitrarily. In addition, it can be implemented so as to change at each activation depending on the contents of the setting file or the like, or can be implemented so as to dynamically change according to the situation at the time of execution. These can be easily realized by arbitrarily applying the algorithm described here as a basis.

  When the generation algorithm is processed on an actual programming language, it is executed by incrementing the counter at each capture timing and calling a function with the numeric value as an argument. As the calculation result, the position of the scan line to be captured at the timing is obtained as a horizontal coordinate value. Since a plurality of scan lines are captured at a single timing, it is necessary to return a plurality of results, and the return value type of the function is an integer array variable.

  FIG. 20 shows pseudo code of the above function. Within this function, a loop is executed for the number of patterns for each resolution calculated by the algorithm. Hereinafter, the calculation process when i = 2 will be described as a specific example. The idea at this time is shown in FIG. 21 and FIG.

  When i = 2 is calculated on this algorithm, the period during which the position of the scan line moves is 4 (assigned to the variable freq), and the resolution corresponds to the third pattern from the finest. Considering the number of pixels in the horizontal direction of the display screen handled in this process in advance, it is assumed that ssize = 10 is set, and the resolution is obtained by calculation based on this (substituted in the variable reso). In this case, the timing is counter = 2791.

  The result “3” obtained by dividing couter by 4 corresponds to “3”, that is, the fourth timing from FIG. 21 with “0” in the resolution pattern as an initial value. From FIG. 22, the coordinate value obtained by shifting the fourth coordinate position to the right by half the resolution when the horizontal direction of the display screen is divided by the resolution width is a numerical value to be calculated. In the above algorithm, the necessary coordinate positions of a plurality of scan lines are calculated by repeating this calculation for the number of resolution patterns to handle i.

Note that the counter variable representing the timing must be a positive integer. Therefore, when an arbitrary multiple (for example, 320 or 2048) of the longest cycle (2 ⁵ = 32 in this example) is exceeded in operating the algorithm, it is necessary to reinitialize to 0.

  The present invention includes a program for realizing the functions shown in the configuration diagram of FIG. 11 or the procedures shown in the flowcharts of FIGS. 12 to 15 and 17 in a known computer via a medium or a communication line. It can also be realized by installing a program.

  1: Computer, 2: Display, 3: Keyboard, 4: Mouse, 5: Hard Disk, 6: Internal Memory, 7: System Bus, 8: CPU, 10: Operating System, 20: Screen Change Detection Device (Screen Change Detection Agent ), 21: scan line pattern generation unit, 22: capture processing unit, 23: capture result accumulation unit, 24: scan line difference comparison unit, 25: screen change notification unit, and 26: detection mode control unit.

JP 2010-204840 A Japanese Patent No. 4476125 Japanese Patent No. 3481515 Japanese Patent No. 3175947 Japanese Patent No. 4394595

Claims (7)

A device for detecting changes in the display screen of a computer,
A scan line pattern generation unit that generates a pattern by a combination of scan lines specified with a predetermined interval in the X direction or the Y direction or both of the display screen;
A capture processing unit that captures a rectangular image on the screen for each scan line;
A capture result storage unit for storing captured image data;
A scanline difference comparison unit that compares the difference between the results accumulated in the previous cycle and the latest results,
A screen change notification unit for notifying the start and end of the screen change continuation state to the outside ,
The scan line pattern generation unit generates a scan line pattern having a predetermined change size sensitivity and change timing resolution,
The scan line difference comparison unit refers to a time stamp of the latest change time associated with each scan line . The screen change detection device according to claim 1,
A screen change detection device, wherein the scan line pattern generation unit generates a pattern in which a plurality of scan lines having different resolutions are superimposed at a predetermined timing of a capture cycle. In the screen change detection device according to claim 1 or 2,
A screen change detection device, characterized in that the scan line pattern generation unit disperses and arranges a set of scan lines of a high resolution pattern at each timing within a capture cycle. In the screen change detection device according to any one of claims 1 to 3,
In addition to the above
A notification mode control unit that stores notification contents of the screen change notification unit and changes and maintains the detection mode according to a combination of the current state on the display screen and / or the state of the external environment,
A screen change detection apparatus, wherein a scan line pattern generation unit changes a pattern to be generated with reference to the detection mode. The screen change detection device according to claim 4,
A screen change detection device, wherein the width of a scan line is increased to a predetermined fixed width by setting, or is dynamically increased according to a situation at the time of execution by a detection mode control unit and a scan line pattern generation unit.   The program for functioning a computer as each part of the screen change detection apparatus in any one of Claims 1 thru | or 5. A method for detecting changes in a display screen of a computer,
A step of generating a pattern by a combination of scan lines specified with a predetermined interval in the X direction or the Y direction or both of the display screen by the scan line pattern generation unit;
A step of capturing a band-shaped rectangular image on the screen for each scan line by the capture processing unit;
A step of storing captured image data by a capture result storage unit;
A step of comparing the difference between the result accumulated in the previous cycle and the latest result by the scan line difference comparison unit,
The screen change notification unit, at least viewed including the step of notifying towards the point beginning and end of the continuous state of the screen changes to the outside,
In the scan line pattern generation step, a scan line pattern having a predetermined change size sensitivity and change timing resolution is generated,
In the scan line difference step, a screen change detection method is characterized in that a time stamp of the latest change time associated with each scan line is referred to .

Images