JP7092364B2 - Business automatic execution system and method - Google Patents

Description

The present invention relates to an automatic business execution system and method.

RPA (Robotic Process Automation) is known as a system for the purpose of automating business using a personal computer. RPA makes it possible to reduce user operations by repeatedly executing recorded user operations. RPA contributes to the automation of work by performing repetitive operations on behalf of the user for 24 hours without the need for human intervention, and thus aims to improve the productivity of the entire work.

In many RPA systems, first, the contents of operating an icon or window arranged on the desktop screen of a personal computer with an input device such as a keyboard or a mouse are recorded as a user's operation history. As the recorded content, a set of "operation target" and "operation content" is recorded as a plurality of steps. This multiple step is stored as an "operation script". When the RPA system re-executes the "operation script", the "operation target" is searched and specified from the desktop of the personal computer for each step. Execute the recorded "operation content" for the found "operation target". Do this step by step.

Specifically, when using RPA to select an icon on the desktop screen to start a specific application or to execute a specific function from the toolbar in the application, the icon displayed on the personal computer screen. The image of the or toolbar is captured in advance, and the image data is registered as the operation target image (see Patent Document 1). More specifically, the operation target image (icon, toolbar, etc.) and the operation content (for example, left button double-click, right-click, etc. of the mouse device) are linked and registered. Then, when RPA is executed, an image matching the registered operation target image is detected from the personal computer screen, and the registered operation content is performed for the detected image to automate the business.

Japanese Unexamined Patent Publication No. 2019-032719

The disclosure of the above prior art document shall be incorporated into this document by citation. The following analysis was made from the point of view of the present invention.

According to the RPA execution method of Patent Document 1, only the desktop screen is displayed on the personal computer screen even though the desktop screen and the window screen are held independently. If the desktop screen is occupied by another window screen that is not the operation target due to a user operation different from that at the time of recording, what was displayed at the time of operation recording, the icon or window to be operated The screen and toolbar may be hidden. At this time, the pre-registered operation target image cannot be found. Even if the toolbar is not hidden, if the resolution or scale of the screen is different from the operation target image due to the environment setting operation by the user, or if the background image of the toolbar is different, the image of the toolbar is accurately displayed. It may not be possible to detect it. In this sense, there are major restrictions on the operating environment in order for RPA to operate normally.

The present invention has been made in view of the above problems, and an object of the present invention is to contribute to more accurately specifying an operation target in response to changes in a desktop screen or the like when RPA is executed.

From the first viewpoint, an automatic execution system for business is provided. This system is an automatic business execution system having an operation registration unit that records user operations using a personal computer and an operation execution unit for re-execution of the operation registration unit. The operation registration unit captures an image of an operation target. An operation target storage unit that is specified and stored as an operation target, an operation content storage unit that stores an operation for the operation target, an operation script storage unit that associates the operation target with the operation content and stores the operation content, and a search for the operation target. It has a virtual screen data creation unit that stores an image image including conditions as virtual screen data, and the operation execution unit reads the recorded operation script and refers to the image image and the virtual screen data for operation. Provided is an automatic business execution system, characterized in that an operation for the operation target is executed after specifying the target.

According to the second viewpoint, a method of automatically executing a business is provided. This method is a method of automatically executing a business, and is an image image including an operation target storage step of specifying and storing an operation target as an image image and a search condition of the operation target in the operation registration unit of a personal computer for performing operation recording. It has a virtual screen data creation step for storing the operation target as virtual screen data, an operation content storage step for storing the operation for the operation target, and a step for saving the operation script for storing the operation content in association with the operation target. In the operation execution unit of the personal computer that executes the operation record, the step of specifying the operation target by searching the desktop screen and the virtual screen data together from the operation script, and the operation of executing the recorded operation content. A method of automatically executing a business having an execution step is provided .

According to each viewpoint of the present invention, it contributes to more accurately specifying the operation target at the time of executing RPA.

It is a block diagram which shows the whole structure of the automatic execution of business which concerns on one Embodiment. It is a figure which shows the problem of the automatic execution of business which concerns on one Embodiment example. It is a flowchart which shows an example of the operation of the operation registration part 10 which concerns on 1st Embodiment example. An example of the storage structure of the virtual screen data 31 according to the first embodiment is shown. It is a flowchart which shows an example of the operation of the operation execution part 20 which concerns on 1st Embodiment. An example of a desktop screen (at the time of operation recording-1) according to the first embodiment is shown. An example of a desktop screen (at the time of operation recording-2) according to the first embodiment is shown. An example of a desktop screen (at the time of operation execution-1) according to the first embodiment is shown. An example of a desktop screen (at the time of operation execution-2) according to the first embodiment is shown. An example of a desktop screen (at the time of operation execution-3) according to the first embodiment is shown. An example of a desktop screen (at the time of operation execution-3 modification) according to the first embodiment is shown. It is a block diagram which shows the whole structure which concerns on 2nd Embodiment example. It is a flowchart which shows an example of the operation target storage part 11 which concerns on 2nd Embodiment example. It is a flowchart which shows an example of the operation execution part 20 which concerns on 2nd Embodiment example. An example of a screen at the time of execution of the operation registration unit 10 and the operation execution unit 20 according to the second embodiment is shown. A description of the tooltip information according to the second embodiment is shown.

First, an outline of one embodiment will be described. It should be noted that the drawing reference reference numerals added to this outline are added to each element for convenience as an example for assisting understanding, and the description of this outline is not intended to limit anything. Further, the connection line between the blocks in each block diagram includes both bidirectional and unidirectional. The one-way arrow schematically shows the flow of the main signal (data), and does not exclude bidirectionality.

As described above, a system that contributes to more accurate execution of a business automatic execution system for automatically executing a personal computer operation is desired.

The present invention focuses on a part of an automatic business execution system for specifying and recording an operation target in an operation registration unit and a part for searching and specifying an operation target in the operation execution unit 20.

The configuration of one embodiment of the present invention will be described in detail with reference to the drawings.

Referring to FIG. 1, an embodiment of the present invention includes an operation registration unit 10 for storing user's operation contents, a stored operation script 30, and an operation execution unit 20 executed at the time of automatic execution. Here, the personal computer having the operation registration unit 10 and the personal computer having the operation execution unit 20 may be the same or different. In FIG. 1, the operation registration personal computer 100 and the operation execution personal computer 200 are tentatively distinguished. However, logically, it must be possible to realize the same function. For example, the operation script 30 performed and recorded by the operation registration unit 10 while referring to the desktop screen of the display device 100a can be executed by the operation execution unit 20 using the desktop screen of the display device 200a (application, library, user). It is assumed that there is no problem with the authority etc.).

The operation script 30 shown in FIG. 1 has a series of user operations as a plurality of steps. The operation script storage unit 13 may name this and save it, or may save it as a script program called from another process. In the present embodiment, for the sake of brevity, "automatic execution record N" (where N is assumed to be counted up for each operation record) is given and saved as the name given to the operation script 30. The operation execution unit 20 shall execute the designated operation script 30 after receiving the designation of "automatic execution record N" which is one of the operation scripts 30. When the operation registration personal computer 100 and the operation execution personal computer 200 are physically different, the operation script 30 is assumed to be shared on the hard disk via the network.

FIG. 2 shows an example of a desktop screen as a problem to be tackled by the present invention. A desktop screen is displayed on FIG. 2, and one of the icons is stored as an operation target. This is the action of the operation registration unit 10. When re-executing again, in the operation execution unit 20, as shown in the lower part of the figure, when the background design of the icon is changed, the image size is changed, or the icon is hidden in another window. There is. These points contribute to improving reproducibility.

Hereinafter, the first embodiment according to the present invention will be described in detail with reference to the drawings.

[Example of the first embodiment]
The configuration in the first embodiment will be described with reference to FIG. The operation registration unit 10 is composed of an operation target storage unit 11, an operation content storage unit 12, and an operation script storage unit 13. In particular, as a feature of this embodiment, a virtual screen data creation unit 14 that gives additional information to the virtual screen data 31 when the operation target storage unit 11 is executed is provided.

When recording a user operation, the operation registration unit 10 stores an "operation target" and an "operation content", and this is set as one step. The operation target storage unit 11 stores the "operation target", and the operation content storage unit 12 stores the "operation content". The "operation target" is an object (icon, window, tooltip) on the desktop specified by the user, and the "operation content" is the operation content by the keyboard device 100b or the mouse device 100c (for example, double-clicking the left button of the mouse device). ).

The operation execution unit 20 reads the operation script 30 and automatically executes the operation. In this embodiment, the virtual screen data 31 is also read. As described above, it operates in the operation execution personal computer 200 having the operation execution unit 20.

First, the operation registration unit 10 will be described. FIG. 3 explains that the processing of the operation target storage unit 11 and the processing of the virtual screen data creation unit 14 are performed in conjunction with each other.

The operation target storage unit 11 changes the virtual screen data 31 to be operated depending on whether the operation target is an icon or a toolbar in the window.

When the operation target is an icon, the icon image is stored as an operation target (S101), sent to the virtual screen data creation unit 14, and the icon image is added to the virtual screen data 31 (S201). Here, the virtual screen data 31 has a structure in which target image data is stored for each type of icon, window, and tooltip, and the relationship between them is managed. The virtual screen data 31 is global data in this system and can be referred to by the operation execution unit 20 or the like.

When a window is generated in response to an operation such as double-clicking an icon (S102), the window image is sent to the virtual screen data creation unit 14, and the window image and the icon image that generated the window image are displayed. Associate and add. Here, the type of the icon image is "icon" and the type of the window image is "window", and the relationship between the icon and the window is associated and stored.

When the operation target is the toolbar in the window (S110), the toolbar image is stored as the operation target (S111), this is sent to the virtual screen data creation unit 14, and the toolbar image and the window are associated with each other and stored in the virtual screen data 31. Add (S203).

Furthermore, in the case of a toolbar, processing is added after that. Here, text information is acquired from the tooltip corresponding to the toolbar (S112). Actually, it is assumed that the mouse cursor is moved over on the toolbar, the image of the displayed tooltip is acquired, and only the text information is extracted from the image by using the image recognition technique. The virtual screen data creation unit 14 associates this tooltip text with the toolbar image and adds it to the virtual screen data 31 (S204).

FIG. 4 shows a structural example of the virtual screen data 31 generated in this way. The virtual screen data 31 is classified by type and has the types of "icon", "window", and "toolbar". Images of icons, windows, and toolbar operations are recorded for each type. In particular, tooltip text information is added and recorded on the toolbar image. The relationships between icons and windows, windows and toolbars, and toolbars and tooltips are associated and stored as shown in the diagram as relationship lines. Information on all pre-recorded operations related icons, windows, and tooltips is recorded here.

Next, the operation execution unit 20 will be described. FIG. 5 shows an example of a processing flowchart of the operation execution unit 20. The operation execution unit 20 reads the operation script 30 and re-executes the "operation content" for the "operation target" for each step recorded in the operation script 30. When the processing of multiple steps is completed, the processing ends. FIG. 5 corresponds to one-step processing. If there are two steps, FIG. 5 will be repeated twice. If there are N steps, repeat N times.

The operation execution unit 20 searches for the target screen from the desktop, which is the operation screen, for the image obtained from the virtual screen data 31 based on the image data (and association) stored in the virtual screen data 31, and executes an operation for the target screen. Do that. For this purpose, the operation target image is specified from the virtual screen data 31 (S300).

Next, the operation execution unit 20 determines whether or not there is a target screen on the desktop screen (S301). If this is present, the specified operation is executed for the found operation target (S302). In this case, the processing of the recorded one step is terminated.

When there is no operation target on the desktop screen, the operation execution unit 20 divides into three cases. When the operation target is an icon, it is a window, and it is a toolbar. In FIG. 5, the transition is made to S310, S320, and S330.

When the operation target is an icon, it is assumed that the window hides the icon as a possibility of hiding it from the desktop screen. Therefore, the process of converting all the desktop windows into icons is performed (S310). When the icon image appears on the desktop screen as a result (S350), the operation is executed for the icon operation target (S370), and the process ends.

When the operation target is a window, it is assumed that another window is displayed on the entire surface as a possibility of hiding this from the desktop screen, so the process of moving the target window to the foreground is performed (S320). As a result, when the window screen appears on the desktop screen (S350), the operation is executed for the window operation target (S370), and the process ends.

When the operation action is a toolbar, the window screen corresponding to the toolbar is moved to the foreground (S330). However, even if the toolbar is the same, the identity is determined based on the fact that the added tooltip information is the same in case the design, scale, or the like is changed (S340). If it is found here, it is confirmed that the window screen appears on the desktop screen (S350), the operation is executed for the tooltip operation target (S370), and the process is terminated.

If the operation target is not found in each case (NO in S350, NO in S340), an error is displayed (S360) and the process is terminated.

The operation of the first embodiment will be described more specifically using a screen example.

First, a screen example of the operation registration unit 10 is shown. FIG. 6 is a desktop screen at the start of recording. There are two icons, one window (window 2), and two toolbars corresponding to window 2. It is assumed that the window 1 shown by the broken line is not displayed at the start of recording.

Now start recording. The recorded content is recorded as an operation script 30 from the operation script storage unit 13.

In the example of FIG. 6, the user selects the icon 1 as the operation target and double-clicks the left button of the mouse device. Then, in FIG. 6, the window 1 inside the broken line is displayed, which is visualized in FIG. 7, and the window 1 and three toolbars included in the window 1 are displayed. These records are recorded as "Record (1)". The icon 1 in FIG. 7 and the window 1 are stored in the virtual screen data 31.

Next, the user shall left-click on the toolbar 11 included in the newly appearing window 1. Here, text information is acquired from the tooltip corresponding to the toolbar by step S112 of FIG. 3, and the relationship between the toolbar image and the window and the association between the tooltip text and the toolbar image are added by steps S203 and S204. .. The virtual screen data 31 stores the toolbar 11 of FIG. 6, the relationship with the window 1, and the tooltip text "font color. Change the color of characters". The records up to this point were recorded as "Record (2)".

Recording ends here. Here, the user record is as follows. Assuming that the operation name is the automatic execution record 1, this name may be automatically given by the system or may be specified by the user. The operation script 30 is as follows.
"Automatic execution record 1:
"Record (1): Double-click the icon 1 (image data) with the left mouse button."
"Record (2): Click the left button of the mouse device on the toolbar 11 (image data) of window 1".

Next, the action of re-executing the automatic execution record 1 will be described. However, since other recordings are also performed, as shown in FIG. 4, not only the images recorded only by the automatic execution record 1 but also the images of other icons, windows, and toolbars may be recorded. .. The present embodiment will continue to be described based on the latter case.

FIG. 8 shows the desktop display screen at the start of the operation. Here, window 2 occupies and displays the majority of the desktop. Here, the operation execution unit 20 searches for the icon 1 image in order to execute the "recording (1)" of the "automatic execution recording 1".

As is clear from the display of FIG. 8, the same icon as the icon 1 in the virtual image is not displayed. Therefore, the process of the operation execution unit 20 executes a process of iconifying all the windows of the desktop (step S310 in FIG. 5). As a result, as shown in FIG. 9, the display of the window 2 area is erased, and the image of the icon 1 is displayed.

Since the icon target screen was found on the desktop screen (step S350 in FIG. 5), an operation (double-clicking the left button of the mouse device) is performed on the icon 1. This completes recording (1).

By the above operation, the window 1 is displayed. However, in this case, two patterns are assumed. One has the same screen configuration as at the time of recording (FIG. 10). The other is the case where the screen resolution, scale, hue, texture, etc. are changed due to some change in the user environment settings (FIG. 11).

First, a case where the screen configuration is the same as that at the time of recording will be described (FIG. 10). Here, the toolbar 11 image of the window 1, which is the operation target of the recording (2), is searched. Here, since the toolbar image is on the desktop image (step S300 in FIG. 5), the toolbar 11 is operated (clicking the left button of the mouse device) to end the process. This completes recording (2).

Next, a case where the screen configuration is different from that at the time of recording will be described (FIG. 11). Here, the toolbar 11 image of window 1, which is the operation target of recording (2), is not found (because the size, color, and texture are different). Therefore, the operation execution unit 20 moves the window corresponding to the toolbar to the foreground (step S330 in FIG. 5), and searches for the same tooltip text among the toolbars in the same window. (Step S340 in FIG. 5). Here, the tooltip text corresponding to the toolbar 11 is "font color. Change the color of characters", which is the same as the one stored in the virtual screen, so that the toolbar 11 is specified as the operation target. .. Therefore, an operation (clicking the left button of the mouse device) is performed on the toolbar 11 to end the process. This completes recording (2).

This completes the execution of "automatic execution record 1". As described above, in the first embodiment, it is shown that the operation records of the icons, windows, and toolbars can be recorded and re-executed.

[Example of the second embodiment]
Next, the configuration in the second embodiment will be described. As shown in FIG. 12, the operation registration unit 10 includes an operation target storage unit 11 that stores icon images, window information, tool chips, etc., which are operation targets, and a mouse device click, double-click, or keyboard for the operation target. It is composed of an operation content storage unit 12 for storing operation contents such as character input, and an operation script storage unit 13 for managing the operation target and the operation content in association with each other and recording the operation content in the operation script 30. The operation target storage unit 11 has an icon information storage unit 11a for storing an operation target icon and an image of a toolbar (hereinafter, a target icon and the like), and a desktop screen including an icon stored by the icon information storage unit 11a. A window information storage unit that stores the virtual screen including the icon and the toolbar stored by the icon information storage unit 11a among the plurality of virtual screens created by the virtual screen data creation unit 14 described later as desktop information. It is composed of 11b and a tool chip information storage unit 11c that stores tool chips corresponding to the icons and toolbars stored by the icon information storage unit 11a.

The operation execution unit 20 has an operation script reading unit 21 that reads information on the operation target and the operation content from the operation script 30, an operation target search unit 22 that searches for the operation target, and information and the operation target read by the operation script reading unit 21. It is composed of an operation content execution unit 23 that executes an operation based on the search result of the search unit 22. Further, the operation target search unit 22 is selected from the icon information search unit 22a for searching the target icon and the like on the computer screen being executed and the virtual screen corresponding to the window information stored in advance by the window information storage unit 11b. A window information search unit 22b that searches for a virtual screen created by the virtual screen data creation unit 14, which will be described later, and a tool chip that matches the tool chip information registered in the tool chip information storage unit 11c for the virtual screen. It is composed of a tool chip information search unit 22c that searches for the existence of an icon in which is displayed.

The operation script 30 is information having a structure having a plurality of steps, with the "operation target (image data)" and the "operation content" as one step, as in the first embodiment.

First, the operation of the operation registration unit 10 will be described in detail.

FIG. 13 is a flowchart showing an example of the operation of the operation registration unit 10.

First, in the operation target storage unit 11 in the operation registration unit 10a, the icon information storage unit 11a stores the target icon and the like as image information (step S01). When the storage is completed, the virtual screen data creation unit 14 is notified to that effect.

Subsequently, the virtual screen data creation unit 14 creates a virtual screen from the window / desktop (step S02). Specifically, the window screen and the desktop screen opened when the software corresponding to the target icon and the like stored in the icon information storage unit 11a are started are created as a single image by image processing. The single image created at this time is the virtual screen data 31 in the present embodiment. When the creation of the virtual screen data 31 is completed, the window information storage unit 11b can store the virtual screen data 31 by transmitting the data to the window information storage unit 11b (step S03). The window information storage unit 11b notifies the tooltip information storage unit 11c that the storage is completed.

Upon receiving the notification from the window information storage unit 11b, the tooltip information storage unit 11c stores the tooltip corresponding to the target icon or the like stored in the icon information storage unit 11a as tooltip information in a text format (step S04). If the corresponding tooltip does not exist, this step is omitted. In any case, the result is notified to the operation target storage unit 11.

Subsequently, the operation content storage unit 12 stores specific operation content to be performed on the operation target, such as clicking or double-clicking the mouse device or inputting characters on the keyboard (step S05). When the storage of the operation content is completed, the operation script storage unit 13 can associate the information stored in the operation target storage unit 11 with the information stored in the operation content storage unit 12, and further, the operation script 30 This information is saved as (step S06). The information stored in each storage unit may be sequentially transmitted to the operation script storage unit 13 at the time of storage, and may be collectively transmitted when the operation content storage unit 12 has completed storage. May be good.

Next, the operation execution unit 20 will be described with reference to FIG.

The operation script reading unit 21 reads information about the operation icon and the like from the operation script 30 and information about the operation content associated with the operation icon (step S11). Then, when the operation script reading unit 21 completes the reading, the operation target search unit 22 is notified to that effect.

Next, the icon information search unit 22a in the operation target search unit 22 determines whether or not the operation icon or the like exists on the desktop screen of the running personal computer (step S12). If the operation icon or the like can be extracted, the process proceeds to step S13, and if the operation icon or the like cannot be extracted, the process proceeds to step S14.

When the operation target search unit 22 returns information such as a matching operation icon to the operation script reading unit 21, the operation script reading unit 21 determines that the operation can be executed, and requests the operation content execution unit 23 to perform a specific operation. This causes the operation to be executed (step S13).

In step S11, the virtual screen data creation unit 14 creates virtual screen data 31 from the desktop screen and window screen of the running personal computer. The window information search unit 22b to be received may hold the created virtual screen data 31, or the virtual screen data creation unit 14 may temporarily hold the created virtual screen data 31. When the virtual screen data creation unit 14 completes the creation of the virtual screen data 31, the virtual screen data creation unit 14 notifies the window information search unit 22b to that effect.

Next, the window information search unit 22b that receives the notification matches the virtual screen data 31 created by the virtual screen data creation unit 14 in step S14 from the virtual screen data 31 stored in the window information storage unit 11b. Search for whether or not there is something (step S14). If there is no matching virtual screen, the process proceeds to step S20.

Subsequently, the window information storage unit 11b notifies the icon information search unit 22a that there is something that matches the virtual screen stored in advance (step S15). In addition, the window information storage unit 11b also notifies the matching predetermined virtual screen data 31.

Upon receiving the notification, the icon information search unit 22a searches for whether or not there is an operation icon or the like in the predetermined virtual screen data 31 (step S16). The execution content here is the same as in step S12. If the operation icon or the like can be extracted, the process proceeds to step S18. That is, even when the window screens overlap and the operation icons and the like are hidden on the desktop screen of the personal computer to be executed, it is possible to execute normally by referring to the virtual screen. If it cannot be extracted, the process proceeds to step S17.

When the icon information search unit 22a extracts an operation icon or the like, the icon information search unit 22a causes the operation script reading unit 21 to display the desktop screen or window screen corresponding to the virtual screen data 31 including the operation icon or the like. Notice. Upon receiving the notification, the operation script reading unit 21 instructs the operation content execution unit 23 to arrange the corresponding desktop screen or window screen in the foreground so that the corresponding desktop screen or window screen can be operated (step S18). When the arrangement in the foreground is completed, the operation content execution unit 23 notifies the operation script reading unit 21 to that effect. Subsequently, the operation script reading unit 21 determines that the operation can be executed, and requests the operation content execution unit 23 for a specific operation to execute the operation. An example of applying this action is shown in FIG.

On the other hand, if the operation icon or the like cannot be extracted in step S16, the window information search unit 22b notifies the tooltip information search unit 22c to that effect. Whether or not the tooltip information search unit 22c that has received the notification has a tooltip corresponding to the operation icon or the like, specifically, whether or not the tooltip information storage unit 11c stores the corresponding tooltip information. Is confirmed (step S17). When the tooltip information search unit 22c determines that the tooltip information storage unit 11c stores the tooltip information, the icon displayed in the virtual screen data 31 determined to match in step S15. Among them, it is searched whether or not there is an icon having the same tooltip information as the tooltip information stored in the tooltip information storage unit 11c (step S19).

If the same tooltip information as the stored tooltip information exists (YES branch in step S19), the process proceeds to step S13 via step S18. That is, even if the registration icon or the like cannot be matched due to the resolution of the personal computer on which the RPA is executed, the background of the desktop screen, or the like, the tooltip information corresponding to the registration icon or the like can be referred to, so that the operation can be performed. Environmental constraints can be relaxed. An example of applying this action is shown in FIG.

On the other hand, when the tooltip information storage unit 11c does not store the tooltip information corresponding to the operation icon or the like, or the same tooltip information as the tooltip information stored in the tooltip information storage unit 11c exists. If not (NO branch in step S19), the process proceeds to step S20.

In step S15, the window information search unit 22b determines that none of the virtual screen data 31 stored in the window information storage unit 11b matches the virtual screen data 31 created by the virtual screen data creation unit 14. In the case, or as described above, in step S19, the tool chip information storage unit 11c does not store the tool chip information corresponding to the operation icon or the like, or is the same as the tool chip information stored in the tool chip information storage unit 11c. When the tool chip information search unit 22c determines that the tool chip information of the above does not exist, the operation content execution unit 23 is notified to that effect via the operation script reading unit 21. Upon receiving the notification, the operation content execution unit 23 outputs a preset error message to the executing personal computer, and ends the process (step S20).

The details of the processing contents of the operation registration unit 10 and the operation execution unit 20 are different between the first embodiment and the second embodiment. However, it is the same in that the virtual screen data creation unit 14 is provided and the virtual screen data 31 is stored with additional information in the information such as the icon and the toolbar to be operated. It has been described as an example of an embodiment in which the same effect can be obtained even if the realization method is different.

The disclosure of the above-mentioned patent documents cited shall be incorporated into this document by citation. Within the framework of the entire disclosure (including the scope of claims) of the present invention, it is possible to change or adjust the embodiments or examples based on the basic technical idea thereof. Further, various combinations or selections of various disclosure elements (including each element of each claim, each element of each embodiment or embodiment, each element of each drawing, etc.) within the framework of all disclosure of the present invention. (Including partial deletion) is possible. That is, it goes without saying that the present invention includes all disclosure including claims, various modifications and modifications that can be made by those skilled in the art in accordance with the technical idea. In particular, with respect to the numerical range described in this document, any numerical value or small range included in the range should be construed as being specifically described even if not otherwise described. Further, each of the disclosed matters of the above-cited documents may be used in combination with the matters described in this document in part or in whole as a part of the disclosure of the present invention, if necessary, in accordance with the purpose of the present invention. It is deemed to be included in the disclosure of this application.

10 Operation registration unit 11 Operation target storage unit 11a Icon information storage unit 11b Window information storage unit 11c Tool chip information storage unit 12 Operation content storage unit 13 Operation script storage unit 14 Virtual screen data creation unit 20 Operation execution unit 21 Operation script reading unit 22 Operation target search unit 22a Icon information search unit 22b Window information search unit 22c Tool chip information search unit 23 Operation content execution unit 30 Operation script 31 Virtual screen data 100 Operation registration computer 100a Display device (device registration computer)
100b keyboard device (operation registration personal computer)
100c mouse device (operation registration personal computer)
200 Operation execution personal computer 200a Display device (device execution personal computer)
200b Keyboard device (operation execution personal computer)
200c mouse device (operation execution personal computer)

Claims (7)

It is an automatic business execution system that has an operation registration unit that records user operations using a personal computer and an operation execution unit for re-execution of this.
The operation registration unit includes an operation target storage unit that identifies and stores an operation target as an image image, and an operation target storage unit.
An operation content storage unit that stores operations for the operation target,
An operation script storage unit that associates the operation target with the operation content and stores it as an operation script,
It has a virtual screen data creation unit that stores an image including the search condition of the operation target as virtual screen data.
In the operation execution unit, the recorded operation script is read, the operation target is specified by referring to the image and the virtual screen data, and then the operation for the operation target is executed .
When the operation target is an icon, a window, or a toolbar included in the window, the operation registration unit stores the type information in the virtual screen data together with the image that distinguishes them.
The operation target storage unit of the operation registration unit stores text information displayed together with the tooltip in the tooltip displayed on the desktop in relation to the toolbar when the operation target is a toolbar. Stored together with the virtual screen data
The operation execution unit is characterized in that when the operation target is a toolbar, the operation target is searched on the condition that the text information displayed as the tooltip information matches the stored text information. ,
Automatic business execution system. The business automatic execution system according to claim 1, wherein the operation registration unit and the operation execution unit are operated by different personal computers. The operation registration unit and the operation execution unit are characterized in that they operate on a personal computer having a different resolution, background, window configuration, and font size.
The automatic execution system for the business according to claim 1 or 2. When the operation target is an icon and the image of the icon is not on the desktop, the operation execution unit converts the window on the desktop into an icon and searches for the icon from the desktop. Features,
The business automatic execution system according to claim 1 . When the operation target is a toolbar and the image of the window including the toolbar is not displayed on the desktop, the operation execution unit moves the window on the desktop to the foreground and then the window. The feature is that the search is performed from the desktop.
The business automatic execution system according to claim 1 . When the operation target is a window and the image of the window is not displayed on the desktop, the operation execution unit moves the window on the desktop to the foreground and then moves the window on the desktop. The feature is that the search is performed from.
The business automatic execution system according to any one of claims 1 to 4 . It is a method to automatically execute business,
In the operation registration section of the personal computer that records the operation,
An operation target storage step that identifies and stores the operation target as an image, and
A virtual screen data creation step for storing an image including the search condition of the operation target as virtual screen data, and
The operation content storage step for storing the operation for the operation target, and
It has a step of saving an operation script that stores the operation target and the operation content in association with each other.
In the operation execution unit of the personal computer that executes the operation recording,
A step of specifying an operation target by searching the desktop screen and the virtual screen data together from the operation script, and
It has an operation execution step for executing the recorded operation content, and has.
In the operation registration unit, when the operation target is an icon, a window, or a toolbar included in the window, the virtual screen data stores the type information together with the image that distinguishes them.
In the operation target storage step, when the operation target is a toolbar, the text information displayed together with the tooltip in the tooltip displayed on the desktop in relation to the toolbar is converted into the virtual screen data. Also remember,
The operation execution unit is characterized in that when the operation target is a toolbar, the operation target is searched on the condition that the text information displayed as the tooltip information matches the stored text information. ,
How to execute business automatically.

Images