JP5477201B2 - GUI analysis apparatus, method, and program - Google Patents

Description

  The present invention relates to a GUI analysis apparatus, method, and program.

  Currently, users use a wide variety of systems (word processors, spreadsheet software, etc.) having a GUI on a PC. There are also many systems with the same function and purpose. In addition, each system will be upgraded at individual timing. In such a case, since the arrangement of operation parts (menus, buttons, etc.) on the screen differs depending on the system, the user may be confused by the operation of the system.

  An example of a related technique for solving such a problem is proposed in Patent Document 1. The GUI screen operation sequence verification device described in Patent Literature 1 includes a storage unit having a component information table, an optimum operation history table, and a test operation history table, and GUI components included in the GUI screen presented by the evaluation target program. In response to a component information extraction means for extracting and storing in the component information table and a first operation performed as an optimum operation on the GUI screen, the first operation is stored in the component information table. In response to the second operation performed as a test operation on the GUI screen and the optimum operation history creating means for saving in the optimum operation history table as the data format of the order of the GUI parts, the second The operation is stored in the test operation history table as a data format of the order of GUI parts stored in the part information table. An evaluation result creating unit that creates an evaluation result focusing on an operation sequence indicating movement between GUI components with reference to the operation history recording unit, the optimum operation history table, and the test operation history table; and the evaluation result Is provided with evaluation result display means for generating output data to be displayed in a predetermined format.

  Another example is described in Patent Document 2. The apparatus described in Patent Document 2 reads GUI part data of a plurality of software having a GUI, compares the read GUI part data between the plurality of softwares, and detects a GUI part having a common property value. A GUI design support apparatus that obtains and outputs the degree of commonality of the detected GUI component among the plurality of software, and the GUI component property includes a GUI component label, a GUI component vertical or horizontal Size, GUI part position, GUI part access key, GUI part shortcut key, menu item position in the menu hierarchy, and GUI part focus setting.

JP 2009-134673 A Japanese Patent No. 3555511

  However, the apparatus described in the above-described patent document has a problem that similar operation components cannot be associated between GUIs of a plurality of systems.

  Therefore, an object of the present invention is to provide an apparatus, a method, and a program capable of associating similar operation components between GUIs of a plurality of systems, which are the problems described above.

  In order to achieve such an object, an aspect of the present invention is a GUI analysis apparatus, which is one of operation components having a hierarchical relationship acquired from the first system and displayed on a screen. The operation component included in the set of operation components located in the upper (lower) hierarchy of the operation component and one of the operation components having a hierarchical relationship acquired from the second system and displayed on the screen. Among the combinations of operation components included in the set of operation components located in the upper (lower) hierarchy of the two operation components, the combination of operation components having the highest operation component similarity is specified, and the highest operation component Dependency similarity calculation means for outputting the evaluation value calculated based on the similarity as the similarity of the dependency between the first operation component and the second operation component, and based on the similarity of the dependency, The first In association Temu operation part of the operation part of the second system includes a component corresponding judging means for outputting the corresponding result.

  In addition, according to the present invention, an operation that is one of the operation components having a hierarchical relationship acquired from the first system and that is positioned on the upper (lower) hierarchy of the first operation component displayed on the screen. The operation component included in the set of components and one of the operation components having a hierarchical relationship acquired from the second system and positioned on the upper (lower) hierarchy of the second operation component displayed on the screen Among the combinations of operation components included in the set of operation components to be performed, the combination of operation components having the highest operation component similarity is specified, and the evaluation value calculated based on the highest operation component similarity is the first value. A dependency similarity calculation step for outputting as a similarity of the dependency relationship between the operation component and the second operation component; and the operation component of the first system and the second system based on the similarity of the dependency relationship Operation of In association with each article, the corresponding results and part corresponding determination step of outputting, the GUI analysis program to be executed by a computer is provided.

  In addition, according to the present invention, an operation that is one of the operation components having a hierarchical relationship acquired from the first system and that is positioned on the upper (lower) hierarchy of the first operation component displayed on the screen. The operation component included in the set of components and one of the operation components having a hierarchical relationship acquired from the second system and positioned on the upper (lower) hierarchy of the second operation component displayed on the screen Among the combinations of operation components included in the set of operation components to be identified, the combination of operation components having the highest operation component similarity is specified, and the evaluation value calculated based on the highest operation component similarity is the first The operation component and the second operation component are output as the dependency similarity, and the operation component of the first system and the operation component of the second system are associated with each other based on the similarity of the dependency relationship. , The corresponding result GUI analysis method outputs are provided.

  According to the present invention, a GUI analysis apparatus, method, and program capable of associating similar operation components among GUIs of a plurality of systems are provided.

It is a block diagram which shows the structure of the 1st Embodiment of this invention. It is a block diagram which shows the more concrete structure of the 1st Embodiment of this invention. It is an operation | movement flowchart of the 1st Embodiment of this invention. It is a figure which shows an example of the calculation method of the similarity of the structure of tree structure data. It is a figure which shows an example of the calculation method of the similarity of the content of tree structure data. It is a figure which shows an example of the GUI display screen of a 1st system. It is a figure which shows an example of the specific example of the operation component information of the operation component to which the text was provided by GUI of the 1st system. It is a figure which shows an example of the specific example of the operation component information of the operation component to which the image was provided by GUI of the 1st system. It is a figure which shows an example of the specific example of the component tree of the operation component of GUI of a 1st system. It is a figure which shows an example of the balloon hint of GUI of a 1st system. It is a figure which shows an example of the file menu of GUI of a 1st system. It is a figure which shows an example of the GUI display screen of a 2nd system. It is a figure which shows an example of the specific example of the operation component information of the operation component to which the text was provided by GUI of the 2nd system. It is a figure which shows an example of the specific example of the operation component information of the operation component to which the image was provided by GUI of the 2nd system. It is a figure which shows an example of the specific example of the component tree of the operation component of GUI of a 2nd system. It is a figure which shows an example of the specific example of calculation of the edit distance of the label character string of an operation component. It is a figure which shows an example of the matching result of the operation component of GUI of a 1st system, and the operation component of GUI of a 2nd system. It is a block diagram which shows the structure of the 2nd Embodiment of this invention. It is an operation | movement flowchart of the 2nd Embodiment of this invention. It is a block diagram which shows the structure of the 3rd Embodiment of this invention. It is an operation | movement flowchart of the 3rd Embodiment of this invention. It is a figure which shows an example of selection of the search menu in the search / replacement menu of a 1st system. It is a figure which shows an example of the highlight display which the corresponding | compatible operation component display part 80 performs. It is a block diagram which shows the structure of the 4th Embodiment of this invention. It is an operation | movement flowchart of the 4th Embodiment of this invention. It is a figure which shows an example of GUI of a 1st system and GUI of a 2nd system. It is a block diagram which shows the structure of the 5th Embodiment of this invention. It is a block diagram which shows the structure of the 6th Embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, the same components are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

  Each unit constituting the information processing apparatus of each embodiment includes a control unit, a memory, a program loaded in the memory, a storage unit such as a hard disk for storing the program, a network connection interface, and the like. Realized by any combination. And unless there is particular notice, the realization method and apparatus are not limited.

  In addition, the block diagram used in the description of each embodiment shows a functional unit block, not a hardware unit configuration. These functional blocks are realized by any combination of hardware and software. Further, in these drawings, the components of each embodiment may be described as being realized by one physically coupled device, but the means for realizing the same is not limited thereto. That is, two or more physically separated devices may be connected by wire or wirelessly, and the system of each embodiment may be realized by the plurality of devices.

<Embodiment 1>
The best mode for carrying out the invention will be described below in detail with reference to the drawings.

  Referring to FIG. 1, the GUI analysis device 10 according to the first exemplary embodiment of the present invention includes an operation component detection unit 20 and a component matching unit 25. As a specific configuration, as shown in FIG. 2, the component matching unit 25 includes a text similarity calculation unit 30, an image similarity calculation unit 40, a dependency relationship similarity calculation unit 50, a component correspondence determination unit 60, Is provided. As shown in FIG. 1, the entire system including the GUI analysis device 10 includes a terminal device 300 that operates by program control, an input device 100 such as a mouse and a keyboard, and an output device such as a display device that includes a display screen. 200.

  The GUI analysis device 10 may be provided in the terminal device 300. For example, the GUI analysis device 10 is constructed by executing a program according to the first embodiment to be described later on the terminal device 300. Each system 400 is a system outside the GUI analysis apparatus 10. Each system 400 may be constructed by executing another program (for example, an application program) on the terminal device 300.

  As shown in FIG. 1, the GUI analysis device 10 according to the present embodiment is connected to an input device 100 and an output device 200 via a terminal device 300. The user can operate the system 400 executed on the terminal device 300 using the input device 100. Further, the output device 200 displays an operation component output by the system 400 executed on the terminal device 300.

  The operation component detection unit 20 operates on the terminal device 300, and specifies information (hereinafter referred to as “operation component information”) for specifying the operation components of the first system and the second system displayed on the display screen of the output device 200. And). For example, the operation component detection unit 20 acquires at least an operation component type, a label character string, a popup hint character string, an icon image, component dependency, and an operation method. The operation component type is the type of operation component such as “push button”, “menu”, and “list box”. The label character string is a display character string assigned to the operation component such as a character string displayed in the button. The pop-up hint character string is a character string that pops up to explain the operation component when the mouse cursor is moved over the operation component. The icon image is an image assigned when the operation component is arranged on a toolbar or the like. The component dependency includes a hierarchical structure of menus and a group structure of components such as grouped buttons. Operation methods include mouse and keyboard combinations such as single-click and double-click with mouse, left-click and right-click, long button press, keyboard input, and click operation while pressing a specific key on the keyboard. For example, a flick, a pinch-in, a pinch-out, and the like are input operation methods for the user's parts.

  The text similarity calculation unit 30 uses the operation component on the GUI of the first system and the second one for text information such as a label character string and a pop-up hint character string among the operation component information acquired by the operation component detection unit 20. The similarity with the operation component on the GUI of the system is calculated. The text similarity calculation unit 30 may use a technique such as “edit distance” for the specific similarity calculation.

  Here, the edit distance is given as the minimum number of steps required to transform a character string into another character string by inserting, deleting, or replacing characters. For example, when transforming "kitten" to "sitting", (1) replace "k" with "s" and replace "kitten" → "sitten", (2) replace "e" with "i" Then, “sitten” → “sittin”, (3) “g” is inserted, “sittin” → “sitting”, and so on, at least three steps are required, so the edit distance is 3. .

  The image similarity calculation unit 40 operates the operation information displayed on the GUI of the first system, that is, on the display screen, with respect to image information such as an icon among the operation component information acquired by the operation component detection unit 20. The similarity between the part and the operation part on the GUI of the second system is calculated. The image similarity calculation unit 40 calculates specific similarity in the field of similar image search, such as a method using a feature amount extraction result for color or a method using a result of extracting a feature amount from color and composition. This technique may be used.

  In addition, the text similarity calculation unit 30 and the image similarity calculation unit 40 may calculate the similarity for each operation method when a plurality of operation methods can be executed for each operation component. For example, the text similarity calculation unit 30 and the image similarity calculation unit 40 allow the user to perform a plurality of operations on the operation component A, such as double-clicking the left mouse button, single-clicking the right mouse button, and dragging and dropping. If possible, the correspondence is determined for each operation and the similarity is calculated. Thereby, the text similarity calculation unit 30 and the image similarity calculation unit 40 can calculate the text similarity and the image similarity of the parts after associating the same parts and the same operation parts. Therefore, even if the GUI analysis device 10 is the same component on the GUI, it can be handled as a different component if the operation method for the component is different.

  The dependency relationship similarity calculation unit 50 includes the operation component on the GUI of the first system and the GUI of the second system based on the dependency relationship of the components in the operation component information acquired by the operation component detection unit 20. The degree of similarity with the operation part (similarity of dependency) is calculated. For example, if the similarity of an operation component having a hierarchical structure is calculated, the dependency similarity calculation unit 50 calculates the similarity of the structure (tree shape) of the tree structure data. Techniques such as “tree editing distance” may be used.

  The tree editing distance is given as the minimum number of steps necessary to transform a tree into another tree by inserting, deleting, or replacing nodes constituting the tree. For example, in the example shown in FIG. 4, in order to transform the tree structure data T1 shown in (a) to the tree structure data T2 shown in (b), (1) node C is deleted, (2) node E is inserted, (3) The edit distance is 3, such as replacing node A with node F.

  In addition, the dependency relationship similarity calculation unit 50 can also use the text similarity and image similarity of the entire operation components that are nodes constituting the tree as a method of calculating the similarity of the operation components having a hierarchical structure. For example, FIG. 5 shows a set of components for two nodes aaa each having the same label character string in a set of operation parts (hereinafter referred to as a part tree) T3 and T4 having two tree structures to which a label character string is assigned. An example of calculating tree similarity is shown. FIG. 5A is a diagram showing the correspondence between nodes of interest. FIG. 5B is a diagram showing the correspondence between the ancestor and descendant nodes of the focused node. At this time, the dependency relationship similarity calculation unit 50 specifies a combination of nodes having the highest similarity between the ancestor node set of the node of interest and the descendant node sets in each component tree. In the example of FIG. 5, the ancestor node set of T3 is (bbb) and the ancestor node set of T4 is (bbc). In this case, there is only one element, so the combination of these two nodes is This is a combination of nodes having the highest similarity among the ancestor node sets. The node set of T3 descendants is (ccc, ddd), and the node set of T4 descendants is (xxx, cbc, yyy, zzz, ddddd). In this case, the combination of ccc and cbc and ddd and ddddd is the combination of nodes having the highest similarity between the descendant node sets.

  From the above results, assuming that the sum of the edit distances of the combination of nodes specified as the similarity of the part tree is used, the similarity of T3 and T4 is the edit distance 1 of bbb and bbc, the edit distance 1 of ccc and cbc, The sum of the edit distance 2 of ddd and ddddd is 4. In this way, the dependency relationship similarity calculation unit 50 has a hierarchical structure of components when there are a plurality of combinations of components having the same or very similar calculation results in the text similarity calculation unit 30 and the image similarity calculation unit 40. The similarity of the operation parts based on it can be calculated.

  In the above example, the text similarity calculation unit 30 and the image similarity calculation unit 40 calculate the similarity of the operation component, and based on this, the dependency similarity is similar for a combination of two nodes aaa having the same label character string. Although the example in which the degree calculation unit 50 calculates the similarity of the parts tree is shown, the present invention is not limited to this. For example, the similarity calculation unit 50 uses the text similarity and image similarity of the entire operation components that are nodes constituting the tree, and the dependency relationship between the operation components of the first system and the operation components of the second system. The degree of similarity may be calculated omnidirectionally for all combinations of operation parts. Further, the dependency relationship similarity calculation unit 50 may calculate the similarity by taking into account the depth of the hierarchical structure in the ancestor node set and descendant node set. For example, in the above example, the node ddd at T3 in FIG. 5 and the ddddd at T4 have one hierarchy as viewed from the reference node aaa. In such a case, 1 may be added as the edit distance. Specifically, in the above example, the edit distance between ddd and ddddd is 2, but the edit distance may be 3 by adding 1 which is the difference in the depth of the hierarchy.

  For example, when there are a plurality of operation parts to which the same label character string “setting” is assigned, the dependency relationship similarity calculation unit 50 has a parent part to which the label character string “format” is assigned. And a part having a parent part to which a label character string “ruled line” is assigned.

  In addition, as a method of calculating the similarity of the operation component having the group structure, the dependency similarity calculation unit 50 uses the above-mentioned hierarchical operation structure of the operation components as the nodes constituting the tree and the text similarity of the entire operation components. And image similarity can be calculated in the same manner. At this time, the parts belonging to the respective groups including the two operation parts to be compared may be regarded as an ancestor node set or descendant node set in the part tree.

  The component correspondence determination unit 60 integrates the output results of the text similarity calculation unit 30, the image similarity calculation unit 40, and the dependency relationship similarity calculation unit 50, and finally on the GUI of the first system. A correspondence relationship between the operation component and the operation component on the GUI of the second system is determined. There are various methods for integrating the similarity calculation results. For example, the part correspondence determination unit 60 includes a text similarity Stext, an image similarity Simage, a part tree structure similarity Stree_struct, and a part tree content similarity Street_content. ΑStext + βSimage + γStree_struct + δStree_content or the like may be used as the weighted sum of. Note that α, β, γ, and δ are weights assigned to the respective similarities.

  Next, the overall operation of the present embodiment will be described in detail with reference to the flowcharts of FIGS.

  First, the operation component detection unit 20 monitors the output device 200, and the operation displayed on the display screen of the output device 200 from the first system and the second system, respectively, by the user operating the input device 100. The component is detected, and the component information of each operation component is acquired (step A1 in FIG. 3).

  FIG. 6 shows an example of the display screen of the first system. FIGS. 7, 8, and 9 show examples of operation component information that the operation component detection unit 20 detects and acquires from the first system.

  FIG. 7 shows an example of operation component information related to text information. Specifically, FIG. 7 shows an operation that is arranged on a window having the window name “X-Editor”, has a component type “menu”, and has a label character string “file”, “edit”, “search”, and the like. Information on the component is acquired by the operation component detection unit 20.

  FIG. 8 shows an example of operation component information given an image. Specifically, FIG. 8 shows that the information of the operation component arranged on the window having the window name “X-Editor” and having the label character strings “new creation”, “open”, “search”, etc. is acquired. An example is shown. In addition, a character string displayed as a pop-up hint for each operation component is also acquired by the operation component detection unit 20. The pop-up hint is information that pops up to explain the operation component when the mouse cursor is moved over the operation component. FIG. 10 shows an example. Specifically, the example of FIG. 10 shows a state where a pop-up hint for the GUI print icon button shown in FIG. 6 is displayed.

  FIG. 9 shows an example of a part tree. Specifically, FIG. 9 shows a window having a window name “X-Editor” as a root window, and menus such as “file”, “edit”, and “search” as child nodes of the root node. Also, icon buttons such as "New", "Open", "Search" are shown, and "New", "Open ...", "Save" as child nodes of "File" menu Menus such as are shown. FIG. 9 shows an example in which the tree structure of these operation components is acquired by the operation component detection unit 20. In this case, FIG. 9 means that there are operation parts such as “New”, “Open ...”, “Overwrite”, etc. under the file menu as shown in FIG.

  FIG. 12 shows an example of the display screen of the second system. In the same manner as in the first system, the operation component detection hand unit 20 detects an operation component on the GUI of the second system, and acquires component information of each operation component. Examples of operation component information on the GUI of the second system are shown in FIG. 13, FIG. 14, and FIG. FIG. 13 shows operation component information related to text information, FIG. 14 shows operation component information to which an image is added, and FIG. 15 shows a component tree.

  Next, the text similarity calculation unit 30 calculates the similarity between the text information of the operation component of the first system and the operation component of the second system acquired by the operation component detection unit 20 (step A2). More specifically, the text similarity calculation unit 30 calculates the similarity for the combinations of the text information shown in FIG. 7 and all the operation components of the text information shown in FIG.

  An example of a specific similarity calculation will be described below. First, the text similarity calculation unit 30 uses, for example, the editing distance described above for calculating the similarity. However, since the edit distance is actually a dissimilarity (a value indicating how dissimilar it is. It means that it is not similar as the value increases), and the reciprocal of the edit distance is used in the calculation formula. In the example shown in FIGS. 7 and 13, there is a combination in which the label character strings completely match for all the operation components. For example, there is a combination of X_T1 and Y_T1 for the label character string “file”, X_T2 and Y_T2 for the label character string “edit”, and in this case, the edit distance is zero. Accordingly, the calculation formula for the text similarity Stext is Stext = 1 / (d + 1), where d is the editing distance. At this time, the maximum similarity 1 is taken when the character strings completely match.

  Using the above calculation formula, in the example shown in FIGS. 7 and 13, for example, the similarity between X_T1 and Y_T1 is calculated to be 1 or the like.

  In addition, in order to prevent the synonyms from being calculated as not being similar at all, such as “paste” and “paste”, the text similarity calculation unit 30 uses a dictionary before the similarity calculation to calculate synonyms. May be converted into the same label character string before calculating the similarity.

  The text similarity calculation unit 30 adds the similarity according to the length of the matching partial character string when the same partial character string exists, for example, “end” and “end”. You may make it do. This processing can also be realized by giving weights to the replacement, insertion, and deletion procedures in the edit distance calculation.

  In addition, the image similarity calculation unit 40 calculates the similarity between the images assigned to the operation component of the first system and the operation component of the second system acquired by the operation component detection unit 20 (step A3). More specifically, the image similarity calculation unit 40 calculates the similarity for the combination of the image information shown in FIG. 8 and all the operation components of the image information shown in FIG.

  The specific image similarity calculation can be performed by using the various known calculation methods described above, and the description thereof will be omitted. In the example of FIGS. 8 and 14, since not only an image but also a label character string or a pop-up hint character string is assigned to the operation component, the similarity of the text character string can be calculated in the same manner as in step A2. it can.

  Note that the order of the step A2 for calculating the text similarity and the step A3 for calculating the image similarity may be interchanged, or the two steps may be processed in parallel.

  Next, the GUI analysis apparatus 10 extracts combinations of operation components each having a value greater than or equal to a certain value from the calculation result of the text similarity in step A2 or the image similarity in step A3 (step A4). For example, with respect to the text similarity in FIGS. 7 and 13, assuming that the threshold is 0.8, only combinations of operation parts having completely matching label character strings such as (X_T1, Y_T1) are extracted. However, in these examples, (X_T14, Y_T15), (X_T14, Y_T16), (X_T17, Y_T15), and (X_T17, Y_T16) are extracted as combinations of operation parts that have the label string "Setting ..." There are operation parts that are extracted redundantly (this will be described later).

  Similarly, for the image similarity in FIGS. 8 and 14, appropriate combinations such as (X_I1, Y_I1), (X_I2, Y_I2), (X_I3, Y_I3), (X_I4, Y_I4) are extracted. .

  Next, the dependency relationship similarity calculation unit 50 calculates the dependency relationship similarity for operation components that are not one-to-one among the combinations of operation components extracted in step A4 (step A5). In the examples so far, the combinations of (X_T14, Y_T15), (X_T14, Y_T16), (X_T17, Y_T15), and (X_T17, Y_T16) are targeted as the operation parts are not one-to-one.

  An example of a specific calculation of the part tree similarity will be described below. First, the GUI analysis apparatus 10 extracts a partial component tree including the operation component and its ancestors and descendants from the component tree of each operation component being combined. For example, for the combination (X_T14, Y_T15), the GUI analysis apparatus 10 uses X-Editor-Edit [X_T2] -Setting ... [X_T14] and Y-Editor-Edit [Y_T2] -Setting ... [Y_T15. ] Is extracted. Similarly, for the combination (X_T14, Y_T16), X-Editor-Edit [X_T2]-Settings ... [X_T14] and Y-Editor-Window [Y_T3]-Settings ... [Y_T16] are extracted ( (See FIG. 16).

  Next, the dependency relationship similarity calculation unit 50 calculates the similarity of the structure of the extracted partial component tree. In the above example, since the tree structure is equal as shown in FIG. 16 for the combination of (X_T14, Y_T15) and (X_T14, Y_T16), the editing distance of the tree is zero. Therefore, the similarity of the structure is 1 for both combinations.

  Next, the dependency relationship similarity calculation unit 50 calculates the similarity of the contents of the extracted partial component tree. First, the dependency relationship similarity calculation unit 50 identifies a combination of nodes having the highest similarity between the ancestor node sets and descendant node sets of X_T14 and Y_T15. An ancestor node includes a window as a root node. However, in this example, since the screen of each system displays only one window, the similarity calculation need not be performed. When the system screen displays a plurality of windows, the dependency relationship similarity calculation unit 50 obtains the similarity by using, for example, the edit distance of the window title character string. As a result, the ancestor node set of X_T14 is {edit ... [X_T2]}, the ancestor node set of Y_T15 is {edit ... [Y_T2]}, and the combination of these nodes has an edit distance of zero, that is, a similarity of 1 Thus, the combination of nodes having the highest similarity is obtained. In this example, since the descendant node has neither X_T14 nor Y_T15, similarity calculation is not performed. In such a case, the similarity may be set to 1 assuming that the descendant nodes are completely matched. As a result, the similarity of the contents of the partial part tree is 1 for the ancestor node and 1 for the descendant node. Similarly, considering the similarity of the contents of the parts tree of X_T14 and Y_T16, the combination of ancestor nodes is "Edit ..." [X_T2] and "Window" [Y_T3], so the similarity of ancestor nodes is the edit distance. 5, that is, the similarity is 1/6 (≈0.1667). Since the similarity of the descendant node is 1, the similarity of the parts tree is 1 + 1 / 6≈1.1667.

  From the above results, if the sum of the structural similarity and the content similarity is taken as the similarity of the partial tree, the combination of (X_T14, Y_T15) is 1 + 2 = 3, and the combination of (X_T14, Y_T16) is 1 + 1.1667 = 2.1667.

  Finally, the component correspondence determination unit 60 determines the correspondence relationship between the operation components by integrating the similarities calculated in the above steps (step A6). In this embodiment, between the operation parts displayed on the display screen of the first system X-Editor and the display screen of the second system Y-Editor, first, “setting ... Except " Next, the part correspondence determination unit 60 responds according to the calculation result of the part tree similarity between the operation parts having the label character string “setting ...” that is not associated one-to-one in the text similarity calculation. Judgment is made. For example, if the association is performed using the combination having the highest similarity, the combination of X_T14 and Y_15 illustrated in FIG. 16 is associated. Similarly, a combination of X_T17 and Y_T16 is associated. From the above results, the operation components on the GUI of the first system X-Editor and the second system Y-Editor are finally associated one-to-one as shown in FIG.

Next, the effect of this embodiment will be described. In the present embodiment, the GUI analysis apparatus 10 determines the correspondence relationship using not only the similarity of the text or image assigned to the operation component but also the similarity of the operation method for the operation component and the dependency relationship of the operation component. It is configured as follows. Therefore, the GUI analysis device 10 can associate similar operation components between GUIs of a plurality of systems. Further, the GUI analysis apparatus 10 can determine the correspondence relationship between the operation parts including not only the apparent similarity but also the meaning of whether or not the parts are parts for executing the same operation. For this reason, the GUI analysis apparatus 10 can associate operation components that execute the same operation among GUIs of a plurality of systems.
<Embodiment 2>
Next, a second embodiment of the present invention will be described in detail with reference to the drawings.

  Referring to FIG. 18, the GUI analysis device 11 in the second embodiment has a configuration different from that of the GUI analysis device 10 shown in FIG. That is, the GUI analysis device 11 includes an operation component information storage unit 21 instead of the operation component detection unit 20 illustrated in FIG.

  Except for the differences described above, the GUI analysis apparatus 11 has the same configuration as the GUI analysis apparatus 10 shown in FIG. The GUI analysis device 11 is also constructed by executing a program on the terminal device 300. Hereinafter, the difference from the first embodiment will be mainly described.

  The operation component information storage unit 21 stores in advance the output result of the operation component detection unit 20 in the first embodiment. Or you may make it memorize | store the information produced manually as shown in FIG. 7 thru | or FIG. 9, FIG. 13 thru | or FIG.

  Next, the overall operation of the present embodiment will be described in detail with reference to the flowcharts of FIGS.

  First, the operation component information storage unit 21 stores operation component information on the GUI of the first system and operation component information on the GUI of the second system in advance (step B1 in FIG. 19). At this time, as the operation component information, information stored in the output result of the operation component detection unit 20 shown in FIG. 2 may be used, or according to the formats of FIGS. 7 to 9 and FIGS. 13 to 145. You may use what was created manually.

  The subsequent steps from Step B2 to Step B6 are the same as the respective steps from Step A2 to Step A6 shown in FIG. The GUI analysis device 11 may be configured to include both the operation component detection unit 20 and the operation component information storage unit 21. In this case, the operation component information of the GUI of the system 400 that operates on the terminal device 300 acquired by the operation component detection unit 20 and the output result of the operation component information detection unit 20 stored in the operation component information storage unit 21. The component correspondence determination unit 60 may associate the operation components using the information (or the operation component information created manually).

  In the present embodiment, the GUI analysis device 11 is configured to associate operation components between GUIs of different systems using operation component information prepared in advance. Therefore, the GUI analysis device 11 can associate the operation components without operating the analysis target system.

  The GUI analysis device 11 according to the present embodiment can associate operation components by manually creating operation component information even in a system that cannot be operated on the terminal device 300. For example, the GUI analysis device 11 can also analyze a system that operates on dedicated hardware such as a mobile phone.

  Further, the GUI analysis device 11 can perform analysis without operating the system on the terminal device 300 by manually creating operation component information from a system manual or the like.

<Embodiment 3>
Next, a third embodiment of the present invention will be described in detail with reference to the drawings.

  Referring to FIG. 20, the GUI analysis device 12 in the third embodiment of the present invention has a configuration different from that of the GUI analysis device 10 shown in FIG. That is, the GUI analysis device 12 includes an operation detection unit 70 and a corresponding operation component display unit 80 in addition to the configuration shown in FIG.

  Except for the differences described above, the GUI analysis device 12 has the same configuration as the GUI analysis device 10 shown in FIG. The GUI analysis device 12 is also constructed by executing a program on the terminal device 300. Hereinafter, the difference from the first embodiment will be mainly described.

  The operation detection unit 70 detects an operation input performed by the user via the input device 100 with respect to the operation components of the system 400 displayed on the display screen of the output device 200. Specifically, the operation detection unit 70 acquires information for specifying the selected operation component when the user selects an operation component on the GUI of the first system, for example. The operation component can be selected by, for example, the user placing a mouse pointer (mouse over) with the input device 100 or giving focus from the keyboard.

  The corresponding operation component display unit 80 first refers to the determination result output by the component correspondence determination unit 60 using the operation component information acquired by the operation detection unit 70. Next, the corresponding operation component display unit 80 specifies an operation component of another system corresponding to the operation component specified by the operation component information. Then, the corresponding operation component display unit 80 makes the specified operation component stand out and displayed on the GUI of another system displayed on the display screen of the output device 200.

  Next, the overall operation of the present embodiment will be described in detail with reference to the flowcharts of FIGS.

  First, the user activates the GUI control unit of the first system 400 and the GUI control unit of the second system 400 on the terminal device 300 via the input device 100, and displays operation components on the display screen of the output device 200. To do. Here, it is assumed that the first system 400 is a system known to the user, for example, and that the GUI is used to the user and sufficiently understands the arrangement of the operation parts. The second system 400 is, for example, a system upgraded from the first system, or a system of the same type as the first system but different from the first system, and the GUI is not used by the user and understands the arrangement of operation components. It is assumed that it has not been completed.

  The GUI analysis device 12 performs the processes of Step C1 to Step C6, which are the same steps as Step A1 to Step A6 shown in FIG. 3, respectively, and the operation components on the GUI of the first system 400 and the second Are associated with the operation parts on the GUI of the system 400 (steps C1 to C6 in FIG. 21). The steps from Step C1 to Step C6 are the same as the steps from Step A1 to Step A6 shown in FIG.

  Next, when the user performs an operation of selecting an operation component of the GUI of the first system 400 displayed on the display screen of the output device 200 via the input device 100, the operation detection unit 70 detects the operation. To do. (YES in step C7 and step C8). When the user is not performing an operation, the display screen is continuously monitored (NO in step C7).

  Next, the corresponding operation component display unit 80 refers to the determination result output by the component correspondence determination unit 60, and the second system 400 corresponding to the operation component on the GUI of the first system 400 identified in step C7. An operation component on the GUI is acquired (step C9). For example, the display screen of the first system 400 is the X-Editor shown in FIG. 6, and the display screen of the second system 400 is the Y-Editor shown in FIG. It is assumed that the mouse is over “Search ...” in the “Search / Replace” menu (see FIG. 22). At this time, the operation detection unit 70 detects the “search” menu of the selected X-Editor. The corresponding operation component display unit 80 refers to the determination result of the component correspondence determination unit 60 shown in FIG. 17, and the “search” menu of the X-Editor (X-Editor → Search / Replace → Search ... in FIG. The "Search ..." menu (Y-Editor-> Edit-> Search ..., Part ID Y_T13 in FIG. 13) of the Y-Editor corresponding to the operation part of the part ID X_T15 in FIG. 7 is acquired.

  Subsequently, the corresponding operation component display unit 80 highlights the operation component specified in step C9 on the display screen of the output device 200 (step C10). FIG. 23 shows an example of highlighting performed by the corresponding operation component display unit 80. FIG. 23 shows a scene in which X-Editor and Y-Editor are displayed on the display screen of the output device 200. In the example of FIG. 23, when the user moves the mouse over "Search ..." in the "Search / Replace" menu of X-Editor, "Search ..." in the "Edit" menu of Y-Editor is highlighted. An example is shown.

  In this embodiment, when a user selects a GUI operation component of one system among GUI operation components of different systems associated with each other, an operation component of another system corresponding to the selected operation component is displayed. The GUI analysis device 12 is configured to be highlighted on the screen. Therefore, the user can grasp operation components corresponding to each other between different systems at a glance.

<Embodiment 4>
Next, a fourth embodiment of the present invention will be described in detail with reference to the drawings.

  Referring to FIG. 24, the GUI analysis device 13 in the fourth embodiment of the present invention has a configuration different from that of the GUI analysis device 10 shown in FIG. That is, the GUI analysis device 13 includes an operation sequence input unit 71 and a GUI evaluation unit 90 in addition to the configuration shown in FIG.

  Except for the differences described above, the GUI analysis device 13 has the same configuration as the GUI analysis device 10 shown in FIG. Also, the GUI analysis device 13 is constructed by executing a program on the terminal device 300. Hereinafter, the difference from the first embodiment will be mainly described.

  The operation sequence input unit 71 designates a series of operation procedures for the operation components of the system 400 operating on the terminal device 300. For example, the operation procedure is specified as “single-click button A with the mouse and then single-click button B with the mouse”. The operation sequence input unit 71 may be configured such that the user actually operates the operation parts of the system 400 via the input device 100 and records the series of operations, or “mouse single click (button A ) → single mouse click (button B) ”may be described in text according to a predetermined format. In this case, one operation is designated by the character string <user operation> + "(" + <operation component> + ")", and the order of a plurality of operations is represented by "→".

  The GUI evaluation unit 90 evaluates the operability of the operation component. Various indexes can be considered as an evaluation index. For example, operational efficiency can be given. The operation efficiency can be expressed as an operation time until the operations are completed when a series of operations are performed on the GUI.

  Fitts' law is known as a model for predicting the time for selecting an operation component using the input device 100 such as a mouse on the GUI. According to Fitts' law, when time is t, the time t for selecting the operation component is calculated as t = a + b * log2 (A / W + 1). Here, A is the distance between the mouse cursor and the target operation component, W is the size of the operation component, and a and b are empirical constants.

  A keystroke level model that predicts the execution time of a series of operations by dividing the operation procedure on the GUI and taking the sum of the required time per unit is also well known. In the keystroke level model, assuming that the execution time is Te, it is calculated as Te = ΣTk + ΣTp + ΣTb + ΣTbb + ΣTh + ΣTm + ΣTr. Where Tk is the key input time, Tp is the time required to move the mouse cursor, Tb is the time required to release the mouse button, Tbb is the time required to click the mouse, Th is the time from the mouse to the keyboard, or from the keyboard to the mouse. Is the time required to move, Tm is the time required to think about the next operation, and Tr is the response time of the system.

  Next, the overall operation of the present embodiment will be described in detail with reference to the flowcharts of FIGS.

  First, the user activates the GUI of the first system 400 and the GUI of the second system 400 on the terminal device 300 via the input device 100 and displays them on the display screen of the output device 200.

  Next, the operation component detection unit 22 detects each operation component from the GUI of the first system and the GUI of the second system displayed on the display screen of the output device 200, and acquires component information of each operation component. (Step D1 in FIG. 25). At this time, the operation component detection unit 22 according to the fourth embodiment of the present invention adds each operation component in addition to the information of FIG. 7, FIG. 8, FIG. 13, and FIG. 14 acquired in the first embodiment. Information such as the position and size is also acquired. For example, the operation component detection unit 22 acquires, as position information, the center coordinates of the operation component in the window where the operation component is arranged, or the upper left and lower right coordinates of the operation component in the window where the operation component is arranged, The size acquires the area of the operation part.

  Next, the GUI analysis device 13 performs the processing from Step D2 to Step D6, which are the same steps as Step A2 to Step A6 shown in FIG. 3, respectively, and operates the operation component on the GUI of the first system 400. Are associated with the operation parts on the GUI of the second system 400 (steps D2 to D6 in FIG. 25). The steps from step D2 to step D6 are the same as the steps from step A2 to step A6 shown in FIG.

  Next, the user uses the operation sequence input unit 71 via the input device 100 to designate a series of operation procedures for the operation components of the first system (step D7). Now, for example, as shown in FIGS. 26A and 26B, a window SystemA is displayed on the display screen of the output device 200 as an operation component of the first system and a window SystemB as an operation component of the second system. Suppose that At this time, the user uses the operation sequence input unit 71 to designate a series of operations “pressing SystemA Button2 after pressing SystemA Button1”. To specify the operation, the user actually uses the mouse to move the mouse cursor onto the button Button1 of the window SystemA and click the mouse, and then moves the mouse cursor onto the button Button2 of the window SystemA and clicks the mouse. Alternatively, the text may be described according to a predetermined format such as “mouse single click (SystemA: ButtonA) → mouse single click (SystemA: ButtonB)”.

  Next, the GUI evaluation unit 90 uses the operation sequence for the operation component of the first system specified in Step D7 to refer to and specify the correspondence relationship of the operation component that is the output result of the component correspondence determination unit 60. An operation sequence for the operation component of the second system corresponding to the operation sequence is acquired (step D8). For example, in the above example, when the output result of the component correspondence determination unit 60 is referred to using the two buttons Button1 and Button2 in the operation sequence of the first system, the label character strings of the second system match each other. Get operation parts Button1 and Button2. As a result, the operation sequence “mouse single click (for the second system operation component) corresponding to the operation sequence“ mouse single click (SystemA: ButtonA) → mouse single click (SystemA: ButtonB) ”for the operation component of the first system. SystemB: ButtonA) → mouse single click (SystemB: ButtonB) ”.

  Next, the GUI evaluation unit 90 calculates an evaluation when the operation sequence acquired in step D8 is executed on the GUI of each system (step D9). For example, it is assumed that the user performs a series of operations on the operation component in order to evaluate the operation efficiency. This sequence of operations is represented by a keystroke level model: (1) The user places his hand on the mouse (2) Moves the mouse cursor to Button1 (3) Clicks on Button1 (4) System response (5) Mouse Move cursor to Button2 (6) Mouse click on Button2 (7) System response. In the keystroke level model, the psychological preparation time (M) is inserted before every key input and click, so that the above-described series of operation procedures is as follows. That is, the operation procedure is as follows: (1) The user places his hand on the mouse (M) Psychological preparation time (2) Moves the mouse cursor to Button1 (M) Psychological preparation time (3) Mouse click on Button1 (4) System Response (M) Psychological preparation time (5) Move mouse cursor to Button2 (M) Psychological preparation time (6) Mouse click on Button2 (7) System response. Now, the time required to move the hand to the mouse is set to Th = 0.4 seconds. Also, a psychological preparation time Tm = 1.35 seconds, a time required for mouse click Tbb = 0.2 seconds, and a system response time Tr = 0.1 seconds. Also, the time Tp_A1 = 1.0 second required to move the mouse cursor from the current position to Button1 in the SystemA window of the GUI of the first system calculated from Fitts's law, and Tp_A2 = 0.3 required to move from Button1 to Button2. Second, time Tp_B1 = 1.1 seconds required to move the mouse cursor from the current position to Button1 in the SystemB window of the GUI of the second system, and Tp_B2 = 0.7 seconds required to move from Button1 to Button2. In this case, the time required to execute the operation sequence in the GUI of the first system is Th + Tm + Tp_A1 + Tm + Tbb + Tr + Tm + Tp_A2 + Tm + Tbb + Tr = 0.4 + 1.35 + 1.0 + 1.35 + 0.2 + 0.1 + 1.35 + 0.3 + 1.35 + 0.2 + 0.1 = 7.7 seconds. Similarly, the time required to execute the operation sequence in the GUI of the second system is Th + Tm + Tp_B1 + Tm + Tbb + Tr + Tm + Tp_B2 + Tm + Tbb + Tr = 0.4 + 1.35 + 1.1 + 1.35 + 0.2 + 0.1 + 1.35 + 0.7 + 1.35 + 0.2 + 0.1 = 8.2 seconds.

  Therefore, the GUI evaluation unit 90 can evaluate that the operation efficiency specified in the GUI of the first system is better than the GUI of the second system for the operation sequence specified in step D7, and the predicted execution time of the operation sequence is It is calculated as 7.7 seconds in the GUI of the first system and 8.2 seconds in the GUI of the second system.

  Further, the GUI analysis apparatus 13 includes a configuration including an operation component information storage unit 21 instead of the operation component detection unit 22 illustrated in FIG. 24, or includes both the operation component detection unit 22 and the operation component information storage unit 21. It may be a configuration. In the case of the former configuration, in the case of the former configuration, the operation component information stored in the operation component information storage unit 21 is used as the output result of the operation component detection unit 22 (or operation component information created manually). The correspondence determination unit 60 may associate the operation components. Then, the GUI evaluation unit 90 may evaluate the GUI using the result. In the case of the latter configuration, the operation component information of the GUI of the system 400 operating on the terminal device 300 acquired by the operation component detection unit 22 and the output result of the operation component information detection unit 22 stored in the operation component information storage unit 21 are used. The part correspondence determination unit 60 may associate the operation parts using certain operation part information (or operation part information created manually). And the GUI evaluation part 90 may perform GUI evaluation using the result.

  When the GUI analysis apparatus 13 according to this embodiment designates an operation sequence for the GUI of the first system when evaluating the GUI, the GUI analysis apparatus 13 of another system corresponding to the GUI operation component of the first system is specified. An operation sequence is replaced with a GUI operation component, and evaluation on a GUI of another system is performed. Therefore, the GUI analysis device 13 can automatically calculate the evaluation of the GUI of another system only by instructing the evaluation of the GUI of one system.

  In addition, the GUI analysis device 13 uses the operation component information manually created, so that the GUI of an incomplete system at the design stage in which the currently operating system is upgraded can be operated on the GUI of the currently operating system. Comparison evaluation can be performed by associating parts.

<Embodiment 5>
Next, a fifth embodiment of the present invention will be described with reference to FIG.

  The GUI analysis device 14 according to the fifth embodiment of the present invention replaces the corresponding operation component display unit 80 of the configuration of the GUI analysis device 12 according to the third embodiment shown in FIG. I have.

  Except for the above differences, the GUI analysis device 14 has the same configuration as the GUI analysis device 10 of the third embodiment shown in FIG. The GUI analysis device 14 is also constructed by executing a program on the terminal device 300. Also in the present embodiment, as in the third embodiment, the first system 400 is a system known to the user, and the GUI is familiar to the user and sufficiently understands the arrangement of the operation components. To do. Also, the second system 400 is a system upgraded from the first system, or a system that is the same type but different from the first system, and the GUI is not familiar to the user and the layout of the operation components can be understood. Shall not.

  The corresponding operation execution unit 81 uses the operation component information acquired by the operation detection unit 70 to acquire the determination result output by the component correspondence determination unit 60. Next, the corresponding operation execution unit 81 specifies an operation component of another system corresponding to the operation component specified by the operation component information. Then, the corresponding operation execution unit 81 performs, on the GUI of another system displayed on the display screen of the output device 200, a user operation (mouse on the operation component of the GUI of the first system with respect to the specified operation component. The same operation as that of the first system GUI and the second system GUI is performed on the GUI operation component of the second system corresponding to the GUI operation component of the first system. Since other parts are the same as those in the third embodiment, description thereof is omitted.

  The GUI analysis device 14 according to the present exemplary embodiment, when a user operates a GUI operation component of one system among GUI operation components of different systems associated with each other, corresponds to the operated operation component. Similar operations can be performed on the operating components of the system. In this way, the user can operate the second system only by operating the GUI of the first system that is familiar to the user.

<Embodiment 6>
The GUI analysis device 15 according to the sixth exemplary embodiment of the present invention includes a dependency relationship similarity calculation unit 50 and a component correspondence determination unit 60. Others are substantially the same as those in the first embodiment, and thus the description thereof is omitted. With this configuration, it is one of the operation components having a hierarchical relationship acquired from the first system, and is a set of operation components located in the upper (lower) hierarchy of the first operation component displayed on the screen. The operation component included and the operation component that is one of the operation components having a hierarchical relationship acquired from the second system and that is located in the upper (lower) hierarchy of the second operation component displayed on the screen. Among the combinations of the operation components included in the set, the combination of the operation components having the highest operation component similarity is identified, and the evaluation value calculated based on the highest operation component similarity is the first operation component. Dependency similarity calculation means for outputting as a similarity of the dependency relationship of the second operation component, and based on the similarity of the dependency relationship, the operation component of the first system and the operation component of the second system Vs Attached to a component correspondence decision means for outputting the corresponding result, GUI analyzer equipped with is provided.

  According to the GUI analysis apparatus 15 according to the present embodiment, similar operation components can be associated among GUIs of a plurality of systems.

(Other expressions of the embodiment)
Part or all of the above-described embodiments can be described as the following supplementary notes (not limited to the following).
(Appendix)
The dependency relationship similarity calculation means uses a part of the dependency relationship including the GUI operation component of the first system to be compared and a part of the dependency relationship including the GUI operation component of the second system. GUI analysis device for calculating the similarity of dependency relationships.

  The operation component information storage means for storing operation component information for specifying the operation components of the GUI of the first system and the GUI of the second system in advance is replaced with the operation component detection means or in addition to the operation component detection means. The component correspondence determining means associates and outputs the GUI operation component of the first system and the GUI operation component of the second system based on the operation component information stored in the operation component information storage means. GUI analysis device.

  (A) The similarity between character strings assigned to each operation component is calculated between the GUI operation component of the first system and the GUI operation component of the second system, which is one of the external systems. And (b) calculating the similarity of images assigned to each operation component between the GUI operation component of the first system and the GUI operation component of the second system; (C) calculating a similarity between the dependency of the GUI operating component of the first system and the dependency of the GUI operating component of the second system; and (d) the steps (a) to (c) ), The overall similarity between the operation components is calculated, and the correspondence between the GUI operation components of the first system and the GUI operation components of the second system is calculated. A step of determining GUI analysis how to.

  (A) The similarity of the character strings assigned to each operation component is calculated between the GUI operation component of the first system and the GUI operation component of the second system, which is one of the external systems. And (b) calculating the degree of similarity of images assigned to the respective operation components between the operation components of the GUI of the first system and the operation components of the GUI of the second system; (C) calculating the similarity between the GUI operating component dependency of the first system and the GUI operating component dependency of the second system; and (d) (a) to (a) By combining the respective output results of step c), the overall similarity between the operation parts is calculated, and the correspondence between the GUI operation parts of the first system and the GUI operation parts of the second system Determining the relationship and Program causing a computer to execute the.

  While the present invention has been described with reference to the embodiments, the present invention is not limited to the above embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

  INDUSTRIAL APPLICABILITY According to the present invention, the present invention can be applied to the use of GUI operation support for an unknown system using the GUI of a known system.

  Further, the present invention can be applied to uses such as comparative evaluation of an operable completed system and an incomplete system at the design stage.

10 GUI analysis device in the first embodiment 11 GUI analysis device in the second embodiment 12 GUI analysis device in the third embodiment 13 GUI analysis device in the fourth embodiment 14 Fifth embodiment GUI analysis device 15 in the embodiment GUI analysis device in the sixth embodiment 20 Operation component detection unit 21 in the first embodiment 21 Operation component information storage unit 22 Operation component detection unit 25 in the fourth embodiment 25 Component matching unit DESCRIPTION OF SYMBOLS 30 Text similarity calculation part 40 Image similarity calculation part 50 Dependency similarity calculation part 60 Component correspondence determination part 70 Operation detection part 71 Operation series input part 80 Corresponding operation part display part 81 Corresponding operation execution part 90 GUI evaluation part 100 Input Device 200 Output device 300 Terminal device 400 System

Claims (8)

An operation component that is one of the operation components having a hierarchical relationship acquired from the first system and that is included in a set of operation components that are located in the upper (lower) hierarchy of the first operation component displayed on the screen. Are included in a set of operation components that are one of the operation components having a hierarchical relationship acquired from the second system and that are located in the upper (lower) hierarchy of the second operation component displayed on the screen. Among the combinations of operation components, the combination of operation components having the highest operation component similarity is specified, and the evaluation value calculated based on the highest operation component similarity is used as the first operation component and the second operation component. Dependency similarity calculation means for outputting as the dependency of the operation component dependency,
Component correspondence determination means for associating the operation component of the first system with the operation component of the second system based on the similarity of the dependency relationship, and outputting the correspondence result;
Operation detection means for detecting an operation input to the operation component of the first system;
Based on the operation component information of the first system that is the target of the operation detected by the operation detection unit, the determination result of the correspondence relationship of the component output by the component correspondence determination unit is referred to, and the first system Corresponding operation component display means for specifying the operation component of the second system corresponding to the operation component of the second system and highlighting the specified operation component of the second system;
Corresponding operation execution means for executing an operation input corresponding to the operation component of the first system with respect to the operation component of the second system specified by the corresponding operation component display means;
A GUI analysis device comprising: Text similarity calculation means for calculating the similarity between character strings assigned to each of the two operation components and outputting the similarity as the similarity between the two operation components;
Image similarity calculation means for calculating the similarity of images assigned to each of the two operation components and outputting the similarity as the similarity between the two operation components;
The operation component similarity is calculated based on at least one of the similarity of the character string or the similarity of the image,
The component correspondence determining unit calculates a total similarity between the operation components by combining the operation component similarity and the dependency similarity, and based on the total similarity, The GUI analysis apparatus according to claim 1, wherein the operation component and the operation component of the second system are associated with each other and the determination result of the correspondence relation is output.   When at least one of the text similarity calculation unit and the image similarity calculation unit can execute a plurality of operation methods for each operation component, the similarity of the character string, or, for each operation method, The GUI analysis apparatus according to claim 2, wherein the similarity of the image is calculated and output. An operation sequence input means for inputting an operation sequence for an operation component of the system;
Referring to the series of operation components that are the target of the operation represented by the operation sequence for the first system input by the operation sequence input means, and the determination result output by the component correspondence determination means, the first system GUI evaluation means for specifying and outputting an operation sequence for the second system corresponding to an operation sequence for
The GUI analysis device according to any one of claims 1 to 3, further comprising: An operation component that is one of the operation components having a hierarchical relationship acquired from the first system and that is included in a set of operation components that are located in the upper (lower) hierarchy of the first operation component displayed on the screen. Are included in a set of operation components that are one of the operation components having a hierarchical relationship acquired from the second system and that are located in the upper (lower) hierarchy of the second operation component displayed on the screen. Among the combinations of operation components, the combination of operation components having the highest operation component similarity is specified, and the evaluation value calculated based on the highest operation component similarity is used as the first operation component and the second operation component. A dependency similarity calculation step for outputting the dependency of the operation component as a dependency,
A component correspondence determination step of associating the operation component of the first system with the operation component of the second system based on the similarity of the dependency relationship, and outputting a correspondence result;
An operation detection step of detecting an operation input to the operation component of the first system;
Based on the operation part information of the first system that is the target of the detected operation, the determination result of the correspondence relationship of the output parts is referred to and the operation part corresponding to the operation part of the first system is referred to. A corresponding operation component display step of identifying an operation component of the second system and highlighting the identified operation component of the second system;
A corresponding operation execution step of executing an operation input corresponding to the operation component of the first system with respect to the specified operation component of the second system;
GUI analysis program for causing a computer to execute. A text similarity calculation step of calculating a similarity between character strings assigned to two operation parts and outputting the similarity as a similarity between the two operation parts;
An image similarity calculation step for calculating the similarity between the images assigned to the two operation components and outputting the similarity as the similarity between the two operation components;
The dependency relationship similarity calculating step for calculating the operation component similarity based on at least one of the similarity of the character string and the similarity of the image;
A total similarity between the operation components is calculated by combining the operation component similarity and the dependency similarity, and the first system operation component and the second system are calculated based on the total similarity. The component correspondence determination step for associating the operation components of the system and outputting the determination result of the correspondence relationship;
The GUI analysis program according to claim 5 , wherein the computer is executed. An operation component that is one of the operation components having a hierarchical relationship acquired from the first system and that is included in a set of operation components that are located in the upper (lower) hierarchy of the first operation component displayed on the screen. Are included in a set of operation components that are one of the operation components having a hierarchical relationship acquired from the second system and that are located in the upper (lower) hierarchy of the second operation component displayed on the screen. Among the combinations of operation components, the combination of operation components having the highest operation component similarity is identified, and the evaluation value calculated based on the highest operation component similarity is used as the first operation component and the second operation. Output as the similarity of component dependency,
Based on the similarity of the dependency relationship, the operation component of the first system and the operation component of the second system are associated with each other, and the corresponding result is output .
Detecting an operation input to the operation component of the first system;
Based on the operation part information of the first system that is the target of the detected operation, the determination result of the correspondence relationship of the output parts is referred to and the operation part corresponding to the operation part of the first system is referred to. Identifying the operating component of the second system, highlighting the identified operating component of the second system,
An operation input corresponding to the operation component of the first system is executed on the identified operation component of the second system.
GUI analysis method. Calculate the similarity of the character strings assigned to the operation parts,
Calculate the similarity of the image assigned to the operation component,
The operation component similarity is calculated based on at least one of the similarity of the character string or the similarity of the image,
A total similarity between the operation components is calculated by combining the operation component similarity and the dependency similarity, and the first system operation component and the second system are calculated based on the total similarity. The GUI analysis method according to claim 7 , wherein the determination result of the correspondence relationship is output by associating with the operation component of the system.

Images