JP7620540B2 - Screen comparison device, screen comparison method, and program - Google Patents

Description

The present invention relates to a screen comparison device, a screen comparison method, and a program.

When updating an application, the screen is visually compared before and after the revision to ensure that there are no unintended changes to text or button placement. For example, a technology for a program that tests web applications has been disclosed (see Patent Document 1).

JP 2018-116496 A

However, in conventional technology, because the screen before and after correction is compared visually, there is a possibility of misdetection of typos or omissions in long strings of characters. In addition, when changing the screen size, it is difficult to visually compare the screen before and after correction, which can lead to errors in the confirmation. In addition, because it is necessary to visually check each screen, the confirmation process takes time.

In view of the above problems, the present invention aims to provide a screen comparison device that can efficiently and accurately check for unintended changes in wording and layout before and after modification when updating an application screen.

To achieve the above object, the present invention provides a screen comparison device for comparing screens displayed by an application to be corrected, the device comprising: an extraction unit that extracts at least one or more characters included in a current screen displayed by the application before correction as a character recognition result for the current screen, and extracts at least one or more characters included in a new screen displayed by the application after correction as a character recognition result for the new screen; and a detection unit that detects errors in characters in the new screen according to pairs of at least one or more character recognition results for the current screen and at least one or more character recognition results for the new screen extracted by the extraction unit.

The present invention also provides a screen comparison method executed by a screen comparison device that compares screens displayed by an application to be modified, which extracts at least one or more characters included in a current screen displayed by the application before modification as a character recognition result for the current screen, and extracts at least one or more characters included in a new screen displayed by the application after modification as a character recognition result for the new screen, and detects errors in characters in the new screen according to pairs of the extracted at least one or more character recognition results for the current screen and at least one or more character recognition results for the new screen.

The present invention is also characterized in that it is a program executed by a screen comparison device that compares screens displayed by an application to be corrected, and extracts at least one or more characters included in a current screen displayed by the application before correction as a character recognition result for the current screen, and extracts at least one or more characters included in a new screen displayed by the application after correction as a character recognition result for the new screen, and executes a process to detect errors in characters on the new screen according to pairs of the extracted at least one or more character recognition results for the current screen and at least one or more character recognition results for the new screen.

According to the present invention, when updating an application screen, it is possible to efficiently and accurately check for unintended changes in wording and layout before and after modification.

1A and 1B are diagrams showing examples of screens to be compared by the screen comparison device of the present embodiment; FIG. 2 is a hardware configuration diagram of the screen comparison device according to the present embodiment. FIG. 2 is a diagram showing an example of functional blocks of the screen comparison device relating to the screen comparison process of the present embodiment. 10 is a flowchart showing an example of a procedure of a screen comparison method by a screen comparison process executed by the screen comparison device. 11 is a flowchart illustrating an example of a procedure of an extraction process executed by the screen comparison device. 11A and 11B are diagrams illustrating an example of an OCR result array of a current screen and an OCR result array of a new screen stored in a storage unit. 13 is a flowchart illustrating an example of a procedure for a single pair extraction process. 13 is a flowchart illustrating an example of a procedure for a multiple pair extraction process. 13 is a diagram showing an example of an outline of a method for acquiring an OCR result of a current screen acquired by a surrounding OCR result acquisition process; FIG. 13 is a diagram showing an example of an outline of a method for acquiring an OCR result of a new screen acquired by a surrounding OCR result acquisition process. FIG. 10 is a flowchart illustrating an example of a procedure for a surrounding OCR result acquisition process. 10 is a flowchart illustrating an example of a procedure for an OCR result update process. 13 is a flowchart showing an example of a procedure for a typographical error and omission check process. 10 is a flowchart illustrating an example of a procedure for a position deviation check process. FIG. 13 is a diagram showing an example of notifying typos and misalignment.

Figure 1 is a diagram showing an example of screens to be compared by the screen comparison device 100 of this embodiment. The screen comparison device 100 is used, for example, as a device for comparing screens displayed by an application to be modified. Hereinafter, the screen displayed by the application before modification is referred to as the current screen 10, and the screen displayed by the application after modification is referred to as the new screen 20. As an application, it can be applied to an ATM (automatic teller machine) application such as an automated teller machine. Applications to which the screen comparison device 100 is applied are not limited to ATMs, and it can be applied to various applications in which it is essential to display a screen.

As shown in FIG. 1, the new screen 20 displayed by the modified application displays wording 300a that was not displayed on the current screen 10 displayed by the application before modification, resulting in a typographical error. Furthermore, on the current screen 10 displayed by the application before modification, wording 300b was displayed in the bottom center of the screen, but on the new screen 20 displayed by the modified application, wording 300b is displayed on the bottom left side of the screen, resulting in a typographical error. Furthermore, the spelling of wording 300c displayed on the current screen 10 displayed by the application before modification, "English Guide," is displayed differently from the spelling of wording 300c displayed on the new screen 20 displayed by the modified application, "English Guid," resulting in a typographical error.

Although the same wording should be displayed on the current screen 10 and the new screen 20, typos and omissions may occur, as shown in Figure 1. When comparing these typos and omissions visually, it is cumbersome to check every time a new screen 20 is created. Furthermore, relying solely on human visual inspection can lead to errors in the confirmation, as it is difficult to compare the screens before and after correction. Furthermore, the confirmation process takes time, as each screen needs to be visually checked one by one.

Therefore, in this embodiment, we will describe a screen comparison device 100 that can efficiently and accurately check for unintended wording or changes in layout before and after modification when updating an application screen.

The screen comparison device 100 has the functions of a computer, such as a workstation, a server, or a personal computer, and executes the screen comparison process described below based on a screen comparison method. The screen comparison device 100 may be connected to other devices, such as a server, so as to be able to communicate with them via a network (not shown).

Fig. 2 is a hardware configuration diagram of the screen comparison device 100 in this embodiment. As shown in Fig. 2, the screen comparison device 100 is configured by connecting a control unit 101, an input device 102, an output device 103, a storage unit 104, and an interface (I/F) 105 to a bus 106. In addition, the screen comparison device 100 may have other hardware configurations not shown in Fig. 2.

The control unit 101 provides overall control over the entire screen comparison device 100 and has a CPU (Central Processing Unit) that loads programs and executes the screen comparison process.

The input device 102 is, for example, a keyboard, joystick, light pen, mouse, touchpad, touch panel, trackball, etc., which inputs various data and signals.

The output device 103 outputs images and other information, for example, various displays such as LCDs (Liquid Crystal Displays), printers, etc.

The storage unit 104 has a RAM (Random Access Memory), a ROM (Read Only Memory), a HDD (Hard Disk Drive), etc. The storage unit 104 stores a program for the screen comparison device 100 to execute the screen comparison process, as well as a control program for controlling and executing each function of the screen comparison device 100, and screen data such as a current screen and a new screen. The storage unit 104 provides a working area for the screen comparison device 100 to execute a program. The storage unit 104 may also store a current screen OCR result array and a new screen OCR result array (described below) that store the results of comparing the character recognition results of the current screen and the character recognition results of the new screen.

The interface 105 is a unit for connecting to an external device (not shown), such as a serial interface such as USB (Universal Serial Bus) or a parallel interface such as Ethernet (registered trademark). The current screen 10 and new screen 20 compared by the screen comparison device 100 are stored in the memory unit 104 via the interface 105.

The screen comparison process in this embodiment is performed by the screen comparison device 100 described above.

Figure 3 is a diagram showing an example of functional blocks of the screen comparison device 100 relating to the screen comparison process of this embodiment. As shown in Figure 3, the control unit 101 of the screen comparison device 100 includes an extraction unit 111, a detection unit 112, and a notification unit 113. The control unit 101 may include other components not shown in Figure 3. The memory unit 104 stores a current screen OCR result array 121 and a new screen OCR result array 122, which will be described later. The memory unit 104 may store information not shown in Figure 3.

The extraction unit 111 extracts at least one or more words contained in the current screen 10 displayed by the application before modification as a character recognition result for the current screen 10. The extraction unit 111 uses existing optical character recognition (OCR) technology to extract at least one or more words from the words contained in the current screen 10 as a character recognition result for the current screen 10 (hereinafter referred to as the "OCR result").

The extraction unit 111 also extracts at least one or more words contained in the new screen 20 displayed by the modified application as a character recognition result for the new screen 20. The extraction unit 111 uses existing optical character recognition technology to extract at least one or more words from the words contained in the new screen 20 as a character recognition result for the new screen 20 (hereinafter referred to as an "OCR result").

The detection unit 112 detects character errors in the new screen 20 according to pairs of at least one or more OCR results of the current screen 10 extracted by the extraction unit 111 and at least one or more OCR results of the new screen 20.

The notification unit 113 notifies the user of the character error on the new screen 20 detected by the detection unit 112 through the display of the output device 103. The notification unit 113 may also notify the user by sound through a speaker (not shown).

Next, the screen comparison process executed by the screen comparison device 100 will be described in detail. FIG. 4 is a flowchart showing an example of the procedure of a screen comparison method using the screen comparison process executed by the screen comparison device 100. The screen extraction process executed by the screen comparison method is executed by the control unit 101 of the screen comparison device 100.

First, the control unit 101 performs an extraction process (step S101). In the extraction process, the OCR results included in the current screen 10 are extracted, and the OCR results included in the new screen 20 are also extracted. The extraction process executed by the screen comparison device 100 will be specifically described with reference to FIG. 5.

Figure 5 is a flowchart showing an example of the procedure of the extraction process executed by the screen comparison device 100 in step S101 of Figure 4. The extraction process executed by the screen comparison device 100 is executed by the control unit 101 of the screen comparison device 100.

First, the control unit 101 performs OCR on the current screen 10 and extracts the OCR result (step S201). In this process, the control unit 101 extracts at least one pre-correction character included in the current screen 10 displayed by the pre-correction application as the OCR result of the current screen 10.

The control unit 101 stores the OCR results of the current screen extracted in step S201 in the current screen OCR result array 121 (step S202). As a result of performing the processes of steps S201 and S202, the control unit 101 determines whether or not all OCR results of the current screen have been extracted (step S203). If all OCR results of the current screen have not been extracted (step S203: NO), the processes of steps S201 to S203 are repeatedly executed until all OCR results of the current screen have been extracted.

If all OCR results for the current screen have been extracted (step S203: YES), the control unit 101 performs OCR on the new screen and extracts the OCR results (step S204). In this process, the control unit 101 extracts at least one corrected phrase contained in the new screen displayed by the corrected application as the OCR result for the new screen. The OCR result includes the characters displayed on the new screen 20.

The control unit 101 stores the OCR results of the new screen extracted in step S204 in the OCR result array of the new screen (step S205). As a result of performing the processes of steps S204 and S205, the control unit 101 determines whether or not all the OCR results of the new screen have been extracted (step S206). If all the OCR results of the new screen have not been extracted (step S206: NO), the processes of steps S204 to S206 are repeatedly executed until all the OCR results of the new screen have been extracted. If all the OCR results of the new screen have been extracted (step S206: YES), the extraction process executed by the screen comparison device 100 is terminated. When this process is terminated, the process proceeds to step S102 in FIG. 4.

Figure 6 shows an example of a current screen OCR result array 121 and a new screen OCR result array 122 stored in the memory unit 104.

In the current screen OCR result array 121, for each ID, the OCR result of the current screen, the coordinates of the OCR result, a counter of the OCR result of the current screen, and information on the ID of the linked new screen are stored. The OCR result of the current screen stores the text that is the result of OCR of the current screen 10. The coordinates of the OCR result store the position information where the OCR result of the current screen is located.

The counter for the OCR result of the current screen stores information on the number of times the OCR result is displayed on the current screen 10. The ID of the linked new screen stores the ID of the OCR result of the new screen 20 that corresponds to the OCR result of the current screen 10.

The new screen OCR result array 122 stores, for each ID, the OCR result of the new screen, the coordinates of the OCR result, a counter of the OCR result of the new screen, the ID of the associated current screen, the check result, and positional deviation information. The OCR result of the new screen stores the text that is the result of OCR on the new screen 20. The coordinates of the OCR result store the positional information where the OCR result of the new screen is located.

The new screen OCR result counter stores information on the number of times the OCR result is displayed on the new screen 20. The linked new screen ID stores the ID of the OCR result of the new screen 20 that corresponds to the OCR result of the current screen 10.

In step S102 of FIG. 4, the control unit 101 performs a single pair extraction process. The details of the single pair extraction process executed in step S102 will be described with reference to FIG. 7. FIG. 7 is a flowchart showing an example of the procedure of the single pair extraction process executed in step S102 of FIG. 4. The single pair extraction process executed by the screen comparison device 100 is executed by the control unit 101 of the screen comparison device 100.

First, the control unit 101 selects one OCR result for the current screen stored in the current screen OCR result array 121 (step S301).

The control unit 101 searches the OCR result array for the new screen for OCR results that match at least a portion of the OCR results for the current screen selected in step S301 (step S302).

As a result of the search in step S302, it is determined whether the OCR results of the current screen selected in step S301 match at least a part of the OCR results of the new screen (step S303). If the OCR results of the selected current screen do not match the OCR results of the new screen (step S303: NO), the process proceeds to step S306.

If the selected OCR result of the current screen matches at least a part of the OCR result of the new screen (step S303: YES), the control unit 101 increments the counter of the current screen OCR result array 121 in which the matching OCR result is stored (step S304). For example, the counter of an OCR result displayed only once on the current screen 10 becomes "1", and the counter of an OCR result displayed twice on the current screen 10 becomes "2". The control unit 101 links the ID of the OCR result of the new screen corresponding to the matching OCR result to the current screen OCR result array 121 (step S305). For example, the OCR result "Deposit" corresponding to ID "1" of the current screen OCR result array corresponds to the OCR result "Deposit" of ID "2" and "Deposit" of ID "7" of the new screen OCR result array. Therefore, "2" and "7" are stored as the IDs of the new screens linked to the OCR results corresponding to ID "1" in the current screen OCR result array.

The control unit 101 determines whether or not all OCR results stored in the current screen OCR result array 121 have been selected (step S306). If all OCR results stored in the current screen OCR result array 121 have not been selected (step S306: NO), the process returns to step S301, and steps S301 to S306 are repeatedly executed until all OCR results stored in the current screen OCR result array 121 have been selected. If all OCR results stored in the current screen OCR result array 121 have been selected (step S306: YES), the process proceeds to step S307.

The control unit 101 selects one of the OCR results for a new screen stored in the OCR result array for the new screen (step S307).

The control unit 101 searches the current screen OCR result array 121 for OCR results of the current screen that match at least a portion of the OCR results of the new screen selected in step S307 (step S308). As a result of the search in step S308, it is determined whether the OCR results of the new screen selected in step S307 match at least a portion of the OCR results of the current screen (step S309).

If the OCR results of the selected new screen do not match the OCR results of the current screen (step S309: NO), the process proceeds to step S312. If the OCR results of the selected new screen match at least a portion of the OCR results of the current screen (step S309: YES), the control unit 101 increments the counter of the OCR result array of the new screen in which the matching OCR results are stored (step S310). For example, the counter of an OCR result that is displayed only once on the new screen 20 is "1", and the counter of an OCR result that is displayed twice on the new screen 20 is "2".

The control unit 101 links the ID of the OCR result of the current screen that corresponds to the matching OCR result to the OCR result array of the new screen (step S311). For example, the OCR result "deposit" corresponding to ID "2" in the new screen OCR result array corresponds to the OCR result "deposit" of ID "1" and "deposit" of ID "6" in the current screen OCR result array. Therefore, "1" and "6" are stored as the ID of the current screen linked to the OCR result corresponding to ID "2" in the new screen OCR result array.

The control unit 101 determines whether all the OCR results stored in the OCR result array of the new screen have been selected (step S312). If all the OCR results stored in the OCR result array of the new screen have not been selected (step S312: NO), the process returns to step S307, and steps S307 to S312 are repeatedly executed until all the OCR results stored in the OCR result array of the new screen have been selected. Then, if all the OCR results stored in the OCR result array of the new screen have been selected (step S312: YES), the single pair extraction process executed by the screen comparison device 100 is terminated. When this process is terminated, the process proceeds to step S103 in FIG. 4.

In step S103 of FIG. 4, the control unit 101 performs a multiple pair extraction process. The multiple pair extraction process executed in step S103 will be described in detail with reference to FIG. 8. FIG. 8 is a flowchart showing an example of the procedure of the multiple pair extraction process executed in step S103 of FIG. 4. The multiple pair extraction process executed by the screen comparison device 100 is executed by the control unit 101 of the screen comparison device 100.

First, the control unit 101 selects one OCR result of the current screen stored in the current screen OCR result array 121 (step S401).

The control unit 101 determines whether the counter of the selected OCR result from the OCR results of the current screen stored in the current screen OCR result array 121 is 2 or more (step S402). In this process, the OCR results of the current screen stored in the current screen OCR result array are processed one item at a time. Then, processing is performed for those with a counter of 2 or more, that is, those with two or more candidates for a pair of OCR results. For example, according to the current screen OCR result array 121 in FIG. 6, the OCR result "Deposit" of ID "1" and the OCR result "Deposit" of ID "6" are to be processed. If the counter of the OCR result of the current screen stored in the current screen OCR result array 121 is 1 (step S402: NO), processing proceeds to step S414.

If the counter of the OCR results of the current screen stored in the current screen OCR result array 121 is 2 or more (step S402: YES), the control unit 101 selects one OCR result stored in the current screen OCR result array 121 (step S403). The control unit 101 specifies the ID of the OCR result of the current screen selected in step S403 as a parameter (step S404).

The control unit 101 performs surrounding OCR result acquisition processing, which will be described later with reference to FIG. 9 (step S405). In this processing, processing is performed to acquire surrounding OCR results for the OCR results of the current screen stored in the current screen OCR result array 121.

Figure 9 is a diagram showing an example of an outline of a method for acquiring the OCR results of the current screen acquired by the surrounding OCR result acquisition process in step S405. As shown in Figure 9, when "Deposit" is selected from the OCR results of the current screen stored in the current screen OCR result array 121, the control unit 101 acquires the OCR result "Withdrawal" to the right of "Deposit" and the OCR result "Enter in bankbook" below as surrounding OCR results. In this case, the output data acquired to be stored in the current screen OCR result array 121 are the OCR result "None" above, the OCR result "Enter in bankbook" below (ID "2"), the OCR result "None" to the left, and the OCR result "Withdrawal" (ID "3") to the right.

The control unit 101 selects one of the OCR results for a new screen stored in the OCR result array for the new screen (step S406). The control unit 101 specifies the ID of the OCR result for the new screen selected in step S406 as a parameter (step S407).

The control unit 101 performs surrounding OCR result acquisition processing, which will be described later with reference to FIG. 9 (step S408). The surrounding OCR result acquisition processing executed in step S408 is substantially the same as the surrounding OCR result acquisition processing executed in step S405, but in the processing executed in step S408, processing is performed to acquire OCR results around the OCR results of the new screen stored in the new screen OCR result array 122.

Figure 10 is a diagram showing an example of an outline of a method for acquiring the OCR results of a new screen acquired by the surrounding OCR result acquisition process in step S408. As shown in Figure 10, when "Deposit" is selected from the OCR results of a new screen stored in the new screen OCR result array 122, the control unit 101 acquires the OCR result "Withdrawal" to the right of "Deposit" and the OCR result "Enter in bankbook" below as surrounding OCR results. In this case, the output data acquired to be stored in the current screen OCR result array 121 are the OCR result "New screen" (ID "1") above, the OCR result "Enter in bankbook" (ID "2") below, the OCR result "None" to the left, and the OCR result "Withdrawal" (ID "4") to the right.

The control unit 101 determines whether the surrounding OCR results match (step S409). If the surrounding OCR results match (step S409: YES), the control unit 101 sets the counter corresponding to the OCR result in the current screen OCR result array 121 to 1 (step S410). That is, in this case, it is determined that a one-to-one OCR result pair that matches the OCR result has been established. The control unit 101 updates the associated screen ID (step S411). When this process is completed, the process proceeds to step S414.

In the process of step S409, if the surrounding OCR results do not match (step S409: NO), the control unit 101 judges whether the process has been executed the number of times equal to the number of counters (step S412). For example, in the case of an OCR result with a counter of "2", it is judged whether the process of steps S406 to S412 has been executed twice. For example, in the case of the OCR result "Deposit" with ID "1" in the current screen OCR result array, the counter is "2", so the process is executed twice for the OCR result "Deposit" with ID [2] in the new screen OCR result array 122 and the OCR result "Deposit" with ID "7". If the process has not been executed the number of times equal to the number of counters, the process returns to step S406, and the process of steps S406 to S412 is executed repeatedly until all the counters have been executed.

When the process has been repeated for all counters (step S412), the counter for the selected OCR result in the current screen OCR result array 121 is set to 0 (step S413). In other words, in this case, it is determined that there is no OCR result item that matches the OCR result.

In step S414, the control unit 101 determines whether or not all OCR results stored in the current screen OCR result array 121 have been selected (step S414). If all OCR results stored in the current screen OCR result array 121 have not been selected (step S414: NO), the process returns to step S402, and steps S402 to S414 are repeatedly executed until all OCR results stored in the current screen OCR result array 121 have been selected. Then, if all OCR results stored in the current screen OCR result array 121 have been selected (step S414: YES), the multiple pair extraction process executed by the screen comparison device 100 is terminated.

The surrounding OCR result acquisition process executed in step S405 or step S408 of FIG. 8 will be described in detail with reference to FIG. 11. FIG. 11 is a flowchart showing an example of the procedure of the surrounding OCR result acquisition process executed in step S405 or step S408 of FIG. 8. The surrounding OCR result acquisition process executed by the screen comparison device 100 is executed by the control unit 101 of the screen comparison device 100.

First, the control unit 101 selects one OCR result stored in the current screen OCR result array 121 (step S501). The control unit 101 determines whether the counter of the selected OCR result is 1 (step S502). In this process, the control unit 101 excludes an OCR result with a counter of "0" from comparison because it is an OCR result for which no one-to-one pair exists, and an OCR result with a counter of "2" or more is an OCR result for which one-to-one pair candidates cannot be narrowed down. If the counter of the selected OCR result is not 1 (step S502: NO), that is, if the counter of the selected OCR result is 0 or 2 or more, the process proceeds to step S507.

If the counter for the selected OCR result is 1 (step S502: YES), the control unit 101 selects one of the directions, up, down, left, and right (step S503). For example, the control unit 101 selects the up direction from the up, down, left, and right directions. The control unit 101 determines whether other OCR results exist in the selected direction (step S504). In this process, the control unit 101 uses the coordinate information stored in the current screen OCR result array 121 and the new screen OCR result array 122 to determine in which direction, up, down, left, and right, other OCR results exist. The control unit 101 makes the determination of up, down, left, and right based on the coordinates (Xs, Ys, Xe, Ye) stored in the current screen OCR result array 121 and the coordinates (Xs, Ys, Xe, Ye) stored in the new screen OCR result array 122.

If other OCR results exist in the selected direction (step S504: YES), the control unit 101 performs an OCR result update process (step S505), which will be described later with reference to FIG. 12. When this process ends, the process proceeds to step S507.

If there are no other OCR results in the selected direction (step S504: NO), the control unit 101 determines whether all directions have been selected (step S506). In this process, the control unit 101 determines whether the processes in steps S503 to S506 have been performed for all directions, including up, down, right, and left. If not all directions have been selected (step S506: NO), the process returns to step S503, and the process repeats the processes in steps S503 to S506 until all directions, including up, down, left, and right, have been selected.

If all the directions, up, down, left and right, have been selected (step S506: YES), the control unit 101 determines whether all the OCR results stored in the current screen OCR result array 121 have been selected (step S507). If all the OCR results stored in the current screen OCR result array 121 have not been selected (step S507: NO), the process returns to step S501, and steps S501 to S507 are repeatedly executed until all the OCR results stored in the current screen OCR result array 121 have been selected. If all the OCR results stored in the current screen OCR result array 121 have been selected (step S507: YES), the surrounding text acquisition process executed by the screen comparison device 100 is terminated. In the case of the surrounding OCR result acquisition process of step S405 in FIG. 8, the process proceeds to step S406 in FIG. 8. In the case of the surrounding OCR result acquisition process in step S408 in FIG. 8, processing proceeds to step S409 in FIG. 8.

The OCR result update process executed in step S505 of FIG. 11 will be described in detail with reference to FIG. 12. FIG. 12 is a flowchart showing an example of the procedure of the OCR result update process executed in step S505 of FIG. 11. The OCR result update process executed by the screen comparison device 100 is executed by the control unit 101 of the screen comparison device 100.

First, the control unit 101 calculates the distance between the OCR result of the current screen selected in the process of step S501 in FIG. 11 and the OCR result of the direction determined to exist in the process of step S504 in FIG. 11 (step S601). In this process, the control unit 101 calculates the distance between the OCR results using the coordinate information stored in the current screen OCR result array 121 and the new screen OCR result array 122.

As a result of calculation in step S601, it is determined whether the distance between the OCR result of the selected current screen and the OCR result in the direction determined to exist is closer than a predetermined distance (step S602). The predetermined distance can be set arbitrarily. For example, a threshold value that determines the predetermined distance can be set arbitrarily depending on the characteristics of the screen, such as the current screen or new screen, displayed by the application. Note that if multiple OCR results exist in the same direction, the closest OCR result is used.

If the distance between the OCR result of the selected current screen and the OCR result of the direction determined to exist is greater than a predetermined distance (step S602: NO), the OCR result update process performed by the screen comparison device 100 ends, and the process proceeds to step S507 in FIG. 11.

If the distance between the OCR result of the selected current screen and the OCR result of the direction determined to exist is closer than a predetermined distance (step S602: YES), the control unit 101 determines whether the OCR result of the direction determined to exist has already been set (step S603).

If the OCR result for the direction determined to exist has not been set (step S603: NO), the control unit 101 sets the OCR result for the current screen for the determined direction (step S604). When this process ends, the OCR result update process executed by the screen comparison device 100 ends, and the process proceeds to step S507 in FIG. 11.

If the OCR result for the direction determined to exist has already been set (step S603: YES), the control unit 101 determines whether the distance is closer than the OCR result for the set direction (step S605). If the distance is farther than the OCR result for the set direction (step S605: NO), the OCR result update process executed by the screen comparison device 100 ends, and the process proceeds to step S507 in FIG. 11.

If the distance is closer than the OCR result for the set direction (step S605: YES), the control unit 101 updates the OCR result for the current screen in the determined direction (step S606). When this process ends, the OCR result update process executed by the screen comparison device 100 ends, and the process proceeds to step S507 in FIG. 11.

In step S104 of FIG. 4, the control unit 101 performs a typographical error and omission check process. The typographical error and omission check process performed in step S104 will be described in detail with reference to FIG. 13. FIG. 13 is a flowchart showing an example of the procedure for the typographical error and omission check process performed in step S104 of FIG. 4. The typographical error and omission check process performed by the screen comparison device 100 is executed by the control unit 101 of the screen comparison device 100.

First, the control unit 101 selects one OCR result stored in the current screen OCR result array 121 (step S701). The control unit 101 compares the OCR result stored in the current screen OCR result array 121, which is linked by an ID, with the OCR result stored in the new screen OCR result array 122 (step S702). In this process, it is determined whether or not at least a portion of the OCR result stored in the current screen OCR result array 121 matches the OCR result stored in the new screen OCR result array 122.

The control unit 101 adds the comparison result obtained in step S702 to the check results of the new screen OCR result array 122 (step S703).

For example, when comparing the OCR result "Deposit" for ID "1" stored in the current screen OCR result array 121 with the OCR result "Deposit" for ID "2" stored in the new screen OCR result array 122, the wording is a perfect match. In this case, the control unit 101 adds information indicating a "perfect match" to the check result of the new screen OCR result array 122.

On the other hand, when comparing the OCR result "English Guide" for ID "7" stored in the current screen OCR result array 121 with the OCR result "English Guid" for ID "8" stored in the new screen OCR result array 122, the wording only matches at the beginning. In this case, the control unit 101 adds information indicating "forward match" to the check result of the new screen OCR result array 122.

The control unit 101 determines whether or not all the OCR results stored in the current screen OCR result array 121 have been selected (step S704). If all the OCR results stored in the current screen OCR result array 121 have not been selected (step S704: NO), the process returns to step S701, and steps S701 to S704 are repeatedly executed until all the OCR results stored in the current screen OCR result array 121 have been selected. If all the OCR results stored in the current screen OCR result array 121 have been selected (step S704: YES), the typo check process executed by the screen comparison device 100 is completed, and the process proceeds to step S105 in FIG. 4.

In step S105 of FIG. 4, the control unit 101 performs a misalignment check process. The details of the misalignment check process performed in step S105 will be described with reference to FIG. 14. FIG. 14 is a flowchart showing an example of the procedure of the misalignment check process performed in step S105 of FIG. 4. The misalignment check process performed by the screen comparison device 100 is executed by the control unit 101 of the screen comparison device 100.

First, the control unit 101 selects one OCR result of the current screen stored in the current screen OCR result array 121 (step S801). The control unit 101 specifies the ID of the OCR result of the current screen selected in step S801 as a parameter (step S802). The control unit 101 performs the surrounding OCR result acquisition process described with reference to FIG. 11 (step S803).

The control unit 101 determines whether or not all the OCR results stored in the current screen OCR result array 121 have been selected (step S804). If all the OCR results stored in the current screen OCR result array 121 have not been selected (step S804: NO), the process returns to step S801, and steps S801 to S804 are repeatedly executed until all the OCR results stored in the current screen OCR result array 121 have been selected.

If all OCR results stored in the current screen OCR result array 121 have been selected (step S804: YES), the control unit 101 selects the OCR result of one new screen stored in the new screen OCR result array 122 (step S805). The control unit 101 specifies the ID of the OCR result of the new screen selected in step S805 as a parameter (step S806). The control unit 101 performs the surrounding OCR result acquisition process described with reference to FIG. 11 (step S807).

The control unit 101 determines whether or not all of the OCR results stored in the new screen OCR result array 122 have been selected (step S808). If all of the OCR results stored in the new screen OCR result array 122 have not been selected (step S808: NO), the process returns to step S805, and steps S805 to S8084 are repeatedly executed until all of the OCR results stored in the new screen OCR result array 122 have been selected.

If all OCR results stored in the new screen OCR result array 122 have been selected (step S808: YES), the control unit 101 compares the surrounding OCR results between the OCR results stored in the current screen OCR result array 121 and the OCR results stored in the new screen OCR result array 122, and stores the comparison results in the new screen OCR result array 122 (step S809).

For example, we will explain the case where the OCR result "Deposit" of ID "1" stored in the current screen OCR result array 121 is compared with the OCR result "Deposit" of ID "2" stored in the new screen OCR result array 122. In this case, the OCR result for the OCR result "Deposit" of ID "1" stored in the current screen OCR result array 121 is "None" to the top, the OCR result for the bottom is "Entry into bankbook", the OCR result for the left is "None", and the OCR result for the right is "Withdrawal". On the other hand, the OCR result for the OCR result "Deposit" of ID "2" stored in the new screen OCR result array 122 is "New screen" to the top, the OCR result for the bottom is "Entry into bankbook", the OCR result for the left is "None", and the OCR result for the right is "Withdrawal". Therefore, when comparing the surrounding OCR results, the comparison results for the OCR results in the upward direction differ, so the control unit 101 stores the information "position misalignment exists" in the comparison result.

We will now explain a comparison between the OCR result "English Guide" for ID "7" stored in the current screen OCR result array 121 and the OCR result "English Guid" for ID "8" stored in the new screen OCR result array 122. In this case, the OCR result for the OCR result "English Guide" for ID "7" stored in the current screen OCR result array 121 is "Utility Bill Payment" in the upward direction, the OCR result for the downward direction is "None", the OCR result for the left direction is "Credit card simple insurance/life insurance", and the OCR result for the right direction is "None". On the other hand, the OCR result for the OCR result "English Guid" for ID "8" stored in the new screen OCR result array 122 is "Utility Bill Payment" in the upward direction, the OCR result for the downward direction is "None", the OCR result for the left direction is "Credit card simple insurance/life insurance", and the OCR result for the right direction is "None". Therefore, when both OCR results are compared, the comparison results of all the OCR results in the top, bottom, left, and right directions match, so the control unit 101 stores the information "no misalignment" in the comparison result. When the process ends, the misalignment check process performed by the screen comparison device 100 ends, and the process proceeds to step S106 in FIG. 4.

In step S106 of FIG. 4, the control unit 101 performs a result notification process. In the result notification process, a process is performed to notify the user of typos and misalignment based on the check results stored in the new screen OCR result array 122 and the comparison results. FIG. 15 is a diagram showing an example of notifying the user of typos and misalignment.

For example, for an OCR result where the check result is "forward match," highlighting 401 is performed to notify the OCR result of typos and omissions by referencing the coordinate information stored in the current screen OCR result array 121 and the coordinate information stored in the new screen OCR result array 122. Highlighting 401 uses, for example, a red frame to highlight. This makes it easy to check whether there are unintended typos and omissions in the text before and after correction when updating the application screen. Therefore, checking for typos and omissions in the text can be performed efficiently and accurately.

In addition, for OCR results where the comparison result is "positional deviation exists," highlighting 402, 403 is performed to notify the user of the positional deviation by referencing the coordinate information stored in the current screen OCR result array 121 and the coordinate information stored in the new screen OCR result array 122. Highlighting 402, 403 is performed by, for example, highlighting with a yellow frame. This makes it easy to check whether there is an unintended change in position before and after the correction when updating the application screen. Therefore, it is possible to check for changes in position efficiently and accurately.

Although not shown, OCR results that are free of typos and misalignment may also be highlighted, for example, with a blue border. Also, highlighting is typically done with red, yellow, and blue borders, but is not limited to this and may be done with blinking, shading, fluorescent colors, etc.

The present invention is not limited to the above-described embodiment as it is, and the components can be modified and embodied without departing from the gist of the invention at the implementation stage. In addition, various inventions can be formed by appropriately combining the multiple components disclosed in the above embodiments. For example, all the components shown in the embodiments may be appropriately combined. Furthermore, components from different embodiments may be appropriately combined. Of course, various modifications and applications are possible without departing from the spirit of the invention.

10: Current screen 20: New screen 100: Screen comparison device 101: Control unit 102: Input device 103: Output device 104: Memory unit 105: Interface 106: Bus 111: Extraction unit 112: Detection unit 113: Notification unit 121: Current screen OCR result array 122: New screen OCR result array 300a: Text 300b: Text 300c: Text 401: Highlight 402: Highlight 403: Highlight

Claims (5)

In a screen comparison device for comparing screens displayed by an application to be modified,
an extraction unit that extracts at least one character included in a current screen displayed by the application before the modification as a character recognition result for the current screen, and extracts at least one character included in a new screen displayed by the application after the modification as a character recognition result for the new screen;
a detection unit that detects an error in a character of the new screen according to a pair of at least one or more character recognition results of the current screen and at least one or more character recognition results of the new screen extracted by the extraction unit;
a means for checking the presence or absence of positional deviation of the character recognition results of the at least one or more new screens by comparing the character recognition results of the at least one or more current screens in the upward, downward, rightward, and leftward directions with the character recognition results of the at least one or more new screens in the upward, downward, rightward, and leftward directions, respectively;
Equipped with
No positional deviation indicates a comparison result in which the character recognition results of the at least one or more current screens in the upward, downward, right, and left directions match the character recognition results of the at least one or more new screens in the upward, downward, right, and left directions, respectively;
A screen comparison device characterized in that a position shift indicates a comparison result in which the character recognition result of the at least one or more current screens in a specified direction among the upward, downward, right, and left directions does not match the character recognition result of the at least one or more new screens in the specified direction among the upward, downward, right, and left directions . 2. The screen comparison device according to claim 1,
The screen comparison device further comprises a notification unit that notifies a user of an error in characters on the new screen detected by the detection unit. 3. The screen comparison device according to claim 1,
The detection unit detects character errors in the new screen based on character recognition results for the upward, downward, right, and left directions of the at least one or more current screens and character recognition results for the upward, downward, right, and left directions of the at least one or more new screens. A screen comparison method executed by a screen comparison device for comparing screens displayed by an application to be modified, comprising:
extracting at least one or more characters included in a current screen displayed by the application before the modification as a character recognition result for the current screen, and extracting at least one or more characters included in a new screen displayed by the application after the modification as a character recognition result for the new screen;
Detecting an error in a character of the new screen according to the extracted pairs of at least one character recognition result of the current screen and at least one character recognition result of the new screen;
checking whether or not there is a positional deviation of the character recognition results of the at least one or more new screens by comparing the character recognition results of the at least one or more current screens in the upward, downward, rightward, and leftward directions with the character recognition results of the at least one or more new screens in the upward , downward, rightward , and leftward directions, respectively;
No positional deviation indicates a comparison result in which the character recognition results of the at least one or more current screens in the upward, downward, right, and left directions match the character recognition results of the at least one or more new screens in the upward, downward, right, and left directions, respectively;
The screen comparison method is characterized in that a position shift indicates a comparison result in which the character recognition result of the at least one or more current screens in a predetermined direction among the upward, downward, right, and left directions does not match the character recognition result of the at least one or more new screens in the predetermined direction among the upward, downward, right, and left directions . A program executed by a screen comparison device for comparing screens displayed by an application to be corrected,
extracting at least one or more characters included in a current screen displayed by the application before the modification as a character recognition result for the current screen, and extracting at least one or more characters included in a new screen displayed by the application after the modification as a character recognition result for the new screen;
Detecting an error in a character of the new screen according to the extracted pairs of at least one character recognition result of the current screen and at least one character recognition result of the new screen;
checking whether or not there is a positional deviation in the character recognition results of the at least one or more new screens by comparing the character recognition results of the at least one or more current screens in the upward, downward, rightward, and leftward directions with the character recognition results of the at least one or more new screens in the upward, downward, rightward, and leftward directions, respectively;
Execute the process ,
No positional deviation indicates a comparison result in which the character recognition results of the at least one or more current screens in the upward, downward, right, and left directions match the character recognition results of the at least one or more new screens in the upward, downward, right, and left directions, respectively;
A program characterized in that a position shift indicates a comparison result in which the character recognition result of the at least one or more current screens in a specified direction among the upward, downward, right, and left directions does not match the character recognition result of the at least one or more new screens in the specified direction among the upward, downward, right, and left directions .