JP5962350B2 - Program, information processing apparatus and test method - Google Patents

Description

  The present invention relates to a program, an information processing apparatus, and a test method.

  In order to verify the quality of software, test data (sometimes referred to as test data) may be input to the software to test its operation. For example, whether or not a correct output can be obtained from the software with respect to the input test data is confirmed. For this purpose, it is conceivable to create a plurality of test data covering patterns of combinations of commands and parameters that can be input and input them to the software to be tested. Analyze the software output for the input test data to find bugs. However, if an exhaustive test is performed, the time and man-hours required for the test may increase.

  Therefore, there is a proposal for supporting the creation of a test plan by preliminarily giving a test priority to an item to be tested and determining an item to be preferentially / intensively tested based on the test priority. In addition, when a combination of parameters to be tested (test pattern) is similar to a combination of parameters that have been tested, there is a proposal of a process for removing the test pattern from the test target.

Japanese Patent Laid-Open No. 2003-256206 JP 2010-3224 A

  As described above, in software testing, a plurality of test data including combinations of character strings such as commands and parameters may be input to the software to be tested. In this test, it is conceivable to test the test data by inputting it into software in the order of generation or the order planned in advance.

  However, what you want to test with priority as you test can change. For example, there is a case where a test using test data that may give an incorrect result is executed first to confirm an error situation at an early stage and to start a countermeasure work. In addition, for example, a test using test data that can give a correct result may be postponed. However, there is a problem that it is difficult to determine which test data is input when an incorrect result is obtained unless the test is actually executed.

  According to one aspect, an object of the present invention is to provide a program, an information processing apparatus, and a test method that can efficiently execute a test.

According to one embodiment, a program executed by a computer is provided. This program allows a computer to sequentially execute an execution test of processing according to test data including a plurality of character strings on each of the plurality of test data, and character strings included in the test data based on the results of the execution test. Execute the process to adjust the priority of each test, and change the execution test execution order for test data that has not been executed from multiple test data based on the priority of each character string included in the test data Let

According to one embodiment, an information processing apparatus is provided. The information processing apparatus includes a storage unit and a calculation unit. The storage unit stores a plurality of character strings. The arithmetic unit sequentially executes the execution test of the process according to the test data including the character string stored in the storage unit for each of the plurality of test data, and the characters included in the test data based on the result of the execution test. The priority for each column is adjusted, and the execution order of the execution tests for the test data for which the execution test has not been executed among the plurality of test data is changed based on the priority for each character string included in the test data.

According to one embodiment, a test method executed by a computer is provided. In this test method, the computer sequentially executes the execution test of the process according to the test data including a plurality of character strings on each of the plurality of test data, and the characters included in the test data based on the result of the execution test. The priority for each column is adjusted, and the execution order of the execution tests for the test data for which the execution test has not been executed among the plurality of test data is changed based on the priority for each character string included in the test data.

  According to one embodiment, the test can be performed efficiently.

It is a figure which shows the information processing apparatus of 1st Embodiment. It is a figure which shows the hardware example of 2nd Embodiment. It is a figure which shows the example of software of 2nd Embodiment. It is a figure which shows the example of data of 2nd Embodiment. It is a figure which shows the example of the definition file of 2nd Embodiment. It is a figure which shows the example (continuation) of the definition file of 2nd Embodiment. It is a figure which shows the example of the code element management table of 2nd Embodiment. It is a figure which shows the example of the combination narrowing-down conditions list | wrist of 2nd Embodiment. It is a figure which shows the example of the combination management table of 2nd Embodiment. It is a figure which shows the example of the order management table of 2nd Embodiment. It is a figure which shows the example of the compilation success / failure reference | standard table of 2nd Embodiment. It is a figure which shows the example of the source code for the test of 2nd Embodiment. It is a figure which shows the example of the test result table of 2nd Embodiment. It is a flowchart which shows the test process of 2nd Embodiment. It is a flowchart which shows the code element combination process of 2nd Embodiment. It is a figure which shows the example of a combination of the code element of 2nd Embodiment. It is a flowchart which shows execution of the objective code of 2nd Embodiment. It is a figure which shows the example of a change of the code element of 2nd Embodiment. It is a flowchart which shows the test order change of 2nd Embodiment. It is a figure which shows the example of a test order change of 2nd Embodiment. It is a figure which shows the example of software of 3rd Embodiment. It is a figure which shows the example of data of 3rd Embodiment. It is a figure which shows the example of the definition file of 3rd Embodiment. It is a figure which shows the example of the code element management table of 3rd Embodiment. It is a figure which shows the example of a test order change of 3rd Embodiment.

Hereinafter, the present embodiment will be described with reference to the drawings.
[First Embodiment]
FIG. 1 is a diagram illustrating the information processing apparatus according to the first embodiment. The information processing apparatus 1 inputs test data to the test target software and tests the processing of the software. The information processing apparatus 1 includes a storage unit 1a and a calculation unit 1b. The storage unit 1a is a memory such as a RAM (Random Access Memory). The arithmetic unit 1b is a processor such as a CPU (Central Processing Unit). For example, the information processing of the first embodiment can be realized by the arithmetic unit 1b executing the program stored in the storage unit 1a.

  The storage unit 1a stores a plurality of character strings. For example, the character string indicates a command or a parameter serving as an argument of the command. For example, the storage unit 1a stores character strings CS1, CS2, CS3, CS4, CS5, and CS6.

  The calculation unit 1b includes the test data based on the result of the execution test while sequentially executing the execution test of the process according to the test data including the character string stored in the storage unit 1a for each of the plurality of test data. Adjust the priority for each string. For example, when the result of the execution test for certain test data is incorrect, the calculation unit 1b may set the priority of the character string included in the test data higher than before the execution test of the test data. Good.

  For example, the software to be tested may be a compiler. The compiler processes the source code and creates a target code. In this case, the test data is data describing the source code. The source code includes a character string for defining a variable, a character string describing a loop, a branch, and the like. The software to be tested may be another application that executes an input command. In this case, the test data includes a character string such as a command and an argument given to the command. Here, the character string is one or more characters. The characters may include special characters such as symbols.

  The computing unit 1b changes the execution order of subsequent execution tests based on the priority for each character string included in the test data. The calculation unit 1b may calculate the priority of the test data based on the priority of the character string included in the test data. The computing unit 1b may change the execution order of the execution tests based on the priority for each test data. For example, an execution test of test data having a high priority may be performed with priority. If it does so, it can control to give priority to the test data with a high possibility that a result will become invalid from the results of the past execution test.

  More specifically, it is as follows. For example, consider a case where test data A, B, C, D, and E are input to the software to be tested in this order to obtain a processing result, and the correctness is confirmed each time the result is obtained. In FIG. 1, a set of test data A, B, C, D, and E is referred to as a test data group. Test data A includes character strings CS1, CS2, and CS3. Test data B includes character strings CS3, CS4 and CS5. Test data C includes character strings CS1, CS2 and CS6. Test data D includes character strings CS2, CS3, CS6. Test data E includes character strings CS3, CS4 and CS6.

  First, the arithmetic unit 1b inputs test data A to the test target software. It is assumed that the result of the processing according to the test data A is correct. Next, the arithmetic unit 1b inputs the test data B to the test target software. It is assumed that the result of processing according to the test data B is invalid. Therefore, the calculation unit 1b sets the priority of the character strings CS3, CS4, and CS5 included in the test data B higher than before the execution test using the test data B is performed. Then, the priority of other test data including the character strings CS3, CS4, and CS5 is increased. In the example of FIG. 1, the test data D and E include character strings CS3 and CS4. Therefore, the priority of the test data D and E is increased.

  As a result, for example, the priority of the test data E becomes higher than the priority of the test data D. In addition, the priority of the test data D is higher than the priority of the test data C. Therefore, the arithmetic unit 1b changes the order of the test data C, D, E to the order of the test data E, D, C. That is, the test with respect to the test data E is carried out with respect to the test data D and C. In addition, the test for the test data D is carried out with respect to the test data C.

  Note that the calculation unit 1b may lower the priority of the character string included in the test data for which a correct result is obtained. If it does so, it can control so that the test data containing the said character string will be put behind other test data.

  Further, the calculation unit 1b may change the execution order at the timing when the execution test is performed on one test data, or the execution order is changed at the timing when the execution test is performed on two or more predetermined numbers of test data. Also good.

  According to the information processing apparatus 1, the character string included in the test data based on the result of the execution test while the execution unit 1b sequentially executes the execution test of the process according to the test data for each of the plurality of test data. Each priority is adjusted. The calculation unit 1b changes the execution order of subsequent execution tests based on the priority for each character string included in the test data.

  Thereby, a test can be performed efficiently. Specifically, the execution order of the tests is dynamically changed while executing the software processing test. Therefore, it is possible to control the execution order according to the test result of each test data. For example, even when it is difficult to determine in advance test data that may have incorrect results before starting the test, give priority to test data that is likely to give incorrect results according to the test results of each test data. Can be tested.

  Here, if the test data or character string causing the error is delayed, the countermeasure work may be delayed. In particular, depending on the number of combinations of character strings included in the test data, it may take several hours to several days from the start to the end of the test of all combinations. In order to improve quality, it is preferable to conduct a comprehensive test. However, the more comprehensive the test is, the longer the test takes and the delay in work can become serious.

  On the other hand, according to the information processing apparatus 1, the possibility that test data and a character string causing an error can be found early by the above-described control increases. Therefore, it becomes possible to quickly investigate the cause of a bug and countermeasures by referring to commands and parameters included in the test data. In this way, software debugging work can be efficiently supported.

  The information processing of the first embodiment can be applied to various software tests. In the following description, first, as a second embodiment, a case where the present invention is applied to an execution test of a compiler is illustrated. Next, as a third embodiment, a case of applying to an SQL execution test in a DBMS (DataBase Management System) will be exemplified. In the following description, a device that executes the test is referred to as a test device. The test apparatus can be realized by a computer, for example.

[Second Embodiment]
FIG. 2 is a diagram illustrating a hardware example of the second embodiment. The test apparatus 100 performs an execution test of compiler compilation processing (translation processing). In the second embodiment, in particular, optimization processing at the time of compilation is tested. Specifically, it is confirmed whether the target code can be created correctly even if optimization is performed at the time of compilation. The optimization process at the time of compiling includes, for example, loop fusion and division, change of loop nesting structure, and the like.

  The test apparatus 100 includes a processor 101, a RAM 102, an HDD (Hard Disk Drive) 103, a communication unit 104, an image signal processing unit 105, an input signal processing unit 106, a disk drive 107, and a device connection unit 108. Each unit is connected to the bus of the test apparatus 100.

  The processor 101 controls information processing of the test apparatus 100. The processor 101 may be a multiprocessor. The processor 101 is, for example, a CPU, a micro processing unit (MPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a programmable logic device (PLD). The processor 101 may be a combination of two or more elements among CPU, MPU, DSP, ASIC, FPGA, and PLD.

  The RAM 102 is a main storage device of the test apparatus 100. The RAM 102 temporarily stores at least part of an OS (Operating System) program and application programs to be executed by the processor 101. The RAM 102 stores various data used for processing by the processor 101.

  The HDD 103 is an auxiliary storage device of the test apparatus 100. The HDD 103 magnetically writes and reads data to and from the built-in magnetic disk. The HDD 103 stores an OS program, application programs, and various data. The test apparatus 100 may include other types of auxiliary storage devices such as flash memory and SSD (Solid State Drive), or may include a plurality of auxiliary storage devices.

  The communication unit 104 is an interface that can communicate with other computers via the network 10. The communication unit 104 may be a wired interface or a wireless interface.

  The image signal processing unit 105 outputs an image to the display 11 connected to the test apparatus 100 in accordance with an instruction from the processor 101. As the display 11, a CRT (Cathode Ray Tube) display, a liquid crystal display, or the like can be used.

  The input signal processing unit 106 acquires an input signal from the input device 12 connected to the test apparatus 100 and outputs the input signal to the processor 101. As the input device 12, for example, a pointing device such as a mouse or a touch panel, a keyboard, or the like can be used.

  The disk drive 107 is a drive device that reads a program and data recorded on the optical disk 13 using a laser beam or the like. As the optical disc 13, for example, a DVD (Digital Versatile Disc), a DVD-RAM, a CD-ROM (Compact Disc Read Only Memory), a CD-R (Recordable) / RW (ReWritable), or the like can be used. For example, the disk drive 107 stores the program and data read from the optical disk 13 in the RAM 102 or the HDD 103 in accordance with an instruction from the processor 101.

  The device connection unit 108 is a communication interface for connecting peripheral devices to the test apparatus 100. For example, the memory device 14 and the reader / writer device 15 can be connected to the device connection unit 108. The memory device 14 is a recording medium equipped with a communication function with the device connection unit 108. The reader / writer device 15 is a device that writes data to the memory card 16 or reads data from the memory card 16. The memory card 16 is a card-type recording medium. For example, the device connection unit 108 stores a program or data read from the memory device 14 or the memory card 16 in the RAM 102 or the HDD 103 in accordance with an instruction from the processor 101.

  In the second embodiment, a test target compiler is referred to as a test compiler. In order to test the optimization function of the test compiler, the following steps (1) to (3) are sequentially executed on a plurality of test source codes (test data).

(1) Compile the source code with the compiler under test. The compiler under test is compiled in two ways: optimization on (optimization) / optimization off (no optimization).
(2) The success or failure of the compilation is confirmed based on the compilation result of the tested compiler. If it is determined that the compilation is successful, two kinds of target codes generated by the optimization on / off compilation process by the tested compiler are executed.

  (3) Based on the execution results of both target codes, it is confirmed whether the target codes can be created correctly even if optimization is performed. Specifically, if the execution results of both target codes match, the target code has been created correctly. On the other hand, if the execution results of both target codes do not match, the target code has not been created correctly.

  FIG. 3 is a diagram illustrating an example of software according to the second embodiment. Each unit illustrated in FIG. 3 can be realized, for example, by the processor 101 executing a program stored in the RAM 102. Each unit in FIG. 3 may be realized by an ASIC, FPGA, or the like.

The test apparatus 100 includes a storage unit 110, a combination generation unit 120, an order management unit 130, a code generation unit 140, test software 150, and a verification unit 160.
The storage unit 110 stores data used for processing of each unit. The storage unit 110 can be mounted using a storage area of the RAM 102 or the HDD 103, for example. The data stored in the storage unit 110 includes a definition file, a combination management table, an order management table, a test result table, and the like.

  The definition file is data defining a character string indicating a command or parameter for generating a source code. The combination management table is data for managing combinations of character strings. The order management table is data for managing the execution order of tests for each combination.

  The combination generation unit 120 generates a combination of character strings based on the definition file stored in the storage unit 110 and registers it in the combination management table. The combination generation unit 120 also performs a process of narrowing down combinations to be used for testing by reducing similar combinations and extra combinations.

  The order management unit 130 determines a test execution order for a plurality of combinations registered in the combination management table stored in the storage unit 110 and registers the test execution order in the order management table. Specifically, a priority is set for each character string included in the combination. The order management unit 130 calculates the priority of the combination unit based on the priority for each character string. The order management unit 130 determines the test execution order based on the priority of the combination unit.

  The code generation unit 140 generates a test source code based on the definition file stored in the storage unit 110. At this time, the code generation unit 140 generates a test source code according to the order determined by the order management unit 130.

  The test software 150 is a software group including software to be tested. In the second embodiment, the software is a compiler. The compiler generates target code from the source code. The test software 150 includes a test compiler 151 and a comparison compiler 152. The tested compiler 151 is a compiler to be tested. The tested compiler 151 can compile with optimization on / off. The comparison compiler 152 is a compiler that can be used to check the success or failure of compilation by the test compiler 151.

  Here, in the second embodiment, Fortran is assumed as an example of a programming language. The test compiler 151 and the comparison compiler 152 compile the source code described in Fortran. However, the information processing of the second embodiment may be applied to the compilation test of the source code described in another programming language such as C, C ++, and Java (registered trademark).

  The verification unit 160 executes the target code for the test source code that has been successfully compiled, and checks whether the execution result is correct. Specifically, when the compilation is successful, two types of object codes generated by the tested compiler 151 are obtained. The first is a target code generated with optimization turned on. The second is a target code generated with optimization off. The verification unit 160 executes both the target codes, and confirms whether a correct result can be obtained even when the optimization is turned on, depending on whether or not both execution results match. The verification unit 160 may determine whether the target code is correct by collating the execution result of the target code generated when optimization is on and the execution result of the target code generated by the comparison compiler. The verification unit 160 registers the confirmation result in the test result table stored in the storage unit 110.

The verification unit 160 adjusts the priority of each code element included in the combination of code elements used for generating the corresponding source code according to the confirmation result.
FIG. 4 is a diagram illustrating an example of data according to the second embodiment. The storage unit 110 stores a definition file 111, a code element management table 112, a combination narrowing condition list 113, a combination management table 114, an order management table 115, a compile success / failure criteria table 116, a source code 117, and a test result table 118.

  As described above, the definition file 111 is data defining a character string indicating a command, a parameter, and the like for generating a source code. In the following description, a character string indicating a command, a parameter, or the like may be referred to as a code element. The definition file 111 is created in advance by the user in accordance with, for example, a test.

  In the definition file 111, a plurality of variations can be defined for one code element. For example, it is possible to define that either the first command or the second command is inserted into a part of the source code. In addition, it is possible to define such that one of the first parameter and the second parameter is inserted for an argument of a certain command.

  In the definition file 111, a priority is set for each code element. For example, when the first and second commands, which are two code elements, are selectively inserted into the source code, the priority of the first command is made higher than the priority of the second command. Then, the first command is selected with priority over the second command, and the source code is created. That is, the test of the source code including the first command can be performed at an earlier stage than the test of the source code including the second command.

  The code element management table 112 is data for managing the code elements defined in the definition file 111 and the priority for each code element. Each code element is associated with an identification symbol (described later).

  The combination narrowing condition list 113 is data defining conditions for narrowing down combinations of code elements. For example, the combination generation unit 120 performs a process of reducing similar combinations. In this case, a criterion for determining a similar combination is set in advance in the combination narrowing condition list 113.

  The combination management table 114 is data for managing combinations of character strings (code elements) as described above. A combination of code elements is managed by a combination of symbols corresponding to the code elements.

  As described above, the order management table 115 is data for managing the execution order of tests for each combination of code elements. The order management table 115 is updated by the order management unit 130.

  The compilation success / failure criteria table 116 is data defining criteria for determining success / failure of compilation of source code. The success or failure of compiling a certain test source code is determined based on the result of the compiling process by the tested compiler 151 or the comparison compiler 152. A criterion for the determination is set in advance in the compilation success / failure criterion table 116.

  The source code 117 is a test source code (an example of test data) generated by the code generation unit 140 based on the definition file 111. A plurality of variations of the source code 117 can be generated for one definition file 111.

  The test result table 118 is data in which the correctness of the execution result of the target code is registered. The test result table 118 is referred to by the order management unit 130 and is used for processing for changing the priority of the code element.

  The storage unit 110 may store a target code corresponding to the source code 117. Specifically, two types of target codes generated by the tested compiler 151 using two types of optimization on / off methods may be stored. Further, the target code generated by the comparison compiler 152 may be stored.

  FIG. 5 is a diagram illustrating an example of a definition file according to the second embodiment. The definition file 111 is data for creating a Fortran source code. Hereinafter, description will be made using line numbers assigned to the definition file 111 for convenience. The definition file 111 is described based on the following rules 1 to 3.

  (Rule 1) A colon “:” at the beginning of the line (the left end of the line) represents one set of code elements in the source code (sometimes referred to as an element set). For example, when the first colon “:” at the beginning of a line exists in a certain line, the definition file 111 is referred to in order from that line, and one of the lines where the second colon “:” appears at the beginning of the line. Identify the previous line. From the first colon “:” line to the specified line is one element set represented by the first colon “:”. A character string written on the same line after “:” at the beginning of the line indicates the name of the element set. One element set may include one or more code elements (however, a description with “*” at the beginning of a line does not correspond to a code element).

  For example, “: program start” on the first line represents the start of one element set. “Program start” is the name of an element set. The element set includes character strings such as “integer i1, i2, i3” and “integer [* 4, 10 | * 8, 20] v1” described in the fourth and subsequent lines. These character strings are code elements.

  (Rule 2) The attribute for the code element in the element set can be defined by “*” at the beginning of the line. Attributes that can be defined include an element ID (IDentifier) and a priority. The element ID is an identification name of the code element. The priority is a priority for each code element. It is assumed that the priority is higher as the value is higher. Following the “*” at the beginning of the line, the element ID and priority are described in a comma-separated list.

  For example, the description “* loop (, 120” on the 16th line defines the attribute of the code element “do i1 = 0, 10, 1” on the 17th line. Specifically, the element ID “loop ( ", Indicating that the priority" 120 "is assigned to the code element. Here, when the element ID includes the character string" loop ", the element set is a block structure in the source code (more details). Indicates a code element related to a loop.

  Similarly, the description “* loop_end” on the 18th line defines the attribute of the code element “end do” on the 19th line. Specifically, it indicates that the element ID “loop_end)” is assigned to the code element. The element ID “loop_end)” indicates that the code element is related to the block structure associated with the code element indicated by the element ID “loop (”) (more specifically, the end of the loop), where priority is given. The code element on the 19th line is attached to the code element on the 17th line, and it is sufficient to define the priority on the code element on the 17th line. The same applies to the code elements of the eyes (both “end do”).

  In addition, the priority may not be defined for the essential code element. Essential code elements are code elements that must be included in the source code. For example, the description “* main” on the second line defines only the element ID “main” (no priority is defined). The element ID “main” indicates that the code element is related to the definition of the data structure.

  (Rule 3) One block may include one or a plurality of character strings indicating commands. A character string indicating each command can be described in a selectable format. Specifically, the character string to be selected is described inside the brackets “[”, “]”, separated by the vertical bar symbol “|”. For each character string, the priority at the time of selecting the character string can be set in a comma separated manner.

  For example, the code element “integer [* 4, 10 | * 8, 20] v1” on the fifth line includes either “integer * 4 v1” or “integer * 8 v1” in the test source code. Indicates selection and description. At this time, according to the definition of the code element, the priority of “integer * 4 v1” is “10”, and the priority of “integer * 8 v1” is “20”. Therefore, in the setting of the definition file 111, “integer * 8 v1” is preferentially selected and the source code is generated. However, this priority can be corrected according to the test result for each source code.

  In addition, the element ID is not defined in the code element on the fifth line. The element ID of the code element is a character string that directly describes the code element. In this case, “integer * 4 v1” and “integer * 8 v1” are element IDs.

  FIG. 6 is a diagram illustrating an example (continued) of the definition file according to the second embodiment. For example, an element set “exper” (description on the 32nd line) is defined. The element set includes, for example, the code element “[v1 | v2] = [v2 | v3]” on the 34th line. This is a definition for describing any one of “v1 = v2”, “v1 = v3”, “v2 = v2”, and “v2 = v3” in the source code. According to the description “* expr1, 50” on the 33rd line, the element ID “expr1” and the priority “50” are assigned to the code element on the 34th line. When the element ID includes the character string “expr”, it indicates that the element set is a code element related to the use of a mathematical expression or API (Application Programming Interface).

  In the definition file 111, the element ID “print” (line 45) and “end” (line 47) are also exemplified in the element set “end” (line 44). The element ID “print” indicates a code element relating to output. The element ID “end” indicates that the code element indicates the end of the program.

  The combination generation unit 120 combines code elements described in the definition file 111. In order to perform the combination, the following conditions (1) and (2) are also used. (1) Number of selected loops. For example, it is designated in advance such as 5 or less. (2) Number of mathematical formulas / APIs to be inserted in the loop. For example, it is designated in advance such as 5 or less. These conditions can be specified in advance for the combination generation unit 120. Alternatively, it may be defined in the definition file 111.

  FIG. 7 illustrates an example of a code element management table according to the second embodiment. In the code element management table 112, code elements that can be selected from the definition file 111 are registered by the combination generation unit 120. The code element management table 112 includes items of code element, element classification, identification symbol, and priority correction value.

  An element ID is registered in the code element item. The element classification is registered in the element classification item. The element classification is a code element classification derived from the element ID. For example, when “integer” is included in the element ID, the element classification is “data structure”. When the element ID includes “loop”, the element classification is “block structure”. Further, when “expr” is included in the element ID, the element classification is “expression, API”.

  An identification symbol is registered in the item of the identification symbol. An identification symbol is a symbol used for managing combinations of code elements. The identification symbol may be specified in the definition file 111 or may be given by the combination generation unit 120. For example, when a symbol following the space is included at the end of the character string of the element ID (for example, “loop (”), the identification symbol of the code element is the symbol (for example, the parenthesis start symbol “(”)). If the ID does not include such a symbol, the combination generation unit 120 assigns a unique identification symbol to the code element.

In the priority correction value item, a correction value for the priority specified in the definition file 111 is registered. The initial value set in the priority correction value item is “0”.
For example, in the code element management table 112, information that the code element is “integer * 4 v1”, the element classification is “data structure”, the identification symbol is “a”, and the priority correction value is “15” is registered. . This indicates that the element classification of the code element “integer * 4 v1” indicates “data structure”. The identification symbol is “a”, which indicates that the correction value “15” is added to the priority set in the definition file 111.

  Here, in the definition file 111, an identification symbol for the code element “integer * 4 v1” or “integer * 8 v1” is not specified. The combination generation unit 120 assigns an identification symbol to such a code element. The identification symbol “a” is given by the combination generation unit 120. The code elements “integer * 4 v1” and “integer * 8 v1” define the same variable “v1”. Therefore, any one definition may be described in the source code.

  Further, for example, in the code element management table 112, information that the code element is “loop (”, the element classification is “block structure”, the identification symbol is “(”, and the priority correction value is “0” is registered. This indicates that the element classification of the code element “loop (” indicates “block structure”. Also, the identification symbol is “(”, and the priority set in the definition file 111 is a correction value. Indicates that “0” is added (leave the initial value).

  Further, for example, in the code element management table 112, information that the code element is “expr1”, the element classification is “expression, API”, the identification symbol is “1”, and the priority correction value is “−1” is registered. Yes. This indicates that the element classification of the code element “expr1” indicates “expression, API”. Further, the identification symbol is “1”, which indicates that the correction value “−1” is added to the priority set in the definition file 111 (“1” is subtracted). Thus, the priority correction value can take a plus / minus value.

  In the following description, a code element may be indicated by an identification symbol set in the code element management table 112. For example, the code element “a” means a code element “integer * 4 v1” whose identification symbol is “a”.

  FIG. 8 is a diagram illustrating an example of a combination narrowing down condition list according to the second embodiment. Conditions for narrowing down combinations of code elements can be set in the combination narrowing condition list 113. This condition can also be called a condition for narrowing down the input data to be tested.

  For example, the information ““ loop <”is similar to“ loop (”” is registered in the combination narrowing-down condition list 113. This is because the code element “do i3 =” indicated by the element ID “loop <” is registered. “0, 10, 2” is similar to the code element “do i1 = 0, 10, 1” indicated by the element ID “loop (”, and indicates that the combination including the former code element can be omitted. Yes.

  For example, in the combination narrowing-down condition list 113, information that “there is a code element of“ formula and API ”” is registered immediately after the “loop_end” symbol. This is a “loop_end” symbol, that is, a combination in which the code element of the element classification “expression, API” is combined after “)”, “]”, “>”, etc. in the example of the definition file 111 It can be omitted.

  Such a combination can be omitted because it is likely not suitable for the compilation optimization test. For example, when it is desired to test the optimization processing for a loop, there is a case where useful information for investigating a bug cannot be obtained by testing a source code having a mathematical expression outside the loop. Therefore, such a combination can be excluded from the test target. This is to enable efficient testing.

As described above, the conditions for determining the combinations that can be excluded from the test target can be registered in the combination narrowing condition list 113 according to the test contents.
FIG. 9 is a diagram illustrating an example of a combination management table according to the second embodiment. In the combination management table 114, combinations of code elements are managed using identification symbols. The combination management table 114 includes items of combination ID, combination contents, and priority.

  In the combination ID item, identification information for identifying the combination is registered. In the item of combination content, the content of the combination expressed using the identification symbol of the code element is registered. For a code element indicating the start of a loop, a code element indicating the end of the loop accompanying the code element is also included in the combination content. This is to facilitate identification of the loop range. The priority of the combination unit is registered in the priority item.

  For example, information indicating that the combination ID is “C1”, the combination content is “bce (<13> <23> <1>)”, and the priority is “704” is registered in the combination management table 114. In this case, the content of the combination indicated by the combination ID “C1” is the combination “bce (<13> <23> <1>)” of the identification symbol of the code element, and the priority of the combination unit is “704”. It shows that.

In the following description, a combination may be indicated using a combination ID. For example, the combination “C1” means a combination whose combination ID is “C1”.
In one combination, the same code element may appear multiple times. However, code elements whose element classification is “data structure” (for example, “integer * 8 v1”) need only appear once in one combination for one variable (for example, “v1”). It is. Therefore, the combination generation unit 120 causes such a code element to appear only once for one variable (for example, “v1”).

  In addition, the priority (priority in combination unit) is obtained by correcting the priority in code element unit defined in the definition file 111 with the priority correction value set in the code element management table 112, and including each of the combinations included in the combination. The priority after the correction of the code element is added.

  In the example of the combination “C1”, the code element “b” corresponds to “integer * 8 v1”. The code element is given a priority “20” in the definition file 111. Further, according to the code element management table 112, the priority correction value of the code element is “5”. Therefore, the priority after correction in units of code elements is “20 + 5 = 25”.

  Similarly, the corrected priority of the code element “c” is “10 + 15 = 25”. The priority after correction of the code element “e” is “10 + 15 = 25”. The priority after correction of the code element “<” is “80 + 20 = 100”. The corrected priority of the code element “(” is “120 + 0 = 120”. The corrected priority of the code element “1” is “50 + (− 1) = 49”. The priority after correction is “40 + 5 = 45.” The priority after correction of the code element “3” is “30 + 3 = 33”.

  The priority of the combination unit of the combination “C1” is the priority after correction of the code elements “b”, “c”, “e”, “(”, “<”, “1”, “2”, “3”. In other words, the priority of the combination “C1” is calculated as “25 + 25 + 25 + 120 + 100 + 49 + 33 + 100 + 45 + 33 + 100 + 49 = 704”.

  FIG. 10 is a diagram illustrating an example of an order management table according to the second embodiment. In the order management table 115, the test order for each combination is managed using the combination ID. The order management table 115 includes items of order and combination ID. The order of execution of tests is registered in the order item. The order “1” is the first (first). The order “2” is the second. The same applies to the third and subsequent times. The combination ID is registered in the combination ID item.

  For example, information indicating that the order is “1” and the combination ID is “C1” is registered in the order management table 115. This indicates that the combination “C1” is most preferentially tested among the combinations registered in the order management table 115.

  FIG. 11 is a diagram illustrating an example of a compilation success / failure criterion table according to the second embodiment. In the compilation success / failure criterion table 116, a criterion for determining whether or not the test source code is successfully compiled is registered.

  If it is determined that the compilation is successful, the process proceeds to the next phase (execution of the target code). This is called “continuation” of the test. On the other hand, if it is determined that the compilation is unsuccessful, there is no target code, so the next phase (execution of the target code) is not performed on the source code (the process proceeds to the next source code test). This is called “skip” of the test.

  The compilation success / failure criteria table 116 includes items of a tested compiler, a comparison compiler, and a test execution policy. Since the test compiler compiles with optimization on / off, the items of the test compiler are further subdivided into optimization on / off items.

  In the item of the test compiler (optimization on / off), either success / failure of compilation by the test compiler 151 is registered. In the comparison compiler item, either success / failure of compilation by the comparison compiler 152 is registered. In the test execution policy item, a test execution policy for the source code corresponding to the success / failure pattern of compilation of the compiler 151 to be tested and the comparison compiler 152 is registered.

  For example, in the compilation success / failure criteria table 116, if compilation is successful by both the optimization on / off methods in the tested compiler 151, the test for the source code is continued, and at least one of optimization on / off is selected. If compilation fails in this way, it is set to “skip” the test against the source code.

  However, if compilation is successful when the test compiler 151 is turned on and compilation is successful with the comparison compiler 152, the test for the source code can be performed even if compilation is failed because the test compiler 151 is off. It may be set to “continue”. This is because the result when the target code generated with optimization on can be verified using the target code generated with optimization on by the tested compiler 151 and the target code generated with the comparison compiler 152. is there.

  FIG. 12 is a diagram illustrating an example of test source code according to the second embodiment. The source code 117 is test data generated by the code generation unit 140 based on the content of the combination specified by the order management unit 130 and the definition file 111. The source code 117 is an example of source code generated for the combination “C1”. The combination content of the combination “C1” is “bce (<13> <23> <1>)”. Hereinafter, description will be made by using line numbers attached to the source code 117 for convenience.

  “Loop0001.f” described in the first line is the file name of the source code 117. The fourth line is a code element “b”. The fifth line is a code element “c”. The sixth line is a code element “e”. The 13th line is a code element “(”.

  The 14th line is a code element “<”. The fifteenth line is a code element “1”. The 16th line is a code element “3”. The 17th line is a code element “>” accompanying the code element “<” on the 14th line.

  The 18th line is a code element “<”. The 19th line is a code element “2”. The 20th line is a code element “3”. The 21st line is a code element “>” accompanying the code element “<” on the 18th line.

The 22nd line is a code element “<”. The 23rd line is a code element “1”. The 24th line is a code element “>” accompanying the code element “<” on the 22nd line.
The 25th line is the code element “(” accompanying the code element “)” of the 13th line.

  Code elements described in other lines are code elements described as essential code elements in the definition file 111. Further, the priority of code elements is not described in the source code.

  FIG. 13 is a diagram illustrating an example of a test result table according to the second embodiment. In the test result table 118, the test result is registered in association with the file name of the source code. The test result table 118 includes items of a combination ID, a source code file name, and a result.

  The combination ID is registered in the combination ID item. In the source code file name item, the file name of the source code is registered. The content of the test result is registered in the result item. As the test result, for example, the following contents can be registered.

  (1) Successful execution. (2) Failure at the time of compilation. (3) An error occurred during execution of the target code. (4) The target code was executed after successful compilation, but the execution result did not match between optimization on / off (or the target code generated by optimization on and the target code generated by the comparison compiler 152). (In other words, the execution result was invalid). When failure or abnormality occurs, debug information indicating the error location of the target code may be registered.

  For example, information indicating that the combination ID is “C1”, the source code file name is “loop0001.f”, and the result is “execution result mismatch” is registered in the test result table 118. This indicates that the file name of the source code generated for the combination “C1” is “loop0001.f”, the compilation was successful, and the target code was executed, but the execution result was incorrect.

  For example, the software developer can refer to the contents of the test result table 118. The software developer can identify and correct the bug location of the tested compiler 151 based on the contents of the test result table 118. The source code file is stored in a predetermined directory. Software developers can browse the contents of the source code and use it for bug analysis.

FIG. 14 is a flowchart illustrating a test process according to the second embodiment. In the following, the process illustrated in FIG. 14 will be described in order of step number.
(Step S <b> 11) The combination generation unit 120 receives an input indicating that a test related to the optimization process of the tested compiler 151 is started. For example, a software developer can perform the input by operating the input device 12. The software developer stores the definition file 111, the combination narrowing condition list 113, and the compilation success / failure criteria table 116 in the storage unit 110 in advance. The combination generation unit 120 executes a combination process of code elements included in the definition file 111, and generates a code element management table 112 and a combination management table 114 (details will be described later). In this step, however, it is assumed that the combination unit priority is not registered in the combination management table 114.

  (Step S12) The order management unit 130 calculates the priority of the combination unit based on the definition file 111, the code element management table 112, and the combination management table 114, and registers the priority in the combination management table 114. The specific calculation method for the combination unit is as described in FIG. The order management unit 130 creates the order management table 115 based on the combination management table 114.

  (Step S <b> 13) The order management unit 130 extracts one combination of code elements to be executed next based on the order management table 115, and outputs the combination ID to the code generation unit 140.

  (Step S <b> 14) The code generation unit 140 acquires the combination content corresponding to the combination ID acquired from the order management unit 130 from the combination management table 114. The code generation unit 140 generates a test source code based on the content of the combination and inputs it to the tested compiler 151 and the comparison compiler 152. Specifically, the code generation unit 140 selects a code element corresponding to an identification symbol included in the combination content from the definition file 111 and generates a source code. For example, if the combination is “C1”, the source code 117 is generated as described in FIG. Here, in the definition file 111, a plurality of character strings may correspond to one code element. For example, for the code element “1” (element ID “expr1”), one of four character strings “v1 = v2”, “v1 = v3”, “v2 = v2”, “v2 = v3” is selected. Is possible. In this case, the code generation unit 140 may select any one at random. Alternatively, a history selected for each code element may be held in the storage unit 110, and a source code may be generated by selecting a code element different from the previous one.

  (Step S15) The tested compiler 151 compiles the source code by two methods of optimization on / off, and outputs the compilation result to the verification unit 160. The tested compiler 151 stores the generated target code in a predetermined directory of the storage unit 110 if the compilation is successful. The comparison compiler 152 compiles the source code and outputs the compilation result to the verification unit 160. The comparison compiler 152 may store the generated target code in a predetermined directory of the storage unit 110 if the compilation is successful.

  (Step S16) Based on the compilation results of the tested compiler 151 and the comparison compiler 152 and the compilation success / failure criterion table 116, the verification unit 160 determines whether or not to continue the subsequent phase test for the source code. To do. When continuing, a process is advanced to step S17. When not continuing (that is, when skipping), the process proceeds to step S19. In the example of the compilation success / failure criteria table 116, if the tested compiler 151 is successfully compiled with both optimization ON / OFF, “Continue” is set, and if any of the compilations fails, “Skip” is set.

  (Step S <b> 17) The verification unit 160 actually executes the target code generated by the test compiler 151 by the two types of optimization on / off methods, and tests whether the target code is generated correctly. Further, the verification unit 160 increases or decreases the priority correction value of each code element included in the definition file 111 based on the determination result of whether or not the target code is correctly generated. Details will be described later.

  (Step S18) The verification unit 160 registers the result of the test performed by executing the target code in the test result table 118. The contents that can be registered are as illustrated in FIG. Then, the process proceeds to step S20.

(Step S19) The verification unit 160 registers in the test result table 118 that the source code generated in step S14 has failed to compile.
(Step S20) The verification unit 160 adds 1 to the number of test executions. Here, the number of test executions is “0” immediately before step S11. The verification unit 160 deletes the tested combination from the combination management table 114.

  (Step S <b> 21) The verification unit 160 determines whether or not all combinations registered in the combination management table 114 have been tested. If all have been tested, the process ends. If there is an untested combination, the process proceeds to step S22.

  (Step S22) The verification unit 160 changes the test order of each combination. Specifically, since the priority correction value for each code element included in the definition file 111 fluctuates due to the process of step S17, the priority of the combination unit is recalculated based on the priority correction value, and the test order To change. Details will be described later. Then, the process proceeds to step S13.

  In this way, the test apparatus 100 changes the order of the source code to be tested later based on the execution result while executing the test of each source code. Next, a process is demonstrated to step S11.

FIG. 15 is a flowchart illustrating a code element combination process according to the second embodiment. In the following, the process illustrated in FIG. 15 will be described in order of step number.
(Step S31) The combination generation unit 120 extracts code elements from the definition file 111 stored in the storage unit 110 and registers them in the code element management table 112. Here, the initial value of the priority correction value is “0”. In this step, the identification symbol of each code element is in an unregistered state.

  (Step S <b> 32) The combination generation unit 120 assigns an identification symbol to each extracted code element and registers it in the code element management table 112. As described above, if an identification symbol of each code element is specified in the definition file 111, the identification symbol is assigned. On the other hand, an identification symbol for uniquely identifying the code element is assigned to a code element to which no identification symbol is assigned in the definition file 111. The identification symbol is preferably a single character, for example. This is to facilitate the management of combinations of code elements.

  (Step S33) The combination generation unit 120 extracts code elements whose element classification is “block structure” and creates a list of block structure patterns. The number of “block structure” code elements to be extracted is given to the combination generation unit 120 in advance. Alternatively, it may be explicitly specified in the definition file 111 and specified to the combination generation unit 120. For example, if the number is 5 or less, select a code element related to a block structure such as a loop within 5 patterns, and execute a plurality of loops in order (without nesting). A plurality of patterns having a block structure such as a pattern to be executed in step 1 and a pattern in which these two types of patterns are mixed are created. The block structure pattern (a pattern that does not include mathematical formulas) is referred to as a block structure pattern.

  (Step S <b> 34) The combination generation unit 120 adds a code element of an element classification other than the element classification “block structure” to the block structure pattern to create a composite pattern. The composite pattern is a combination of code elements created in consideration of the code elements of all element categories. The term composite pattern is used to distinguish it from a block structure pattern (a combination created considering only code elements of the element classification “block structure”). For example, one code element of the element classification “data structure” is selected for one variable and added to the block structure pattern. For example, the code element of the element section “data structure” is added before entering the first loop. Also, for example, the code element of the element classification “expression, API” is inserted in the loop. The number of insertions in one loop is given to the combination generation unit 120 in advance. Alternatively, it may be explicitly specified in the definition file 111 and specified to the combination generation unit 120. For example, if the number is 5 or less, the code element of “expression, API” to be inserted into the loop is selected within 5 and inserted into each loop included in each block structure pattern. Thereby, a plurality of composite patterns are created.

  (Step S35) The combination generation unit 120 creates a post-mask pattern by masking code elements other than the block structure in the composite pattern. “Mask” means replacing with another character (for example, the character “x”). The mask character may be changed for each element classification (for example, “x” is used for the element classification “expression / API”, and “y” is masked for the element classification “data structure”).

(Step S36) The combination generation unit 120 excludes the same post-mask pattern from the combination to be tested, leaving only one.
(Step S37) The combination generation unit 120 further narrows down the post-mask pattern based on the combination narrowing condition list 113. In the combination narrowing-down condition list 113, for example, “loop <” and “loop (” are similar code elements. In this case, the structure of the pattern is the same, and the identification symbol of the code element used in the loop is “ If there is only a difference between <”or“ (”, and“> ”or“) ”, the masked pattern using the identification symbols“ <”and“> ”is excluded from the test target (specific example) In addition, in the combination narrowing-down condition list 113, for example, when there is a code element of the element classification “expression, API” immediately after the identification symbol of “loop_end”, the post-mask pattern is excluded from the test target. In this case, for example, a post-mask pattern in which the character “x” is present after the identification symbol “)” is excluded from the test target (a specific example will be described later).

  (Step S38) The combination generation unit 120 registers the composite pattern corresponding to the remaining post-mask pattern in the combination management table 114. For a code element of the element classification “data structure”, two types of identification symbols can exist for one mask character (for example, identification symbols “a” and “b” exist for one variable “v1”) To do). Therefore, at least two composite patterns corresponding to the remaining post-mask pattern are registered.

  In this way, the combination generation unit 120 generates a combination of code elements to be tested and registers them in the combination management table 114. Next, a specific example of steps S33 to S37 will be described.

  FIG. 16 is a diagram illustrating a combination example of code elements according to the second embodiment. FIG. 16 illustrates each pattern list processed in steps S33 to S37 described in FIG. Each pattern list includes a plurality of lines, and one line corresponds to one pattern. One pattern may be indicated by a symbol (such as a1) attached to each line for convenience. For example, the block structure pattern a1 indicates the pattern “(<[]>)” in the row “a1” of the block structure pattern list 210.

  First, the combination generation unit 120 creates a block structure pattern list 210 (step S33). The combination generation unit 120 creates a composite pattern list 220 by adding code elements of other element sections to each block structure pattern of the block structure pattern list 210 (step S34).

  The combination generation unit 120 performs mask processing on the composite pattern list 220 to create a post-mask pattern list 230 (step S35). Then, the same post-mask pattern is excluded except for one, and the post-mask pattern list 240 is created (step S36). For example, in the post-mask pattern list 230, the post-mask patterns c2 and c3 are the same. Therefore, for example, the post-mask pattern c3 is deleted while leaving the post-mask pattern c2.

  The combination generation unit 120 further narrows down the post-mask pattern list 240 based on the combination narrowing condition list 113, and creates the post-mask pattern list 250 (step S37). For example, according to the post-mask pattern list 240, the post-mask pattern d4 is “xxx [xx (xx)]”. The post-mask pattern d5 is “xxx [xx <xx>]”. The post-mask patterns d4 and d5 have the same pattern structure (the arrangement of the characters x and the arrangement of symbols indicating the block structure match). The only difference is that the portion described as “(xx)” in the post-mask pattern d4 is described as “<xx>” in the post-mask pattern d5. According to the combination narrowing-down condition list 113, these are handled as being similar. In this case, the post-mask pattern d5 is deleted while leaving the post-mask pattern d4.

  For example, according to the post-mask pattern list 240, the post-mask pattern d1 is “xxx (x <x [x] x> x)”. The post-mask pattern d3 is “xxx (x <xx [x] x>)”. According to the combination narrowing-down condition list 113, if there is a code element of “expression, API” immediately after the “loop_end” symbol, it is excluded from the test target. The “loop_end” symbol includes “)”, “]”, “>”, and the like. Further, according to the processing in step S34, the character “x” in the loop is “expression, API”. Therefore, the post-mask patterns d1 and d3 correspond to the narrowing-down conditions. This is because “] x” (character “x” immediately after the square bracket symbol) and “> x” (character “x” immediately after the angle bracket symbol) are included. Therefore, the combination generation unit 120 deletes the post-mask patterns d1 and d3.

  In this manner, the combination generation unit 120 eliminates duplicate tests and eliminates combinations that are not useful for the optimization test. As a result, it is possible to suppress the unnecessary execution of the test and improve the efficiency of the test. In step S37, deletion of a similar pattern is exemplified, but other methods are also conceivable. For example, when the first pattern and the second pattern are similar, control may be performed so that the test of any one pattern is postponed (for example, last).

Next, the process of step S17 in FIG. 14 will be described.
FIG. 17 is a flowchart illustrating the execution of the object code according to the second embodiment. In the following, the process illustrated in FIG. 17 will be described in order of step number.

  (Step S41) The verification unit 160 executes the target code. The verification unit 160 executes two types of target codes for one source code. One is a target code generated by the tested compiler 151 with optimization turned on. The other is, for example, a target code generated by the tested compiler 151 with optimization off.

  (Step S42) Based on the execution results of the two types of target codes, the verification unit 160 determines whether or not the execution results of the target code generated with optimization on are correct. If it is correct, the process proceeds to step S43. If not, the process proceeds to Step S45. As described above, whether the execution result of the target code is correct or not can be determined based on whether the execution results of the two types of target code match, for example. Correct if they match. If they do not match, it is illegal. An error may also occur during execution of the target code generated with optimization turned on. Also in this case, it is determined that the execution result of the target code is invalid.

  (Step S43) The verification unit 160 determines whether or not the number of times the target code has been normally executed continuously is greater than or equal to a threshold value. That the target code is normally executed continuously means that, for example, “normal execution” is continuously registered in the test result table 118. That is, “continuous” is interrupted when contents other than “normal execution” such as “execution result mismatch” and “compilation failure” are recorded. The threshold value is given to the verification unit 160 in advance, for example, 50 times, 100 times, and the like. For example, assume that the threshold is 50 times. When “normal execution” is recorded 49 times continuously in the test result table 118 immediately before, if the execution result of the current target code is normal execution, the threshold value reaches 50 times. If the number of normal executions in succession is equal to or greater than the threshold, the process proceeds to step S44. If it is smaller than the threshold value, the process is terminated.

  (Step S44) The verification unit 160 lowers the priority of code elements included in a combination that has been normally executed normally. For example, if the threshold value in step S43 is 50 and the number of times of normal execution is 50 consecutively, the priority of each code element included in the combination tested in the last 50 times is lowered. If the threshold value is 50 and the number of times of normal execution is 51 consecutively, the priority of each code element included in the combination tested in the last 51 past times is lowered. For example, 1 is subtracted from the priority correction value for each code element. The value to be subtracted may be changed according to the number of times of normal execution in succession. For example, 1 may be subtracted when 50 to 59 times can be normally executed, and 2 may be subtracted when 60 to 69 times can be normally executed. The verification unit 160 registers the priority correction value after the subtraction of each code element in the priority correction value item of the code element management table 112. Then, the process ends.

  (Step S45) The verification unit 160 instructs the code generation unit 140 to regenerate the source code whose code element has been changed. The code generation unit 140 regenerates the source code according to the instruction from the verification unit 160. For example, a source code in which the variable definition is changed with respect to the original source code is created. Also, for example, a source code in which an addition operation is changed to a subtraction operation is created. A specific method will be described later.

  (Step S46) The code generation unit 140 inputs the regenerated source code to the tested compiler 151. The tested compiler 151 compiles the source code using both methods of optimization on / off, and generates two target codes. The verification unit 160 executes the two target codes and verifies whether the result is correct. Changing the code element may identify the code element that caused the invalid result (error cause). For example, if an error occurs due to a certain code element, a correct result may be obtained by replacing the code element with another code element that performs similar processing. In that case, it can be inferred that the code element before the change is likely to be the cause of the error. Therefore, in that case, the code element before the change is identified as the cause of the error.

  (Step S47) The verification unit 160 determines whether or not the code element causing the error has been identified by the process of step S46. If it can be specified, the process proceeds to step S48. If not specified, the process proceeds to step S49.

  (Step S48) The verification unit 160 excludes a combination including a code element causing the error from the test target. Specifically, the verification unit 160 deletes the combination from the combination management table 114. For example, when the code element “4” is identified as the cause of the error, all combinations including the code element “4” are deleted. Then, the process ends.

  (Step S49) The verification unit 160 increases the priority of each code element included in the combination in error. For example, 1 is added to the priority correction value for each code element. However, values other than 1 may be added. The verification unit 160 registers the priority correction value after the addition of each code element in the priority correction value item of the code element management table 112. Then, the process ends.

  In this way, the verification unit 160 updates the priority correction value of each code element according to the execution result while executing the target code. That is, the verification unit 160 adjusts the priority of each code element. The priority correction value of each code element may be changeable at an arbitrary timing by a user (for example, a software developer).

  In step S48, the reason for excluding the combination including the code element causing the error from the test target is as follows. If one code element is identified as the cause of the error, there is a high possibility that the execution result of the corresponding target code is incorrect in the combination including the one code element. In this case, the software developer may concentrate on reviewing the optimization process related to the code element. Therefore, in this case, a combination including the code element is excluded from the test target so that a useless test is not performed. As a result, the test can be made more efficient.

  In step S48, when the number of times that the same code element causes an error reaches a predetermined number (for example, twice), the combination including the code element may be excluded from the test target. For example, when it is first identified that a certain code element is the cause of the error, the combination including the code element is not excluded in step S48. Instead, the verification unit 160 stores the number of times “1” that the code element caused an error. Thereafter, when the code element is next identified as the cause of error in testing another target code, the combination including the code element is excluded from the test target. This is because, when the same code element is identified as the cause of the error multiple times, it is highly likely that the execution result of the target code corresponding to the combination including the code element is incorrect.

  In step S48, instead of excluding the combination including the code element from the test target, the priority of the code element identified as the cause of the error is significantly reduced, or the test of the combination is postponed (for example, the last combination) ). For example, when there are a plurality of combinations that have been postponed (turned last), the priorities of the combinations that have been postponed are compared, and the order is further determined. Also, lowering the priority of the code element can be said to be control for postponing the test of the combination of the code elements.

  FIG. 18 is a diagram illustrating a modification example of the code element according to the second embodiment. In step S45 of FIG. 17, the code generation unit 140 changes the code element of the source code 117 as follows, for example. FIG. 18A illustrates a case where the definition of a variable is changed. FIG. 18B illustrates a case where subtraction is changed to addition.

  For example, the code generation unit 140 changes “integer * 8 v1” on the fourth line to “integer * 4 v1”. Alternatively, “v1 = v1−v3” on the 16th line is changed to “v1 = v1 + v3”. For each code element of the source code 117, the code elements are searched one by one in order and changed one by one. When one code element is changed, the compile execution and target code execution procedures are repeated. For example, changes such as changing a double precision variable to single precision (or vice versa), changing addition to subtraction (or vice versa), changing multiplication to division (or vice versa), and the like are possible.

Next, the process of step S22 of FIG. 14 will be described.
FIG. 19 is a flowchart illustrating a test order change according to the second embodiment. In the following, the process illustrated in FIG. 19 will be described in order of step number.

  (Step S51) The order management unit 130 determines whether or not the number of test executions has reached a predetermined value. If the number of test executions reaches a predetermined value, the process proceeds to step S52. If the number of test executions is smaller than the predetermined value, the process is terminated. As the predetermined value, for example, a value indicating one or more times such as 10 times, 50 times, and 100 times is given to the order management unit 130 in advance.

  (Step S52) The order management unit 130 recalculates the priority of the combination unit based on the definition file 111, the code element management table 112, and the combination management table 114. The order management unit 130 registers the recalculated priority in the combination management table 114. Further, the order management unit 130 recreates the order management table 115 based on the combination management table 114.

(Step S53) The order management unit 130 resets the number of test executions to “0”.
In this manner, when the number of test executions reaches a predetermined value, the order management unit 130 recalculates the priority for each combination. Then, the order management table 115 is recreated based on the priority. This updates the order of testing code element combinations. As the “predetermined value” in step S51, a value corresponding to the test can be set. For example, when there are many types of code elements, changing the test order every time can increase the processing cost for changing the test order. Therefore, if the test order is set to be changed every time the test is performed a plurality of times, an increase in processing cost can be suppressed.

  Here, in step S52, for a certain combination of code elements, the combination unit priority may be lower than a predetermined minimum value. In that case, it may be controlled to exclude the combination of the code elements from the test target.

  FIG. 20 is a diagram illustrating a test order change example according to the second embodiment. FIG. 20 illustrates source codes 117k, 117l, 117p, 117q, and 117r. Source codes 117k, 117l, 117p, 117q, and 117r are variations of the source code 117 generated by the code generation unit 140 based on the definition file 111.

  Assume that the test sequence at the beginning of the test is as follows. Source code 117k will be tested Kth. Source code 117l will be tested Lth (L> K). Source code 117p will be tested Pth (P> L). Source code 117q will be tested Qth (Q> P). Source code 117r will be tested Rth (R> Q).

  For example, the first cycle includes the test for the predetermined number of times in step S51 of FIG. The Kth and Lth tests are included in the first cycle. The second cycle includes a test group that is executed after the first cycle. The Pth, Qth, and Rth tests are included in the second cycle.

  It is assumed that the execution result of the target code is incorrect in the Kth test in the first cycle. In addition, even if the code element is changed and the test is performed again, the cause of the error cannot be identified. Then, 1 is added to the priority correction value for the code element included in the combination corresponding to the target code. Similarly, it is assumed that the execution result of the target code is incorrect in the Lth test. In addition, even if the code element is changed and the test is performed again, the cause of the error cannot be identified. Then, 1 is added to the priority correction value for the code element included in the combination corresponding to the target code.

  Thus, whenever the execution result of the target code is invalid, the priority of the code element included in the combination corresponding to the target code increases, and the priority of the test of the combination including the code element increases.

  When the first cycle ends, the order management unit 130 updates the test order. For example, it is assumed that the source codes 117q and 117r include a relatively large number of code elements included in the source codes 117k and 117l in which an error has occurred in the first cycle. It is assumed that the priority of the combination corresponding to the source code 117q, 117r is higher than the priority of the combination corresponding to the source code 117p. Then, the source codes 117q and 117r are executed by moving up the order before the source code 117p. In this way, the test apparatus 100 controls the test execution order.

  By testing the source code that is likely to be invalid at an early stage, it is possible to obtain an input sample that causes an error and the contents of each error at an early stage. Therefore, the software developer can deal with bugs in the tested compiler 151 at an early stage. In this way, the test apparatus 100 can execute the test efficiently. In addition, software debugging work can be efficiently supported.

[Third Embodiment]
Hereinafter, a third embodiment will be described. Differences from the second embodiment will be mainly described, and description of similar matters will be omitted.

  In the second embodiment, the compiler is exemplified as the test target software. In the third embodiment, a DBMS is exemplified as other software. Specifically, it is assumed that a SQL execution test is performed in the DBMS. Here, the hardware example of the test apparatus according to the third embodiment is the same as that of the test apparatus 100 described with reference to FIG. The test apparatus according to the third embodiment is indicated by using the same name / symbol as the test apparatus 100 unless otherwise specified.

  FIG. 21 is a diagram illustrating an example of software according to the third embodiment. Each unit illustrated in FIG. 21 can be realized, for example, by a processor included in the test apparatus 100a executing a program stored in the RAM. Each unit in FIG. 21 may be realized by an ASIC, FPGA, or the like.

  The test apparatus 100 includes a storage unit 110, a combination generation unit 120, an order management unit 130, a code generation unit 140, test software 150, and a verification unit 160. The description of the storage unit 110, the combination generation unit 120, the order management unit 130, the code generation unit 140, the test software 150, and the verification unit 160 is the same as that in the second embodiment.

  However, the combination generation unit 120 and the order management unit 130 are different from the second embodiment in that a combination of a command and a parameter for generating a test SQL is processed instead of the test source code. The code generation unit 140 is different from the second embodiment in that it generates a test SQL instead of the test source code.

  The test software 150 is different from the second embodiment in that the test DBMS 151a is included as test target software. The test software 150 is different from the second embodiment in that the test software 150 includes a comparison DBMS 152a as software for confirming whether SQL execution by the test DBMS 151a is correct.

  Furthermore, the verification unit 160 is different from the second embodiment in that the verification unit 160 verifies whether the SQL execution by the test DBMS 151a is correct based on the SQL execution result by the test DBMS 151a and the SQL execution result by the comparison DBMS 152a. For example, if the SQL execution result by the test DBMS 151a matches the SQL execution result by the comparison DBMS 152a, the test DBMS 151a has correctly executed SQL. On the other hand, if the SQL execution result by the test DBMS 151a does not match the SQL execution result by the comparison DBMS 152a, the test DBMS 151a cannot execute the SQL correctly.

  FIG. 22 is a diagram illustrating an example of data according to the third embodiment. The storage unit 110 stores a definition file 111a, a code element management table 112a, a combination narrowing condition list 113, a combination management table 114, an order management table 115, a test result table 118, and a test SQL 119.

  Here, the description of the combination narrowing-down condition list 113, the combination management table 114, the order management table 115, and the test result table 118 is the same as the narrowing-down condition list 113, the combination management table 114, the order management table and the second embodiment. This is the same as the test result table 118. However, in the combination narrowing-down condition list 113, narrowing-down conditions taking into account the SQL execution test are registered in advance. In the test result table 118, test results such as SQL execution correctness are registered by the verification unit 160.

  The definition file 111a is data defining a character string indicating a command, a parameter, or the like for generating an SQL. In the following description, a character string indicating a command or parameter for generating SQL may be referred to as a code element. This is because SQL can also be called a code describing an instruction to be executed by the DBMS. The contents that can be defined for each code element are the same as those in the definition file 111 of the second embodiment. For example, the definition file 111a can also define the priority for each code element.

  The code element management table 112a is data for managing the code elements defined in the definition file 111a and the priority for each code element. Each code element is associated with an identification symbol, as in the second embodiment.

  The test SQL 119 is a test SQL (an example of test data) generated by the code generation unit 140. In the test SQL 119, a plurality of variations can be generated for one definition file 111a.

  FIG. 23 is a diagram illustrating an example of a definition file according to the third embodiment. Hereinafter, description will be made using line numbers assigned to the definition file 111a for convenience. The description rule of the definition file 111a is the same as the description rule of the definition file 111a described in FIG. However, the definition file 111a has no concept of element set. For this reason, there is no line in which “:” is described at the beginning of the line. What is specified in the definition file 111a is a command and an argument used to execute the command.

  For example, the description “* command1, 100” on the first line defines the attribute of the code element “select [row1] from [table1] [where1 | where2 |...]” On the second line. Specifically, the element ID “command1” and the priority “100” are assigned. Among the code elements, “select... From...” Indicates one syntax of the select command in SQL.

  For other parts surrounded by “[” and “]” (such as “[row1]”), select a column name, table name, conditional statement, subquery statement, etc. used in the select command from multiple options. Indicates that it can be inserted. For example, in the case of “[row1]”, the code element “[ID | NAME |...]” Corresponding to the element ID “row1” is described in the 16th line (the element ID of “row1” is 15). Is given in line). This indicates that any one of “ID”, “NAME”,... Can be selected and inserted as “[row1]”.

  Descriptions such as “[table1]” and “[where1 | where2 |...]” On the second line have the same meaning. However, the description “[where1 | where2 |...]” Means that “[where1]”, “[where2]”,... Is selected. For “[where1]”, a code element having an element ID “where1” is defined on the 20th line. There may be a plurality of options for “where1”. In this case, when “[where1]” is selected in the code element “command1”, any one of a plurality of options of “where1” is further selected.

  For other SQL commands such as the insert command and the update command, variations of code elements given as arguments can be defined in the same manner as the select command. A priority can be defined for each code element.

  FIG. 24 is a diagram illustrating an example of a code element management table according to the third embodiment. The code element management table 112a includes items of code element, element classification, identification symbol, and priority correction value. Here, the description of the items of the code element, the element classification, the identification symbol, and the priority correction value is the same as the description of each item of the code element management table 112 of the second embodiment.

  For example, information indicating that the code element is “command1”, the element classification is “command type”, the identification symbol is “!”, And the priority correction value is “5” is registered in the code element management table 112a. This is because the element classification of the element ID “command1” is “command type”, the identification symbol used for combination management is “!”, And the priority correction value set in the definition file 111a is “5”. Show.

  As described above, the code elements for generating the SQL can be managed in the same manner as in the definition file 111. With the above configuration, the same procedure as that of the second embodiment described with reference to FIGS. However, instead of generating source code, SQL is generated. Further, the steps corresponding to the confirmation of the success or failure of compilation (for example, steps S15, S16, S19 in FIG. 14 or the compiling process in step S46 in FIG. 17) need not be performed. Further, instead of executing the target code, the test DBMS 151a and the comparison DBMS 152a are caused to execute SQL. Depending on the execution result, the priority correction value of the code element included in the combination corresponding to the tested SQL is adjusted.

  FIG. 25 is a diagram illustrating a test order change example according to the third embodiment. In FIG. 25, test SQLs 119k, 119l, 119m, 119n, 119p, 119q, and 119r are illustrated. Test SQLs 119k, 119l, 119m, 119n, 119p, 119q, and 119r are variations of the test SQL 119 generated by the code generation unit 140 based on the definition file 111a.

  Assume that the test sequence at the beginning of the test is as follows. The test SQL 119k will be tested Kth. The test SQL 119l will be tested Lth (L> K). The test SQL 119m is scheduled to be tested Mth (M> L). The test SQL 119n will be tested Nth (N> M). The test SQL 119p is scheduled to be tested Pth (P> N). The test SQL 119q is scheduled to be tested Qth (Q> P). The test SQL 119r will be tested Rth (R> Q).

  For example, the first cycle includes a predetermined number of tests. The Kth, Lth, Mth and Nth tests are included in the first cycle. The second cycle includes a test group that is executed after the first cycle. The Pth, Qth, and Rth tests are included in the second cycle.

  Assume that the SQL execution result is incorrect in the K, L, M, and Nth tests in the first cycle. Further, in any test, it is assumed that the cause of the error cannot be identified even if the code element is changed and the test is performed again. Then, 1 is added to the priority correction value for each code element included in the combination corresponding to the SQL of each test.

  Thus, whenever the SQL execution result is invalid, the priority of the code element included in the combination corresponding to the SQL increases, and the priority of the test of the combination including the code element increases.

  When the first cycle ends, the order management unit 130 updates the test order. For example, it is assumed that the test SQLs 119q and 119r include a relatively large number of code elements included in the test SQLs 119k, 119l, 119m, and 119n in which an error has occurred in the first cycle. Assume that the priority of the combination corresponding to the test SQL 119q, 119r is higher than the priority of the combination corresponding to the test SQL 119p. Then, the test SQLs 119q and 119r are executed in order before the test SQL 119p. In this way, the test apparatus 100 controls the test execution order.

  By testing the SQL that is likely to have an incorrect result at an early stage, it is possible to acquire a large number of input samples and the contents of each error in the early stage. The software developer can deal with bugs in the DBMS 151a to be tested at an early stage. In this way, the test apparatus 100 can execute the test efficiently. In addition, software debugging work can be efficiently supported.

  As described above, the information processing of the first embodiment can be realized by causing the computing unit 1b to execute a program. The information processing according to the second embodiment can be realized by causing the processor 101 to execute a program. The program can be recorded on a computer-readable recording medium (for example, the optical disc 13, the memory device 14, and the memory card 16).

  When distributing the program, for example, a portable recording medium in which the program is recorded is provided. It is also possible to store the program in a storage device of another computer and distribute the program via a network. The computer stores, for example, a program recorded on a portable recording medium or a program received from another computer in a storage device, and reads and executes the program from the storage device. However, a program read from a portable recording medium may be directly executed, or a program received from another computer via a network may be directly executed.

  In addition, at least a part of the information processing described above can be realized by an electronic circuit such as a DSP, ASIC, or PLD.

DESCRIPTION OF SYMBOLS 1 Information processing apparatus 1a Memory | storage part 1b Operation part A, B, C, D, E Test data CS1, CS2, CS3, CS4, CS5, CS6 Character string

Claims (7)

On the computer,
While executing the execution test of the process according to the test data including multiple character strings sequentially for each of the multiple test data, the priority for each character string included in the test data is adjusted based on the result of the execution test. ,
A program for executing a process for changing an execution order of execution tests for test data for which execution tests have not been executed among the plurality of test data based on a priority for each character string included in the test data. If the result of the execution test of the process according to the test data is invalid, adjust the priority for each character string so that the priority for each character string included in the test data increases,
The program according to claim 1, wherein the priority of each test data unit is obtained based on the priority for each character string included in the test data, and the execution order is changed based on the priority for each test data. If the result of the execution test of the process according to the test data is correct, adjust the priority for each character string so that the priority for each character string included in the test data decreases,
The program according to claim 1 or 2, wherein a priority of a test data unit is obtained based on a priority for each character string included in test data, and the execution order is changed based on a priority for each test data.   When the character string whose result is invalid can be identified, the test data including the character string is excluded from the target of the execution test among the test data in which the execution test is not executed, or the test data including the character string The program according to any one of claims 1 to 3, wherein control is performed so that an execution order of execution tests is postponed.   5. The priority of each character string included in the tested test data is adjusted each time an execution test is performed on each of the plurality of test data, and the execution order is changed every time the execution test is performed a predetermined number of times. The program according to any one of the above. A storage unit for storing a plurality of character strings;
While sequentially executing the execution test of the process according to the test data including the character string stored in the storage unit for each of the plurality of test data, for each character string included in the test data based on the result of the execution test An arithmetic unit that adjusts the priority and changes the execution order of the execution test with respect to the test data in which the execution test is not executed among the plurality of test data based on the priority for each character string included in the test data; Information processing apparatus. Computer
While executing the execution test of the process according to the test data including multiple character strings sequentially for each of the multiple test data, the priority for each character string included in the test data is adjusted based on the result of the execution test. ,
A test method for changing an execution order of execution tests with respect to test data for which execution tests have not been executed among the plurality of test data based on a priority for each character string included in the test data.

Images