JP7792325B2 - Test system and test method - Google Patents

Description

The present invention relates to a test tool.

To improve the efficiency of software development, standardization is being promoted in various fields and standards are being established. As a result, the impact of standardization is not limited to the software itself; it also applies to software testing, with testing based on standard items being required.

For testing targeting AUTOSAR (AUTomotive Open System ARchitecture), AUTOSAR defines and provides acceptance test (AT) specifications for each of AUTOSAR's functions. However, since the tests targeted in acceptance tests are limited to AUTOSAR's basic functions, it is necessary to create test scenarios for the configuration to be implemented in the product.

In addition, with AUTOSAR, when configuration parameters are changed, all source code is regenerated, and although the amount of source code generated is enormous, all patterns must be verified when testing to ensure quality.

The following prior art exists as background technology in this technical field. Patent Document 1 (JP 2000-000000) describes a test tool that includes configuration parameters that set the environment of an abstraction layer that abstracts hardware, API parameters, a combination extraction means that comprehensively extracts combinations of configuration parameters, and a comprehensive combination of API parameters, a test case generation means that generates test cases based on the processes included in the processing sequence, an interface model that models the interface of the abstraction layer, and a test execution means that applies combinations of configuration parameters, combinations of API parameters, or test cases to the abstraction layer to test the abstraction layer.

JP 2011-221803 A

The test tool described in Patent Document 1 uses the all-pairs method for testing AUTOSAR, using configuration parameters as factors in the method, selecting two factors from four target factors, and constructing a system that ensures 100% combinations of tests for the two selected factors. However, as mentioned above, this method cannot verify all patterns, and there are concerns about reduced test coverage and missing test scenarios due to bias in the selected configuration parameter/API parameter combinations.

Therefore, there is a need to build an automated testing environment that has the ability to convert and standardize non-standard configuration parameters based on additional information, and the ability to automatically generate test scenarios based on the additional information.

A representative example of the invention disclosed in this application is as follows: a test system for testing software in compliance with a software architecture based on a predetermined standard, the test system comprising a computer having an arithmetic unit that executes predetermined processing and a storage device accessible by the arithmetic unit, the arithmetic unit executing the following steps: a test target selection step that sorts software to be tested into a first test target that complies with the software architecture and a second test target that does not comply with the software architecture and generates accompanying information indicating the second test target; a standard conversion step that converts configuration parameters included in the second test target that do not comply with the standard into a format that complies with the standard; a first test step that creates a first test scenario from a pre-registered first base test scenario and tests the first test target; and a second test step that creates a second test scenario from a pre-registered second base test scenario and tests the second test target.

According to one aspect of the present invention, it is possible to create an environment in which tests can be performed on test subjects for which tests are not provided in standards such as AUTOSAR by selecting test subjects and taking individual measures. Issues, configurations, and effects other than those mentioned above will become clear from the description of the following examples.

FIG. 1 is an overall configuration diagram of a test tool according to a first embodiment. FIG. 10 is a diagram illustrating the configuration of a list according to the first embodiment. FIG. 10 is a diagram showing configuration parameters recorded in a list according to the first embodiment. FIG. 10 is a diagram showing configuration parameters recorded in a list according to the first embodiment. FIG. 10 is a diagram showing configuration parameters recorded in a list according to the first embodiment. FIG. 10 is a diagram showing configuration parameters recorded in a list according to the first embodiment. FIG. 10 is a diagram showing configuration parameters recorded in a list according to the first embodiment. FIG. 10 is a diagram showing configuration parameters recorded in a list according to the first embodiment. FIG. 10 is a diagram showing configuration parameters recorded in a list according to the first embodiment. FIG. 1 is a diagram showing a test subject selection step in Example 1. FIG. 1 is a diagram illustrating an overall configuration of test scenario creation steps according to a first embodiment. FIG. 10 is a diagram illustrating the contents of a data table for creating a test scenario according to the first embodiment. FIG. 10 is a diagram illustrating the contents of a test definition file according to the first embodiment. FIG. 10 is a diagram showing a test method and information on whether or not the test can be performed in the first embodiment. FIG. 4 is a diagram illustrating CAN definition information according to the first embodiment. FIG. 1 is a diagram showing test results of Example 1.

Example 1
Hereinafter, the embodiments will be described in detail with reference to the drawings.

Figure 1 is a diagram showing the overall configuration of a test tool according to the first embodiment of the present invention.

The test tool of Example 1 will be explained below, divided into the test target selection step 104, the standard conversion step 107, the test scenario creation step 113, and the test step 117.

In the test target selection step 104, the test tool in this embodiment targets the AUTOSAR DCM (Diagnostic Communication Manager), and testing is required for each configuration parameter, with all pattern verification being mandatory. As a result, tests are required for each configuration parameter, resulting in a huge number of test scenarios. Therefore, the test targets are selected as either configuration parameters that determine the behavior of the entire function or configuration parameters specific to the requirement, making the test targets easier to handle.

Next, referring to Figure 2, we will explain list 101, which contains the configuration parameters used by the test tool in this embodiment.

Figure 2 shows the configuration of list 101. List 101 includes Parameters 202, session-service 203, session-localId 204, DID_List(OBD) 205, DID_List(OBD) 206, RID_List 207, and TID_List 208. Each list records the configuration parameters used by the test tool of this embodiment, and contains the information shown in Figures 3A to 3G.

Figure 4 shows the test target selection step 104.

In the test target selection step 104, the list 101 is separated into a first test target 102 and a second test target 103. For each selected test target, the first test target 102 is designated as the common part, and the second test target 103 is designated as the specific part. The first test target 102, which is the common part, contains basic configuration parameters required for operation as an AUTOSAR DCM, such as the basic communication parts and constant sizes used by AUTOSAR. On the other hand, the second test target 103, which is the specific part, contains detailed configuration parameters associated with AUTOSAR's basic configuration parameters and configuration parameters that change depending on product-specific specifications.

Next, we will explain the type determination step 201, which separates common and specific parts in the test object selection step 104. The "Specification" included in Parameters 202 shown in Figure 3A is used as additional information, and if any of session-service 203, session-localId 204, DID_List(UDS) 205, DID_List(OBD) 206, RID_List 207, and TID_List 208 is set in this item, it is determined to be a specific part. Then, additional information indicating that it is a specific function is entered in Parameters 202 of Figure 2, which is included in the second test object 103. As a result of this separation, as shown in Figure 4, list 101 is separated into the first test object 102, which is a common part, and the second test object 103, which is a specific part. Because the number of tests performed and the tests to be implemented change dynamically depending on the product's specific specifications, the specific part is separated as a specific part to create an easier-to-manage structure.

In this embodiment, as a measure for automatically executing the test tool, a list compliant with AUTOSAR is created in standard conversion step 107. Standard conversion step 107 identifies the unique configuration parameters for each test platform included in the aforementioned list 101 and creates a mechanism for converting them into configuration parameters compliant with AUTOSAR.

The overall configuration of the standard conversion step 107 will be explained with reference to Figure 1. For the first test target 102 and second test target 103 selected in the test target selection step 104, a standard conversion tool 105 is used to combine multiple lists to create a single configuration list 106 that complies with AUTOSAR. For example, the standard conversion tool 105 has a correspondence table between unique configuration parameters and common configuration parameters, and converts the unique configuration parameters into common configuration parameters according to the designer's operations.

In the standard conversion step 107, the conversion targets are configuration parameters for utilizing the unique specifications of each test platform. If these configuration parameters were reflected in the list as is, it would no longer be a standard list for AUTOSAR.

Regarding the configuration parameters described above, the designer looks at the additional information listed in Parameters 202 in Figure 2, considers the setting values of each of the configuration parameters shown in Figure 2 for the configuration parameters of the specific functions, and converts them to the values considered when creating the configuration list 106.

Also, as mentioned above, if a unique specification is enabled, it will no longer be a standardized list, but since it may be used, it is possible to enable the setting itself. On the other hand, for configuration parameters that do not need to be converted, both the common and unique parts are transcribed into the configuration list 106.

Next, in test scenario creation step 113, test scenarios are created for each of the first test target 102 and the second test target 103 using the additional information in the configuration list 106. For example, it is advisable to create a base test scenario in advance and then create test scenarios based on the contents registered in the configuration list 106.

In test step 117, expected values are pre-built into each base test scenario to automatically judge the results obtained by the tool. The expected values of the base test scenarios should be created by the designer based on the ISO (International Organization for Standardization) or AUTOSAR standard 301. The expected values to be used are automatically set according to the configuration parameters registered in the configuration list 106.

Figure 5 shows the overall configuration of the test scenario creation step 113. Based on the configuration list 106, a first base test scenario 109 created from a pre-prepared AUTOSAR standard 301 is used to create a first test scenario 111 for the first test target 102 (common part), and a second base test scenario 110 created from the test scenario creation data table 302 is used to create a second test scenario 112 for the second test target 103 (specific part). The respective test scenario creation methods are explained below.

The first test scenario 111 is a test scenario that targets the configuration parameters required for AUTOSAR behavior in the test target selection step, and is created based on the first base test scenario 109 created from the AUTOSAR standard 301. Furthermore, the test scenario creation tool 108 selects the necessary test scenarios and expected values from the first base test scenario 109 according to the configuration parameters registered in the configuration list 106, and creates the first test scenario 111.

The second test scenario 112 is created using the second base test scenario 110 based on the information in the configuration list 106. The second base test scenario 110 prepares a test scenario creation data table 302 for each of the lists included in the configuration list 106: session-service 203, DID_List(UDS) 205, DID_List(OBD) 206, RID_List 207, and TID_List 208.

Here, we will explain the test scenario creation data table 302. Figure 6 shows the contents of the test scenario creation data table 302 for session-service 203, DID_List(UDS) 205, DID_List(OBD) 206, RID_List 207, and TID_List 208. As shown in Figure 6, each test scenario creation data table 302 has a structure that can be defined as a structure based on the number of valid IDs, data size, and whether it exists in the configuration list 106. Therefore, configuration parameters with invalid values entered in the configuration list 106 are determined to be unnecessary test scenarios. Furthermore, since session-service 203 defines the test area to be implemented, IDs that will not be implemented are excluded from the test scenario.

Furthermore, in test step 117, it is necessary to define the ID and addressing method for the CAN (Controller Area Network) communication used by the test tool, so a test definition file 303 as shown in FIG. 7 is created. The test definition file 303 shown in FIG. 7 includes test information 401, an ID list 402, and a CAN definition list 403. Test information 401 is created based on the test method and test implementation information shown in FIG. 8, and the test scenario name is registered in test information 401. The test scenario registered in test information 401 is executed in the subsequent test step 117. The CAN ID list 402 and CAN definition list 403 are CAN definition information used for communication with the tool in test step 117. This definition information defines the CAN ID and addressing method for each test scenario based on the CAN definition information shown in FIG. 9. This information is also incorporated into the test scenario creation data table 302, and the test definition file 303 is created.

Next, the related information extraction step will be described. The related information extraction step is performed by referencing the configuration list 106. The configuration list 106 is written in a way that makes the characteristics of each listed configuration parameter clear, so when a designer modifies or checks a test scenario, they can easily select the configuration parameters for the relevant test scenario. Furthermore, because the configuration list 106 is a list organized by AUTOSAR configuration parameter, the designer can easily check test scenario-related information.

In this embodiment, in test step 117, the test scenario created in test scenario creation step 113 is executed. The test is executed based on the test definition file 303 and in accordance with the contents of the file. The test results 116 obtained by executing the test are compared with the expected values defined in the base test scenario, and the test judgment results are notified for each test scenario.

The overall configuration of test step 117 will be explained with reference to Figure 5. In test step 117, the first test scenario 111 created in test scenario creation step 113 is tested in first test step 114 based on test definition file 303, and the expected values described in the first base test scenario 109 are compared with test results 116 to make a test judgment. Furthermore, the second test scenario 112 created in test scenario creation step 113 is tested in second test step 115 based on test definition file 303, and the expected values described in the second base test scenario 110 are compared with test results 116 to make a test judgment.

Figure 10 shows test results 116, a file that records the results of the test. The values below the expected values in the figure are the expected values obtained from the base test scenario, and the actual measured values are the test results. The results are determined by comparing the CAN data and using a tool to determine whether the sent and received data match, and the result is recorded in the determination section in the table as normal or invalid.

In addition, since all test scenarios are associated with the configuration list 106, the correspondence is clear regarding tests of configuration parameters that resulted in invalid test judgment results, and the information in the configuration list 106 can be used to analyze the invalid setting values and their causes.

Furthermore, in test step 117, it is preferable to extract test objects whose test results indicate an abnormality based on the test results of the first test step and the test results of the second test step. In this way, test objects whose test results indicate an abnormality can be extracted, improving the efficiency of designers' work to address defects.

Next, we will explain the computer on which the test tool of this embodiment runs. The test tool of this embodiment is a program executed on a computer that has a processor (CPU), memory, auxiliary storage device, and a communication interface. The computer on which the test tool runs constitutes a test system.

The processor is a computing device that executes programs stored in memory. The processor executes various programs to realize the various functions of the test tool. Note that some of the processing performed by the processor when executing programs may be executed by another computing device (e.g., hardware such as an ASIC or FPGA).

Memory includes ROM, a non-volatile storage element, and RAM, a volatile storage element. ROM stores unchanging programs (e.g., BIOS). RAM is a high-speed, volatile storage element such as DRAM (Dynamic Random Access Memory), and temporarily stores programs executed by Processor 1 and data used when executing programs.

The auxiliary storage device is a large-capacity, non-volatile storage device such as a magnetic storage device (HDD) or flash memory (SSD). The auxiliary storage device also stores data used by the processor when executing programs, as well as the programs executed by the processor. In other words, programs are read from the auxiliary storage device, loaded into memory, and executed by the processor to realize the various functions of the test tool.

A communication interface is a network interface device that controls communication with other devices according to a specific protocol. A computer may have an input interface and an output interface.

The program executed by the processor is provided to the computer via removable media (CD-ROM, flash memory, etc.) or a network, and is stored in a non-volatile auxiliary storage device, which is a non-temporary storage medium. For this reason, the computer should have an interface for reading data from removable media.

The test tool executed by the computer in this embodiment is a computer system configured on a single physical computer, or on multiple logically or physically configured computers, and may operate on a virtual computer constructed on multiple physical computer resources. For example, each functional unit may operate on a separate physical or logical computer, or multiple functional units may be combined to operate on a single physical or logical computer.

As explained above, the test tool of this embodiment proposes a method and environment for efficiently creating test scenarios to verify all patterns of a huge number of test scenarios targeting AUTOSAR. The computing device selects the software to be tested into a first test target 102 that conforms to a software architecture (for example, the standard AUTOSAR) and is a test target targeting the basic configuration parameters 203-208 of the software architecture, and a second test target 103 that does not conform to the software architecture and is a test target targeting detailed configuration parameters associated with the basic configuration parameters 203-208, and generates additional information indicating that it is the second test target in a test target selection step 104. The computing device also selects the software to be tested from the second test target 103. The system includes a standard conversion step 107 that converts non-standard configuration parameters into a format that complies with the standard, a first test step 114 in which the computing device creates a first test scenario 111 from a pre-registered first base test scenario 109 and tests a first test target 102, and a second test step 115 in which the computing device creates a second test scenario 112 from a pre-registered second base test scenario 110 and tests a second test target 103. This allows the test targets in the list 101 to be selected based on whether they comply with the software architecture, and further converts test targets that do not comply with the software architecture into a format that complies with the software architecture, thereby expanding the scope of software test automation, reducing the man-hours required to create test scenarios, reducing the burden on designers, and shortening the testing period.

In addition, test scenarios are created from base test scenarios according to the settings in the configuration list, making it possible to verify all patterns, preventing test scenario omissions and reducing test coverage. In other words, verifying all patterns for each configuration parameter would result in an enormous number of test scenarios, but by taking measures in creating configuration parameters and test scenarios, testing can be made more efficient.

In addition, the test target selection step 104 includes a type determination step 201 in which the calculation device determines the type of test target based on the additional information of the first test target 102 and the additional information of the second test target 103, allowing for simple and highly accurate selection based on the additional information.

Furthermore, in the standard conversion step 107, the computing device converts non-standard-compliant configuration parameters for the test target whose type has been determined into configuration parameters that comply with the standard, and creates a configuration list 106 that includes the characteristics of the converted configuration parameters. For example, in the case of a DCM that uses CAN communication, the CAN ID, etc. is converted into a format that complies with the software architecture. Therefore, conversion can be performed according to the type of test target, improving conversion accuracy and conversion efficiency.

In addition, in the test scenario creation step 113, the computing device selects the tests required for the first test target 102 based on the additional information in the common section of the configuration list 106, creates a first test scenario 111 using a pre-registered first base test scenario 109, and selects the tests required for the second test target 103 based on the additional information in the specific section of the configuration list 106, and creates the second test scenario using a pre-registered second base test scenario 110.Therefore, the necessary tests can be automatically selected based on whether they are listed in the configuration list 106, and test scenarios can be created.

In addition, in the related information extraction step, the computing device extracts information related to the base test scenario used to create the second test scenario 112, so when the designer creates the second test scenario by hand-coding, the automatically extracted related information can be used, improving work efficiency.

In addition, in the test target extraction step, test targets whose test results indicate an abnormality are extracted based on the test results in the first test step 114 and the test results in the second test step 115.This makes it possible to extract test targets whose test results indicate an abnormality, thereby improving the efficiency of designers' work in addressing defects.

The present invention is not limited to the above-described embodiments, and includes various modifications and equivalent configurations within the spirit of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to configurations that include all of the described configurations. Furthermore, part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Furthermore, the configuration of another embodiment may be added to the configuration of one embodiment. Furthermore, part of the configuration of each embodiment may be added to, deleted from, or replaced with other configurations.

Furthermore, the aforementioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole in hardware, for example by designing them as integrated circuits, or in software, by a processor interpreting and executing a program that realizes each function.

Information such as programs, tables, and files that implement each function can be stored in storage devices such as memory, hard disks, and solid-state drives (SSDs), or in recording media such as IC cards, SD cards, and DVDs.

Furthermore, the control lines and information lines shown are those considered necessary for explanation, and do not necessarily represent all control lines and information lines necessary for implementation. In reality, it is safe to assume that almost all components are interconnected.

101 Configuration list 102 First test target 103 Second test target 104 Test target selection step 105 Configuration parameter standard conversion tool 106 Configuration list 107 Standard conversion step 108 Test scenario creation tool 109 First base test scenario file 110 Second base test scenario file 111 First test scenario file 112 Second test scenario file 113 Test scenario creation step 114 First test step 115 Second test step 116 Test result 117 Test step 201 Type determination step 202, 202, 203, 204, 205, 206, 207, 208 Configuration parameter file 301 AUTOSAR standard 302 Test scenario creation data table 303 Test definition file 401 Test information 402 CAN ID list 403 CAN definition list

Claims (7)

A test system for testing software in compliance with a software architecture based on a predetermined standard,
The test system includes:
a computer having an arithmetic unit that executes predetermined processing and a storage device that can be accessed by the arithmetic unit;
The computing device
a test target selection step of sorting the software to be tested into a first test target that conforms to the software architecture and a second test target that does not conform to the software architecture, and generating supplemental information indicating that the software is the second test target;
A standard conversion step of converting configuration parameters that do not comply with the standard included in the second test target into a format that complies with the standard;
a first test step of creating a first test scenario from a pre-registered first base test scenario and testing the first test target;
a second test step for creating a second test scenario from a second base test scenario registered in advance and testing the second test target. 10. The test system of claim 1,
A test system characterized in that in the test target selection step, the calculation device determines the type of test target based on additional information of the first test target and additional information of the second test target. 3. The test system of claim 2,
In the standard conversion step, the arithmetic device converts configuration parameters that do not comply with the standard included in the second test target into configuration parameters that comply with the standard, and creates a configuration list including characteristics of the converted configuration parameters. A test system characterized by the above. 4. The test system of claim 3,
the computing device executes a test scenario creation step of creating a test scenario;
In the test scenario creation step,
the computing device selects tests necessary for the first test target based on additional information in the common part of the configuration list, and creates the first test scenario using a first base test scenario registered in advance;
A test system characterized in that the calculation device selects tests necessary for the second test target based on additional information in the unique part of the configuration list, and creates the second test scenario using a pre-registered second base test scenario. 5. The test system of claim 4,
A test system, comprising: a related information extraction step in which the computing device extracts information related to a base test scenario used to create the second test scenario. 10. The test system of claim 1,
A test system comprising a test object extraction step of extracting test objects whose test results show an abnormality based on the test results in the first test step and the test results in the second test step. A testing method in which a computer tests software in compliance with a software architecture based on a predetermined standard, comprising:
the computer has an arithmetic unit that executes predetermined processing and a storage device that can be accessed by the arithmetic unit;
The test method comprises:
a test target selection step in which the computing device sorts the software to be tested into a first test target that conforms to the software architecture and a second test target that does not conform to the software architecture, and writes additional information indicating that the software is the second test target;
A standard conversion step in which the arithmetic device converts configuration parameters that do not comply with the standard and are included in the second test target into a format that complies with the standard;
a first test step in which the computing device creates a first test scenario from a pre-registered first base test scenario and tests the first test target;
a second test step in which the computing device creates a second test scenario from a pre-registered second base test scenario and tests the second test target.