The objective is to evaluate the usability and effectiveness of two approaches to automated real-time testing: Manually assisted test purpose based test generation and full automatic randomized online testing. Each group is given a UppAal specification of the light controller, and an (assumed!) correct implementation in java, including two example mutants. The experiment proceeds in two phases: Preparation phase and evaluation phase.
In the preparation phase, each group will independently generate a set of test cases and a set of mutated implementations based on the given specification and implementation. Each group runs their test suite on their own mutants, and possibly extend it as desired.
In the evaluation phase, each group swaps mutants (java bye-code) and runs
their own test suite against the groups own and other groups mutants, collecting statistics with the results (which own/goreign mutants were killed).
applies T-UppAal (randomized online testing) to the collected mutants (e.g 10 runs per mutant)., again collecting statistics about the percentage of runs killing the mutants.
Each group is to hand in the following deliverables to be evaluated.
10 to 20 error-seeded mutants of the LightController application, with a short description explaining the seeded errors.
The test suite, including the material needed to generate it, e.g a list of timed traces, uppaal (annotated) models, reachability properties, simulation traces, java sources. Each test must be accompanied by a short description in the style of an informal test purpose (one line of text) for each test case.
The statistics of the evaluation phase for manually generated tests
The statistics collected by runnning T-Uppaal.
A (max) 1 page summary with the lessons learned.
The deliverables must be signed by each student who, by his signature, declares active and equal participation in preparation of the deliverables
To execute application use (0 indicates mutant 0)
>java LightController 0
To execute a programmed test case (number 1 with the application running on localhost at port 9999) use
>java TestLightController localhost 9999 1
Brian