ITEA3 TESTOMAT Project ended at the turn of the year 2020/21. In the project VTT’s research focused on improving effectiveness and efficiency of software testing. In particular VTT developed techniques and tools that (1) improve efficiency of mutant generation for industrial strength C/C++ software, (2) allow transpartent tool integration into development and build environments, (3) allow straightforward integration into test execution environments, and (4) make management of mutants easy. The tool is available for interested parties (tool installation packages).
TESTOMAT-project - The next level of test automation
The motivation of TESTOMAT Project was the observation that nowadays quality software has often come to mean “easy to adapt” because of the constant pressure to change. Consequently, modern software teams seek a delicate balance between two opposing forces: striving for reliability and striving for agility. The aim of the project was to support software teams to strike the right balance by increasing development speed without sacrificing quality. The project developed a wide range of methods and tools starting from a Test Automation Improvement Model to specific tools improving efficiency of test automation.
Knowing when to stop
A fundamental problem in software testing is to decide whether a test set explores the software under test thoroughly enough. The software team has to be able to answer a simple question: “Do we have enough test cases so that we can be confident that the software works correctly? If the answer is yes, then we say that the test set is adequate for the software. If the answer is no, then there is a follow-up question: “What areas of the software need more testing?”
The answers to these questions must not be based on gut feelings. Software teams need means to answer the questions in a consistent way. For this reason there must be well-defined methods of measuring whether a test set explores the software thoroughly enough. Such a well-defined method is called test adequacy criterion. During the past decades researchers have define a plethora of test adequacy criteria. The most well known and widely used criterion is statement coverage, which stipulates that each statement of the software must be executed by at least one test case. The criterion is very intuitive and tool support is easy to develop. Unfortunately, it has turned out that the criterion is not that effective. It very effectively identifies the areas that are not tested at all. But it may show full coverage for areas that are rather poorly tested. This is because the criterion does not take into account interaction and dependencies between program elements.
Mutation testing
Mutation testing is regarded as one of the most effective testing method. The idea in mutation testing is to create variations of the original software. The variations are called mutants. Each mutant is created by modifying a statement of the original software. We say that a test case kills a mutant if the outputs of the mutant and the original software differ on the test case. Mutation testing adequacy criterion stipulates that the test cases in a test set must kill all mutants. Intuitively, a test case that kills a mutant demonstrates that it is important that the statement chosen for mutation appears in its original form as the mutated form computes incorrect output on the test case and the original form computes correct output.
Mutation testing is not yet widely used by software teams. Impediments to wider acceptance of mutation testing for compiled languages such as C and C++ includes:
- The number of mutants can be huge as all statements of the software are mutated. Hence compiling each mutant separately takes too much computing resources.
- Storing all mutants to hard disks takes too much disk space.
- Management of mutants is complex, in particular for large software that is composed of several modules and libraries
- Integration of mutant generation into build process is laborious
- Integration of mutants into test execution system is hard
In TESTOMAT Project VTT has developed techniques to resolve the problems above, and a mutant generation tool for industrial strength C/C++ software. The tool is a compiler extension (on top of the LLVM/Clang compiler) that generates a special all mutations form of the software. All mutations form contains all mutations of the software. Hence there is only one compiled version of the software. The software can be executed so that one of the mutations is activated. Then it behaves exactly the same way as a mutant containing exactly that mutation. If none of the mutations is activated, then the program behaves exactly the same way as the original software.
The tool provides the following benefits:
- Shorter compilation time. Speed-up factor of 100 is typical. In a detailed analysis of OpenCV software builds showed speed-up factor of 160 (1.5h vs. 240h).
- Management of mutations and mutants is easy
- Consumes significantly less disk space
- Easy to integrate into build process
- Straightforward to integrate into test automation
- Simple to use
You can watch a short video illustrating the use of the tool. The video uses LLVM/Clang compiler as example software.
The tool removes several major roadblocks standing in the way of wider acceptance of mutation testing. The tool is available from the project's VTT home page.
The research was funded in part by Business Finland grant 3025/31/2017
