About Test Optimization
This page explains Clover's Test Optimization feature and gives a brief explanation of how it works. See the Test Optimization Quick Start Guide for Ant for practical instructions.
On this page:
What is Test Optimization?
Test optimization will make a build potentially complete a lot faster than a full build and test run. It should do this without substantially compromising the quality of the feedback it gives; in other words a quicker pass or fail result, but a reasonably accurate pass or fail.
There are two ways of ensuring a build completes quickly:
1. Run only the tests required to confirm the validity of the changes that triggered the build.
2. Run all the tests but in an optimal order: any failed tests from the previous build, all tests covering modified code, then in ascending order by test invocation time.
Since Clover records which tests covered which lines of code, it can tell the build to only run tests that cover code modified since the last build.
How Test Optimization Works
The following is a general outline of what is required to enable Clover to optimize the test and build process.
- A full clean build is performed. Any existing Clover databases are removed.
- Clover performs instrumentation on all Java source files. The Clover registry is created.
- javac compiles the instrumented sources
- All unit tests are run. Coverage data is stored next to the Clover registry.
- A Clover snapshot is saved - this is essentially a mapping of application source files to the set of tests which hit each file.
- Zero or more Java source files are modified, added or removed
- Clover re-instruments either only the modified source files or all source files (depending on how Clover is invoked). The Clover registry is updated, and any files modified or added since the last snapshot are marked as such.
- Clover uses this information and the snapshot from the previous test run to determine which tests need rerunning.
- The test runner is invoked. Only tests obtained in #9 and any tests unknown to Clover (perhaps excluded from instrumentation) are run. Tests are ordered to encourage early failures.
- A snapshot is saved.
- Go to step 7 unless a maximum number of optimized builds has reached,
clover.jarhas changed between builds or some other build-specific condition signifies a full rebuild is required (e.g. configuration file changes) in which case go to step 1.
Supported Test Environments for Test Optimization
The following environments are supported for Test Optimization.
Your unit tests must be completely standalone and have no inter-test dependencies.
Clover's Test Optimization features currently do not support Groovy.
Ant
- Junit - using the <batchset/> element nested in the Ant <junit/> task to select tests to be run.
- Junit TestSuites are currently not supported.
- TestNG is unsupported by Test Optimization for Clover for Ant.
- If your tests run in a separate JVM to your application, you will need to enable distributed coverage.
Maven 2 & 3
- Maven version 2.0.8+
- maven-surefire-plugin.
- JUnit TestSuites are currently not supported.
- TestNG test suites are currently not supported. (NB Will not work in Maven 2.0.9)
- Parallel test execution is unsupported.
- If your tests run in a separate JVM to your application (e.g. in a forked web server), you will need to enable distributed coverage.
Eclipse
- JUnit test classes are supported but TestSuites are not currently supported.
- TestNG is not currently supported.
- Optimization of tests where they reside in different project to the application code is not currently supported (to be addressed in a future release).
IDEA
- JUnit test classes are supported but TestSuites are not currently supported.
- TestNG is not currently supported.
Exemplary results
The following is a sample from development of one of Atlassian products. Optimization results in your project(s) may vary.
To see the benefits of Clover's Test Optimization, we have tracked build times on one Atlassian software development project. Over a 10 day period a development team committed 142 changesets as part of their ongoing development effort. For each changeset, two builds were triggered - a "normal" build, where all tests were executed, and a test-optimized build, where only relevant tests were executed. The following chart shows cumulative times for both the normal and test-optimized builds.
Clover's test optimization was configured to perform full test run every 10 builds.
This is a reason why you can see regular peaks in optimized build time.
We have gained about 70% reduction of cumulative build time thanks to test optimization.
Less than 10% of test cases (on average) were executed in optimized runs
(i.e. after a peak with a full test run).