This page explains Clover's Test Optimization feature and gives a brief explanation of how it works. See the Test Optimization Quick Start Guide for Ant for practical instructions.
On this page:
Test optimization will make a build potentially complete a lot faster than a full build and test run. It should do this without substantially compromising the quality of the feedback it gives; in other words a quicker pass or fail result, but a reasonably accurate pass or fail.
There are two ways of ensuring a build completes quickly:
1. Run only the tests required to confirm the validity of the changes that triggered the build.
2. Run all the tests but in an optimal order: any failed tests from the previous build, all tests covering modified code, then in ascending order by test invocation time.
Since Clover records which tests covered which lines of code, it can tell the build to only run tests that cover code modified since the last build.
The following is a general outline of what is required to enable Clover to optimize the test and build process.
clover.jarhas changed between builds or some other build-specific condition signifies a full rebuild is required (e.g. configuration file changes) in which case go to step 1.
The following environments are supported for Test Optimization.
Your unit tests must be completely standalone and have no inter-test dependencies.
Clover's Test Optimization features currently do not support Groovy.
The following is a sample from development of one of Atlassian products. Optimization results in your project(s) may vary.
To see the benefits of Clover's Test Optimisation, we have tracked build times on one Atlassian software development project. Over a 10 day period a development team committed 142 changesets as part of their ongoing development effort. For each changeset, two builds were triggered - a "normal" build, where all tests were executed, and a test-optimised build, where only relevant tests were executed. The following chart shows cumulative times for both the normal and test-optimised builds.
Clover's test optimization was configured to perform full test run every 10 builds.
This is a reason why you can see regular peaks in optimized build time.
We have gained about 70% reduction of cumulative build time thanks to test optimization.
Less than 10% of test cases (on average) were executed in optimized runs