This page contains few sample performance results based on a synthetic and a real code. Results in your project may be different.

Comparison of statement and method instrumentation level

Sample code

A following open source libraries were tested using statement- and method-level instrumentation.

• Apache Commons IO version 2.4 (I/O operations)
• Apache Commons Math version 3.2 (CPU-intensive calculations)
• Guava version 14.0.1 (data collections)

Standard test of unit tests was executed and a total time of test execution was measured.

Results

(*) only 200 test classes were executed for Commons Math

Conclusion

Performance penalty for a method and statement instrumentation level may vary significantly, especially for CPU-intensive applications.

Comparison of different per-test coverage recording strategies

Sample code

The following class:

import junit.framework.TestCase;
public class PerformanceTest extends TestCase {
    static int hitsPerTest;
    static int numberOfTests;
    
    public void testPerformance() {
        for (int i = 0; i < hitsPerTest; i++); // empty loop, one R.inc(...) call per loop
    }
    
    public static void main(String[] args) { 
        numberOfTests = Integer.valueOf(args[0]);
        hitsPerTest = Integer.valueOf(args[1]);
        
        PerformanceTest pt = new PerformanceTest();
        for (int i = 0; i < numberOfTests; i++) {
            pt.testPerformance();
        }
    }
}

was instrumented using Fixed Coverage Recorder and executed with different per-test recording strategies. In order to have roughly 10'000'000 hits recorded by Clover, application was executed with following arguments:

PerformanceTest 10 1000000
PerformanceTest 20 500000
PerformanceTest 50 200000
PerformanceTest 100 100000
PerformanceTest 200 500000
PerformanceTest 500 200000
PerformanceTest 1000 10000
PerformanceTest 10000 1000

Results

Test environment: JDK 1.5, Windows 7; Core i7 2670QM 2.2 GHz; 8GB RAM; HDD 750GB 7200RPM.

Conclusion

For a large number of tests, performance is mainly affected by a fact that the coverage recording file is being created on a hard disk. If you have more than 1000 tests there is practically no difference which strategy is used.

For CPU intensive applications the "diffing" strategy is slightly faster than the "single-threaded" or "volatile".