Measuring per-test coverage for manual tests

Clover can measure code not only code coverage from whole application run but also a coverage generated by a single test case (JUnit / TestNG). It is possible to "hack" Clover and measure per-test coverage from manual test cases too. What has to be done is to "tell" Clover when manual test case starts and ends.

The easiest way to provide this information is to write a JUnit test case, with one test method for each manual test case. Clover will add "start test / end test" instrumentation to such methods. Next it's necessary run such JUnit test together with the application - thanks to the Clover's Distributed Coverage feature it's possible to run JUnit test and the application under test in separate JVMs.

Steps

1) Write JUnit test case having following features:

• tests do not start until application under test is launched
• one test method per one manual test case
• single test method starts just before corresponding manual test case is started
• single test method ends just after corresponding manual test case is finished

Example:

import junit.framework.TestCase;
 
public class MyManualTest extends TestCase {
 
    public static void main(String args[]) {
        MyManualTest myTest = new MyManualTest();
        myTest.waitUntilYourApplicationStarts();
        int testNo = myTest.getTestNumber();
        switch (testNo) {
            case 1:
                myTest.testManualTest1();
                break;
            case 2:
                myTest.testManualTest2();
                break;
        }
    }
 
    private int getTestNumber() {
        // e.g. entered by user / read from commandline arg ...
        return 1;
    }
 
    private void waitUntilYourApplicationStarts() {
        // e.g. "Press any key when ready" / check for existence of some marker file
    }
 
    private void waitUntilTestEnds() {
        // e.g. "Press any key when test is finished" / check for some marker file
    }
 
    public void testManualTest1() {
        // Clover will add "test start" here
        waitUntilTestEnds();
        // Clover will add "test end" here
    }
 
    public void testManualTest2() {
        // Clover will add "test start" here
        waitUntilTestEnds();
        // Clover will add "test end" here
    }
 
}

2) Compile JUnit test case together with the application under test (produce two jars, for instance) with a Distributed Coverage feature enabled.

• flushpolicy = interval or threaded might be needed

Example:

<clover-setup initstring="/path/to/my/clover.db" flushpolicy="interval" flushinterval="1000">
    <distributedCoverage/>
</clover-setup>

3) Run unit test and application:

a) unit test shall be launched with -Dclover.server=true parameter, e.g.

java -cp ...;clover.jar -Dclover.server=true MyManualTest

b) application shall be launched as usual (note that distributed coverage configuration is already compiled into instrumented classes), e.g.

java -cp ...;clover.jar MyApplication

Browsing per-test coverage

4) Generate coverage report. In the HTML report it's possible to browse coverage contributed by given test case(s) by clicking "Show tests" link on the class summary page.

Per-test optimization for manual tests

Because of fact that all manual test cases were wrapped into JUnit tests it's possible use per-test optimization:

• create test optimization snapshot file after test execution (<clover-snaphsot/> task)
• after code base changes, run optimized test set (using the <clover-optimized-testset/>selector for <junit/> task)
  • note that Clover-for-Ant optimization is based on test classes, not test methods (it's a limitation of <junit> <batchtest>) so it' might be worth to have one test method per test class

It is also possible to browse "source file - test case" and "class file - test case" mapping using the SnapshotPrinter tool:

java -cp clover.jar com.atlassian.clover.optimization.SnapshotPrinter clover.snapshot.file

SnapshotPrinter can print mapping in plain text or in JSON format (since 3.1.11).