Coverage Plugin

Name Coverage Plugin Version 1.0.4 Product Versions 1.2.x (see Version Compatibility for complete list) Author(s) Dan Grabowski Homepage http://confluence.atlassian.com/display/BAMEXT/Coverage+Plugin Price free License BSD JavaDocs bamboo-coverage-plugin-1.0.4-javadoc.jar IssueTracking http://developer.atlassian.com/jira/browse/BCOV Source Repository http://svn.atlassian.com/svn/public/contrib/bamboo/bamboo-coverage-plugin/ or browse in FishEye Download JAR bamboo-coverage-plugin-1.0.4.jar Download Source bamboo-coverage-plugin-1.0.4-sources.jar and bamboo-coverage-plugin-1.0.4-test-sources.jar

Description/Features

The coverage plugin provides tracking of, and insight into, project code coverage for Bamboo builds. It supports code coverage data generated by Clover, Cobertura, and EMMA.

Features

Here are all of the reasons why you would want to download, install, and try out the coverage plugin. If a feature description uses a term you are not familiar with, consider looking in the Concepts section for an explanation. You could start there, if you like, but it doesn't have any pretty screenshots.

Multiple data source support

Not matter what tool you use to collect code coverage data, the coverage plugin can provide insight into the project's code coverage in Bamboo. By default, the coverage plugin has support for coverage data generated by Clover, Cobertura, and EMMA. Support for other code coverage tools can easily be added by implementing a coverage provider. The coverage provider tells the coverage plugin what data your code coverage tool can collect and how to parse the data. All of the great code coverage tracking features come along for the ride.

Details about the support for each code coverage tool. For an explanation of terms, see the Concepts section.

Coverage Tool	Supported Scopes	Supported Granularities	Preferred Granularity
Clover	project, package, class	element, method, conditional, statement	element
Cobertura	project, package, class	line, branch	line
EMMA	project, package, class	class, method, line, block	line

How well is the project being tested?

You just want a quick status check on the code coverage of the project. The coverage plugin is there for you. It provides a build coverage status box on the Code Coverage build plan tab that gives you just the facts: project coverage percentage, change since last build, and number of builds since coverage decreased.

Are the project's tests getting better or worse?

One way to answer this question is to look at how the project's code coverage has changed over time. The coverage plugin allows you to see this information by charting code coverage over time. Coverage for the project scope is charted for all available granularities and the result is placed on the Code Coverage tab on the build plan page.

Who is helping the project's testing, and who is holding it back?

To help answer this question, the coverage plugin charts code coverage change by author over time. For each author, the chart shows whether they have tended to increase or decrease the code coverage of the project over a period of time. The chart is displayed on the new Code Coverage tab on the build plan page.

Where is the project's testing changing?

You just noticed that your project's code coverage has increased (or decreased) dramatically in the last few builds. Now you want to know where the project coverage has improved (or declined). The coverage plugin makes this easy by displaying the 5 classes with the best and worst change in code coverage for each build. This information is displayed on the Code Coverage tab of the build results page. Each class listed is hightlighted in red or green (depending on the nature of the change) and shaded based on the magnitude of the change.

What's that you say? Too many classes have changed to get an accurate picture of what is going on? The coverage plugin also displays the 5 packages with the best and worst change in code coverage for each build. This information is also displayed on the Code Coverage tab of the build results page.

Still can't find out why the project's coverage is changing? It could be due to new classes that were added to the project in this build. Since there are no coverage statistics for these classes in the previous build, their change since the last build cannot be calculated. This means that they will not be displayed in the list of best and worst coverage change. The coverage plugin does display a list of new classes for the build, along with their code coverage percentage, so that you can determine if new classes are the culprit (or benefactor). Just in case you did some major refactoring and there are lots of new classes, there list is limited to 25 entries. Don't worry, it will tell you if it is being filtered. This information is displayed on the Code Coverage tab of the build results page.

How does this project's testing compare to others?

Now that you have the project's code coverage heading in the right direction, you probably want to know how all the other projects stack up against it. The coverage plugin provides two reports that let you do exactly that. The Build-Specific Code Coverage report charts the code coverage of multiple projects over time.

The Build-Specific Code Coverage Delta By Author (wow, that's a mouthful) report tracks code coverage change by author over time. It combines this data from multiple projects onto the same chart for easy comparison.

How well are all projects being tested?

There's probably one last question that you're asking yourself. What does total code coverage look like across all projects? The code coverage plugin includes two reports that help answer this very question. The Code Coverage report sums up the code coverage across multiple projects for all available granularities and charts it over time.

The Code Coverage Change By Author report sums up the coverage deltas for each author across multiple projects and charts it over time.

Configurable level of detail

The scope of coverage data tracked can be limited to the project, package, or class level. Generally, the default setting of class level detail is the best choice, as this setting will provide the most insight into the code coverage of your project. Certain features, such as identifying the classes or packages with the greatest code coverage change, only work when a particular level of code coverage data is collected.

Level	Statistics Included	Advantages	Disadvantages
class (recommended)	class, project, and package level	Provides greatest insight into project code coverage.	May result in performance and/or storage issues for large projects.
package	project and package level	Compromise on both data storage requirements and level of code coverage insight.	Compromise on both data storage requirements and level of code coverage insight.
project	project level only	Requires storing minimal data. May have performance benefits for large builds.	Greatly limits code coverage insight the plugin can provide.

Concepts

This section describes some of the high-level concepts that the implementation of the coverage plugin is based on. This model abstracts code coverage data away from the tool that generated it, allowing for support of multiple code coverage tools. It also allows for views of the code coverage data to be implemented quickly.

Code coverage scope

Code coverage scope defines what code was taken into account when collecting code coverage data. The three scopes supported by the coverage plugin are project, package, and class.

Code coverage granularity

Code coverage granularity defines how precise the code coverage data collected is. The granularities supported by the coverage plugin are class, method, line, conditional, branch, statement, block, and element.

Code coverage statistic

A code coverage statistic tracks a specific piece of data about the code coverage of a unit of code. It has a scope, a granularity, the name of the unit of code being reported on, and a coverage percentage. The code coverage data for a project is represented by a collection of code coverage statistics. The code coverage data for a particular build will typically contain the following:

• One code coverage statistic with project scope for each granularity supported by the coverage provider.
• A set of code coverage statistics for each package in the project, with the set containing one statistic for each granularity supported by the provider.
• A set of code coverage statistics for each class in the project, with the set containing one statistic for each granularity supported by the provider.

Code coverage provider

A code coverage provider defines how the coverage plugin retrieves code coverage data from the build results, as well as what data is provided. A code coverage provider implementation must provide a parser for reading coverage report files, a list of supported scopes, a list of supported code coverage granularities, and a preferred granularity. These properties allow the coverage plugin to get the code coverage data from the code coverage tool's output file and properly display the data. The preferred granularity is for functionalities such as listing classes with the greatest change in coverage. Doing this requires choosing a granularity to compare and preferred granularity specified by the provider is used.

Using the Plugin

Setup

1. Ensure that your build is setup to collect code coverage statistics and that the tool being used will produce an XML code coverage report. Most of these tools will only produce HTML reports by default, so you may need to modify your build slightly to get the XML output. Refer to your tool's documentation for details. Some commonly used code coverage tools are listed below, with hints on where to look for each tool.

   Tool	Configuration Point	URL
   Clover	Home page	http://www.cenqua.com/clover/
   Clover	General Ant task setup	http://www.cenqua.com/clover/doc/ant/tasksintro.html
   Clover	Specifying report formats in the <clover-report> ant task.	http://www.cenqua.com/clover/doc/ant/reporttask.html#formatElement
   Clover	Maven 2 plugin setup	http://maven.apache.org/plugins/maven-clover-plugin/usage.html
   Clover	Specifying report formats in the Maven 2 plugin	http://maven.apache.org/plugins/maven-clover-plugin/usage.html#Specifying%20report%20formats
   Cobertura	Home page	http://cobertura.sourceforge.net
   Cobertura	General Ant task setup	http://cobertura.sourceforge.net/anttaskreference.html
   Cobertura	Maven 1 plugin setup	http://maven-plugins.sourceforge.net/maven-cobertura-plugin/
   Cobertura	Maven 2 plugin setup	http://mojo.codehaus.org/cobertura-maven-plugin/
   Cobertura	Specifying report formats in the Maven 2 plugin	http://mojo.codehaus.org/cobertura-maven-plugin/cobertura-mojo.html#format
   EMMA	Home Page	http://emma.sourceforge.net
   EMMA	General Ant task setup	http://emma.sourceforge.net/userguide/ar01s03.html
   EMMA	Specifying report formats in the <emmajava> ant task	http://emma.sourceforge.net/reference/ch02s02s02.html
   EMMA	Specifying report formats in the <report> ant task	http://emma.sourceforge.net/reference/ch02s04s02.html
   EMMA	Maven 1 plugin setup	http://emma.sourceforge.net/maven-emma-plugin/

2. Place the bamboo-coverage-plugin-x.x.jar file in <Bamboo install directory>/webapp/WEB-INF/lib. Start or restart the Bamboo server.
3. If you are configuring a new build plan, follow the normal process until you reach the Builder Configuration tab. If you are enabling the plugin for an existing build plan, go to the Configuration tab and then the Builder Configuration subtab. Change the subtab to edit mode.
   
4. You should see a section on this page with the header "Where should Bamboo look for the code coverage output file?". Select the "Code coverage output will be produced" checkbox and some additional fields will appear.
   
5. Select the code coverage tool that you are using from the "Code Coverage Provider" dropdown.
6. Select the desired value for "Code Coverage Scope". It is recommended that this be set to "class", since this setting will provide the most detailed insight into the code coverage of your build. However, this plugin has not been tested extensively on large projects and it may be necessary to select a less detailed level of reporting in order to conserve resources.
7. Enter the location of your build's XML code coverage report in the "Code Coverage Output Directory" field. The field will accept the path to a directory, the path to a specific file, an Ant-style pattern that identifies one or more files, or a comma-separate list of these values. All values entered in this field are relative to the root directory of your project. Typically, entering the relative path to the directory containing the coverage output file is sufficient. If there are other XML files in the same directory as the coverage output file, these files will be examined by the coverage plugin, but not processed. The coverage plugin checks the format of the coverage files before loading them and will skip any files that do not appear to be valid. However, if you would like to avoid having other files examined, especially if they are large files, then a pattern that specifically identifies the coverage report file can be provided. For example, assume you have a project located in the directory /projects/cool_app and the build for this project produces the file /projects/cool_app/target/emma/coverage.xml containing code coverage data. The output directory field should be set to target/emma for this build. If there are other XML files in the same directory that should not be parsed, then the output directory should be set to target/emma/coverage.xml.
   
8. Configure any other required build plan settings and save your changes.
9. Execute your build plan.
10. Check the Atlassian Developer Blog to see if any interesting stuff is going on.
11. Look for the Code Coverage tabs on the build plan summary and build result pages.
    
    
12. Smile, knowing that you'll be writing better code in no time

Version Compatibility

Compatibility of Coverage Plugin versions (side) versus Bamboo versions (top).

1.0-1.0.4     1.0.5     1.1-1.1.2     1.2-1.2.4     1.0-1.0.3                     1.0.4

= compatible = not compatible = compatibility unknown

Developing the Plugin

Building From Source

It is strongly recommended that you use Maven 2 to build the code coverage plugin. It will make retrieving the necessary libraries, generating IDE project files, and life in general much easier. There is an Ant build file included with the project, but it was generated from Maven and its primary purpose is to allow generation of EMMA code coverage data, since there does not appear to be a stable working EMMA plugin for Maven 2. Information about installing Maven 2 can be found here. These directions will assume that you are building using Maven.

1. Get the source code. This can be done by downloading the source zip file or checking it out from the Atlassian Subversion repository. Both can be located using the information at the top of this page.
2. Navigate to the root directory of the project.
3. Run mvn package. This should build the bamboo-coverage-plugin-x.x.jar file in <project root>/target.
4. Depending on your IDE of choice, run mvn idea:idea for IntelliJ IDEA or mvn -Declipse.workspace=<path-to-eclipse-workspace> eclipse:add-maven-repo followed by mvn eclipse:eclipse for Eclipse. There are additional parameters that can be specified in both cases. See Maven Eclipse plugin or Maven IDEA plugin for details. There are already IntelliJ IDEA project files included in the source, but you will want to regenerate your own so that the paths to the libraries in Maven's repository match your environment.

Extending

You are encouraged to extend and improve the code coverage plugin. The general process to follow is:

1. Get the source code for the plugin and get your development environment setup as described in Building From Source.
2. Read through Concepts to become familiar with some of the principles behind how the coverage plugin is organized.
3. Come up with a great idea or check Jira for one that someboday has already requested. If your idea is not already in Jira, enter it so that other people can provide suggestions or ideas.
4. Write some code.
5. Contribute your improvement back to the project by submitting a patch to the project owner, or by getting access to Atlassian's Subversion repository.

Packaging a Release

These are the general steps necessary to package new release of the coverage plugin. This assumes you have already gone through the setup described in Building From Source.

1. Copy the version of the project to be release to the /contrib/bamboo/bamboo-coverage-plugin/tags directory in the subversion repository. For example, to release the trunk version of the plugin as version 1.0, run the command

   svn copy http://svn.atlassian.com/svn/public/contrib/bamboo/bamboo-coverage-plugin/trunk/
       http://svn.atlassian.com/svn/public/contrib/bamboo/bamboo-coverage-plugin/tags/bamboo-coverage-plugin-1.0/
   

2. Update your local copy of the project to include the new tagged version. This assumes that you have checked out everything in the /contrib/bamboo/bamboo-coverage-plugin directory locally, which includes all tags, branches and the trunk version of the plugin. If you have only checked out a specific version of the plugin, then check out a new copy of the project for the tag that was just created.
3. Run the following from the root directory of the version of the project tagged for release:
   1. mvn package
   2. mvn javadoc:jar
   3. mvn source:jar
   4. mvn source:test-jar
4. Attach the files bamboo-coverage-plugin-1.0.4.jar, bamboo-coverage-plugin-1.0.4-javadoc.jar, bamboo-coverage-plugin-1.0.4-sources.jar, and bamboo-coverage-plugin-1.0.4-test-sources.jar from the directory <project root>/target to this page.
5. Update the links to these files in the table at the top of this page.
6. "Release" the version of the plugin tracking the release in Jira.
7. Add details about the release to the Version History section of this page.

Version History

1.0.4

Added support for Bamboo 1.2

1.0.3

Fixed problems parsing Cobertura 1.9 reports.

Atlassian Developer Network JIRA (1 issues)
Key	Summary	Pr	Status
BCOV-27	Plugin doesn't work with cobertura 1.9		Closed

1.0.2

Fixed a few problematic bugs and annoyances. Did a lot of refactoring to improve testability and expanded the unit testing coverage.

Atlassian Developer Network JIRA (5 issues)
Key	Summary	Pr	Status
BCOV-22	NPE after a build or so of cobertura errors		Closed
BCOV-23	EMMA coverage data generated with non-debug compile not properly supported		Closed
BCOV-28	Coverage plugin causes crash		Closed
BCOV-19	Better error reporting when no coverage file is found		Resolved
BCOV-20	Coverage values clipped on code coverage tab for build		Closed

1.0.1

Fixed a NullPointerException when the plugin was installed but not enabled for a build.

Atlassian Developer Network JIRA (1 issues)
Key	Summary	Pr	Status
BCOV-17	Null pointer exception when plugin installed, but not enabled		Closed

1.0

Final release for the Codegeist competition. Fixed some issues with Cobertura and Clover report parsing, expanded test coverage (especially on report parsing), got cross-project reports working, and added a few new features.

Atlassian Developer Network JIRA (4 issues)
Key	Summary	Pr	Status
BCOV-11	Verify compatibility with Bamboo 1.1		Closed
BCOV-15	Cross project reports not reading data		Closed
BCOV-16	Clover code coverage percentages 1/100th of actual values		Closed
BCOV-8	Summarize most recent build's coverage status on code coverage plan tab		Closed

0.9

This release cleans up issues with Cobertura and Clover support, along with expanding the information provided in the charts and build results. Some UI niceties have been added, too.

Atlassian Developer Network JIRA (7 issues)
Key	Summary	Pr	Status
BCOV-7	Limit number of new classes displayed on build results tab		Closed
BCOV-6	Modify project coverage graph to display all available granularities		Closed
BCOV-5	Only display top/bottom 5 tables when data is present		Closed
BCOV-4	Graphs and top/bottom 5 lists show no data when using Clover		Closed
BCOV-3	Show coverage percentage for new classes on build results tab		Closed
BCOV-2	Make top/bottom 5 rows on build results tab red or green depending on coverage delta		Closed
BCOV-1	Add top/bottom 5 coverage delta packages to build results tab		Closed

0.8

Initial release with basic infrastructure and some of the reporting in place. Support for Clover and Cobertura may be rough in spots, since this plugin started out as an EMMA plugin.

Possible Future Changes

Unscheduled

Atlassian Developer Network JIRA (8 issues)
Key	Summary	Pr	Status
BCOV-24	Allow user to configure preferred coverage granularity		Open
BCOV-26	Provide help links to the plugin's documentation		Open
BCOV-12	Link to html coverage report generated during build		Open
BCOV-9	Code coverage data browsing tab on build results page		Open
BCOV-31	Exception raised trying to view Code Coverage By Author report		Open
BCOV-18	Code coverage output label isn't accurate		Open
BCOV-38	Support deloying plugin on Bamboo 2.0		Open
BCOV-21	Move inline styles to css file		Open

Screenshots