Child pages
  • Coverage Plugin
Skip to end of metadata
Go to start of metadata

Description/Features

The coverage plugin provides tracking of, and insight into, project code coverage for Bamboo builds. It supports code coverage data generated by Clover, Cobertura, and EMMA.

Features

Here are all of the reasons why you would want to download, install, and try out the coverage plugin. If a feature description uses a term you are not familiar with, consider looking in the Concepts section for an explanation. You could start there, if you like, but it doesn't have any pretty screenshots.

Multiple data source support

Not matter what tool you use to collect code coverage data, the coverage plugin can provide insight into the project's code coverage in Bamboo. By default, the coverage plugin has support for coverage data generated by Clover, Cobertura, and EMMA. Support for other code coverage tools can easily be added by implementing a coverage provider. The coverage provider tells the coverage plugin what data your code coverage tool can collect and how to parse the data. All of the great code coverage tracking features come along for the ride.

Details about the support for each code coverage tool. For an explanation of terms, see the Concepts section.

Coverage Tool

Supported Scopes

Supported Granularities

Preferred Granularity

Clover

project, package, class

element, method, conditional, statement

element

Cobertura

project, package, class

line, branch

line

EMMA

project, package, class

class, method, line, block

line

How well is the project being tested?

You just want a quick status check on the code coverage of the project. The coverage plugin is there for you. It provides a build coverage status box on the Code Coverage build plan tab that gives you just the facts: project coverage percentage, change since last build, and number of builds since coverage decreased.

Are the project's tests getting better or worse?

One way to answer this question is to look at how the project's code coverage has changed over time. The coverage plugin allows you to see this information by charting code coverage over time. Coverage for the project scope is charted for all available granularities and the result is placed on the Code Coverage tab on the build plan page.

Who is helping the project's testing, and who is holding it back?

To help answer this question, the coverage plugin charts code coverage change by author over time. For each author, the chart shows whether they have tended to increase or decrease the code coverage of the project over a period of time. The chart is displayed on the new Code Coverage tab on the build plan page.

Where is the project's testing changing?

You just noticed that your project's code coverage has increased (or decreased) dramatically in the last few builds. Now you want to know where the project coverage has improved (or declined). The coverage plugin makes this easy by displaying the 5 classes with the best and worst change in code coverage for each build. This information is displayed on the Code Coverage tab of the build results page. Each class listed is hightlighted in red or green (depending on the nature of the change) and shaded based on the magnitude of the change.

What's that you say? Too many classes have changed to get an accurate picture of what is going on? The coverage plugin also displays the 5 packages with the best and worst change in code coverage for each build. This information is also displayed on the Code Coverage tab of the build results page.

Still can't find out why the project's coverage is changing? It could be due to new classes that were added to the project in this build. Since there are no coverage statistics for these classes in the previous build, their change since the last build cannot be calculated. This means that they will not be displayed in the list of best and worst coverage change. The coverage plugin does display a list of new classes for the build, along with their code coverage percentage, so that you can determine if new classes are the culprit (or benefactor). Just in case you did some major refactoring and there are lots of new classes, there list is limited to 25 entries. Don't worry, it will tell you if it is being filtered. This information is displayed on the Code Coverage tab of the build results page.

How does this project's testing compare to others?

Now that you have the project's code coverage heading in the right direction, you probably want to know how all the other projects stack up against it. The coverage plugin provides two reports that let you do exactly that. The Build-Specific Code Coverage report charts the code coverage of multiple projects over time.

The Build-Specific Code Coverage Delta By Author (wow, that's a mouthful) report tracks code coverage change by author over time. It combines this data from multiple projects onto the same chart for easy comparison.

How well are all projects being tested?

There's probably one last question that you're asking yourself. What does total code coverage look like across all projects? The code coverage plugin includes two reports that help answer this very question. The Code Coverage report sums up the code coverage across multiple projects for all available granularities and charts it over time.

The Code Coverage Change By Author report sums up the coverage deltas for each author across multiple projects and charts it over time.

Configurable level of detail

The scope of coverage data tracked can be limited to the project, package, or class level. Generally, the default setting of class level detail is the best choice, as this setting will provide the most insight into the code coverage of your project. Certain features, such as identifying the classes or packages with the greatest code coverage change, only work when a particular level of code coverage data is collected.

Level

Statistics Included

Advantages

Disadvantages

class (recommended)

class, project, and package level

Provides greatest insight into project code coverage.

May result in performance and/or storage issues for large projects.

package

project and package level

Compromise on both data storage requirements and level of code coverage insight.

Compromise on both data storage requirements and level of code coverage insight.

project

project level only

Requires storing minimal data. May have performance benefits for large builds.

Greatly limits code coverage insight the plugin can provide.

Concepts

This section describes some of the high-level concepts that the implementation of the coverage plugin is based on. This model abstracts code coverage data away from the tool that generated it, allowing for support of multiple code coverage tools. It also allows for views of the code coverage data to be implemented quickly.

Code coverage scope

Code coverage scope defines what code was taken into account when collecting code coverage data. The three scopes supported by the coverage plugin are project, package, and class.

Code coverage granularity

Code coverage granularity defines how precise the code coverage data collected is. The granularities supported by the coverage plugin are class, method, line, conditional, branch, statement, block, and element.

Code coverage statistic

A code coverage statistic tracks a specific piece of data about the code coverage of a unit of code. It has a scope, a granularity, the name of the unit of code being reported on, and a coverage percentage. The code coverage data for a project is represented by a collection of code coverage statistics. The code coverage data for a particular build will typically contain the following:

  • One code coverage statistic with project scope for each granularity supported by the coverage provider.
  • A set of code coverage statistics for each package in the project, with the set containing one statistic for each granularity supported by the provider.
  • A set of code coverage statistics for each class in the project, with the set containing one statistic for each granularity supported by the provider.

Code coverage provider

A code coverage provider defines how the coverage plugin retrieves code coverage data from the build results, as well as what data is provided. A code coverage provider implementation must provide a parser for reading coverage report files, a list of supported scopes, a list of supported code coverage granularities, and a preferred granularity. These properties allow the coverage plugin to get the code coverage data from the code coverage tool's output file and properly display the data. The preferred granularity is for functionalities such as listing classes with the greatest change in coverage. Doing this requires choosing a granularity to compare and preferred granularity specified by the provider is used.

Using the Plugin

Setup

Caution

The Coverage plug-in stores large amounts of data in your Bamboo database. There are two consequences of this:

  1. Using this plug-in with an embedded Bamboo database may result in poor performance, OutOfMemoryErrors, and/or Bamboo start-up failures. If you are currently using an embedded database, Atlassian strongly recommends that you migrate to a supported external database before installing this plug-in. If you have already encountered such problems while using the plug-in with an embedded database, please consult this article.
  2. After installing this plug-in, you may encounter difficulties with Bamboo's import, export and backup features, such as OutOfMemoryErrors and corrupted export and backup files. If you install this plug-in, Atlassian strongly recommends that you regularly backup your bamboo-home and external database using external tools. We intend to address the underlying issue in a future release of Bamboo.
  1. Ensure that your build is setup to collect code coverage statistics and that the tool being used will produce an XML code coverage report. Most of these tools will only produce HTML reports by default, so you may need to modify your build slightly to get the XML output. Refer to your tool's documentation for details. Some commonly used code coverage tools are listed below, with hints on where to look for each tool.

    Tool

    Configuration Point

    URL

    Clover

    Home page

    http://atlassian.com/clover

    Clover

    General Ant task setup

    http://confluence.atlassian.com/display/CLOVER/6.+Ant+Task+Reference

    Clover

    Specifying report formats in the <clover-report> ant task.

    http://confluence.atlassian.com/display/CLOVER/clover-report#clover-report-Format

    Clover

    Maven 2 plugin setup

    http://confluence.atlassian.com/display/CLOVER/Clover-for-Maven+2+User%27s+Guide

    Clover

    Specifying report formats in the Maven 2 plugin

    http://confluence.atlassian.com/display/CLOVER/Clover-for-Maven+2+User%27s+Guide#Clover-for-Maven2User%27sGuide-ChoosingReportFormats

    Cobertura

    Home page

    http://cobertura.sourceforge.net

    Cobertura

    General Ant task setup

    http://cobertura.sourceforge.net/anttaskreference.html

    Cobertura

    Maven 1 plugin setup

    http://maven-plugins.sourceforge.net/maven-cobertura-plugin/

    Cobertura

    Maven 2 plugin setup

    http://mojo.codehaus.org/cobertura-maven-plugin/

    Cobertura

    Specifying report formats in the Maven 2 plugin

    http://mojo.codehaus.org/cobertura-maven-plugin/cobertura-mojo.html#format

    EMMA

    Home Page

    http://emma.sourceforge.net

    EMMA

    General Ant task setup

    http://emma.sourceforge.net/userguide/ar01s03.html

    EMMA

    Specifying report formats in the <emmajava> ant task

    http://emma.sourceforge.net/reference/ch02s02s02.html

    EMMA

    Specifying report formats in the <report> ant task

    http://emma.sourceforge.net/reference/ch02s04s02.html

    EMMA

    Maven 1 plugin setup

    http://emma.sourceforge.net/maven-emma-plugin/

  2. Place the bamboo-coverage-plugin-x.x.jar file in <Bamboo install directory>/webapp/WEB-INF/lib. Start or restart the Bamboo server.
  3. If you are configuring a new build plan, follow the normal process until you reach the Builder Configuration tab. If you are enabling the plugin for an existing build plan, go to the Configuration tab and then the Builder Configuration subtab. Change the subtab to edit mode.
  4. You should see a section on this page with the header "Where should Bamboo look for the code coverage output file?". Select the "Code coverage output will be produced" checkbox and some additional fields will appear.
  5. Select the code coverage tool that you are using from the "Code Coverage Provider" dropdown.
  6. Select the desired value for "Code Coverage Scope". It is recommended that this be set to "class", since this setting will provide the most detailed insight into the code coverage of your build. However, this plugin has not been tested extensively on large projects and it may be necessary to select a less detailed level of reporting in order to conserve resources.
  7. Enter the location of your build's XML code coverage report in the "Code Coverage Output Directory" field. The field will accept the path to a directory, the path to a specific file, an Ant-style pattern that identifies one or more files, or a comma-separate list of these values. All values entered in this field are relative to the root directory of your project. Typically, entering the relative path to the directory containing the coverage output file is sufficient. If there are other XML files in the same directory as the coverage output file, these files will be examined by the coverage plugin, but not processed. The coverage plugin checks the format of the coverage files before loading them and will skip any files that do not appear to be valid. However, if you would like to avoid having other files examined, especially if they are large files, then a pattern that specifically identifies the coverage report file can be provided. For example, assume you have a project located in the directory /projects/cool_app and the build for this project produces the file /projects/cool_app/target/emma/coverage.xml containing code coverage data. The output directory field should be set to target/emma for this build. If there are other XML files in the same directory that should not be parsed, then the output directory should be set to target/emma/coverage.xml.
  8. Configure any other required build plan settings and save your changes.
  9. Execute your build plan.
  10. Check the Atlassian Developer Blog to see if any interesting stuff is going on.
  11. Look for the Code Coverage tabs on the build plan summary and build result pages.

  12. Smile, knowing that you'll be writing better code in no time(smile)

Version Compatibility

Compatibility of Coverage Plugin versions (side) versus Bamboo versions (top).

 

  1.0-1.0.4  

  1.0.5  

  1.1-1.1.2  

  1.2-1.2.4  

  1.0-1.0.3  

  (question)
 

  (tick)
 

  (tick)
 

  (error)
 

  1.0.4  

  (error)
 

  (error)
 

  (error)
 

  (tick)
 

Unknown macro: {roundrect}

= compatible
= not compatible
= compatibility unknown

Developing the Plugin

Building From Source

It is strongly recommended that you use Maven 2 to build the code coverage plugin. It will make retrieving the necessary libraries, generating IDE project files, and life in general much easier. There is an Ant build file included with the project, but it was generated from Maven and its primary purpose is to allow generation of EMMA code coverage data, since there does not appear to be a stable working EMMA plugin for Maven 2. Information about installing Maven 2 can be found here. These directions will assume that you are building using Maven.

  1. Get the source code. This can be done by downloading the source zip file or checking it out from the Atlassian Subversion repository. Both can be located using the information at the top of this page.
  2. Navigate to the root directory of the project.
  3. Run mvn package. This should build the bamboo-coverage-plugin-x.x.jar file in <project root>/target.
  4. Depending on your IDE of choice, run mvn idea:idea for IntelliJ IDEA or mvn -Declipse.workspace=<path-to-eclipse-workspace> eclipse:add-maven-repo followed by mvn eclipse:eclipse for Eclipse. There are additional parameters that can be specified in both cases. See Maven Eclipse plugin or Maven IDEA plugin for details. There are already IntelliJ IDEA project files included in the source, but you will want to regenerate your own so that the paths to the libraries in Maven's repository match your environment.

Extending

You are encouraged to extend and improve the code coverage plugin. The general process to follow is:

  1. Get the source code for the plugin and get your development environment setup as described in Coverage Plugin.
  2. Read through Concepts to become familiar with some of the principles behind how the coverage plugin is organized.
  3. Come up with a great idea or check Jira for one that someboday has already requested. If your idea is not already in Jira, enter it so that other people can provide suggestions or ideas.
  4. Write some code.
  5. Contribute your improvement back to the project by submitting a patch to the project owner, or by getting access to Atlassian's Subversion repository​.

Packaging a Release

These are the general steps necessary to package new release of the coverage plugin. This assumes you have already gone through the setup described in Coverage Plugin.

  1. Copy the version of the project to be release to the /contrib/bamboo/bamboo-coverage-plugin/tags directory in the subversion repository. For example, to release the trunk version of the plugin as version 1.0, run the command
    svn copy http://svn.atlassian.com/svn/public/contrib/bamboo/bamboo-coverage-plugin/trunk/
        http://svn.atlassian.com/svn/public/contrib/bamboo/bamboo-coverage-plugin/tags/bamboo-coverage-plugin-1.0/
    
  2. Update your local copy of the project to include the new tagged version. This assumes that you have checked out everything in the /contrib/bamboo/bamboo-coverage-plugin directory locally, which includes all tags, branches and the trunk version of the plugin. If you have only checked out a specific version of the plugin, then check out a new copy of the project for the tag that was just created.
  3. Run the following from the root directory of the version of the project tagged for release:
    1. mvn package
    2. mvn javadoc:jar
    3. mvn source:jar
    4. mvn source:test-jar
  4. Attach the files bamboo-coverage-plugin-1.0.4.jar, bamboo-coverage-plugin-1.0.4-javadoc.jar, bamboo-coverage-plugin-1.0.4-sources.jar, and bamboo-coverage-plugin-1.0.4-test-sources.jar from the directory <project root>/target to this page.
  5. Update the links to these files in the table at the top of this page.
  6. "Release" the version of the plugin tracking the release in Jira.
  7. Add details about the release to the Coverage Plugin section of this page.

Version History

1.0.4

Added support for Bamboo 1.2

1.0.3

Fixed problems parsing Cobertura 1.9 reports.

type key summary priority status

Data cannot be retrieved due to an unexpected error.

View these issues in JIRA

1.0.2

Fixed a few problematic bugs and annoyances. Did a lot of refactoring to improve testability and expanded the unit testing coverage.

type key summary priority status

Data cannot be retrieved due to an unexpected error.

View these issues in JIRA

1.0.1

Fixed a NullPointerException when the plugin was installed but not enabled for a build.

type key summary priority status

Data cannot be retrieved due to an unexpected error.

View these issues in JIRA

1.0

Final release for the Codegeist competition. Fixed some issues with Cobertura and Clover report parsing, expanded test coverage (especially on report parsing), got cross-project reports working, and added a few new features.

type key summary priority status

Data cannot be retrieved due to an unexpected error.

View these issues in JIRA

0.9

This release cleans up issues with Cobertura and Clover support, along with expanding the information provided in the charts and build results. Some UI niceties have been added, too.

type key summary priority status

Data cannot be retrieved due to an unexpected error.

View these issues in JIRA

0.8

Initial release with basic infrastructure and some of the reporting in place. Support for Clover and Cobertura may be rough in spots, since this plugin started out as an EMMA plugin.

Possible Future Changes

Unscheduled

type key summary priority status

Data cannot be retrieved due to an unexpected error.

View these issues in JIRA

Screenshots

24 Comments

  1. Nice plugin. I'm just trying this with the JIRA bamboo instance now, and I'm getting the following exception for builds where there's no coverage defined:

    java.lang.NullPointerException
    	at com.atlassian.bamboo.plugins.coverage.CoverageBuildProcessor.run(CoverageBuildProcessor.java:57)
    	at com.atlassian.bamboo.build.pipeline.tasks.ExecuteBuildTask.performCustomBuildProcess(ExecuteBuildTask.java:97)
    	at com.atlassian.bamboo.build.pipeline.tasks.ExecuteBuildTask.call(ExecuteBuildTask.java:53)
    	at edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:176)
    	at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:987)
    	at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:528)
    	at edu.emory.mathcs.backport.java.util.concurrent.helpers.ThreadHelpers$1.run(ThreadHelpers.java:34)
    	at java.lang.Thread.run(Thread.java:595)
    

    Any ideas? The 'Code coverage output will be produced' checkbox is NOT checked. These errors started happening as soon as I installed the plugin.

    1. There is a bug in the coverage build processor. It does not check to make sure a custom configuration property is present before checking its value. If the coverage plugin is installed, this causes any build where the 'Code coverage output will be produced' is not selected to fail with the exception that you provided. I created BCOV-17 to track this issue.

      I could easily commit a fix to the trunk version, but I'm not sure what is allowed with the contest closed and judging still to take place. I suppose I'll email Jonathan Nolen and see what he has to say. At worst I'll commit a fix after the judging is complete.

    2. There is a 1.0.1 version of the coverage plugin tagged in subversion that fixes this bug. Once this page has moved and I can edit it again, I'll post a new jar file.

  2. Any chance of that 1.0.1 release? It would be very nice to see this bug fixed.

    1. The 1.0.1 jars are now posted.

  3. I get this exception after adding the plugin and restarting Tomcat when I view an existing build plan. Any idea which class is missing ?

    --------------------------------------/
    2007-07-05 13:53:19,786 ERROR http-8080-Processor25 CompoundRootAccessor Caught an Ognl exception while getting property combinedByTimeChart
    ognl.OgnlException: combinedByTimeChart java.lang.NoClassDefFoundError
    at ognl.OgnlRuntime.getMethodValue(OgnlRuntime.java:908)
    at ognl.ObjectPropertyAccessor.getPossibleProperty(ObjectPropertyAccessor.java:54)
    ............
    /-- Encapsulated exception ------------\
    java.lang.NoClassDefFoundError
    at org.jfree.chart.ChartFactory.createStackedXYAreaChart(ChartFactory.java:1431)
    at com.atlassian.bamboo.charts.timeperiod.SuccessRatioStackedAreaChart.getChart(SuccessRatioStackedAreaChart.java:66)
    at com.atlassian.bamboo.charts.CombinedBuildSummaryByTimeChart.generateChartParams(CombinedBuildSummaryByTimeChart.java:31)

    1. I'm not sure that this exception is caused by the coverage plugin. Based on the stack trace, it appears that this error occurs while building the "% Successful Builds & Avg Duration per Time Period" chart on the build plan summary page. The coverage plugin should not affect the functionality of this chart or page.

      What version of Bamboo did this error occur with? I was trying to look at the JFreeChart source for the code that raised the NoClassDefFoundError (ChartFactory.java:1431), but I can't seem to find the version of JFreeChart that matches up with this stack trace.

  4. Any ETA for 1.2 support?

  5. 1.2 support would be really helpful!

  6. Hi all, I've just added a 1.0.4 version that should support Bamboo 1.2

    Cheers,

    Ross

  7. Hi!

    Great plugin, once I got it working. Since we had a few JUnit tests that were failing, the coverage plugin did not collect data (as failed JUnit-tests breaks the build).
    It took a while and some looking in the plugin source/bamboo.log and the atlassian-plugin.xml before I could figure out why it did not run the coverage collector.

    """
    <buildProcessor key="coverageResultsCollector" name="Code Coverage Results Collector"
                      class="com.atlassian.bamboo.plugins.coverage.CoverageBuildProcessor">

        <skipIfFailed>true</skipIfFailed>

        <resource type="freemarker" name="edit" location="templates/buildProcessor/coverageProcessorEdit.ftl"/>
        <resource type="freemarker" name="view" location="templates/buildProcessor/coverageProcessorView.ftl"/>
        <description>A code coverage report parser and data collector</description>
      </buildProcessor>
    """

    It would be benefical to document this behaviour, so others won't curse over their own inability to install the plugin. Or even better if this were configurable (if its doable) rather than hardcoded into the plugin.

    Thank You

    1. I agree this should be configurable but probably discouraged. And at the least, documented. This took some time to figure out (after recovering from the initial letdown of not seeing code coverage numbers).

  8. When using this plugin on a clover enabled multi-module project we found that the results are completely inaccurate. When we let the plugin analyse and aggregate the 10 different clover.xml reports that made up our test project we found it reported a total plan coverage of 77%. However when we aggregated the individual report files into one aggregated clover xml report via clover itself and then configured the coverage plugin to process only that, pre-aggregated report, the coverage was reported correctly at 28%. Seems to us that whatever logic is being used in aggregation bares no relation to the truth.

    1. We're experiencing the same. I'm still going to look into the source code to see whether there is anything blatantly wrong, but in the meantime - would you mind sharing how you aggregated the reports? Concatinating all the <project> and <test-project> tags doesn't seem to work as far as I can tell - summary is still off.

      1. There is a known issue with statistic calculation for projects that output multiple coverage files. See http://developer.atlassian.com/jira/browse/BCOV-37 for details. The issue has been resolved in the trunk version of the plugin in subversion. I have not built a release including this fix yet, because the resolution breaks the backwards compatibility of data stored by the plugin and I am still working on fixing another issue that will make further changes in this area.

        In short, the issue with calculating the coverage statistics is that the percentages from each file are averaged, which is inaccurate because it gives equal weight to each file, even if they do not represent equal amounts of code. In the trunk, the plugin has been reworked to track raw counts and only calculate percentages once the ultimate totals have been determined.

  9. Hi Dan!

    is any update scheduled to support Bamboo v2???

    Thanks in advance,

    Fernando

  10. Dan,

    Do you need help getting this upgraded to Bamboo 2.0.x ? I wrote the Mercurial repo plugin so might have a fighting chance to figure out how to upgrade this codebase too....

    Brett

  11. The plug-in descriptor for this plug-in incorrectly names Atlassian as the plug-in vendor.

    Can you please correct this in the next release of the plug-in? Thanks.

  12. is this available on bamboo 2.0?

    This feature will be very useful

    1. This plug-in hasn't yet been upgraded by its author for Bamboo 2.0.

      However, Bamboo 2.0 includes built-in support for Clover, which might meet your needs.

      1. yeaa i had look at that, what i was after is the code coverage (coverage percentage) for each project / namespace

        my nant script produce xml and html report using ncover, is there anyway of using that html report in this section of report?

        1. I've logged a feature request for the Ncover tab to use the HTML report to track user interest in this, so I recommend that you sign up for a free account there and vote. You may also like to comment or set a watch on the issue to be emailed if there is any progress.

          If this is urgent, you may wish to add the HTML report independently by developing your own report plugin.

        2. Hi, the NAnt Builder plugin includes support for parsing and displaying NCover reports, does this provide what you're after?

          Cheers,

          Ross