Customise JIRA For Test Case Management

This content has been contributed by Matt Doar, author of Practical Development Environments.

These rough notes describe how I implemented a test case and test result tracking system using Jira.
Note: the changes below mean that the Enterprise edition is necessary.
Tool Requirements

A test case manager must support the following requirements:

1. Test cases need to be defined before they can be run.
2. Changes to the definitions of test cases may be made in parallel for different branches and their releases.
3. The results of running the test cases have to be accumulated somewhere.
4. Reports should contain information about many tests run over a period of time on a particular release.
5. The reports should be easy to update when the underlying information changes, e.g. as ongoing tests pass and fail, or new tests are added and old tests are removed.
6. The reports should contain a clear summary of the state of the testing activity for each release.
7. The format of the reports should be such that they can be archived as self-contained collections of files such as test logs.

The following requirements are useful but not essential:

1. If a test case is automated, then it should be possible to record the result automatically as well.
2. If a large number of test cases are defined, then automatic import into the test case manager is important.

"If data is hard to enter, it won't be entered"
Existing Test Case Managers

• TestLink - an open source test case manager
• Mercury Test Directory
• Rational Test Manager
• TestRun - a new Web 2.0 application, just out of beta

Implementation Overview

Define a new issue type named "TestCase", with the following custom fields:

• Date Last Run
• Build Run Against

A new test case is defined by creating a new TestCase issue in Jira and filling in the required fields such as Summary, Description and Component.

For each release where this test is going to be run, a subtask of the issue is created and the Affects Version field is set to the release. The priority of the test for this release can also be added here.

When a test is run, the results are entered or updated in the subtask,not in the parent TestCase issue. Comments about the test run can also be entered if appropriate.

When a report is created for the current state of a release, we search for all TestCase issues with the Affects Version field set to the required release. The resulting set of issues can then be sorted and counted by number passed, failed, not run etc. Producing historical reports can be done with the Timecharts plugin for Jira, which shows a graph of how the results in a report change over time.

We can also create a new link type to connect test cases to the bugs that they created or are verifying. This is the biggest advantage that I see of using an issue tracker to track test cases and bugs together. We could also add a box to all bugs to indicate whether a Test Case is expected by someone for a bug.
Example

Test Case TC-1234
------------------

Summary: Check that the Citrix installer works

Description: instructions on how to check that the installer works,
including the environment and other assumptions.

Links:
TC-1235
TC-1236
TC-1237

TC-1235
-------

Affects Version: 3.0
Date Last Run: 8/8/07
Build Run Against: 3.0.0.4322
State: PASSED

TC-1236
-------

Affects Version: 3.1
Date Last Run: 12/12/07
Build Run Against: 3.1.0.9992
State: FAILED

TC-1237
-------

Affects Version: 3.2
Date Last Run:
Build Run Against:
State: Not Yet Run
Example Report

Date: 12/15/07
Release: 3.1
Number of test cases: 200
Number of tests not run: 20 (10%)

Number of tests passed: 90 (50%)
Number of tests failed: 90 (50%)

1. Created TestCase and TestResult (subtask so can only create from TestCsse)
2. Reordered fields in Issue Types Scheme

I yhought about putting tests in their own project, but linking wouldn't be as good?
Field Changes

You could add a new tab for tests. The field changes I made were:

1) Test Case

Required: Summary, Component, Description, but not Affects Version or Finder

No Affects or Fixed Version

Assignee is person who defines the test

2) Test Result

Required: Summary, Component, Description, but not Finder

Component should be inherited from the parent test case, or not used?

Assignee is person who runs the test, or is this QA contact?

No Fixed Version for test result, just Affects Version
Resolution not used?

Add the following custom fields:

Date Last Run
Build

Custom issue icon images exist too somewhere on this wiki
Screens

Create a Field Configuration for each issue type

Create a new Field Configuration scheme and add each issue type with its Field Configuration

Associate the new Field Configuration scheme with the project PS.
Workflow

Add new statuses: Not Wanted or Inactive, Not Yet Run, Passed, Failed, Error, Needs Rerunning

The intent is that a test should Pass or Fail, and an Error is when you couldn't decide whether it passed or failed because of a problem with the test. Or, "error is when the test itself fails, not the device under test"

Add new workflow scheme

Add new workflow for TestResults

Define new workflow

To remove the default initial state of Open, edit the Open step, and
change the transition to use the new state

Associate a project with new workflow scheme, migrate (takes a couple of minutes)
This is hard to change while active!
Link Types

Test, "is a test for", "is tested by"
Running Reports

Choose the project and Issue Type.
Conclusion

This hasn't been through a release cycle yet, but is attractive since it:

1. Uses an existing and known tool
2. Provides an easy way to link bugs to test cases and results
3. Provides decent reporting of the state of testing
4. Supports multiple releases