Diagnosing runtime issues using the Java Flight Recorder

Accurately diagnosing and troubleshooting problems with Java apps can be challenging and may require digging through gigabytes of logs to figure out what exactly went wrong and when. Instead of spending countless hours on manual investigation, you can use the Java Flight Recorder, which provides easy after-the-fact diagnostic and troubleshooting features, allowing you to quickly identify and resolve runtime problems as they occur.

The Java Flight Recorder uses a circular buffer to continuously collect events from the Java Virtual Machine runtime and periodically records a snapshot of the collected data to a binary file.

Additionally, you can include the data collected by the Java Flight Recorder in a support zip archive that you can send to Atlassian Support for assistance.

The Java Flight Recorder can take a snapshot automatically at specified intervals or when:

• an admin creates a support zip with the In-app diagnostics option enabled on the Troubleshooting and support tools administration page

• an admin disables the In-app diagnostics option on the Troubleshooting and support tools administration page or by making an appropriate call to the REST API

• the Java Virtual Machine shuts down gracefully

You can review collected runtime data in JDK Mission Control, a free tool provided by Oracle that lets you efficiently and accurately analyze the data collected by the Java Flight Recorder.

Supported Data Center and Server products

The Java Flight Recorder in-app diagnostics are available in the following Atlassian Data Center and Server products:

• Jira Data Center and Server 8.13 and later

• Bitbucket Data Center and Server 8.3 and later

• Confluence Data Center and Server 7.4.0 and later

• Bamboo Data Center 9.6.2 and later

Jira Data Center and Server

The Java Flight Recorder supports the following releases of Jira Server

• In Jira Data Center and Server 9.2 and later, the Java Flight recorder is bundled with the software. Continuous data recording begins as soon as you start up your instance.

• In Jira Data Center and Server 9.0 and 9.1, the Java Flight Recorder is active by default. Continuous data recording begins as soon as you start up your Jira instance.

• In Jira Data Center and Server releases 8.13 through 9.0, the Java Flight Recorder is an opt-in feature. To use it, upgrade the Troubleshooting and Support Tools.

In Jira Server and Data Center 9.0, the recording begins automatically once you upgrade Troubleshooting and Support Tools to version 1.39.0. In earlier Jira releases, you have to start the recording manually.

Take a look at the following table for a summary of the information above:

Jira release	Feature availability	Bundled with Jira	Automatic recording
8.12 and older		N/A	N/A
8.13–8.22	(opt-in)
9.0	(opt-in)
9.1 and later

Bitbucket Data Center and Server

In Bitbucket Data Center and Server 8.3 and later, the Java Flight recorder is bundled with the software. Continuous data recording begins as soon as you start up your instance.

Confluence Data Center and Server

• In Confluence Data Center and Server 7.20 and later, the Java Flight Recorder is bundled with the software. Continuous data recording begins as soon as you start up your instance.

• In Confluence Data Center and Server releases 7.4.0 through 7.20, the Java Flight Recorder is an opt-in feature. To use it, upgrade the Troubleshooting and Support Tools.

In Confluence Server and Data Center 8.0, the recording begins automatically once you upgrade Troubleshooting and Support Tools to version 1.41.0. In earlier Confluence releases, you have to start the recording manually.

Bamboo Data Center

In Bamboo Data Center 9.6.2 and later, the Java Flight recorder is bundled with the software. Continuous data recording begins as soon as you start up your instance.

On this page:

System requirements

To use the Java Flight Recorder, make sure that you are running a supported Data Center and Server product, and that your instance meets the following Java version requirements:

Before the recording is started, make sure your disk has enough free space to store JFR dumps.

The space required for the recording is calculated according to the following formula: jfr.recording.max_size * jfr.recording.files_to_remain.

To reduce the space that the recording will take, edit the properties jfr.recording.max_size and jfr.recording.files_to_remain as needed. Learn how to edit the properties

Enabling continuous data recording

Your data recording is started automatically in the following Data Center applications:

• Jira Data Center and Server 9.0 and later

• Bitbucket Data Center and Server 8.3 and later

• Confluence Data Center and Server 7.20 and later

• Bamboo Data Center 9.6.2 and later

If you're running an earlier release of any of those products, make sure to enable in-app diagnostics to allow the Java Flight Recorder to continuously record runtime data.

To enable continuous runtime data recording:

1. In the upper-right corner of the screen, select Administration, and then System.
2. From the left menu, select Troubleshooting and support tools.
3. On the Troubleshooting and support tools page, enable  In-app diagnostics.

Default event collection settings

We've based the default event collection settings on the default Java Flight Recorder configuration included in the Oracle JDK. While we left most of the default configuration unchanged, we’ve disabled gathering garbage collection events and changed thread dump events to be collected every 10 seconds.

All active event collection settings are listed in the active_configuration.jfc file. This file contains default settings if no custom configuration is applied, so you can use it as a template to create your own custom configuration.

The following table lists the configuration of event type collection in the template file:

Overriding the default event collection settings

If you want to override the default event collection settings, you can create a custom configuration based on the active_configuration.jfc template file located at <APP_HOME>/log/jfr.

To override the default event collection settings:

1. In JDK Mission Control, create a custom Java Flight Recorder configuration based on the included template and save it under a unique name with the .jfc extension (for example, custom.jfc.)

2. Copy the file to the app's local home directory.

   In clustered Data Center configurations, copy the file to the shared home directory, where it can be accessed by all nodes in the system.

3. If not already created, create a custom jfr.properties file.

4. Edit the jfr.properties file by uncommenting the jfr.configuration.template.path property and changing the value to the path to your custom template.
5. Restart your app instance for the changes to take effect.

Default data recording settings

Data recording settings such as thresholds, target file paths, or the number of snapshot files to retain are defined in the embedded jfr.properties file. You can change some or all of the values depending on your needs.

The following table lists the properties and values that control the data recording process:

Overriding the default data recording properties

You can override the default data recording properties by creating a custom configuration file, through the REST API, or your application's user interface.

The priority of the properties override is the following (from the highest to the lowest priority):

1. Custom jfr.properties and *.jfc files (while all other default settings are ignored)

2. Settings modified in the REST API or UI runtime

3. Default settings from the embedded jfr.properties and default.jfc files

What properties can be overridden

The table shows the minimum and maximum values for properties that can be overridden.

 The values are stored in the database. The database tables differ depending on the product:

Overriding properties through the REST API

To override a property value through the REST API, call the following endpoint and pass a specific value in the request body:

PUT <domain>/jfr/properties/

For example:

curl --location --request PUT 'http://<host>:<port>/<product>/rest/troubleshooting/latest/jfr/properties/jfr.recording.max_size' \
--header 'Authorization: <Authorization>' \
--header 'Content-Type: text/plain' \
--header 'Cookie: <Cookie>' \
--data-raw '300'

Overriding properties through the user interface

To override a property value through the user interface:

1. In the upper-right corner of the screen, select Administration, and then System.
2. Under the System support  settings group, select Troubleshooting and support tools.
3. Select Diagnostic settings.

4. Under Java Flight Recorder settings, set the property values to your preference.

5. Select Apply.

Overriding data recording properties through a custom configuration file

To override the default data recording properties through a custom configuration file:

1. Create an empty jfr.properties file in the app's local home directory.

   In clustered Data Center configurations, copy the file to the shared home directory, where it can be accessed by all nodes in the system.

2. Edit the custom file by overriding all or only some of the properties depending on your needs.
   See the following example of the file content based on the default configuration:

   # How long data is to be kept in the disk repository before it is thrown away (in millis)
   jfr.recording.max_age=600000
   # The max size, measured in bytes, at which data will no longer be kept in the disk repository (in bytes)
   jfr.recording.max_size=52428800
   # How many JFR historical files are kept on the disk
   jfr.recording.files_to_remain=1
   # Path to store JFR recording files (relative to local app's home)
   jfr.recording.recordings_path=log/jfr
   # Path to store extracted thread dumps (relative to JFR path above)
   jfr.recording.threaddumps_path=threaddumps
   
   # Cron expression for periodical dump of JFR recording
   jfr.recording.dump_cron_expression=0 */5 * ? * *
   
   # Path to a custom JFR configuration template  (relative to an app's shared home)
   #jfr.configuration.template.path=jfr/custom.jfc

3. Edit the custom file by overriding all or only some of the properties depending on your needs.

4. Set the -Djfr.config.location=/<APP_HOME>/<FILE_NAME>.properties, where:
   • <APP_HOME> is the full path to your app's local or shared home directory
   • <FILE_NAME> is the name of the custom properties file.
     
5. Restart your app instance for the changes to take effect.
   Learn how to set properties and options on startup

Auditing JFR operations

The following JFR operations are audited:

Category: System

The operations respectively take place when:

• JFR starts recording when the application starts up.

• JFR is turned on.

• JFR is turned off.

Snapshot retention policy

When the Java Flight Recorder takes a snapshot of collected runtime data (either on schedule, when an admin creates a support zip), it also removes any previously saved recordings. By default, the tool retains only one snapshot at a time.

The number of snapshots to keep in storage is controlled by the jfr.default.recording.files_to_remain  property in the jfr.properties file. If you want to override the default value, see Overriding the default data recording properties.

Periodical snapshot files are named according to the following scheme:

atst_in_product_diagnostic_yyyy_MM_dd_HH_mm_ss.jfr

Snapshots taken on Java Virtual Machine shutdown are named according to the following scheme:

atst_in_product_diagnostic_dump_on_exit_yyyy_MM_dd_HH_mm_ss.jfr

Downloading the latest snapshot

By default, a snapshot of runtime events collected in the Java Flight Recorder buffer is automatically saved to a binary file every five minutes. The latest binary file becomes part of the support zip archive that you can send to Atlassian Support for assistance. To retrieve the file, create and download the support zip.

To download the snapshot:

1. In the upper-right corner of the screen, select Administration, and then System.
2. In the System  tab, select  Troubleshooting and support tools.
3. Select the Create support zip  tab.
4. Select  Create zip.
5. When the zip archive becomes available for download, select  Download zip.
6. Extract the snapshot and open it in JDK Mission Control to view the data in a human-friendly format.

Changing the snapshot creation frequency

By default, a snapshot of runtime events collected in the Java Flight Recorder buffer is automatically saved to a binary file every five minutes. You can change the frequency to a different value depending on your needs.

To change the frequency, edit the jfr.recording.dump_cron_expression property in your custom jfr.properties file in a text editor.