Diagnosing runtime issues using the Java Flight Recorder

Accurately diagnosing and troubleshooting problems with Java apps can be challenging and may require digging through gigabytes of logs to figure out what exactly went wrong and when. Instead of spending countless hours on manual investigation, you can use the Java Flight Recorder, which provides easy after-the-fact diagnostic and troubleshooting features, allowing you to quickly identify and resolve runtime problems as they occur.

The Java Flight Recorder uses a circular buffer to continuously collect events from the Java Virtual Machine runtime and periodically records a snapshot of the collected data to a binary file.

Additionally, you can include the data collected by the Java Flight Recorder in a support zip archive that you can send to Atlassian Support for assistance.

The Java Flight Recorder can take a snapshot automatically at specified intervals or when:

  • an admin creates a support zip with the In-app diagnostics option enabled on the Troubleshooting and support tools administration page
  • an admin disables the In-app diagnostics option on the Troubleshooting and support tools administration page or by making an appropriate call to the REST API
  • the Java Virtual Machine shuts down gracefully

You can review collected runtime data in JDK Mission Control, a free tool provided by Oracle that lets you efficiently and accurately analyze the data collected by the Java Flight Recorder.

Currently, the Java Flight Recorder in-app diagnostics are available in:

  • Jira Server and Data Center 8.13 or newer
  • Bitbucket Server and Data Center 8.3 or newer
  • Confluence Server and Data Center 7.4.0 or newer

Jira Server and Data Center

The Java Flight Recorder

  • In Jira Server and Data Center 9.2, the Java Flight recorder is bundled with the software and continuous data recording begins as soon as you start up your instance.

  • In Jira Server and Data Center 9.0 and 9.1, the Java Flight Recorder is active by default and continuous data recording begins as soon as you start up your Jira instance.

  • In Jira Server and Data Center releases 8.13 through 9.0, the Java Flight Recorder is an opt-in feature. To use it, upgrade the Troubleshooting and Support Tools app to version 1.39.0 from the Atlassian Marketplace.

    In Jira Server and Data Center 9.0, the recording begins automatically once you upgrade Troubleshooting and Support Tools to version 1.39.0. In earlier Jira releases, you have to start the recording manually.

Take a look at the following table for a summary of the information above:

Jira releaseFeature availabilityBundled with JiraAutomatic recording
8.12 and older(error)N/AN/A
8.13–8.22(tick) (opt-in)(error)(error)
9.0(tick) (opt-in)(error)(tick)
9.1 and newer(tick)(tick)(tick)

Bitbucket Server and Data Center

In Bitbucket Server and Data Center 8.3 or newer, the Java Flight recorder is bundled with the software and continuous data recording begins as soon as you start up your instance.

Confluence Server and Data Center

  • In Confluence Server and Data Center 7.20, the Java Flight Recorder is bundled with the software and continuous data recording begins as soon as you start up your instance.

  • In Confluence Server and Data Center releases 7.4.0 through 7.20, the Java Flight Recorder is an opt-in feature. To use it, upgrade the Troubleshooting and Support Tools app to version 1.41.0 from the Atlassian Marketplace.

    In Confluence Server and Data Center 8.0, the recording begins automatically once you upgrade Troubleshooting and Support Tools to version 1.41.0. In earlier Confluence releases, you have to start the recording manually.

On this page:

System requirements

To use the Java Flight Recorder, make sure that you are running Jira Server and Data Center 8.13 or newer, or Bitbucket Server and Data Center 8.3 or newer, and that your instance meets the following Java version requirements:

Oracle JRE, JDK:

  • Java 8
  • Java 11

In Oracle Java versions earlier than 11, the Java Flight Recorder is available only in commercial packages based on the Java Platform, Standard Edition (Oracle Java SE Advanced and the Oracle Java SE Suite). Enabling the feature requires setting the -XX:+UnlockCommercialFeatures startup option. Learn how to set properties and options on startup

OpenJDK:

  • Java 1.8 (build 8u282 or newer)
  • Java 11 (build 11.0.5 or newer)

Supported products versions

Check Atlassian’s products that support JFR in-app diagnostics and the products' versions where the feature is available:

Product and supported version

JFR active by default

JFR recording starts automatically

JFR bundled

Jira 8.13 and newer

8.13 and newer

9.0 and newer

9.2

Bitbucket 8.3 and newer

7.17 and newer

8.3 and newer

N/A

Confluence 7.4.0 and newer

7.4.0 and newer

7.20.0

7.20.0

Enabling continuous data recording

Your data recording is started automatically in:

  • Jira Data Center and Server 9.0 and newer
  • Bitbucket Data Center and Server 8.3 or newer
  • Confluence Data Center and Server 7.20 and newer

However, if you're running an earlier release of Jira (at least 8.13), make sure to enable in-app diagnostics to allow the Java Flight Recorder to record runtime data continuously.

To enable continuous runtime data recording:

  1. From the app header, select Administration > System.
  2. From the left menu, select Troubleshooting and support tools.
  3. On the Troubleshooting and support tools page, enable In-app diagnostics.

Default event collection settings

We've based the default event collection settings on the default Java Flight Recorder configuration included in the Oracle JDK. While we left most of the default configuration included as part of the Oracle JDK unchanged, there are some essential differences; we've disabled gathering garbage collection events and changed thread dump events to be collected every 10 seconds.

The default event collection settings are listed in the default.jfc template file in the <APP_HOME>/jfr directory. You can use the template to create your own custom configuration. 

The following table lists the configuration of event type collection in the template file:

OptionValue
Garbage collectorOff
Memory profilingOff
CompilerNormal
Method samplerEvery 10 s
Thread dumpErrors only
ExceptionsObject types
Memory leak detection20 ms
Synchronization threshold20 ms
File I/O threshold20 ms
Socket I/O threshold20 ms

Overriding the default event collection settings

If you want to override the default event collection settings, you can create a custom configuration file based on the default.jfc template file located at <APP_HOME>/jfr.

To override the default event collection settings:

  1. In JDK Mission Control, create a custom Java Flight Recorder configuration based on the included template and save it under a unique name with the .jfc extension (for example, custom.jfc.)
  2. Copy the file to the <APP_HOME>/jfr/ directory.

    In clustered Data Center configurations, copy the file to the shared home directory, where it can be accessed by all nodes in the system.

  3. Edit the <APP_HOME>/jfr/jfr.properties file by uncommenting the jfr.configuration.template.path property and changing the value to the path to your custom template.
  4. Restart your app instance for the changes to take effect.

Default data recording properties

Data recording properties such as thresholds, target file paths, or the number of snapshot files to retain are defined in the <APP_HOME>/jfr/jfr.properties file. You can change some or all of the values depending on your needs.

The following table lists the properties and values that control the data recording process:

PropertyValueDescription
jfr.recording.max_age600000The maximum snapshot file age threshold (in milliseconds), beyond which the snapshot is deleted.
jfr.recording.max_size52428800The maximum snapshot size threshold (in bytes), beyond which the snapshot is deleted.
jfr.recording.files_to_remain1The number of previously captured snapshots to retain.
jfr.recording.recordings_pathlog/jfrThe location where snapshots are saved when the recording stops or a snapshot is taken.
jfr.recording.threaddumps_paththreaddumpsThe location where thread dumps are extracted relative to the app's local home path.
jfr.recording.dump_cron_expression0 */5 * ? * *A CRON expression controlling the snapshot creation frequency (by default, every five minutes).
jfr.configuration.template.pathCommented out by defaultThe location of the custom Java Flight Recorder event collection configuration template.

To reduce memory overhead, the Java Virtual Machine doesn't stop all threads immediately when the threshold is reached. This means that the stored data may exceed the configured maximum age and maximum size property values. If your system's resources are under heavy load, the total amount of recorded data may be considerably greater than the thresholds defined in the properties.

Overriding the properties through REST API and user interface

You can override the most used properties through REST API endpoints or the user interface, instead of restarting your instance and editing the .jfc template.

New properties won’t be applied to a currently running recording. If you want to apply them, stop the current recording and start a new one.

The priority of the properties override is the following (from the highest to the lowest priority):

  1. Custom config file (.jfc), while all other default settings are ignored

  2. Settings modified in the REST API or UI runtime

  3. Default settings from the embedded jfr.properties and default.jfc files

What properties can be overridden and how

The table shows the minimum and maximum values for properties that can be overridden.

Property name

Description and units

Minimum value

Maximum value

jfr.recording.max.age

Indicates how long data should be kept in the disk repository before it is thrown away (milliseconds)

10 minutes

5 days

jfr.recording.max.size

The maximum size of data that can be kept in the disk repository (bytes)

10 MB

2GB

jfr.thread.dump.interval

The interval in which a thread dump is captured (milliseconds)

1 second

10 minutes

jfr.configuration.template.path

The path to a custom configuration template

N/A

N/A

 The values are stored in the database. The database tables differ depending on the product:

Product

Database table

Jira

propertyentry

Confluence

bandana

Bitbucket

plugin_setting

Overriding properties through REST API

Properties should be overridden one by one, but not in a batch.

The following endpoint can be used to modify property values:
PUT <domain>/jfr/properties/{propertyName},
where specific value should be passed inside the request body.

See the example
curl --location --request PUT 'http://<host>:<port>/<product>/rest/troubleshooting/latest/jfr/properties/jfr.recording.max_size' \
--header 'Authorization: <Authorization>' \
--header 'Content-Type: text/plain' \
--header 'Cookie: <Cookie>' \
--data-raw '300'

Overriding properties through the user interface

This instruction is for Jira Server and Data Center. In Confluence and Bitbucket, the setting menus may have different names.

To use the UI for properties override:

  1. Error rendering macro 'excerpt-include'

    User 'null' does not have permission to view the page.

  2. Under SYSTEM SUPPORT settings group, select Troubleshooting and support tools.
  3. Go to Diagnostic settings.
  4. Under Java Flight Recorder settings, you’ll find the available properties. Set the values to your preference.

  5. Select Apply.

You won’t see these settings if the Java Runtime Environment isn’t compatible with the JFR.

After you select Apply and the properties are updated, current JFR recording will stop automatically and restart with new property values.

Overriding the default data recording properties

To override the default data recording properties:

  1. Create a custom jfr.properties file in the <APP_HOME>/jfr/ directory by copying the existing file and saving it under a new name.

    In clustered Data Center configurations, copy the file to the shared home directory, where it can be accessed by all nodes in the system.

  2. Edit the custom file by overriding all or only some of the properties depending on your needs.
  3. Set the -Djfr.config.location=/<APP_HOME>/jfr/<FILE_NAME>.properties, where <APP_HOME> is the full path to your app's local or shared home directory and <FILE_NAME> is the name of the custom properties file.
  4. Restart your app instance for the changes to take effect. Learn how to set properties and options on startup

Auditing JFR operations

The following JFR operations are audited:

Category: System

Coverage level

Events logged

Base

  • JFR recording started

  • JFR recording stopped

The operations respectively take place when:

  • JFR starts recording when the application starts up.

  • JFR is turned on.

  • JFR is turned off.

Snapshot retention policy

When the Java Flight Recorder takes a snapshot of collected runtime data (either on schedule, when an admin creates a support zip), it also removes any previously saved recordings. By default, the tool retains only one snapshot at a time.

The number of snapshots to keep in storage is controlled by the jfr.default.recording.files_to_remain  property in the jfr.properties file. If you want to override the default value, see 

Periodical snapshot files are named according to the following scheme:

atst_in_product_diagnostic_yyyy_MM_dd_HH_mm_ss.jfr

Snapshots taken on Java Virtual Machine shutdown are named according to the following scheme:

atst_in_product_diagnostic_dump_on_exit_yyyy_MM_dd_HH_mm_ss.jfr

Downloading the latest snapshot

By default, a snapshot of runtime events collected in the Java Flight Recorder buffer is automatically saved to a binary file every five minutes. The latest binary file becomes part of the support zip archive that you can send to Atlassian Support for assistance. To retrieve the file, create and download the support zip.

Only the latest snapshot file is included in the support zip. You can find the file in the jfr-bundle directory. Other files (for example, snapshots created on exit and older periodical files) remain on disk storage and don't become part of the support zip.

To download the snapshot:

  1. From the app header, select Administration > System.
  2. In the System tab, select Troubleshooting and support tools.
  3. Select the Create support zip tab.
  4. Select Create zip.
  5. When the zip archive becomes available for download, select Download zip.
  6. Extract the snapshot and open it in JDK Mission Control to view the data in a human-friendly format.

Changing the snapshot creation frequency

By default, a snapshot of runtime events collected in the Java Flight Recorder buffer is automatically saved to a binary file every five minutes. You can change the frequency to a different value depending on your needs.

To change the frequency, edit the jfr.default.recording.dump_cron_expression property in the <APP_HOME>/jfr/jfr.properties file in a text editor or by opening it in JDK Mission Control.


Last modified on Sep 13, 2022

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.