Diagnosing runtime issues using the Java Flight Recorder

Accurately diagnosing and troubleshooting problems with Java apps can be challenging and may require digging through gigabytes of logs to figure out what exactly went wrong and when. Instead of spending countless hours on manual investigation, you can use the Java Flight Recorder, which provides easy after-the-fact diagnostic and troubleshooting features, allowing you to quickly identify and resolve runtime problems as they occur.

The Java Flight Recorder uses a circular buffer to continuously collect events from the Java Virtual Machine runtime and periodically records a snapshot of the collected data to a binary file.

Additionally, you can include the data collected by the Java Flight Recorder in a support zip archive that you can send to Atlassian Support for assistance.

The Java Flight Recorder can take a snapshot automatically at specified intervals or when:

  • an admin creates a support zip with the In-app diagnostics option enabled on the Troubleshooting and support tools administration page

  • an admin disables the In-app diagnostics option on the Troubleshooting and support tools administration page or by making an appropriate call to the REST API

  • the Java Virtual Machine shuts down gracefully

You can review collected runtime data in JDK Mission Control, a free tool provided by Oracle that lets you efficiently and accurately analyze the data collected by the Java Flight Recorder.

Supported Data Center and Server products

The Java Flight Recorder in-app diagnostics are available in the following Atlassian Data Center and Server products:

  • Jira Data Center and Server 8.13 and later

  • Bitbucket Data Center and Server 8.3 and later

  • Confluence Data Center and Server 7.4.0 and later

  • Bamboo Data Center 9.6.2 and later

Jira Data Center and Server

The Java Flight Recorder supports the following releases of Jira Server

  • In Jira Data Center and Server 9.2 and later, the Java Flight recorder is bundled with the software. Continuous data recording begins as soon as you start up your instance.

  • In Jira Data Center and Server 9.0 and 9.1, the Java Flight Recorder is active by default. Continuous data recording begins as soon as you start up your Jira instance.

  • In Jira Data Center and Server releases 8.13 through 9.0, the Java Flight Recorder is an opt-in feature. To use it, upgrade the Troubleshooting and Support Tools.

In Jira Server and Data Center 9.0, the recording begins automatically once you upgrade Troubleshooting and Support Tools to version 1.39.0. In earlier Jira releases, you have to start the recording manually.

Take a look at the following table for a summary of the information above:

Jira releaseFeature availabilityBundled with JiraAutomatic recording
8.12 and older(error)N/AN/A
8.13–8.22(tick) (opt-in)(error)(error)
9.0(tick) (opt-in)(error)(tick)
9.1 and later(tick)(tick)(tick)

Bitbucket Data Center and Server

In Bitbucket Data Center and Server 8.3 and later, the Java Flight recorder is bundled with the software. Continuous data recording begins as soon as you start up your instance.

Confluence Data Center and Server

  • In Confluence Data Center and Server 7.20 and later, the Java Flight Recorder is bundled with the software. Continuous data recording begins as soon as you start up your instance.

  • In Confluence Data Center and Server releases 7.4.0 through 7.20, the Java Flight Recorder is an opt-in feature. To use it, upgrade the Troubleshooting and Support Tools.

In Confluence Server and Data Center 8.0, the recording begins automatically once you upgrade Troubleshooting and Support Tools to version 1.41.0. In earlier Confluence releases, you have to start the recording manually.

Bamboo Data Center

In Bamboo Data Center 9.6.2 and later, the Java Flight recorder is bundled with the software. Continuous data recording begins as soon as you start up your instance.

On this page:

System requirements

To use the Java Flight Recorder, make sure that you are running a supported Data Center and Server product, and that your instance meets the following Java version requirements:


Oracle JRE & JDK

  • Java 11 (build 11.0.5 or newer)

OpenJDK

  • Java 1.8 (build 8u282 or newer)

  • Java 11 (build 11.0.5 or newer)

Before the recording is started, make sure your disk has enough free space to store JFR dumps.

The space required for the recording is calculated according to the following formula: jfr.recording.max_size * jfr.recording.files_to_remain.

To reduce the space that the recording will take, edit the properties jfr.recording.max_size and jfr.recording.files_to_remain as needed. Learn how to edit the properties

Enabling continuous data recording

Your data recording is started automatically in the following Data Center applications:

  • Jira Data Center and Server 9.0 and later

  • Bitbucket Data Center and Server 8.3 and later

  • Confluence Data Center and Server 7.20 and later

  • Bamboo Data Center 9.6.2 and later

If you're running an earlier release of any of those products, make sure to enable in-app diagnostics to allow the Java Flight Recorder to continuously record runtime data.

To enable continuous runtime data recording:

  1. In the upper-right corner of the screen, select Administration, and then System.
  2. From the left menu, select Troubleshooting and support tools.
  3. On the Troubleshooting and support tools page, enable  In-app diagnostics.

Default event collection settings

We've based the default event collection settings on the default Java Flight Recorder configuration included in the Oracle JDK. While we left most of the default configuration unchanged, we’ve disabled gathering garbage collection events and changed thread dump events to be collected every 10 seconds.

All active event collection settings are listed in the active_configuration.jfc file. This file contains default settings if no custom configuration is applied, so you can use it as a template to create your own custom configuration.

The following table lists the configuration of event type collection in the template file:

OptionValue
Garbage collectorOff
Memory profilingOff
CompilerNormal
Method samplerEvery 10 s
Thread dumpErrors only
ExceptionsObject types
Memory leak detection20 ms
Synchronization threshold20 ms
File I/O threshold20 ms
Socket I/O threshold20 ms

Overriding the default event collection settings

If you want to override the default event collection settings, you can create a custom configuration based on the active_configuration.jfc template file located at <APP_HOME>/log/jfr.

To override the default event collection settings:

  1. In JDK Mission Control, create a custom Java Flight Recorder configuration based on the included template and save it under a unique name with the .jfc extension (for example, custom.jfc.)
  2. Copy the file to the app's local home directory.

    In clustered Data Center configurations, copy the file to the shared home directory, where it can be accessed by all nodes in the system.

  3. If not already created, create a custom jfr.properties file.

  4. Edit the jfr.properties file by uncommenting the jfr.configuration.template.path property and changing the value to the path to your custom template.
  5. Restart your app instance for the changes to take effect.

Default data recording settings

Data recording settings such as thresholds, target file paths, or the number of snapshot files to retain are defined in the embedded jfr.properties file. You can change some or all of the values depending on your needs.

The following table lists the properties and values that control the data recording process:

PropertyValueDescription
jfr.recording.max_age600000The maximum snapshot file age threshold (in milliseconds), beyond which the snapshot is deleted.
jfr.recording.max_size52428800The maximum snapshot size threshold (in bytes), beyond which the snapshot is deleted.
jfr.recording.files_to_remain1The number of previously captured snapshots to retain.
jfr.recording.recordings_pathlog/jfrThe location where snapshots are saved when the recording stops or a snapshot is taken.
jfr.recording.threaddumps_paththreaddumpsThe location where thread dumps are extracted relative to the app's local home path.
jfr.recording.dump_cron_expression0 */5 * ? * *A CRON expression controlling the snapshot creation frequency (by default, every five minutes).
jfr.configuration.template.pathCommented out by defaultThe location of the custom Java Flight Recorder event collection configuration template.

To reduce memory overhead, the Java Virtual Machine doesn't stop all threads immediately when the threshold is reached. This means that the stored data may exceed the configured maximum age and maximum size property values. If your system's resources are under heavy load, the total amount of recorded data may be considerably greater than the thresholds defined in the properties.

Overriding the default data recording properties

You can override the default data recording properties by creating a custom configuration file, through the REST API, or your application's user interface.

tip/resting Created with Sketch.

Instead of restarting your instance every time you edit the custom file, override the most frequently used properties by using the available REST API endpoints or the user interface (preferred).

New property values won't be applied to a running recording. If you want to apply the new values, stop the current recording and start a new one.

The priority of the properties override is the following (from the highest to the lowest priority):

  1. Custom jfr.properties and *.jfc files (while all other default settings are ignored)

  2. Settings modified in the REST API or UI runtime

  3. Default settings from the embedded jfr.properties and default.jfc files

What properties can be overridden

The table shows the minimum and maximum values for properties that can be overridden.

Property name

Description and units

Minimum value

Maximum value

jfr.recording.max.age

Indicates how long data should be kept in the disk repository before it is thrown away (milliseconds)

10 minutes

5 days

jfr.recording.max.size

The maximum size of data that can be kept in the disk repository (bytes)

10 MB

2GB

jfr.thread.dump.interval

The interval in which a thread dump is captured (milliseconds)

1 second

10 minutes

jfr.configuration.template.path

The path to a custom configuration template

N/A

N/A

 The values are stored in the database. The database tables differ depending on the product:

Product

Database table

Jira

propertyentry

Confluence

bandana

Bitbucket

plugin_setting

Overriding properties through the REST API

To override a property value through the REST API, call the following endpoint and pass a specific value in the request body:

PUT <domain>/jfr/properties/

Don’t override multiple property values in a batch. Instead, override each property individually.

For example:

curl --location --request PUT 'http://<host>:<port>/<product>/rest/troubleshooting/latest/jfr/properties/jfr.recording.max_size' \
--header 'Authorization: <Authorization>' \
--header 'Content-Type: text/plain' \
--header 'Cookie: <Cookie>' \
--data-raw '300'

Overriding properties through the user interface

To override a property value through the user interface:

These instructions are for Jira Data Center and Server. In Confluence and Bitbucket, the settings and menus may have different names.

  1. In the upper-right corner of the screen, select Administration, and then System.
  2. Under the System support  settings group, select Troubleshooting and support tools.
  3. Select Diagnostic settings.
  4. Under Java Flight Recorder settings, set the property values to your preference.

  5. Select Apply.

You won't see these settings if the Java Runtime Environment isn't compatible with the JFR.

After you select Apply, the current JFR recording will stop and automatically restart with new property values applied.

Overriding data recording properties through a custom configuration file

To override the default data recording properties through a custom configuration file:

  1. Create an empty jfr.properties file in the app's local home directory.

    In clustered Data Center configurations, copy the file to the shared home directory, where it can be accessed by all nodes in the system.

  2. Edit the custom file by overriding all or only some of the properties depending on your needs.
    See the following example of the file content based on the default configuration:

    # How long data is to be kept in the disk repository before it is thrown away (in millis)
    jfr.recording.max_age=600000
    # The max size, measured in bytes, at which data will no longer be kept in the disk repository (in bytes)
    jfr.recording.max_size=52428800
    # How many JFR historical files are kept on the disk
    jfr.recording.files_to_remain=1
    # Path to store JFR recording files (relative to local app's home)
    jfr.recording.recordings_path=log/jfr
    # Path to store extracted thread dumps (relative to JFR path above)
    jfr.recording.threaddumps_path=threaddumps
    
    # Cron expression for periodical dump of JFR recording
    jfr.recording.dump_cron_expression=0 */5 * ? * *
    
    # Path to a custom JFR configuration template  (relative to an app's shared home)
    #jfr.configuration.template.path=jfr/custom.jfc
  3. Edit the custom file by overriding all or only some of the properties depending on your needs.

  4. Set the -Djfr.config.location=/<APP_HOME>/<FILE_NAME>.properties, where:
    • <APP_HOME> is the full path to your app's local or shared home directory
    • <FILE_NAME> is the name of the custom properties file.
  5. Restart your app instance for the changes to take effect.
    Learn how to set properties and options on startup

Auditing JFR operations

The following JFR operations are audited:

Category: System

Coverage level

Events logged

Base

  • JFR recording started

  • JFR recording stopped

The operations respectively take place when:

  • JFR starts recording when the application starts up.

  • JFR is turned on.

  • JFR is turned off.

Snapshot retention policy

When the Java Flight Recorder takes a snapshot of collected runtime data (either on schedule, when an admin creates a support zip), it also removes any previously saved recordings. By default, the tool retains only one snapshot at a time.

The number of snapshots to keep in storage is controlled by the jfr.default.recording.files_to_remain  property in the jfr.properties file. If you want to override the default value, see Overriding the default data recording properties.

Periodical snapshot files are named according to the following scheme:

atst_in_product_diagnostic_yyyy_MM_dd_HH_mm_ss.jfr

Snapshots taken on Java Virtual Machine shutdown are named according to the following scheme:

atst_in_product_diagnostic_dump_on_exit_yyyy_MM_dd_HH_mm_ss.jfr

Downloading the latest snapshot

By default, a snapshot of runtime events collected in the Java Flight Recorder buffer is automatically saved to a binary file every five minutes. The latest binary file becomes part of the support zip archive that you can send to Atlassian Support for assistance. To retrieve the file, create and download the support zip.

Only the latest snapshot file is included in the support zip. You can find the file in the jfr-bundle directory. Other files (for example, snapshots created on exit and older periodical files) remain on disk storage and don't become part of the support zip.

To download the snapshot:

  1. In the upper-right corner of the screen, select Administration, and then System.
  2. In the System  tab, select  Troubleshooting and support tools.
  3. Select the Create support zip  tab.
  4. Select  Create zip.
  5. When the zip archive becomes available for download, select  Download zip.
  6. Extract the snapshot and open it in JDK Mission Control to view the data in a human-friendly format.

Changing the snapshot creation frequency

By default, a snapshot of runtime events collected in the Java Flight Recorder buffer is automatically saved to a binary file every five minutes. You can change the frequency to a different value depending on your needs.

To change the frequency, edit the jfr.recording.dump_cron_expression property in your custom jfr.properties file in a text editor.


Last modified on May 6, 2024

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.