This article describes how to expose JMX MBeans within Jira for monitoring with a JMX client and how to use JMX MBeans for in-product diagnostics.

What is JMX?

JMX (Java Management Extensions) is a technology for monitoring and managing Java applications. JMX uses objects called MBeans (Managed Beans) to expose data and resources from your application. For large instances of Jira Server or Jira Data Center, enabling JMX allows you to more easily monitor the consumption of application resources and diagnose performance issues related to indexing. This enables you to make better decisions about how to maintain and optimize machine resources. 

Metrics collected by Jira

The following table lists metrics (MBeans) that are collected by Jira. All of them are grouped in the com.atlassian.jira property.

The following metrics have been added to Jira as of 8.1 and are exposed under com.atlassian.jira/metrics. All the following metrics will be reset after restarting Jira. 

Metrics collected by Assets in Jira Service Management

The following table lists metrics (MBeans) that are collected by Assets in Jira Service Management. 

Monitoring Jira

Before you can monitor Jira, you should enable JMX monitoring and then use a JMX client to view the metrics.

Enabling JMX monitoring in Jira

All of the metrics are collected by default, but you should enable JMX monitoring to expose them. You can do it in Jira but you must be a Jira admin.

1. From the top navigation bar select Administration  > System. 
2. Go to JMX monitoring.
3. Toggle Enable JMX monitoring.

Monitoring with JConsole

After you enabled JMX monitoring, you can use any JMX client to view the metrics. To make it quick and easy, we've described how to view them by using JConsole. You can monitor your Jira instance either locally or remotely:

• Monitoring Jira locally is good if you're troubleshooting a particular issue or only need to monitor Jira for a short time. Local monitoring can have a performance impact on your server, so it's not recommended for long-term monitoring of your production system.

  Show me how to do this...

   To monitor locally:

  1. Start JConsole. You'll find it in the bin directory of the JDK installation directory. (JConsole is only available as part of the JDK.)
  2. Select Local Process.
  3. Select the Jira process. It'll be called something like org.apache.catalina.startup.Bootstrap start
  4. After connecting, expand the com.atlassian.jira property that groups all the metrics.

  See Using JConsole for more information on local monitoring.

• Monitoring Jira remotely is recommended for production systems as it does not consume resources on your Jira server.

  Show me how to do this...

   To monitor remotely:

  1. Add the following properties to your setenv.sh / setenv.bat file. The port can be any port that is not in use.
     
     CATALINA_OPTS

     Windows

     SET CATALINA_OPTS="-Dcom.sun.management.jmxremote.authenticate=true ${CATALINA_OPTS}"
     SET CATALINA_OPTS="-Dcom.sun.management.jmxremote.password.file=/atlassian-jira-software-x.y.z-standalone/jmxremote.password ${CATALINA_OPTS}"
     SET CATALINA_OPTS="-Dcom.sun.management.jmxremote.access.file=/atlassian-jira-software-x.y.z-standalone/jmxremote.access ${CATALINA_OPTS}"

     x.y.z stands for the Jira version you're using.

     Linux

     CATALINA_OPTS="-Dcom.sun.management.jmxremote.authenticate=true ${CATALINA_OPTS}"
     CATALINA_OPTS="-Dcom.sun.management.jmxremote.password.file=/atlassian-jira-software-x.y.z-standalone/jmxremote.password ${CATALINA_OPTS}"
     CATALINA_OPTS="-Dcom.sun.management.jmxremote.access.file=/atlassian-jira-software-x.y.z-standalone/jmxremote.access ${CATALINA_OPTS}"

     x.y.z stands for the Jira version you're using.
     
     For more details, see the Using Password Authentication section in Monitoring and Management Using JMX Technology.
     
     Additionally, to access the JMX properties, you might need to configure SSL. For more details, see the SSL sections in Monitoring and Management Using JMX Technology.
     
     JAVA_OPTS

     Using JAVA_OPTS works to expose the JMX MBeans for remote access but causes errors to be thrown during Jira shutdown.

     Windows

     JAVA_OPTS=-Dcom.sun.management.jmxremote %JAVA_OPTS%
     JAVA_OPTS=-Dcom.sun.management.jmxremote.port=8099 %JAVA_OPTS%
     JAVA_OPTS=-Dcom.sun.management.jmxremote.authenticate=false %JAVA_OPTS%

     Linux

     JAVA_OPTS="-Dcom.sun.management.jmxremote ${JAVA_OPTS}"
     JAVA_OPTS="-Dcom.sun.management.jmxremote.port=8099 ${JAVA_OPTS}"
     JAVA_OPTS="-Dcom.sun.management.jmxremote.authenticate=false ${JAVA_OPTS}"
     export JAVA_OPTS

  2. Decide how you will secure your remote connection. See Remote Monitoring and Management for more information. 
     Although it is possible to disable authentication, we don't recommend doing this on a production system.
  3. Start JConsole. You'll find it in the bin directory of the JDK installation directory.
  4. Select Remote Process.
  5. Enter your hostname and port. This is the port you specified earlier, not the Jira port.
  6. Select Connect.
  7. After connecting, expand the com.atlassian.jira property that groups all the metrics.

  See Using JConsole for more information on remote monitoring.

Known security issues

We’re providing a robust fix for a potential security vulnerability that may be caused by an RCE (remote code execution) JMX attack. During this attack, a remote user with valid credentials for JMX monitoring can execute arbitrary code on Jira Data Center via Java Deserialization, even if this user’s account is readOnly (montiorRole).

To prevent fabricated data from getting into the system through requests, we're using a blocklist deserialization filter based on ObjectInputFilter from JVM.

If you use a custom JDK and miss appropriate classes in your classpath based on the Java version, your Jira node won’t be started.

atlassian.jira.log will contain the following error: BlocklistDeserializationFilter has not been set up. It means that your Java environment has some security issues.

In-product diagnostics available through JMX

Since Jira 9.3, we've introduced a set of database connectivity metrics for in-product diagnostics available through JMX.

In-product diagnostics (IPD) provides greater insights for you and our Support into how running instances are operating.

IPD uses additional metrics handling Jira’s interactions with its database. Using database connectivity metrics, you’ll efficiently identify what in your environment or infrastructure might cause the performance issues. 

In Jira 9.5, in-product diagnostics has been complemented with more new metrics: HTTP connection metrics and mail queue metrics. In Jira 9.8, we've added a few new mail queue metrics allowing you to get a more detailed picture of mail queue contents and to collect more data for better performance monitoring.

The feature is disabled by default. Live metrics are available in the following formats:

• as new JMX MBeans
• as a history of snapshots of the JMX values in the new IPD log file atlassian-jira-ipd-monitoring.log

The log file is available in the {jira_home}\log folder where you can find all the existing log files. The log file is also included in the Support Zip file, created in the ATST plugin. If needed, you can generate the Support Zip file in the Atlassian troubleshooting & support tools plugin and send the file to Atlassian Support, where we have internal tools to interpret it. Learn more about the plugin

Communication

The feature communicates in the following ways:

• JMX: JMX MBeans are updated periodically based on an internal schedule.
• The log file atlassian-jira-ipd-monitoring.log: JMX values are snapshotted and recorded to the log file on a configurable schedule. By default, the JMX values are polled and written to the log file every 60 seconds. (This parameter is up to date since Jira 9.3 EAP 02.)

In-product diagnostics metrics

Expand the following sections to learn more about the metrics available for in-product diagnostics.

Enabling In-product diagnostics monitoring

IPD monitoring is enabled by default. To manage it:

1. From the top navigation bar select Administration  > System. 

2. In the left-side panel, go to System Support and select Monitoring.

3. Use the In-product diagnostics monitoring toggle to enable or disable IPD monitoring.
   The toggle is also available in version 9.4.3. Check Jira Software release notes for updates.

REST API

In Jira 9.5, we've introduced a new REST API endpoint for managing the IPD monitoring, specifically the In-product diagnostics monitoring toggle in the user interface: /rest/api/2/monitoring/ipd.

{
  "enabled": true
}

{
  "enabled": false
}

Log formatting

Writing to atlassian-jira-ipd-monitoring.log is done via log4j. Its configuration is managed in log4j.properties.

#####################################################
# In-product diagnostics monitoring logging
#####################################################

log4j.appender.ipd=com.atlassian.jira.logging.JiraHomeAppender
log4j.appender.ipd.File=atlassian-jira-ipd-monitoring.log
log4j.appender.ipd.MaxFileSize=20480KB
log4j.appender.ipd.MaxBackupIndex=5
log4j.appender.ipd.layout=com.atlassian.logging.log4j.NewLineIndentingFilteringPatternLayout
log4j.appender.ipd.layout.ConversionPattern=%d %m%n

log4j.logger.ipd-monitoring = INFO, filelog
log4j.additivity.ipd-monitoring = false
log4j.logger.ipd-monitoring-data-logger = INFO, ipd
log4j.additivity.ipd-monitoring-data-logger = false

Log contents

By default, a concise set of data is included in each log entry. An extended set of data can be logged by enabling the com.atlassian.jira.in.product.diagnostics.extended.logging feature flag.

To enable the extended data:

1. Go to <JIRA_URL>/secure/admin/SiteDarkFeatures!default.jspa, where <JIRA_URL> is the base URL of your Jira instance.
2. In the Enable dark feature text area, enter com.atlassian.jira.in.product.diagnostics.extended.logging.enabled. Select Add. Learn how to manage dark features
   1. To disable the extended data, in the Site Wide Dark Features panel, find com.atlassian.jira.in.product.diagnostics.extended.logging.enabled and select Disable.

In the following tables, see the structures of the concise vs extended logging formats.

Concise data

2022-09-06 18:37:48,011 IPDMONITORING {"timestamp":"1662453468","label":"DB.CONNECTION.LATENCY.STATISTICS","attributes":{"_mean":"6.704470250010645E-25","_max":"1.0","_99thPercentile":"0.0","_min":"0.0"}}

Extended data

2022-09-06 18:38:48,015 IPDMONITORING {"timestamp":"1662453528","label":
"DB.CONNECTION.LATENCY.STATISTICS","objectName":
"com.atlassian.jira:category00\u003ddb,category01\u003dconnection,category02\
u003dlatency,name\u003dstatistics,type\u003dmetrics",
"attributes":{"_oneMinuteRate":"0.02012497818617073","_50thPercentile":"0.0",
"_mean":"1.9379304604014412E-25","_max":"1.0","_stdDev":"4.40219315841711E-13",
"_98thPercentile":"0.0","_meanRate":"0.003612560785169162","_rateUnit":
"events/second","_99thPercentile":"0.0","_count":"16","_durationUnit":
"milliseconds","_75thPercentile":"0.0","_fiveMinuteRate":
"0.005912972095043379","_fifteenMinuteRate":"0.0037696657500141968",
"_999thPercentile":"0.0","_95thPercentile":"0.0","_min":"0.0"}}

Definitions of metric attributes

Expand the following sections to learn more about metric attributes.

Processing properties

• JMX logging polling interval is set to 60 seconds and can't be modified.

• Log file polling interval is set to 60 seconds and can be changed by using the system property jira.diagnostics.ipdlog.poll.seconds.

• By default, the JMX values are polled and written to atlassian-jira-ipd-monitoring.log.