Live monitoring using the JMX interface

On this page

Still need help?

The Atlassian Community is here for you.

Ask the community

This article describes how to expose JMX MBeans within Jira for monitoring with a JMX client and how to use JMX MBeans for in-product diagnostics.

This guide provides a basic introduction to the JMX interface and is provided as is. Our support team can help you troubleshoot a specific Jira problem but can't help you set up your monitoring system or interpret the results.

On this page:

This article describes how to expose JMX MBeans within Jira for monitoring with a JMX client and how to use JMX MBeans for in-product diagnostics.

What is JMX?

JMX (Java Management Extensions) is a technology for monitoring and managing Java applications. JMX uses objects called MBeans (Managed Beans) to expose data and resources from your application. For large instances of Jira Server or Jira Data Center, enabling JMX allows you to more easily monitor the consumption of application resources and diagnose performance issues related to indexing. This enables you to make better decisions about how to maintain and optimize machine resources. 

Details of the JMX metrics available in Jira

JMX attribute name

Description

50thPercentile

The value at the 50th percentile in the distribution of measured times (the median value)

75thPercentile

The value at the 75th percentile in the distribution of measured times

95thPercentile

The value at the 95th percentile in the distribution of measured times

98thPercentile

The value at the 98th percentile in the distribution of measured times

99thPercentile

The value at the 99th percentile in the distribution of measured times

999thPercentile

The value at the 99.9th percentile in the distribution of measured times

Count

The number of invocations since node startup

DurationUnit

The time unit used to report percentile values, min, max, mean and standard deviation. The default unit is milliseconds.

FifteenMinuteRate

The fifteen-minute moving average rate of invocations since node startup. This rate uses the same exponential decay factor as is used for the fifteen-minute load average in Unix’s top command.

FiveMinuteRate

The five-minute moving average rate of invocations since node startup. This rate uses the same exponential decay factor as is used for the five-minute load average in Unix’s top command.

Max

The highest measured time since node startup

Mean

The mean measured time since node startup

MeanRate

The mean rate of invocations since node startup

Min

The lowest measured time since node startup

OneMinuteRate

The one-minute moving average rate of invocations since node startup. This rate uses the same exponential decay factor as is used for the one-minute load average in Unix’s top command.

RateUnit

The unit in which MeanRate, OneMinuteRate, FiveMinuteRate, and FifteenMinuteRate are reported. The default unit is events/second.

StdDev

The standard deviation in the measured times since node startup

Metrics collected by Jira

The following table lists metrics (MBeans) that are collected by Jira. All of them are grouped in the com.atlassian.jira property.

MetricDescriptionReset after restarting Jira
dashboard.view.count

The number of times all dashboards were viewed by users

Yes
entity.attachments.totalThe number of attachmentsN/A
entity.components.totalThe number of componentsN/A
entity.customfields.totalThe number of custom fieldsN/A
entity.filters.totalThe number of filtersN/A
entity.groups.totalThe number of user groupsN/A
entity.issues.totalThe number of issuesN/A
entity.users.totalThe number of usersN/A
entity.versions.totalThe number of versions createdN/A
issue.assigned.count

The number of times issues were assigned or reassigned to users (counts each action)

Yes
issue.created.count

The number of issues that you created after starting your Jira instance

Yes
issue.link.count

The number of issue links created after starting your Jira instance

Yes
issue.search.count

The number of times you searched for issues

Yes
issue.updated.count

The number of times you updated issues (each update after adding or changing some information)

Yes
issue.worklogged.count

The number of times you logged work on issues

Yes
jira.licenseThe types of licenses you have, the number of active users, and the maximum number of users available for each license typeN/A
quicksearch.concurrent.search

The number of concurrent searches that are being performed in real-time by using the quick search. You can use it to determine whether the limit set for concurrent searches is sufficient or should be increased.

Yes
web.requests

The number of requests (invocation.count) and the total response time (total.elapsed.time)

Yes

The following metrics have been added to Jira as of 8.1 and are exposed under com.atlassian.jira/metrics. All the following metrics will be reset after restarting Jira. 

Metric path

Description

comment

Metrics related to comment operations

comment/Create

A comment being created

comment/Delete

A comment being deleted

comment/Update

A comment being updated

indexing

Metrics related to issue, comment, worklog, and change indexing

indexing/CreateChangeHistoryDocument

Index documents being created for Change History entities. Note that many Change History Documents can be created for every issue.

indexing/CreateCommentDocument

Index documents being created for Comment entities

indexing/CreateIssueDocument

Index documents being created for Issue entities

indexing/IssueAddFieldIndexers

FieldIndexer modules enrich Issue documents as part of Index document creation. Plugins can register custom FieldIndexer modules. These metrics provide insight into how much time is spent in FieldIndexer and can be used to track down indexing performance issues caused by them. The metrics describe how much time was spent in all FieldIndexers combined per Issue document created.

indexing/LuceneAddDocument

How much time was spent adding a document to the Lucene index

indexing/LuceneDeleteDocument

How much time was spent deleting one or more documents matching a term from the Lucene index

indexing/LuceneOptimize

Metrics about Lucene index optimization (triggered manually from Jira)

indeding/LuceneUpdateDocument

How much time was spent adding a created document to the Lucene index

indexing/ReplicationLatency

Replication latency is the time between an issue, comment, or worklog being indexed on the node where the change was made and the indexing operation being replayed on the current node

indexing/WaitForLucene

Documents are written to the Lucene index asynchronously. This metric captures how much time Jira’s indexing thread spent waiting for Lucene to complete the write.

indexing/issueAddSearchExtractors

EntitySearchExtractor enriches issue documents as part of Index document creation. Plugins can register custom EntitySearchExtractor modules. These metrics provide insight into how much time is spent in EntitySearchExtractor and can be used to track down indexing performance issues they cause. The metrics describe how much time was spent in all EntitySearchExtractors combined per issue document created.

issue

Metrics about issue operations

issue/Create

Issue being created

issue/Delete

Issue being deleted

issue/Index

An issue being added to the Lucene index. This covers issue document creation and adding the document to the index.

issue/DeIndex

An issue being removed from the Lucene index

issue/ReIndex

An issue being re-indexed as a result of issue updates. This covers: creating an issue document, deleting the old document from the index, and adding new documents to the index.

issue/Update

An issue being updated

Monitoring Jira

Before you can monitor Jira, you should enable JMX monitoring and then use a JMX client to view the metrics.

Good to know

Viewing the metrics will always have some performance impact on Jira. We recommend that you don't refresh them more than once a second.

Enabling JMX monitoring in Jira

All of the metrics are collected by default, but you should enable JMX monitoring to expose them. You can do it in Jira but you must be a Jira admin.

  1. In Jira, go to Administration (> System > JMX monitoring.
  2. Toggle Enable JMX monitoring.

Monitoring with JConsole

After you enabled JMX monitoring, you can use any JMX client to view the metrics. To make it quick and easy, we've described how to view them by using JConsole. You can monitor your Jira instance either locally or remotely:

  • Monitoring Jira locally is good if you're troubleshooting a particular issue or only need to monitor Jira for a short time. Local monitoring can have a performance impact on your server, so it's not recommended for long-term monitoring of your production system.

    Show me how to do this...

     To monitor locally:

    1. Start JConsole. You'll find it in the bin directory of the JDK installation directory. (JConsole is only available as part of the JDK.)
    2. Select Local Process.
    3. Select the Jira process. It'll be called something like org.apache.catalina.startup.Bootstrap start
    4. After connecting, expand the com.atlassian.jira property that groups all the metrics.

    See Using JConsole for more information on local monitoring.

  • Monitoring Jira remotely is recommended for production systems as it does not consume resources on your Jira server.

    Show me how to do this...

     To monitor remotely:

    1. Add the following properties to your setenv.sh / setenv.bat file. The port can be any port that is not in use.

      CATALINA_OPTS

      Windows
      SET CATALINA_OPTS="-Dcom.sun.management.jmxremote.authenticate=true ${CATALINA_OPTS}"
      SET CATALINA_OPTS="-Dcom.sun.management.jmxremote.password.file=/atlassian-jira-software-x.y.z-standalone/jmxremote.password ${CATALINA_OPTS}"
      SET CATALINA_OPTS="-Dcom.sun.management.jmxremote.access.file=/atlassian-jira-software-x.y.z-standalone/jmxremote.access ${CATALINA_OPTS}"

      (info) x.y.z stands for the Jira version you're using.

      Linux
      CATALINA_OPTS="-Dcom.sun.management.jmxremote.authenticate=true ${CATALINA_OPTS}"
      CATALINA_OPTS="-Dcom.sun.management.jmxremote.password.file=/atlassian-jira-software-x.y.z-standalone/jmxremote.password ${CATALINA_OPTS}"
      CATALINA_OPTS="-Dcom.sun.management.jmxremote.access.file=/atlassian-jira-software-x.y.z-standalone/jmxremote.access ${CATALINA_OPTS}"

      (info) x.y.z stands for the Jira version you're using.

      For more details, see the Using Password Authentication section in Monitoring and Management Using JMX Technology.

      Additionally, to access the JMX properties, you might need to configure SSL. For more details, see the SSL sections in Monitoring and Management Using JMX Technology.

      JAVA_OPT

      Using JAVA_OPTS works to expose the JMX MBeans for remote access but causes errors to be thrown during Jira shutdown.

      Windows
      JAVA_OPTS=-Dcom.sun.management.jmxremote %JAVA_OPTS%
      JAVA_OPTS=-Dcom.sun.management.jmxremote.port=8099 %JAVA_OPTS%
      JAVA_OPTS=-Dcom.sun.management.jmxremote.authenticate=false %JAVA_OPTS%
      Linux
      JAVA_OPTS="-Dcom.sun.management.jmxremote ${JAVA_OPTS}"
      JAVA_OPTS="-Dcom.sun.management.jmxremote.port=8099 ${JAVA_OPTS}"
      JAVA_OPTS="-Dcom.sun.management.jmxremote.authenticate=false ${JAVA_OPTS}"
      export JAVA_OPTS
    2. Decide how you will secure your remote connection. See Remote Monitoring and Management for more information. 
      Although it is possible to disable authentication, we don't recommend doing this on a production system.
    3. Start JConsole. You'll find it in the bin directory of the JDK installation directory.
    4. Select Remote Process.
    5. Enter your hostname and port. This is the port you specified earlier, not the Jira port.
    6. Select Connect.
    7. After connecting, expand the com.atlassian.jira property that groups all the metrics.

    See Using JConsole for more information on remote monitoring.

In-product diagnostics with database connectivity metrics available through JMX

Since version 9.3, we've introduced a set of database connectivity metrics for in-product diagnostics available through JMX.

In-product diagnostics (IPD) provides greater insights for you and our Support into how running instances are operating.

IPD provides additional metrics handling Jira’s interactions with its database. Using database connectivity metrics, you’ll efficiently identify what in your environment or infrastructure might cause the performance issues. 

The feature is enabled by default. Live metrics are available in the following formats:

  • as new JMX MBeans
  • as a history of snapshots of the JMX values in the new IPD log file atlassian-jira-ipd-monitoring.log

The log file is available in the {jira_home}\log folder where you can find all the existing log files. The log file is also included in the Support Zip file, created in the ATST plugin. If needed, you can generate the Support Zip file in the Atlassian troubleshooting & support tools plugin and send the file to Atlassian Support, where we have internal tools to interpret it. Learn more about the plugin

Communication

The feature communicates in the following ways:

  • JMX: JMX MBeans are updated periodically based on an internal schedule.
  • The log file atlassian-jira-ipd-monitoring.log: JMX values are snapshotted and recorded to the log file on a configurable schedule. By default, the JMX values are polled and written to the log file every 60 seconds. (This parameter is up to date since Jira 9.3 EAP 02.) This polling interval can be changed by using the system property jira.diagnostics.ipdlog.poll.seconds.

The system property jira.diagnostics.ipdlog.poll.seconds will be removed in Jira 9.6.

Database connectivity metrics

See the metrics provided by the IPD and their descriptions in the following table.

To use the metrics, make sure you’ve enabled JMX.

MBean ObjectNameMetric description
com.atlassian.jira:type=metrics,
category00=db,category01=connection,
category02=failures,name=counter

db.connection.failures.counter

  • The count of database connection failures since the last restart
com.atlassian.jira:type=metrics,
category00=db,category01=connection,
category02=latency,name=statistics

db.connection.latency.statistics

  • Aggregated statistics of latency since the last restart

com.atlassian.jira:type=metrics,
category00=db,category01=connection,
category02=latency,name=value

db.connection.latency.value

  • The latest measure of latency when querying the database

com.atlassian.jira:type=metrics,
category00=db,category01=connection,
category02=pool,category02=numActive,
name=statistics

db.connection.pool.numActive.statistics

  • Aggregated statistics of the number of active connections in the database connection pool since the last restart

com.atlassian.jira:type=metrics,
category00=db,category01=connection,
category02=pool,category02=numActive,
name=value

db.connection.pool.numActive.value

  • The latest measure of the number of active connections in the database connection pool

com.atlassian.jira:type=metrics,
category00=db,category01=connection,
category02=pool,category02=numIdle,
name=statistics

db.connection.pool.numIdle.statistics

  • Aggregated statistics of the number of idle connections in the database connection pool since the last restart

com.atlassian.jira:type=metrics,
category00=db,category01=connection,
category02=pool,category02=numIdle,
name=value

db.connection.pool.numIdle.value

  • The latest measure of the number of idle connections in the database connection pool

com.atlassian.jira:type=metrics,
category00=db,category01=connection,
category02=state,name=value

db.connection.state.value

  • The latest indicator of the state of the connection to the database

To find more details of cross-product database connectivity metrics, check the article Interpreting cross-product metrics for in-product diagnostics.

Disabling the IPD

The IPD is enabled by default. To disable it:

  1. Go to <JIRA_URL>/secure/admin/SiteDarkFeatures!default.jspa, where <JIRA_URL> is the base URL of your Jira instance.
  2. In the Enable dark feature text area, enter com.atlassian.jira.in.product.diagnostics.disabled. Select Add. Learn how to manage dark features
    1. To re-enable the IPD, in the Site Wide Dark Features panel, find com.atlassian.jira.in.product.diagnostics.disabled and select Disable.

In Jira 9.4.3, you can enable and disable the IPD with a toggle in the user interface. To use the toggle:

  1. Administration_menu_System_
  2. In the left-side panel, go to System Support and select Monitoring.

  3. Use the In-product diagnostics monitoring toggle to enable or disable IPD monitoring.

Enable IPD monitoring


Log formatting

Writing to atlassian-jira-ipd-monitoring.log is done via log4j. Its configuration is managed in log4j.properties.

#####################################################
# In-product diagnostics monitoring logging
#####################################################

log4j.appender.ipd=com.atlassian.jira.logging.JiraHomeAppender
log4j.appender.ipd.File=atlassian-jira-ipd-monitoring.log
log4j.appender.ipd.MaxFileSize=20480KB
log4j.appender.ipd.MaxBackupIndex=5
log4j.appender.ipd.layout=com.atlassian.logging.log4j.NewLineIndentingFilteringPatternLayout
log4j.appender.ipd.layout.ConversionPattern=%d %m%n

log4j.logger.ipd-monitoring = INFO, filelog
log4j.additivity.ipd-monitoring = false
log4j.logger.ipd-monitoring-data-logger = INFO, ipd
log4j.additivity.ipd-monitoring-data-logger = false

Log contents

By default, a concise set of data is included in each log entry. An extended set of data can be logged by enabling the com.atlassian.jira.in.product.diagnostics.extended.logging feature flag.

To enable the extended data:

  1. Go to <JIRA_URL>/secure/admin/SiteDarkFeatures!default.jspa, where <JIRA_URL> is the base URL of your Jira instance.
  2. In the Enable dark feature text area, enter com.atlassian.jira.in.product.diagnostics.extended.logging.enabled. Select AddLearn how to manage dark features
    1. To disable the extended data, in the Site Wide Dark Features panel, find com.atlassian.jira.in.product.diagnostics.extended.logging.enabled and select Disable.

In the following tables, see the structures of the concise vs extended logging formats.

The metrics in JMX always go in the extended format.

Learn more about the metric attributes

Concise data

MBean Type

Properties

Attributes

Counter

timestamp

label

attributes

_count

Value

_value

Statistics

_99thPercentile

_max

_min

_mean

See the example of the concise log line format
2022-09-06 18:37:48,011 IPDMONITORING {"timestamp":"1662453468","label":"DB.CONNECTION.LATENCY.STATISTICS","attributes":{"_mean":"6.704470250010645E-25","_max":"1.0","_99thPercentile":"0.0","_min":"0.0"}}

Extended data

The metrics in JMX always go in the extended format.

Learn more about the metric attributes

MBean Type

Properties

Attributes

Counter

timestamp

label

attributes

objectName

_count

_fifteenMinuteRate

_fiveMinuteRate

_meanRate

_oneMinuteRate

_rateUnit

Value

_value

_number

Statistics

_50thPercentile

_75thPercentile

_95thPercentile

_98thPercentile

_99thPercentile

_999thPercentile

_count

_min

_max

_mean

_stdDev

_durationUnit

_fifteenMinuteRate

_fiveMinuteRate

_meanRate

_oneMinuteRate

_rateUnit

See the example of the extended log line format
2022-09-06 18:38:48,015 IPDMONITORING {"timestamp":"1662453528","label":
"DB.CONNECTION.LATENCY.STATISTICS","objectName":
"com.atlassian.jira:category00\u003ddb,category01\u003dconnection,category02\
u003dlatency,name\u003dstatistics,type\u003dmetrics",
"attributes":{"_oneMinuteRate":"0.02012497818617073","_50thPercentile":"0.0",
"_mean":"1.9379304604014412E-25","_max":"1.0","_stdDev":"4.40219315841711E-13",
"_98thPercentile":"0.0","_meanRate":"0.003612560785169162","_rateUnit":
"events/second","_99thPercentile":"0.0","_count":"16","_durationUnit":
"milliseconds","_75thPercentile":"0.0","_fiveMinuteRate":
"0.005912972095043379","_fifteenMinuteRate":"0.0037696657500141968",
"_999thPercentile":"0.0","_95thPercentile":"0.0","_min":"0.0"}}

Definitions of metric attributes

Expand the following sections to learn more about metric attributes.

Counter metric attributes
AttributeDefinition

_count

The number of occurrences of a metric within the current time window
_fifteenMinuteRateThe number of occurrences of a metric over the last 15 minutes
_fiveMinuteRateThe number of occurrences of a metric over the last five minutes
_meanRateThe mean rate at which events have occurred since the meter was created
_oneMinuteRateThe number of occurrences of a metric over the last one minute
_rateUnitThe unit of measure used for rates

Pay attention to the following attributes:  _oneMinuteRate, _fiveMinuteRate, and _fifteenMinuteRate.

The _count gives no indication of how the measurements have changed over time. A sense of recency is provided with the minute rates.

Value metric attributes
AttributeDefinition

_value

A most recently sampled value of the metric
_numberContains the same value as the _value attribute
Statistics metric attributes

The metrics of the statistics MBean type are also known as aggregated values. They provide statistics for the items that have been subjected to any changes over a period of time. For example, for the items that have been processed in a mail queue or added to an error mail queue.

Time window

Unless stated, aggregated values are calculated over a sliding time window. It covers the last five minutes, approximately.

Percentile values are calculated using a reservoir sampling technique. This technique uses a small, manageable set of values that is statistically representative of the data stream as a whole, hence reducing the quantity of data that must be held in memory.

Resets

Outside of the sliding time window, all aggregated values are reset:

  • After each system restart.

  • After each time JMX monitoring or in-product diagnostic metrics are enabled.

Learn more about JMX monitoring and in-product diagnostic in other Data Center products:

In the following table, find the definitions of statistics metric attributes.

AttributeDefinition

_50thPercentile

A measured value below which 50% of all measurements can be found within the current time window; also referred to as the median value.

This attribute provides an alternative to the mean as a representation of the middle measurement. The median is less likely to be skewed by outlier values than the mean.

_75thPercentileThe measured value below 75% of all measurements that can be found within the current time window; the third quartile value
_95thPercentileThe measured value below 95% of all measurements that can be found within the current time window
_98thPercentileThe measured value below 98% of all measurements that can be found within the current time window
_99thPercentileThe measured value below 99% of all measurements that can be found within the current time window
_999thPercentileThe measured value below 999% of all measurements that can be found within the current time window
_countThe number of occurrences of a metric within the current time window
_minThe minimum measured value within the current time window
_maxThe maximum measure value within the current time window; the statistical range between _max and _min provides a measure of values variability
_mean

The average value within the current time window.

This attribute can be skewed by large outlier measurements. In such cases, the _50thPercentile provides a better measure of the middle value.

_stdDev

A measure of the data variability.

A low standard deviation indicates that the values tend to be close to the mean of the set, while a high standard deviation indicates that the values are spread out over a wider range of values.

_durationUnitThe unit of measure used for durations
_fifteenMinuteRateThe number of occurrences of a metric over the last 15 minutes
_fiveMinuteRateThe number of occurrences of a metric over the last five minutes
_meanRateThe mean rate at which events have occurred since the meter was created
_oneMinuteRateThe number of occurrences of a metric over the last one minute
_rateUnitThe unit of measure used for rates

Pay attention to the following attributes:  _oneMinuteRate, _fiveMinuteRate, and _fifteenMinuteRate.

The _count gives no indication of how the measurements have changed over time. A sense of recency is provided with the minute rates.

Processing properties

  • JMX logging polling interval is set to 60 seconds and can't be modified.

  • Log file polling interval is set to 60 seconds and can be changed by using the system property jira.diagnostics.ipdlog.poll.seconds.

  • By default, the JMX values are polled and written to atlassian-jira-ipd-monitoring.log.

Last modified on Jun 15, 2023

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.