Live monitoring using the JMX interface

This article describes how to expose JMX MBeans within Jira for monitoring with a JMX client and how to use JMX MBeans for in-product diagnostics.

This guide provides a basic introduction to the JMX interface and is provided as is. Our support team can help you troubleshoot a specific Jira problem but can't help you set up your monitoring system or interpret the results.

On this page:

This article describes how to expose JMX MBeans within Jira for monitoring with a JMX client and how to use JMX MBeans for in-product diagnostics.

What is JMX?

JMX (Java Management Extensions) is a technology for monitoring and managing Java applications. JMX uses objects called MBeans (Managed Beans) to expose data and resources from your application. For large instances of Jira Server or Jira Data Center, enabling JMX allows you to more easily monitor the consumption of application resources and diagnose performance issues related to indexing. This enables you to make better decisions about how to maintain and optimize machine resources. 

Details of the JMX metrics available in Jira

JMX attribute name

Description

50thPercentile

The value at the 50th percentile in the distribution of measured times (the median value)

75thPercentile

The value at the 75th percentile in the distribution of measured times

95thPercentile

The value at the 95th percentile in the distribution of measured times

98thPercentile

The value at the 98th percentile in the distribution of measured times

99thPercentile

The value at the 99th percentile in the distribution of measured times

999thPercentile

The value at the 99.9th percentile in the distribution of measured times

Count

The number of invocations since node startup

DurationUnit

The time unit used to report percentile values, min, max, mean and standard deviation. The default unit is milliseconds.

FifteenMinuteRate

The fifteen-minute moving average rate of invocations since node startup. This rate uses the same exponential decay factor as is used for the fifteen-minute load average in Unix’s top command.

FiveMinuteRate

The five-minute moving average rate of invocations since node startup. This rate uses the same exponential decay factor as is used for the five-minute load average in Unix’s top command.

Max

The highest measured time since node startup

Mean

The mean measured time since node startup

MeanRate

The mean rate of invocations since node startup

Min

The lowest measured time since node startup

OneMinuteRate

The one-minute moving average rate of invocations since node startup. This rate uses the same exponential decay factor as is used for the one-minute load average in Unix’s top command.

RateUnit

The unit in which MeanRate, OneMinuteRate, FiveMinuteRate, and FifteenMinuteRate are reported. The default unit is events/second.

StdDev

The standard deviation in the measured times since node startup

Metrics collected by Jira

The following table lists metrics (MBeans) that are collected by Jira. All of them are grouped in the com.atlassian.jira property.

MetricDescriptionReset after restarting Jira
dashboard.view.count

The number of times all dashboards were viewed by users

Yes
entity.attachments.totalThe number of attachmentsN/A
entity.components.totalThe number of componentsN/A
entity.customfields.totalThe number of custom fieldsN/A
entity.filters.totalThe number of filtersN/A
entity.groups.totalThe number of user groupsN/A
entity.issues.totalThe number of issuesN/A
entity.users.totalThe number of usersN/A
entity.versions.totalThe number of versions createdN/A
issue.assigned.count

The number of times issues were assigned or reassigned to users (counts each action)

Yes
issue.created.count

The number of issues that you created after starting your Jira instance

Yes
issue.link.count

The number of issue links created after starting your Jira instance

Yes
issue.search.count

The number of times you searched for issues

Yes
issue.updated.count

The number of times you updated issues (each update after adding or changing some information)

Yes
issue.worklogged.count

The number of times you logged work on issues

Yes
jira.licenseThe types of licenses you have, the number of active users, and the maximum number of users available for each license typeN/A
quicksearch.concurrent.search

The number of concurrent searches that are being performed in real-time by using the quick search. You can use it to determine whether the limit set for concurrent searches is sufficient or should be increased.

Yes
web.requests

The number of requests (invocation.count) and the total response time (total.elapsed.time)

Yes

The following metrics have been added to Jira as of 8.1 and are exposed under com.atlassian.jira/metrics. All the following metrics will be reset after restarting Jira. 

Metric path

Description

comment

Metrics related to comment operations

comment/Create

A comment being created

comment/Delete

A comment being deleted

comment/Update

A comment being updated

indexing

Metrics related to issue, comment, worklog, and change indexing

indexing/CreateChangeHistoryDocument

Index documents being created for Change History entities. Note that many Change History Documents can be created for every issue.

indexing/CreateCommentDocument

Index documents being created for Comment entities

indexing/CreateIssueDocument

Index documents being created for Issue entities

indexing/IssueAddFieldIndexers

FieldIndexer modules enrich Issue documents as part of Index document creation. Plugins can register custom FieldIndexer modules. These metrics provide insight into how much time is spent in FieldIndexer and can be used to track down indexing performance issues caused by them. The metrics describe how much time was spent in all FieldIndexers combined per Issue document created.

indexing/LuceneAddDocument

How much time was spent adding a document to the Lucene index

indexing/LuceneDeleteDocument

How much time was spent deleting one or more documents matching a term from the Lucene index

indexing/LuceneOptimize

Metrics about Lucene index optimization (triggered manually from Jira)

indeding/LuceneUpdateDocument

How much time was spent adding a created document to the Lucene index

indexing/ReplicationLatency

Replication latency is the time between an issue, comment, or worklog being indexed on the node where the change was made and the indexing operation being replayed on the current node

indexing/WaitForLucene

Documents are written to the Lucene index asynchronously. This metric captures how much time Jira’s indexing thread spent waiting for Lucene to complete the write.

indexing/issueAddSearchExtractors

EntitySearchExtractor enriches issue documents as part of Index document creation. Plugins can register custom EntitySearchExtractor modules. These metrics provide insight into how much time is spent in EntitySearchExtractor and can be used to track down indexing performance issues they cause. The metrics describe how much time was spent in all EntitySearchExtractors combined per issue document created.

issue

Metrics about issue operations

issue/Create

Issue being created

issue/Delete

Issue being deleted

issue/Index

An issue being added to the Lucene index. This covers issue document creation and adding the document to the index.

issue/DeIndex

An issue being removed from the Lucene index

issue/ReIndex

An issue being re-indexed as a result of issue updates. This covers: creating an issue document, deleting the old document from the index, and adding new documents to the index.

issue/Update

An issue being updated

Metrics collected by Assets in Jira Service Management

The following table lists metrics (MBeans) that are collected by Assets in Jira Service Management. 

Metric

Description

assets.objectindeximpl.objects_load_on_add_ms



How long to load objects when a message is received.

This metric tells you how long it took to populate the index from the database.

assets.objectindeximpl.missing_objects_reload_retry_count_on_add

The number of attempts to reload an object when adding the object to the database.

The metric gives an indication on how many attempts it took to load the object from the database due to the delays in writing to the database.

assets.objectindeximpl.missing_objects_count_on_add

The number of missing objects on the first attempt at adding them to the database.

This metric indicates the number of missing objects.

assets.objectindeximpl.missing_objects_reload_on_add_ms

The time in milliseconds that was spent trying to reload from the database when an object wasn’t available.

This metric shows how much time was spent looping and waiting for the object to be available in the database.

assets.objectindeximpl.objects_load_on_update_ms

The time in milliseconds to load updates for on the first attempt.

onUpdate(final int id, final long updateTime)


assets.objectindeximpl.missing_objects_reload_on_update_ms

The time in milliseconds that is spent waiting for updates to appear in the database.

reloadObjectFromDatabaseUntilUpdateTimeMatches

assets.objectindeximpl.missing_objects_reload_retry_count_on_update

The number of attempts to reload an object on update until the right version was found.

assets.objectindeximpl.missing_objects_reload_retry_count_on_update

The number of missing object updates that were not ready in the database when an update started.

assets.objectindeximpl.objects_removal_ms

The time in milliseconds that indicates how long it took to remove objects from Assets.

onRemove(final Collection<Integer> ids)

assets.assetsbatchreplicationmessageworkqueuepoller.process_ms

The number of milliseconds that indicates how long to process a replication message.

assets.assetsbatchreplicationmessageworkqueuepoller.number_of_create_failures

The number of failures in a message batch when an object is created.

assets.assetsbatchreplicationmessageworkqueuepoller.number_of_update_failures

The number of failures in a message batch when an object is updated.

assets.cachemessageworkqueuepoller.process_batch_size

The number of object changes that were batched together to be sent across the cluster.

assets.cachemessageworkqueuepoller.wait_for_clearance_ms

The time in milliseconds that is spent waiting for database updates to be drained from the application or submitted to the database.

assets.cachemessageworkqueuepoller.process_ms

The time in milliseconds that is the total duration of the process of batching and sending a replication message.

assets.assetsbatchreplicationmessagereceiver.work_queue_size

The size of the work queue in the replication message receiver.

assets.assetsbatchreplicationmessagereceiver.work_queue_gauge

The current size of the work queue in the replication message receiver

assets.assetsobjectreplicationbatchmanager.work_queue_size

The work queue size in the batch manager collecting individual changes to batch and send across the cluster.

assets.assetsobjectreplicationbatchmanager.work_queue_gauge

The current work queue size in the batch manager collecting individual changes to batch and send across the cluster

assets.defaultassetsbatchmessagesender.send_message

The amount of time to dispatch the message and continue processing the next message.

assets.insightcachereplicatorimpl.legacy_object_receiver_queue_size

The queue size on the legacy replication mechanism using cluster messages.

assets.insightcachereplicatorimpl.object_replication_dispatch

The amount of time it takes to handleMessage or dispatch a message by calling send for the legacy index replication using the cluster message cache.

assets.insightcachereplicatorimpl.legacy_object_send_queue_size

The queue size after offering a message for legacy index replication using the cluster message cache.

messageToSend.offer(cacheMessage)

assets.assetsreplicationretryqueuepoller.retry_queue_size

The size of the retry/failure queue.

assets.assetsreplicationretryqueuepoller.create_retry_attempts

The number of attempts to get successful processing for creating a message.

assets.assetsreplicationretryqueuepoller.update_retry_attempts

The number of attempts to get successful processing for updating a message.

assets.assetsreplicationretryqueuepoller.replay_wait_time

The amount of waiting time that was added before retrying the update.

This metric helps to see if processing is keeping up with the queue, or if updates are backlogged and being processed immediately. If there is no waiting time applied, the queue is not processed quickly enough.

assets.assetsreplicationretryqueuepoller.dead_letter_queue_size

The number of items in the dead letter queue.

assets.assetsreplicationretryqueuepoller.process_retry_excluding_wait_ms

The time in milliseconds that it takes to process the failures, excluding any wait time prior to processing.

This metric is an indication of how long the batches of failures are taking to be processed. It will also help to see if more delays are being added in the onAdd or onUpdate wait loops. The value of this metric should be low, unless more retries are needed, in which case the delay could be increased.

assets.assetsreplicationretryqueuepoller.dead_letter_queue_gauge

The current number of messages in the dead letter queue.

assets.assetsreplicationretryqueuepoller.retry_queue_gauge

The current number of messages in the retry queue.

Monitoring Jira

Before you can monitor Jira, you should enable JMX monitoring and then use a JMX client to view the metrics.

Good to know

Viewing the metrics will always have some performance impact on Jira. We recommend that you don't refresh them more than once a second.

Enabling JMX monitoring in Jira

All of the metrics are collected by default, but you should enable JMX monitoring to expose them. You can do it in Jira but you must be a Jira admin.

  1. In the upper-right corner of the screen, select Administration  > System
  2. Go to JMX monitoring.
  3. Toggle Enable JMX monitoring.

Monitoring with JConsole

After you enabled JMX monitoring, you can use any JMX client to view the metrics. To make it quick and easy, we've described how to view them by using JConsole. You can monitor your Jira instance either locally or remotely:

  • Monitoring Jira locally is good if you're troubleshooting a particular issue or only need to monitor Jira for a short time. Local monitoring can have a performance impact on your server, so it's not recommended for long-term monitoring of your production system.

    Show me how to do this...

     To monitor locally:

    1. Start JConsole. You'll find it in the bin directory of the JDK installation directory. (JConsole is only available as part of the JDK.)
    2. Select Local Process.
    3. Select the Jira process. It'll be called something like org.apache.catalina.startup.Bootstrap start
    4. After connecting, expand the com.atlassian.jira property that groups all the metrics.

    See Using JConsole for more information on local monitoring.

  • Monitoring Jira remotely is recommended for production systems as it does not consume resources on your Jira server.

    Show me how to do this...

     To monitor remotely:

    1. Add the following properties to your setenv.sh / setenv.bat file. The port can be any port that is not in use.

      CATALINA_OPTS

      Windows
      SET CATALINA_OPTS="-Dcom.sun.management.jmxremote.authenticate=true ${CATALINA_OPTS}"
      SET CATALINA_OPTS="-Dcom.sun.management.jmxremote.password.file=/atlassian-jira-software-x.y.z-standalone/jmxremote.password ${CATALINA_OPTS}"
      SET CATALINA_OPTS="-Dcom.sun.management.jmxremote.access.file=/atlassian-jira-software-x.y.z-standalone/jmxremote.access ${CATALINA_OPTS}"

      (info) x.y.z stands for the Jira version you're using.

      Linux
      CATALINA_OPTS="-Dcom.sun.management.jmxremote.authenticate=true ${CATALINA_OPTS}"
      CATALINA_OPTS="-Dcom.sun.management.jmxremote.password.file=/atlassian-jira-software-x.y.z-standalone/jmxremote.password ${CATALINA_OPTS}"
      CATALINA_OPTS="-Dcom.sun.management.jmxremote.access.file=/atlassian-jira-software-x.y.z-standalone/jmxremote.access ${CATALINA_OPTS}"

      (info) x.y.z stands for the Jira version you're using.

      For more details, see the Using Password Authentication section in Monitoring and Management Using JMX Technology.

      Additionally, to access the JMX properties, you might need to configure SSL. For more details, see the SSL sections in Monitoring and Management Using JMX Technology.

      JAVA_OPTS

      Using JAVA_OPTS works to expose the JMX MBeans for remote access but causes errors to be thrown during Jira shutdown.

      Windows
      JAVA_OPTS=-Dcom.sun.management.jmxremote %JAVA_OPTS%
      JAVA_OPTS=-Dcom.sun.management.jmxremote.port=8099 %JAVA_OPTS%
      JAVA_OPTS=-Dcom.sun.management.jmxremote.authenticate=false %JAVA_OPTS%
      Linux
      JAVA_OPTS="-Dcom.sun.management.jmxremote ${JAVA_OPTS}"
      JAVA_OPTS="-Dcom.sun.management.jmxremote.port=8099 ${JAVA_OPTS}"
      JAVA_OPTS="-Dcom.sun.management.jmxremote.authenticate=false ${JAVA_OPTS}"
      export JAVA_OPTS
    2. Decide how you will secure your remote connection. See Remote Monitoring and Management for more information. 
      Although it is possible to disable authentication, we don't recommend doing this on a production system.
    3. Start JConsole. You'll find it in the bin directory of the JDK installation directory.
    4. Select Remote Process.
    5. Enter your hostname and port. This is the port you specified earlier, not the Jira port.
    6. Select Connect.
    7. After connecting, expand the com.atlassian.jira property that groups all the metrics.

    See Using JConsole for more information on remote monitoring.

Known security issues

We’re providing a robust fix for a potential security vulnerability that may be caused by an RCE (remote code execution) JMX attack. During this attack, a remote user with valid credentials for JMX monitoring can execute arbitrary code on Jira Data Center via Java Deserialization, even if this user’s account is readOnly (montiorRole).

To prevent fabricated data from getting into the system through requests, we're using a blocklist deserialization filter based on ObjectInputFilter from JVM.

If you use a custom JDK and miss appropriate classes in your classpath based on the Java version, your Jira node won’t be started.

atlassian.jira.log will contain the following error: BlocklistDeserializationFilter has not been set up. It means that your Java environment has some security issues.

To eliminate the error and boost the security of your Jira instance, make sure your JDK contains the following classes:

  • For JDK 8: the class sun.misc.ObjectInputFilter must be enabled in the classpath.

  • For JDK 11 and later: the class java.io.ObjectInputFilter must be enabled in the classpath.


In-product diagnostics available through JMX

Since Jira 9.3, we've introduced a set of database connectivity metrics for in-product diagnostics available through JMX.

Learn about the updates in Jira 9.5 and later

In Jira 9.5, in-product diagnostics was complemented with more new metrics: HTTP connection metrics and mail queue metrics.

In Jira 9.8, we added a few new mail queue metrics allowing you to get a more detailed picture of mail queue contents and to collect more data for better performance monitoring.

In Jira 9.11, we introduced new infrastructure metrics for monitoring the health and performance of your instance infrastructure: outgoing and incoming mail servers, external user directories, shared and local home directories, and node communication for Data Center instances.

In-product diagnostics (IPD) provides greater insights for you and our Support into how running instances are operating.

IPD uses additional metrics handling Jira’s interactions with its database. Using database connectivity metrics, you’ll efficiently identify what in your environment or infrastructure might cause the performance issues. 

The feature is disabled by default. Live metrics are available in the following formats:

  • as new JMX MBeans
  • as a history of snapshots of the JMX values in the new IPD log file atlassian-jira-ipd-monitoring.log

The log file is available in the {jira_home}\log folder where you can find all the existing log files. The log file is also included in the Support Zip file, created in the ATST plugin. If needed, you can generate the Support Zip file in the Atlassian troubleshooting & support tools plugin and send the file to Atlassian Support, where we have internal tools to interpret it. Learn more about the plugin

Communication

The feature communicates in the following ways:

  • JMX: JMX MBeans are updated periodically based on an internal schedule.
  • The log file atlassian-jira-ipd-monitoring.log: JMX values are snapshotted and recorded to the log file on a configurable schedule. By default, the JMX values are polled and written to the log file every 60 seconds. (This parameter is up to date since Jira 9.3 EAP 02.)

In-product diagnostics metrics

Expand the following sections to learn more about the metrics available for in-product diagnostics.

To use the metrics, make sure you’ve enabled JMX.

Database connectivity metrics

See the metrics provided by the IPD and their descriptions in the following table.

MBean ObjectNameMetric description
com.atlassian.jira:type=metrics,
category00=db,category01=connection,
category02=failures,name=counter

db.connection.failures.counter

  • The count of database connection failures since the last restart
com.atlassian.jira:type=metrics,
category00=db,category01=connection,
category02=latency,name=statistics

db.connection.latency.statistics

  • Aggregated statistics of latency since the last restart

com.atlassian.jira:type=metrics,
category00=db,category01=connection,
category02=latency,name=value

db.connection.latency.value

  • The latest measure of latency when querying the database

com.atlassian.jira:type=metrics,
category00=db,category01=connection,
category02=pool,category02=numActive,
name=statistics

db.connection.pool.numActive.statistics

  • Aggregated statistics of the number of active connections in the database connection pool since the last restart

com.atlassian.jira:type=metrics,
category00=db,category01=connection,
category02=pool,category02=numActive,
name=value

db.connection.pool.numActive.value

  • The latest measure of the number of active connections in the database connection pool

com.atlassian.jira:type=metrics,
category00=db,category01=connection,
category02=pool,category02=numIdle,
name=statistics

db.connection.pool.numIdle.statistics

  • Aggregated statistics of the number of idle connections in the database connection pool since the last restart

com.atlassian.jira:type=metrics,
category00=db,category01=connection,
category02=pool,category02=numIdle,
name=value

db.connection.pool.numIdle.value

  • The latest measure of the number of idle connections in the database connection pool

com.atlassian.jira:type=metrics,
category00=db,category01=connection,
category02=state,name=value

db.connection.state.value

  • The latest indicator of the state of the connection to the database

HTTP connection metrics

See the metrics provided by the IPD and their descriptions in the following table.

MBean ObjectNameMetric description
com.atlassian.jira:type=metrics,
category00=http,category01=connection,
category02=pool,category03=numActive,
name=value

http.connection.pool.numActive.value

  • The latest measure of the number of active connections in the HTTP connection pool
com.atlassian.jira:type=metrics,
category00=http,category01=connection,
category02=pool,category03=numIdle,
name=value

http.connection.pool.numIdle.value

  • The latest measure of the number of idle connections in the HTTP connection pool
com.atlassian.jira:type=metrics,
category00=http,category01=connection,
category02=pool,category03=numMax,
name=value

http.connection.pool.numMax.value

  • The maximum number of threads to be created by the connector and made available for requests
com.atlassian.jira:type=metrics,
category00=http,category01=connection,
category02=sessions,category03=active,
name=value

http.connection.sessions.active.value

  • The latest measure of the number of active user sessions
com.atlassian.jira:type=metrics,
category00=http,category01=connection,
category02=sessions,category03=active,
name=statistics

http.connection.sessions.active.statistics

  • Aggregated statistics of the number of active user sessions
com.atlassian.jira:type=metrics,
category00=http,category01=connection,
category02=sessions,category03=recent,
name=value

http.connection.sessions.recent.value

  • The latest measure of the number of recent user sessions. A recent session is the session that has been active in the one last hour.
com.atlassian.jira:type=metrics,
category00=http,category01=requests,
name=value

http.requests.value

  • The latest measure of the total number of HTTP requests per minute
com.atlassian.jira:type=metrics,
category00=http,category01=requests,
name=statistics

http.requests.statistics

  • Aggregated statistics of the total number of HTTP requests per minute
Mail queue metrics

See the metrics provided by the IPD and their descriptions in the following table.

MBean ObjectNameMetric description
com.atlassian.jira:type=metrics,
category00=mail,category01=queue,
category02=numErrors,name=value

mail.queue.numErrors.value

  • The latest measure of the number of items in an error mail queue
com.atlassian.jira:type=metrics,
category00=mail,category01=queue,
category02=numItems,name=value

mail.queue.numItems.value

  • The latest measure of the number of items in a mail queue
com.atlassian.jira:type=metrics,
category00=mail,category01=queue,
category02=numErrors,name=statistics

mail.queue.numErrors.statistics

  • Aggregated statistics of the number of items in an error mail queue
com.atlassian.jira:type=metrics,
category00=mail,category01=queue,
category02=numItems,name=statistics

mail.queue.numItems.statistics

  • Aggregated statistics of the number of items in a mail queue
com.atlassian.jira:type=metrics,
category00=mail,category01=queue,
category02=numItemsAddedPerMin,
name=value

mail.queue.numItemsAddedPerMin.value

  • The latest measure of the number of items added to a mail queue per minute
com.atlassian.jira:type=metrics,
category00=mail,category01=queue,
category02=numErrorsAddedPerMin,
name=value

mail.queue.numErrorsAddedPerMin.value

  • The latest measure of the number of items added to an error mail queue per minute

com.atlassian.jira:type=metrics,
category00=mail,category01=queue,
category02=numItemsAddedPerMin,
name=statistics

mail.queue.numItemsAddedPerMin.statistics

  • Aggregated statistics of the number of items added to a mail queue per minute
com.atlassian.jira:type=metrics,
category00=mail,category01=queue,
category02=numErrorsAddedPerMin,
name=statistics

mail.queue.numErrorsAddedPerMin.statistics

  • Aggregated statistics of the number of items added to an error mail queue per minute
com.atlassian.jira:type=metrics,
category00=mail,category01=queue,
category02=numItemsProcessedPerMin,
name=value

mail.queue.numItemsProcessedPerMin.value

  • The latest measure of the number of items processed by a mail queue per minute
com.atlassian.jira:type=metrics,
category00=mail,category01=queue,
category02=numItemsProcessedPerMin,
name=statistics

mail.queue.numItemsProcessedPerMin.statistics

  • Aggregated statistics of the number of items processed by a mail queue per minute
com.atlassian.jira:type=metrics,
category00=mail,category01=queue,
category02=numEmailsSentPerMin,
name=value

mail.queue.numEmailsSentPerMin.value

  • The latest measure of the number of emails sent by the SMTP server per minute
com.atlassian.jira:type=metrics,
category00=mail,category01=queue,
category02=numEmailsSentPerMin,
name=statistics

mail.queue.numEmailsSentPerMin.statistics

  • Aggregated statistics of the number of emails sent by the SMTP server per minute
com.atlassian.jira:type=metrics,
category00=mail,category01=queue,
category02=jobRunning,name=value

mail.queue.jobRunning.value

  • The latest indicator of the state of a mail queue job
    • 1 – the mail queue job is currently running

    • 0 – the mail queue job is currently not running

Infrastructure metrics

See the metrics provided by the IPD and their descriptions in the following table.

MBean ObjectNameMetric description
Cluster metrics
com.atlassian.
jira:type=metrics,
category00=node,
category01=latency,
name=statistics,
tag.destNode<nodeId>

node.latency.statistics
destNode=<nodeId>

  • Aggregated statistics of communication latency to the node (<nodeId>).

  • There's a metric for all cluster nodes except for itself.

    • These metrics are emitted on every node in the cluster. While latency to all other nodes is measured,
      <nodeID> will never be the same as the node where the metric is emitted.
com.atlassian.
jira:type=metrics,
category00=node,
category01=connection,
category02=state,
name=custom,
tag.destNode=<nodeId>

node.connection.state.custom
destNode=<nodeId>

  • The state of communication with another node (<nodeId>).

  • There's a metric for all cluster nodes except for itself.

    • These metrics are emitted on every node in the cluster. While latency to all other nodes is measured,
      <nodeID> will never be the same as the node where the metric is emitted.
Mail server metrics
com.atlassian.jira:
type=metrics,
category00=mail,
category01=outgoing,
category02=connection,
category03=state,
name=custom

mail.outgoing.connection.state.custom

  • The state of connection to an outgoing SMTP mail server.

  • The metric is available if the SMTP server is configured.

com.atlassian.jira:
type=metrics,
category00=mail,
category01=incoming,
category02=connection,
category03=state,
name=custom,tag.serverName=<mailName>

mail.incoming.connection.state.custom
serverName=<mailName>

  • The state of connection to an incoming mail server (<mailName>).

  • There's a separate metric for each configured incoming mail server.

Shared storage metrics
com.atlassian.
jira:type=metrics,
category00=home,
category01=shared,
category02=write,
category03=latency,
name=value

home.shared.write.latency.value

  • The median latency of writing a small file (~30 bytes) to the shared home

com.atlassian.
jira:type=metrics,
category00=home,
category01=shared,
category02=write,
category03=latency,
name=statistics

home.shared.write.latency.statistics

  • Aggregated latency statistics of writing a small file (~30 bytes) to the shared home

Local storage metrics
com.atlassian.
jira:type=metrics,
category00=home,
category01=local,
category02=write,
category03=latency,
category04=synthetic,
name=value

home.local.write.latency.synthetic.value

  • The median latency of writing a small file (~30 bytes) to the local home with guaranteed persistence

com.atlassian.
jira:type=metrics,
category00=home,
category01=local,
category02=write,
category03=latency,
category04=synthetic,
name=statistics

home.local.write.latency.synthetic.statistics

  • Aggregated latency statistics of writing a small file (~30 bytes) to the local home with guaranteed persistence

com.atlassian.
jira:type=metrics,
category00=home,
category01=local,
category02=write,
category03=latency,
category04=indexwriter,
name=statistics

home.local.write.latency.indexwriter.statistics

  • Aggregated latency statistics to persist the current Lucene index buffer

  • The metric is updated only when Lucene persists latest index changes.

User directory metrics
com.atlassian.
jira:type=metrics,
category00=user,
category01=directory,
category02=connection,
category03=latency,
name=value,
tag.userDirName=<directoryName>

user.directory.connection.latency.value
userDirName=<directoryName>

  • The latest value of latency to search a single user in the external user directory (<directoryName>).

  • There's a metric for every external user directory.

com.atlassian.
jira:type=metrics,
category00=user,
category01=directory,
category02=connection,
category03=latency,
name=statistics,
tag.userDirName=<directoryName>

user.directory.connection.latency.statistics
userDirName=<directoryName>

  • Aggregated latency statistics to search a single user in the external user directory (<directoryName>).

  • There's a metric for every external user directory.

com.atlassian.
jira:type=metrics,
category00=user,
category01=directory,
category02=connection,
category03=state,
name=custom,
tag.userDirName=<directoryName>

user.directory.connection.state.custom
userDirName=<directoryName>

  • The state of connection to an external user directory (<directoryName>).

  • Checks if a connection to a remote server can be established.

  • There's a metric for every external user directory.

To learn more details about the infrastructure metrics, check the article Interpreting infrastructure metrics for in-product diagnostics.

To find more details on cross-product metrics, check the article Interpreting cross-product metrics for in-product diagnostics.

To check the more detailed definitions of Jira-specific metrics, check the article Interpreting Jira-specific metrics for in-product diagnostics.

Enabling In-product diagnostics monitoring

IPD monitoring is enabled by default. To manage it:

  1. In the upper-right corner of the screen, select Administration  > System
  2. In the left-side panel, go to System Support and select Monitoring.

  3. Use the In-product diagnostics monitoring toggle to enable or disable IPD monitoring.
    The toggle is also available in version 9.4.3. Check Jira Software release notes for updates.

Enabling In-product diagnostics monitoring in the user interface

REST API

In Jira 9.5, we've introduced a new REST API endpoint for managing the IPD monitoring, specifically the In-product diagnostics monitoring toggle in the user interface: /rest/api/2/monitoring/ipd.

GET

Returns the state of the IPD functionality:

  • true: In-product diagnostics monitoring is enabled.

  • false: In-product diagnostics monitoring is disabled.

Sample response:

{
  "enabled": true
}
POST

Sets the state of IPD functionality. The response is empty and the successful result is confirmed with an HTTP 20x status response.

Sample response:

{
  "enabled": false
}

Log formatting

Writing to atlassian-jira-ipd-monitoring.log is done via log4j. Its configuration is managed in log4j.properties.

#####################################################
# In-product diagnostics monitoring logging
#####################################################

log4j.appender.ipd=com.atlassian.jira.logging.JiraHomeAppender
log4j.appender.ipd.File=atlassian-jira-ipd-monitoring.log
log4j.appender.ipd.MaxFileSize=20480KB
log4j.appender.ipd.MaxBackupIndex=5
log4j.appender.ipd.layout=com.atlassian.logging.log4j.NewLineIndentingFilteringPatternLayout
log4j.appender.ipd.layout.ConversionPattern=%d %m%n

log4j.logger.ipd-monitoring = INFO, filelog
log4j.additivity.ipd-monitoring = false
log4j.logger.ipd-monitoring-data-logger = INFO, ipd
log4j.additivity.ipd-monitoring-data-logger = false

Log contents

By default, a concise set of data is included in each log entry. An extended set of data can be logged by enabling the com.atlassian.jira.in.product.diagnostics.extended.logging feature flag.

To enable the extended data:

  1. Go to <JIRA_URL>/secure/admin/SiteDarkFeatures!default.jspa, where <JIRA_URL> is the base URL of your Jira instance.
  2. In the Enable dark feature text area, enter com.atlassian.jira.in.product.diagnostics.extended.logging.enabled. Select AddLearn how to manage dark features
    1. To disable the extended data, in the Site Wide Dark Features panel, find com.atlassian.jira.in.product.diagnostics.extended.logging.enabled and select Disable.

In the following tables, see the structures of the concise vs extended logging formats.

The metrics in JMX always go in the extended format.

Learn more about the metric attributes

Concise data

MBean Type

Properties

Attributes

Counter

timestamp

label

attributes

_count

Value

_value

Statistics

_99thPercentile

_max

_min

_mean

See the example of the concise log line format
2022-09-06 18:37:48,011 IPDMONITORING {"timestamp":"1662453468","label":"DB.CONNECTION.LATENCY.STATISTICS","attributes":{"_mean":"6.704470250010645E-25","_max":"1.0","_99thPercentile":"0.0","_min":"0.0"}}

Extended data

The metrics in JMX always go in the extended format.

Learn more about the metric attributes

MBean Type

Properties

Attributes

Counter

timestamp

label

attributes

objectName

_count

_fifteenMinuteRate

_fiveMinuteRate

_meanRate

_oneMinuteRate

_rateUnit

Value

_value

_number

Statistics

_50thPercentile

_75thPercentile

_95thPercentile

_98thPercentile

_99thPercentile

_999thPercentile

_count

_min

_max

_mean

_stdDev

_durationUnit

_fifteenMinuteRate

_fiveMinuteRate

_meanRate

_oneMinuteRate

_rateUnit

See the example of the extended log line format
2022-09-06 18:38:48,015 IPDMONITORING {"timestamp":"1662453528","label":
"DB.CONNECTION.LATENCY.STATISTICS","objectName":
"com.atlassian.jira:category00\u003ddb,category01\u003dconnection,category02\
u003dlatency,name\u003dstatistics,type\u003dmetrics",
"attributes":{"_oneMinuteRate":"0.02012497818617073","_50thPercentile":"0.0",
"_mean":"1.9379304604014412E-25","_max":"1.0","_stdDev":"4.40219315841711E-13",
"_98thPercentile":"0.0","_meanRate":"0.003612560785169162","_rateUnit":
"events/second","_99thPercentile":"0.0","_count":"16","_durationUnit":
"milliseconds","_75thPercentile":"0.0","_fiveMinuteRate":
"0.005912972095043379","_fifteenMinuteRate":"0.0037696657500141968",
"_999thPercentile":"0.0","_95thPercentile":"0.0","_min":"0.0"}}


Definitions of metric attributes

Expand the following sections to learn more about metric attributes.

Counter metric attributes

Attribute

Definition

_count

The number of occurrences of a metric within the current time window

_fifteenMinuteRate

The number of occurrences of a metric over the last 15 minutes

_fiveMinuteRate

The number of occurrences of a metric over the last five minutes

_meanRate

The mean rate at which events have occurred since the meter was created

_oneMinuteRate

The number of occurrences of a metric over the last one minute

_rateUnit

The unit of measure used for rates

Pay attention to the following attributes:  _oneMinuteRate, _fiveMinuteRate, and _fifteenMinuteRate.

The _count gives no indication of how the measurements have changed over time. A sense of recency is provided with the minute rates.

Value metric attributes

Attribute

Definition

_value

A most recently sampled value of the metric

_number

Contains the same value as the _value attribute

Statistics metric attributes

The metrics of the statistics MBean type are also known as aggregated values. They provide statistics for the items that have been subjected to any changes over a period of time. For example, for the items that have been processed in a mail queue or added to an error mail queue.

Time window

Unless stated, aggregated values are calculated over a sliding time window. It covers the last five minutes, approximately.

Percentile values are calculated using a reservoir sampling technique. This technique uses a small, manageable set of values that is statistically representative of the data stream as a whole, hence reducing the quantity of data that must be held in memory.

Resets

Outside of the sliding time window, all aggregated values are reset:

  • After each system restart.

  • After each time JMX monitoring or in-product diagnostic metrics are enabled.

Learn more about JMX monitoring and in-product diagnostic in other Data Center products:

In the following table, find the definitions of statistics metric attributes.

Attribute

Definition

_50thPercentile

A measured value below which 50% of all measurements can be found within the current time window; also referred to as the median value.

This attribute provides an alternative to the mean as a representation of the middle measurement. The median is less likely to be skewed by outlier values than the mean.

_75thPercentile

The measured value below 75% of all measurements that can be found within the current time window; the third quartile value

_95thPercentile

The measured value below 95% of all measurements that can be found within the current time window

_98thPercentile

The measured value below 98% of all measurements that can be found within the current time window

_99thPercentile

The measured value below 99% of all measurements that can be found within the current time window

_999thPercentile

The measured value below 999% of all measurements that can be found within the current time window

_count

The number of occurrences of a metric within the current time window

_min

The minimum measured value within the current time window

_max

The maximum measure value within the current time window; the statistical range between _max and _min provides a measure of values variability

_mean

The average value within the current time window.

This attribute can be skewed by large outlier measurements. In such cases, the _50thPercentile provides a better measure of the middle value.

_stdDev

A measure of the data variability.

A low standard deviation indicates that the values tend to be close to the mean of the set, while a high standard deviation indicates that the values are spread out over a wider range of values.

_durationUnit

The unit of measure used for durations

_fifteenMinuteRate

The number of occurrences of a metric over the last 15 minutes

_fiveMinuteRate

The number of occurrences of a metric over the last five minutes

_meanRate

The mean rate at which events have occurred since the meter was created

_oneMinuteRate

The number of occurrences of a metric over the last one minute

_rateUnit

The unit of measure used for rates

Pay attention to the following attributes:  _oneMinuteRate, _fiveMinuteRate, and _fifteenMinuteRate.

The _count gives no indication of how the measurements have changed over time. A sense of recency is provided with the minute rates.

Processing properties

  • JMX logging polling interval is set to 60 seconds and can't be modified.

  • Log file polling interval is set to 60 seconds and can be changed by using the system property jira.diagnostics.ipdlog.poll.seconds.

  • By default, the JMX values are polled and written to atlassian-jira-ipd-monitoring.log.

Last modified on Aug 14, 2023

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.