Enabling JMX counters for performance monitoring

This article describes how to expose JMX MBeans within Bitbucket Server for monitoring with a JMX client.

What is JMX?

JMX (Java Management eXtensions) is a technology for monitoring and managing Java applications. JMX uses objects called MBeans (Managed Beans) to expose data and resources from your application.

Why would I want to enable JMX monitoring within Bitbucket Server?

On this page:

Related reading:

For large instances of Bitbucket Server or Bitbucket Data Center, enabling JMX allows you to more easily monitor the consumption of application resources. This enables you to make better decisions about how to maintain and optimize machine resources.

What can I monitor with JMX?

It is possible to monitor various statistics using JMX counters within Bitbucket Server. Below are some examples of some statistics that can be monitored.

Mail statistics (com.atlassian.bitbucket:name=MailStatistics)

Name Description
AverageMessageSize Average size (in bytes) of messages sent
LargestMessageSent Largest message that has been sent (in bytes)
LastMessageFailure Date of the last failure to send a message
LastMessageSuccess Date of the last successful message send operation
LastQueueFullEvent Last time the message queue was full
QueueFullEventCount Number of times the message queue was full
QueueUsage Queue usage as a fraction, 0.0d indicates empty and 1.0d indicates full
QueuedMessagesCount Current count of queued (unsent) messages
QueuedMessagesSize Current size (in bytes) of the queued (unsent) messages
TotalMailDataSent Total size (in bytes) of messages sent
TotalMessagesFailed Total number of messages that failed to send
TotalMessagesSent Total number of messages sent

Diagnostics

Alerts Total (com.atlassian.diagnostics:type=Alerts,name=Total)

Name Description
LatestAlertTimestamp Timestamp of the most recent alert
TotalCount Total number of alerts since the JVM was started
ErrorCount Number of alerts of severity ERROR since the JVM was started
InfoCount Number of alerts of severity INFO since the JVM was started
WarningCount Number of alerts of severity WARNING since the JVM was started

Plugin (com.atlassian.diagnostics:type=Alerts,Category=Plugin,name=${PLUGIN_NAME})

Name Description
LatestAlertTimestamp Timestamp of the most recent alert
TotalCount Total number of alerts since the JVM was started
ErrorCount Number of alerts of severity ERROR since the JVM was started
InfoCount Number of alerts of severity INFO since the JVM was started
WarningCount Number of alerts of severity WARNING since the JVM was started
PluginName Plugin name, if available

Issue (com.atlassian.diagnostics:type=Alerts,Category=Issue,name=${ISSUE_ID})

Name Description
LatestAlertTimestamp Timestamp of the most recent alert
Component Component the issue is defined for
Count Number of alerts for the issue since the JVM was started
Severity Issue's severity
Description Issue's description


Hosting statistics

Protocol Object name
SSH com.atlassian.bitbucket:name=SshHostingStatistics
HTTP com.atlassian.bitbucket:name=HttpHostingStatistics

Hosting statistic attributes

All the hosting statistics attributes are monotonically increasing since the JVM was restarted

Name Description
cloneCacheBypass clone requests that have bypassed the scm-cache
cloneCacheHit clone requests served from the scm-cache
cloneCacheMiss clone requests that could not be served from the scm-cache
cloneRead bytes read from clients during clone operations
cloneRequestCount number of clone requests served
cloneWritten bytes written to clients during <i>clone</i> operations
fetchRead bytes read from clients during <i>fetch</i> operations
fetchRequestCount number of fetch requests served
fetchWritten bytes written to clients during fetch operations
requests total number of requests served
totalBytesRead total bytes read from clients
totalBytesWritten total bytes written to clients


Webhooks statistics (com.atlassian.webhooks:name=Webhooks)

Name Description
PublishCount A count of the total number of events that could trigger webhooks
(A publish may create many dispatches)
DispatchSuccessCount Total number of webhooks to fire successfully with a successful HTTP response
DispatchRejectedCount A count of the number of webhook dispatches that were rejected for execution
DispatchLastRejectedTimestamp The last time a webhook was rejected, either from circuit breaking, or due to too many webhooks being in flight
DispatchInFlightCount Total number of dispatches that have been triggered and are awaiting resolution
DispatchFailureCount Total number of webhooks that fired successfully, but the HTTP response indicates a failure
(non 2xx code)
DispatchErrorCount Total number of webhooks to have had an error while they were being dispatched
DispatchCount Total number of webhooks to have been dispatched

Thread pools

Thread pool Description Object name
IoPumpThreadPool Threads that handle external process IO com.atlassian.bitbucket.thread-pools:name=IoPumpThreadPool
ScheduledThreadPool Thread pool that takes care of several miscellaneous scheduled tasks com.atlassian.bitbucket.thread-pools:name=ScheduledThreadPool
EventThreadPool Threads that dispatch events to @EventListenermethods com.atlassian.bitbucket.thread-pools:name=EventThreadPool

Thread pool attributes

Name

Description

ActiveCount Returns the approximate number of threads that are actively executing tasks
MaximumPoolSize Returns the maximum allowed number of threads
PoolSize Returns the current number of threads in the pool
QueueLength The number of tasks awaiting execution by the thread pool
LargestPoolSize The largest number of threads that have ever been simultaneously in the pool
CompletedTaskCount The approximate total number of tasks that have completed execution. Because the states of tasks and threads may change dynamically during computation, the returned value is only an approximation, but one that does not ever decrease across successive calls

Repositories (com.atlassian.bitbucket:name=Repositories)

Name Description
Count Number of repositories currently configured in Bitbucket server across all projects

Scm Statistics (com.atlassian.bitbucket:name=ScmStatistics)

Name Description
Pulls Number of scm pulls serviced by this instance since it was started
Pushes Number of scm pushes received by this instance is it was started


Ticket statistics

Bitbucket server uses 'tickets' as a mechanism for creating back-pressure to prevent the system from being overloaded with requests. There are two types of tickets used by Bitbucket server, hosting tickets and command tickets. 
Hosting tickets (com.atlassian.bitbucket:name=HostingTickets): Limits the number of SCM hosting operations, meaning pushes and pulls over HTTP or SSH, which may be running concurrently. 
Command tickets(com.atlassian.bitbucket:name=CommandTickets):  Limits the number of SCM commands, such as: `git diff`, `git blame`, or `git rev-list`, which may be running concurrently.

Bitbucket server supports the following metrics for each ticket type.

Name Description
Available The number of tickets available for acquisition (lower number means higher load)
LastRejection The timestamp of the last rejected ticket, or null if no tickets have been rejected
Name The name of the ticket bucket either 'scm-command' or 'scm-hosting'
OldestQueuedRequest The timestamp at which the oldest queued request started waiting, or null if there are no queued requests
QueuedRequests The number of requests currently waiting for an available ticket
Total The maximum number of tickets that can be acquired concurrently before back-pressure is applied
Used The number of tickets that have been acquired (higher number means higher load)

Event Statistics (com.atlassian.bitbucket:name=EventStatistics)

Name Description
DispatchedCount Total number of listener callbacks that have been performed. An event that is delivered to 10 listeners counts as 10 dispatches
LastRejection Date of the last event being rejected, or null if no event has been rejected
PublishedCount Total number of events delivered. An event that is delivered to 10 listeners counts as 1 event
QueueCapacity Maximum number of event callbacks that can be queued before events are rejected
QueueLength Number of event callbacks that have been queued but haven't been dispatched yet
RejectedCount Total number of events that were not dispatched because the event queue was full
RemainingQueueCapacity Remaining number of event callbacks that can be queued before events are rejected

Cluster Lock Statistics (com.atlassian.bitbucket:name=ClusterLocks)

Name Description

LockedCount

Number of cluster locks that are currently held by this node

QueuedThreadCount

Number of threads on this node that are currently blocked waiting for a lock

TotalAcquiredCount

Total number of times a cluster lock was acquired on this node since startup

TotalAcquireErrorCount

Number of times an exception was thrown while trying to acquire a cluster lock on this node since startup

TotalAcquireTimeMillis

Total time in milliseconds that any thread on this node has spent acquiring a lock (including time blocked waiting for a lock to become available)

TotalReleasedCount

Total number of times a cluster lock was released on this node since startup

TotalReleaseErrorCount

Total number of times an exception was thrown while releasing a cluster lock on this node since startup

SSH Session Statistics (com.atlassian.bitbucket:name=SshSessions)

Name Description
ActiveSessionCount Number of currently active SSH session
MaxActiveSessionCount Highest number of concurrently active SSH sessions since the last startup
SessionClosedCount Total number of SSH sessions that have been closed since the last startup
SessionCreatedCount Total number of SSH sessions that have been created since the last startup
SessionExceptionCount Total number of SSH sessions that have been terminated because an exception was thrown from the SSH command run


Interesting 3rd party library attributes

Bitbucket Server exposes the JMX attributes from number of third party libraries. Listed below is a sample of the attributes that are particularly interesting from an operations perspective. 

HikariCP - (com.zaxxer.hikari:type=Pool (bitbucket))

Name

Description

ActiveConnections

Active Connections (in use)

IdleConnections

Idle Connection count

ThreadsAwaitingConnection

The number of threads waiting for a connection (when all available connections are in use)

TotalConnections Total Connections
Hibernate - (org.hibernate.core:sessionFactory=bitbucket.core,serviceRole=org.hibernate.stat.Statistics,serviceType=org.hibernate.stat.internal.ConcurrentStatisticsImpl)
Name Description

QueryCacheHitCount

Global number of cached queries successfully retrieved from cache

QueryCacheMissCount

Global number of cached queries not found in cache

SecondLevelCacheHitCount

Global number of cacheable entities/collections successfully retrieved from the cache

SecondLevelCacheMissCount

Global number of cacheable entities/collections not found in the cache and loaded from the database

Expose JMX MBeans within Bitbucket Server

To enable Bitbucket Server to publish specific statistics using JMX you need to

  1. Locate and open the bitbucket.properties file in the <Bitbucket home directory>/shared directory. 
    1. Add this property to the file.

      jmx.enabled=true
    2. Save and close the file.
       
  2. Create a JMX password file for secure access to JMX monitoring.
     
  3. Modify the set-jmx-opts.sh file to enable Bitbucket Server to expose JMX Mbeans.


These changes will not take effect until Bitbucket Server has been restarted.

Set up the JMX password file

To set up a JMX password file to secure access to JMX monitoring

  1. Create a file named jmx.access.

    This file will contain password information. Ensure the file is only readable by the secure user Bitbucket Server will run under. However, note that if the Bitbucket Server user cannot read the file Bitbucket Server will fail to start.

  2. Edit the jmx.access file to include this property and save the file.

    monitorRole=<password>

    If you wish to use a username other than monitorRole or controlRole you will need to modify the jmxremote.access file located in the /lib/management/ directory of the installed Java.

Modify the Bitbucket Server environment file

To modify the set-jmx-opts.sh (for Windows set-jmx-opts.bat) files to enable JMX monitoring for Bitbucket Server

  1. Within the bin directory, locate the file set-jmx-opts.sh (for Windows set-jmx-opts.bat) and change these properties.

    JMX_REMOTE_AUTH=password
    JMX_REMOTE_PORT=3333
    RMI_SERVER_HOSTNAME=-Djava.rmi.server.hostname=<hostname>
    JMX_PASSWORD_FILE=<path>/jmx.access
  2. Restart Bitbucket Server.

Expose JMX MBeans when Bitbucket Server is run as a Windows service

To expose JMX MBeans when Bitbucket Server is run as a Windows service

  1. Stop the Bitbucket Server service.
  2. Open the command line prompt and enter.

    cmd
  3. Navigate to the Bitbucket Server bin directory.

    cd <Bitbucket Server installation dir>\bin
  4. Run this command.

    tomcat8w //ES//AtlassianBitbucket Server
  5. In the window that appears, click on the Java tab to see the list of current startup options. Under "Java Options:" form, input the value

    -Dcom.sun.management.jmxremote.port=<JMX_REMOTE_PORT>   
    -Djava.rmi.server.hostname=<hostname>
    -Dcom.sun.management.jmxremote.ssl=false 
    -Dcom.sun.management.jmxremote.password.file=<JMX_PASSWORD_FILE> 

    Ensure the owner of this password file is the secure user Bitbucket Server will run as. If the Bitbucket Server user cannot read the file, Bitbucket Server will fail to start.

  6. Replace the values within the < > characters.

    JMX_REMOTE_PORT=3333
    JMX_PASSWORD_FILE=<path>\jmx.access
  7. Restart Bitbucket Server Service.


Verify JMX is configured correctly

These steps use JConsole to test that JMX has been configured correctly. JConsole is a utility that ships with the Oracle JDK.

  1. To start the jconsole utility, from a command line prompt enter

    jconsole

  2. Create a new JConsole connection with similar connection settings.

    bitbucket the hostname of the instance of Bitbucket Server to monitor

    3333

    the JMX port number previously configured.

    username, password values configured within the JMX password file jmx.access.
  3. Click Connect.


When configured correctly, you will see these properties.

com.atlassian.bitbucket
  • CommandTickets
  • HostingTickets
  • Projects
  • Repositories
  • ScmStatistics
  • Tickets
  • EventStatistics
  • ClusterLocks

  • SshSessions

com.atlassian.bitbucket.thread-pools
  • EventThreadPool
  • IoPumpThreadPool
  • ScheduledThreadPool

Example performance dashboard

This dashboard was generated using Java Mission Control that ships with the Oracle JDK (since 1.7u40). See the documentation that comes with your JMX client of choice for more information. 

Configuring JMX to use SSL

You can find information about the options for configuring JMX to use SSL in the set-jmx-opts files. Comprehensive documentation is available from Oracle.


Last modified on Sep 27, 2018

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.