Enabling JMX counters for performance monitoring

On this page

Still need help?

The Atlassian Community is here for you.

Ask the community

This article describes how to expose JMX MBeans within Bitbucket Data Center and Server for monitoring with a JMX client.

What is JMX?

JMX (Java Management eXtensions) is a technology for monitoring and managing Java applications. JMX uses objects called MBeans (Managed Beans) to expose data and resources from your application.

Why would I want to enable JMX monitoring within Bitbucket?

On this page:

Related reading:

For large Bitbucket instances, enabling JMX allows you to more easily monitor the consumption of application resources. This enables you to make better decisions about how to maintain and optimize machine resources.

What can I monitor with JMX?

It is possible to monitor various statistics using JMX counters within Bitbucket. Below are some examples of some statistics that can be monitored.

Mail statistics (com.atlassian.bitbucket:name=MailStatistics)

NameDescription
AverageMessageSizeAverage size (in bytes) of messages sent
LargestMessageSentLargest message that has been sent (in bytes)
LastMessageFailureDate of the last failure to send a message
LastMessageSuccessDate of the last successful message send operation
LastQueueFullEventLast time the message queue was full
QueueFullEventCountNumber of times the message queue was full
QueueUsageQueue usage as a fraction, 0.0d indicates empty and 1.0d indicates full
QueuedMessagesCountCurrent count of queued (unsent) messages
QueuedMessagesSizeCurrent size (in bytes) of the queued (unsent) messages
TotalMailDataSentTotal size (in bytes) of messages sent
TotalMessagesFailedTotal number of messages that failed to send
TotalMessagesSentTotal number of messages sent

Diagnostics

Alerts Total (com.atlassian.diagnostics:type=Alerts,name=Total)

NameDescription
LatestAlertTimestampTimestamp of the most recent alert
TotalCountTotal number of alerts since the JVM was started
ErrorCountNumber of alerts of severity ERROR since the JVM was started
InfoCountNumber of alerts of severity INFO since the JVM was started
WarningCountNumber of alerts of severity WARNING since the JVM was started

Plugin (com.atlassian.diagnostics:type=Alerts,Category=Plugin,name=${PLUGIN_NAME})

NameDescription
LatestAlertTimestampTimestamp of the most recent alert
TotalCountTotal number of alerts since the JVM was started
ErrorCountNumber of alerts of severity ERROR since the JVM was started
InfoCountNumber of alerts of severity INFO since the JVM was started
WarningCountNumber of alerts of severity WARNING since the JVM was started
PluginNamePlugin name, if available

Issue (com.atlassian.diagnostics:type=Alerts,Category=Issue,name=${ISSUE_ID})

NameDescription
LatestAlertTimestampTimestamp of the most recent alert
ComponentComponent the issue is defined for
CountNumber of alerts for the issue since the JVM was started
SeverityIssue's severity
DescriptionIssue's description


Hosting statistics

ProtocolObject name
SSHcom.atlassian.bitbucket:name=SshHostingStatistics
HTTPcom.atlassian.bitbucket:name=HttpHostingStatistics

Hosting statistic attributes

All the hosting statistics attributes are monotonically increasing since the JVM was restarted

NameDescription
CloneCacheBypassClone requests that have bypassed the scm-cache
CloneCacheHitClone requests served from the scm-cache
CloneCacheMissClone requests that could not be served from the scm-cache
CloneReadBytes read from clients during clone operations
CloneRequestCountNumber of clone requests served
CloneWrittenbytes written to clients during <i>clone</i> operations
FetchReadBytes read from clients during <i>fetch</i> operations
FetchRequestCountNumber of fetch requests served
FetchWrittenBytes written to clients during fetch operations
RequestsTotal number of requests served
TotalBytesReadTotal bytes read from clients
TotalBytesWrittenTotal bytes written to clients


Webhooks statistics (com.atlassian.webhooks:name=Webhooks)

NameDescription
PublishCountA count of the total number of events that could trigger webhooks
(A publish may create many dispatches)
DispatchSuccessCountTotal number of webhooks to fire successfully with a successful HTTP response
DispatchRejectedCountA count of the number of webhook dispatches that were rejected for execution
DispatchLastRejectedTimestampThe last time a webhook was rejected, either from circuit breaking, or due to too many webhooks being in flight
DispatchInFlightCountTotal number of dispatches that have been triggered and are awaiting resolution
DispatchFailureCountTotal number of webhooks that fired successfully, but the HTTP response indicates a failure
(non 2xx code)
DispatchErrorCountTotal number of webhooks to have had an error while they were being dispatched
DispatchCountTotal number of webhooks to have been dispatched

Thread pools

Thread poolDescriptionObject name
BuildActionsThreadPool

Threads that handle Integrated CI/CD build actions

com.atlassian.bitbucket.thread-pools:name=BuildActionsThreadPool

EventThreadPoolThreads that dispatch events to @EventListenermethodscom.atlassian.bitbucket.thread-pools:name=EventThreadPool
IoPumpThreadPoolThreads that handle blocking process I/Ocom.atlassian.bitbucket.thread-pools:name=IoPumpThreadPool
NioPumpThreadPoolThreads that handle nonblocking process I/Ocom.atlassian.bitbucket.thread-pools:name=NioPumpThreadPool
ScheduledThreadPoolThread pool that takes care of several miscellaneous scheduled taskscom.atlassian.bitbucket.thread-pools:name=ScheduledThreadPool

Thread pool attributes

Name

Description

ActiveCountReturns the approximate number of threads that are actively executing tasks
MaximumPoolSizeReturns the maximum allowed number of threads
PoolSizeReturns the current number of threads in the pool
QueueLengthThe number of tasks awaiting execution by the thread pool
LargestPoolSizeThe largest number of threads that have ever been simultaneously in the pool
CompletedTaskCountThe approximate total number of tasks that have completed execution. Because the states of tasks and threads may change dynamically during computation, the returned value is only an approximation, but one that does not ever decrease across successive calls

Repositories (com.atlassian.bitbucket:name=Repositories)

NameDescription
CountNumber of repositories currently configured across all projects

Scm Statistics (com.atlassian.bitbucket:name=ScmStatistics)

NameDescription
PullsNumber of scm pulls serviced by this instance since it was started
PushesNumber of scm pushes received by this instance is it was started


Ticket statistics

Bitbucket uses 'tickets' as a mechanism for creating back-pressure to prevent the system from being overloaded with requests. There are two types of tickets, hosting tickets and command tickets. 
Hosting tickets (com.atlassian.bitbucket:name=HostingTickets): Limits the number of SCM hosting operations, meaning pushes and pulls over HTTP or SSH, which may be running concurrently. 
Command tickets(com.atlassian.bitbucket:name=CommandTickets):  Limits the number of SCM commands, such as: `git diff`, `git blame`, or `git rev-list`, which may be running concurrently.

Bitbucket supports the following metrics for each ticket type.

NameDescription
AvailableThe number of tickets available for acquisition (lower number means higher load)
LastRejectionThe timestamp of the last rejected ticket, or null if no tickets have been rejected
NameThe name of the ticket bucket either 'scm-command' or 'scm-hosting'
OldestQueuedRequestThe timestamp at which the oldest queued request started waiting, or null if there are no queued requests
QueuedRequestsThe number of requests currently waiting for an available ticket
TotalThe maximum number of tickets that can be acquired concurrently before back-pressure is applied
UsedThe number of tickets that have been acquired (higher number means higher load)

Event Statistics (com.atlassian.bitbucket:name=EventStatistics)

NameDescription
DispatchedCountTotal number of listener callbacks that have been performed. An event that is delivered to 10 listeners counts as 10 dispatches
LastRejectionDate of the last event being rejected, or null if no event has been rejected
PublishedCountTotal number of events delivered. An event that is delivered to 10 listeners counts as 1 event
QueueCapacityMaximum number of event callbacks that can be queued before events are rejected
QueueLengthNumber of event callbacks that have been queued but haven't been dispatched yet
RejectedCountTotal number of events that were not dispatched because the event queue was full
RemainingQueueCapacityRemaining number of event callbacks that can be queued before events are rejected

Cluster Lock Statistics (com.atlassian.bitbucket:name=ClusterLocks)

NameDescription

LockedCount

Number of cluster locks that are currently held by this node

QueuedThreadCount

Number of threads on this node that are currently blocked waiting for a lock

TotalAcquiredCount

Total number of times a cluster lock was acquired on this node since startup

TotalAcquireErrorCount

Number of times an exception was thrown while trying to acquire a cluster lock on this node since startup

TotalAcquireTimeMillis

Total time in milliseconds that any thread on this node has spent acquiring a lock (including time blocked waiting for a lock to become available)

TotalReleasedCount

Total number of times a cluster lock was released on this node since startup

TotalReleaseErrorCount

Total number of times an exception was thrown while releasing a cluster lock on this node since startup

SSH Session Statistics (com.atlassian.bitbucket:name=SshSessions)

NameDescription
ActiveSessionCountNumber of currently active SSH session
MaxActiveSessionCountHighest number of concurrently active SSH sessions since the last startup
SessionClosedCountTotal number of SSH sessions that have been closed since the last startup
SessionCreatedCountTotal number of SSH sessions that have been created since the last startup
SessionExceptionCountTotal number of SSH sessions that have been terminated because an exception was thrown from the SSH command run

Rate limiting statistics (com.atlassian.bitbucket:name=RateLimitStatistics)

NameDescription
RejectedRequestCountThe number of rate limited requests
UserMapSizeThe number of token buckets currently in memory

Interesting 3rd party library attributes

Bitbucket exposes the JMX attributes from number of third party libraries. Listed below is a sample of the attributes that are particularly interesting from an operations perspective. 

HikariCP - (com.zaxxer.hikari:type=Pool (bitbucket))

Name

Description

ActiveConnections

Active Connections (in use)

IdleConnections

Idle Connection count

ThreadsAwaitingConnection

The number of threads waiting for a connection (when all available connections are in use)

TotalConnectionsTotal Connections
Hibernate - (org.hibernate.core:sessionFactory=bitbucket.core,serviceRole=org.hibernate.stat.Statistics,serviceType=org.hibernate.stat.internal.ConcurrentStatisticsImpl)
NameDescription

QueryCacheHitCount

Global number of cached queries successfully retrieved from cache

QueryCacheMissCount

Global number of cached queries not found in cache

SecondLevelCacheHitCount

Global number of cacheable entities/collections successfully retrieved from the cache

SecondLevelCacheMissCount

Global number of cacheable entities/collections not found in the cache and loaded from the database

Expose JMX MBeans within Bitbucket

To enable Bitbucket to publish specific statistics using JMX:

  1. Locate and open the bitbucket.properties file in the <Bitbucket home directory>/shared directory. 
    1. Add this property to the file.

      jmx.enabled=true
    2. Save and close the file.
       
  2. Create a JMX password file for secure access to JMX monitoring.
     
  3. Modify the set-jmx-opts.sh file to enable Bitbucket to expose JMX Mbeans.


These changes will not take effect until Bitbucket is restarted.

Set up the JMX password file

To set up a JMX password file to secure access to JMX monitoring

  1. Create a file named jmx.access.

    This file will contain password information. Ensure the file is only readable by the secure user Bitbucket will run under. However, note that if the user cannot read the file Bitbucket will fail to start.

  2. Edit the jmx.access file to include this property and save the file.

    monitorRole <password>

    If you wish to use a username other than monitorRole or controlRole you will need to modify the jmxremote.access file located in the /lib/management/ directory of the installed Java.

  3. Change ownership of jmx.access file,

    chown bitbucket:bitbucket <path>/jmx.access

    where bitbucket is the user that runs Bitbucket service.

  4. Change file permissions of jmx.access file.

    chmod 600 <path>/jmx.access

Modify the Bitbucket environment file

Modify the set-jmx-opts.sh (for Windows set-jmx-opts.bat) files to enable JMX monitoring:

  1. Within the bin directory, locate the file set-jmx-opts.sh (for Windows set-jmx-opts.bat) and change these properties.

    JMX_REMOTE_AUTH=password
    JMX_REMOTE_PORT=3333
    RMI_SERVER_HOSTNAME=-Djava.rmi.server.hostname=<hostname>
    JMX_PASSWORD_FILE=<path>/jmx.access
  2. Restart Bitbucket.

Docker

For Docker deployments, the properties can be passed as Docker environment variables:

Sample:

docker run -v /data/bitbucket:/var/atlassian/application-data/bitbucket \
  --name="bitbucket" \
  -d -p 7990:7990 -p 7999:7999 -p 3333:3333 \
  -e JMX_ENABLED=true \
  -e JMX_REMOTE_AUTH=password \
  -e JMX_REMOTE_PORT=3333 \
  -e JMX_REMOTE_RMI_PORT=3333 \
  -e RMI_SERVER_HOSTNAME=<hostname> \
  -e JMX_PASSWORD_FILE=<path>/jmx.access \
  atlassian/bitbucket

Expose JMX MBeans when Bitbucket is run as a Windows service

To expose JMX MBeans when Bitbucket is run as a Windows service

  1. Stop the Bitbucket service.
  2. Open the command line prompt and enter.

    cmd
  3. Navigate to the Bitbucket bin directory.

    cd <Bitbucket installation dir>\bin
  4. Run this command.

    tomcat8w //ES//AtlassianBitbucket Server
  5. In the window that appears, click on the Java tab to see the list of current startup options. Under "Java Options:" form, input the value

    -Dcom.sun.management.jmxremote.port=<JMX_REMOTE_PORT>   
    -Djava.rmi.server.hostname=<hostname>
    -Dcom.sun.management.jmxremote.ssl=false 
    -Dcom.sun.management.jmxremote.password.file=<JMX_PASSWORD_FILE> 

    Ensure the owner of this password file is the secure user Bitbucket will run as. If the user cannot read the file, Bitbucket will fail to start.

  6. Replace the values within the < > characters.

    JMX_REMOTE_PORT=3333
    JMX_PASSWORD_FILE=<path>\jmx.access
  7. Restart the Bitbucket service.


Verify JMX is configured correctly

These steps use JConsole to test that JMX has been configured correctly. JConsole is a utility that ships with the Oracle JDK.

  1. To start the jconsole utility, from a command line prompt enter

    jconsole

  2. Create a new JConsole connection with similar connection settings.

    bitbucketthe hostname of the instance to monitor

    3333

    the JMX port number previously configured.

    username, passwordvalues configured within the JMX password file jmx.access.
  3. Click Connect.


When configured correctly, you will see these properties.

com.atlassian.bitbucket
  • CommandTickets
  • HostingTickets
  • Projects
  • Repositories
  • ScmStatistics
  • Tickets
  • EventStatistics
  • ClusterLocks

  • SshSessions

com.atlassian.bitbucket.thread-pools
  • EventThreadPool
  • IoPumpThreadPool
  • ScheduledThreadPool

Example performance dashboard

This dashboard was generated using Java Mission Control that ships with the Oracle JDK (since 1.7u40). See the documentation that comes with your JMX client of choice for more information. 

Configuring JMX to use SSL

You can find information about the options for configuring JMX to use SSL in the set-jmx-opts files. Comprehensive documentation is available from Oracle.


Last modified on Mar 2, 2022

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.