Enabling JMX counters for performance monitoring

This article describes how to expose JMX MBeans within Bitbucket Server for monitoring with a JMX client.

What is JMX?

JMX (Java Management eXtensions) is a technology for monitoring and managing Java applications. JMX uses objects called MBeans (Managed Beans) to expose data and resources from your application.

Why would I want to enable JMX monitoring within Bitbucket Server?

For large instances of Bitbucket Server or Bitbucket Data Center, enabling JMX allows you to more easily monitor the consumption of application resources. This enables you to make better decisions about how to maintain and optimize machine resources.

What can I monitor with JMX?

It is possible to monitor various statistics using JMX counters within Bitbucket Server. Below are some examples of some statistics that can be monitored.

Bitbucket Server repository statistics

• Total number of projects
• Total number of repositories
• Git pushes and pulls
• Various thread pools and attributes

Thread pools

Thread pool attributes

Ticket statistics

Bitbucket server uses 'tickets' as a mechanism for creating back-pressure to prevent the system from being overloaded with requests. There are two types of tickets used by Bitbucket server, hosting tickets and command tickets. 
Hosting tickets: Limits the number of SCM hosting operations, meaning pushes and pulls over HTTP or SSH, which may be running concurrently. 
Command tickets:  Limits the number of SCM commands, such as: `git diff`, `git blame`, or `git rev-list`, which may be running concurrently.

Bitbucket server supports the following metrics for each ticket type.

 

Interesting 3rd party library attributes

Bitbucket Server exposes the JMX attributes from number of third party libraries. Listed below is a sample of the attributes that are particularly interesting from an operations perspective. 

HikariCP - com.zaxxer.hikari.Pool (bitbucket)

Hibernate - org.hibernate.core/org.hibernate.stat.Statistics/org.hibernate.stat.internal.ConcurrentStatisticsImpl/bitbucket.core

Expose JMX MBeans within Bitbucket Server

To enable Bitbucket Server to publish specific statistics using JMX you need to

1. Modify the bitbucket.properties file.
2. Create a JMX password file for secure access to JMX monitoring.
3. Modify the setenv.sh file to enable Bitbucket Server to expose JMX Mbeans.
   

These changes will not take effect until Bitbucket Server has been restarted.

Modify the bitbucket.properties file

To modify (or create) the bitbucket.properties file

1. Create the bitbucket.properties file, in the shared folder of your Bitbucket home directory. Take care to use the standard format for Java properties files.

   The bitbucket.properties file is created automatically if you previously performed a database migration.

2. Add this property to the file.

   jmx.enabled=true

Set up the JMX password file

To set up a JMX password file to secure access to JMX monitoring

1. Create a file named jmx.access.

   This file will contain password information. Ensure the file is only readable by the secure user Bitbucket Server will run under. However, note that if the Bitbucket Server user cannot read the file Bitbucket Server will fail to start.

2. Edit the jmx.access file to include this property and save the file.
   

   monitorRole=<password>

   If you wish to user a username other than monitorRole or controlRole you will need to modify the jmxremote.access file that is bundled with bitbucket in the .install4j/jre.bundle/Contents/Home/jre/lib/management/ directory.

Modify the Bitbucket Server environment file

To modify the setenv.sh (for Windows setenv.bat) files to enable JMX monitoring for Bitbucket Server

1. Within the bin directory, locate the file setenv.sh (for Windows setenv.bat) and change these properties.

   JMX_REMOTE_AUTH=password
   JMX_REMOTE_PORT=3333
   RMI_SERVER_HOSTNAME=-Djava.rmi.server.hostname=<hostname>
   JMX_PASSWORD_FILE=<path>/jmx.access

2. Restart Bitbucket Server.

---

Expose JMX MBeans when Bitbucket Server is run as a Windows service

To expose JMX MBeans when Bitbucket Server is run as a Windows service.

1. Stop the Bitbucket Server service.

2. Open the command line prompt and enter.

   cmd

3. Navigate to the Bitbucket Server bin directory.

   cd <Bitbucket Server Installation dir>\bin

4. Run this command.

   tomcat8w //ES//AtlassianBitbucket Server

5. In the window that appears, click on the Java tab to see the list of current startup options. Under "Java Options:" form, input the value

   -Dcom.sun.management.jmxremote.port=<JMX_REMOTE_PORT>   
   -Djava.rmi.server.hostname=<hostname>
   -Dcom.sun.management.jmxremote.ssl=false 
   -Dcom.sun.management.jmxremote.password.file=<JMX_PASSWORD_FILE> 

   Ensure the owner of this password file is the secure user Bitbucket Server will run as. If the Bitbucket Server user cannot read the file, Bitbucket Server will fail to start.

6. Replace the values within the < > characters.

   JMX_REMOTE_PORT=3333
   JMX_PASSWORD_FILE=<path>\jmx.access

7. Restart Bitbucket Server Service.

---

Verify JMX is configured correctly

These steps use JConsole to test that JMX has been configured correctly. JConsole is a utility that ships with the Oracle JDK.

1. To start the jconsole utility, from a command line prompt enter

   jconsole

   

2. Create a new JConsole connection with similar connection settings.

   bitbucket	the hostname of the instance of Bitbucket Server to monitor
   3333	the JMX port number previously configured.
   username, password	values configured within the JMX password file jmx.access.

3. Click Connect.

 

When configured correctly, you will see these properties.

Example performance dashboard

This dashboard was generated using Java Mission Control that ships with the Oracle JDK (since 1.7u40). See the documentation that comes with your JMX client of choice for more information. 

Configuring JMX to use SSL

You can find information about the options for configuring JMX to use SSL in the setenv files. Comprehensive documentation is available from Oracle.