Enabling JMX counters for performance monitoring

This article describes how to expose JMX MBeans within Stash for monitoring with a JMX client.

What is JMX?

JMX (Java Management eXtensions) is a technology for monitoring and managing Java applications. JMX uses objects called MBeans (Managed Beans) to expose data and resources from your application.

Why would I want to enable JMX monitoring within Stash?

For large instances of Stash or Stash Data Center, enabling JMX allows you to more easily monitor the consumption of application resources. This enables you to make better decisions about how to maintain and optimize machine resources.

On this page

Related reading

What can I monitor with JMX?

It is possible to monitor various statistics using JMX counters within Stash. Below are some examples of some statistics that can be monitored.

Stash repository statistics

  • Total number of projects
  • Total number of repositories
  • Git pushes and pulls
  • Various thread pools and attributes

 

Thread pools

Thread pool Description
IoPumpThreadPool Threads that handle external process IO
ScheduledThreadPool Thread pool that takes care of several miscellaneous scheduled tasks
EventThreadPool Threads that dispatch events to @EventListenermethods

 

Thread pool attributes

Name
Description
ActiveCount Returns the approximate number of threads that are actively executing tasks.
MaximumPoolSize Returns the maximum allowed number of threads.
PoolSize Returns the current number of threads in the pool.
QueueLength The number of tasks awaiting execution by the thread pool.
LargestPoolSize The largest number of threads that have ever been simultaneously in the pool.
CompletedTaskCount The approximate total number of tasks that have completed execution. Because the states of tasks and threads may change dynamically during computation, the returned value is only an approximation, but one that does not ever decrease across successive calls.

Ticket statistics

Stash uses 'tickets' as a mechanism for creating back-pressure to prevent the system from being overloaded with requests. There are two types of tickets used by stash, hosting tickets and command tickets. 
Hosting tickets: Limits the number of SCM hosting operations, meaning pushes and pulls over HTTP or SSH, which may be running concurrently. 
Command tickets:  Limits the number of SCM commands, such as: `git diff`, `git blame`, or `git rev-list`, which may be running concurrently.

Stash supports the following metrics for each ticket type.

Name Description
Total The total number of tickets allowed for this bucket
Available The number of tickets currently available, out of the configured total
LastRejection The timestamp of the last rejected ticket, or null if no tickets have been rejected
Name The name of the ticket bucket either 'scm-command' or 'scm-hosting'
OldestQueuedRequest The timestamp at which the oldest queued request started waiting, or null if there are no queued requests
QueuedRequests The number of requests currently waiting for an available ticket

 

Interesting 3rd party library attributes

Stash exposes the JMX attributes from number of third party libraries. Listed below is a sample of the attributes that are particularly interesting from an operations perspective. 

BoneCP  (com.jolbox.bonecp.BoneCP)

Name
Description

TotalLeased

Returns total number of connections currently in use by stash

ConnectionWaitTimeAvg

Return the average time it takes for a getConnection request to be services (in ms).

StatementExecutionTimeAvg

Return the average execution time for prepared statements to execute (in ms).

Hibernate (org.hibernate.com.stash.core.org.hibernate.stat.Statistics.org.hibernate.stat.internal.ConcurrentStatisticsImpl)

Name Description

QueryCacheHitCount

Get the global number of cached queries successfully retrieved from cache

QueryCacheMissCount

Get the global number of cached queries *not* found in cache

SecondLevelCacheHitCount

Global number of cacheable entities/collections successfully retrieved from the cache

SecondLevelCacheMissCount

Global number of cacheable entities/collections not found in the cache and loaded from the database.


Expose JMX MBeans within Stash

To enable Stash to publish specific statistics using JMX you need to

  1. Modify the stash-config.properties file.
  2. Create a JMX password file for secure access to JMX monitoring.
  3. Modify the setenv.sh file to enable Stash to expose JMX Mbeans.

These changes will not take effect until Stash has been restarted.

Modify the stash-config properties file

To modify (or create) the stash-config.properties file

  1. Create the stash-config.properties file, in the shared folder of your Stash home directory. Take care to use the standard format for Java properties files.

    The  stash-config.properties file is created automatically if you previously performed a  database migration.
  2. Add this property to the file.

    jmx.enabled=true

Set up the JMX password file

To set up a JMX password file to secure access to JMX monitoring

  1. Create a file named jmx.access.

    This file will contain password information. Ensure the file is only readable by the secure user Stash will run under. However, note that if the Stash user cannot read the file Stash will fail to start.

  2. Edit the jmx.access file to include this property and save the file.

    monitorRole=<password>

    If you wish to user a username other than monitorRole or controlRole you will need to modify the jmxremote.access file that is bundled with stash in the .install4j/jre.bundle/Contents/Home/jre/lib/management/ directory.

Modify the Stash environment file

To modify the setenv.sh (for Windows setenv.bat) files to enable JMX monitoring for Stash

  1. Within the bin directory, locate the file setenv.sh (for Windows setenv.bat) and change these properties.

    JMX_REMOTE_AUTH=password
    JMX_REMOTE_PORT=3333
    RMI_SERVER_HOSTNAME=-Djava.rmi.server.hostname=<hostname>
    JMX_PASSWORD_FILE=<path>/jmx.access
  2. Restart Stash.

Expose JMX MBeans when Stash is run as a Windows service

To expose JMX MBeans when Stash is run as a Windows service.

  1. Stop the Stash service.
  2. Open the command line prompt and enter.

    cmd
  3. Navigate to the Stash bin directory.

    cd <Stash Installation dir>\bin
  4. Run this command.

    tomcat8w //ES//AtlassianStash
  5. In the window that appears, click on the Java tab to see the list of current startup options. Under "Java Options:" form, input the value

    -Dcom.sun.management.jmxremote.port=<JMX_REMOTE_PORT>   
    -Djava.rmi.server.hostname=<hostname>
    -Dcom.sun.management.jmxremote.ssl=false 
    -Dcom.sun.management.jmxremote.password.file=<JMX_PASSWORD_FILE> 

    Ensure the owner of this password file is the secure user Stash will run as. If the Stash user cannot read the file, Stash will fail to start.

  6. Replace the values within the < > characters.

    JMX_REMOTE_PORT=3333
    JMX_PASSWORD_FILE=<path>\jmx.access
  7. Restart Stash Service.


Verify JMX is configured correctly

These steps use JConsole to test that JMX has been configured correctly. JConsole is a utility that ships with the Oracle JDK.

  1. To start the jconsole utility, from a command line prompt enter

    jconsole

  2. Create a new JConsole connection with similar connection settings.

    stash the hostname of the instance of Stash to monitor

    3333

    the JMX port number previously configured.

    username, password values configured within the JMX password file jmx.access.
  3. Click Connect.

 

When configured correctly, you will see these properties.

com.atlassian.stash
  • Projects
  • Repositories
  • ScmStatistics
  • Tickets
com.atlassian.stash.thread-pools
  • EventThreadPool
  • IoPumpThreadPool
  • ScheduledThreadPool

Example performance dashboard

This dashboard was generated using Java Mission Control that ships with the Oracle JDK (since 1.7u40). See the documentation that comes with your JMX client of choice for more information. 

Configuring JMX to use SSL

You can find information about the options for configuring JMX to use SSL in the setenv files. Comprehensive documentation is available from Oracle.

Redirection notice

This page will redirect to /display/BitbucketServer/Enabling+JMX+counters+for+performance+monitoring .

This article describes how to expose JMX MBeans within Stash for monitoring with a JMX client.

What is JMX?

JMX (Java Management eXtensions) is a technology for monitoring and managing Java applications. JMX uses objects called MBeans (Managed Beans) to expose data and resources from your application.

Why would I want to enable JMX monitoring within Stash?

For large instances of Stash or Stash Data Center, enabling JMX allows you to more easily monitor the consumption of application resources. This enables you to make better decisions about how to maintain and optimize machine resources.

On this page

Related reading

What can I monitor with JMX?

It is possible to monitor various statistics using JMX counters within Stash. Below are some examples of some statistics that can be monitored.

Stash repository statistics

  • Total number of projects
  • Total number of repositories
  • Git pushes and pulls
  • Various thread pools and attributes

 

Thread pools

Thread pool Description
IoPumpThreadPool Threads that handle external process IO
ScheduledThreadPool Thread pool that takes care of several miscellaneous scheduled tasks
EventThreadPool Threads that dispatch events to @EventListenermethods

 

Thread pool attributes

Name
Description
ActiveCount Returns the approximate number of threads that are actively executing tasks.
MaximumPoolSize Returns the maximum allowed number of threads.
PoolSize Returns the current number of threads in the pool.
QueueLength The number of tasks awaiting execution by the thread pool.
LargestPoolSize The largest number of threads that have ever been simultaneously in the pool.
CompletedTaskCount The approximate total number of tasks that have completed execution. Because the states of tasks and threads may change dynamically during computation, the returned value is only an approximation, but one that does not ever decrease across successive calls.

Ticket statistics

Stash uses 'tickets' as a mechanism for creating back-pressure to prevent the system from being overloaded with requests. There are two types of tickets used by stash, hosting tickets and command tickets. 
Hosting tickets: Limits the number of SCM hosting operations, meaning pushes and pulls over HTTP or SSH, which may be running concurrently. 
Command tickets:  Limits the number of SCM commands, such as: `git diff`, `git blame`, or `git rev-list`, which may be running concurrently.

Stash supports the following metrics for each ticket type.

Name Description
Total The total number of tickets allowed for this bucket
Available The number of tickets currently available, out of the configured total
LastRejection The timestamp of the last rejected ticket, or null if no tickets have been rejected
Name The name of the ticket bucket either 'scm-command' or 'scm-hosting'
OldestQueuedRequest The timestamp at which the oldest queued request started waiting, or null if there are no queued requests
QueuedRequests The number of requests currently waiting for an available ticket

 

Interesting 3rd party library attributes

Stash exposes the JMX attributes from number of third party libraries. Listed below is a sample of the attributes that are particularly interesting from an operations perspective. 

BoneCP  (com.jolbox.bonecp.BoneCP)

Name
Description

TotalLeased

Returns total number of connections currently in use by stash

ConnectionWaitTimeAvg

Return the average time it takes for a getConnection request to be services (in ms).

StatementExecutionTimeAvg

Return the average execution time for prepared statements to execute (in ms).

Hibernate (org.hibernate.com.stash.core.org.hibernate.stat.Statistics.org.hibernate.stat.internal.ConcurrentStatisticsImpl)

Name Description

QueryCacheHitCount

Get the global number of cached queries successfully retrieved from cache

QueryCacheMissCount

Get the global number of cached queries *not* found in cache

SecondLevelCacheHitCount

Global number of cacheable entities/collections successfully retrieved from the cache

SecondLevelCacheMissCount

Global number of cacheable entities/collections not found in the cache and loaded from the database.


Expose JMX MBeans within Stash

To enable Stash to publish specific statistics using JMX you need to

  1. Modify the stash-config.properties file.
  2. Create a JMX password file for secure access to JMX monitoring.
  3. Modify the setenv.sh file to enable Stash to expose JMX Mbeans.

These changes will not take effect until Stash has been restarted.

Modify the stash-config properties file

To modify (or create) the stash-config.properties file

  1. Create the stash-config.properties file, in the shared folder of your Stash home directory. Take care to use the standard format for Java properties files.

    The  stash-config.properties file is created automatically if you previously performed a  database migration.
  2. Add this property to the file.

    jmx.enabled=true

Set up the JMX password file

To set up a JMX password file to secure access to JMX monitoring

  1. Create a file named jmx.access.

    This file will contain password information. Ensure the file is only readable by the secure user Stash will run under. However, note that if the Stash user cannot read the file Stash will fail to start.

  2. Edit the jmx.access file to include this property and save the file.

    monitorRole=<password>

    If you wish to user a username other than monitorRole or controlRole you will need to modify the jmxremote.access file that is bundled with stash in the .install4j/jre.bundle/Contents/Home/jre/lib/management/ directory.

Modify the Stash environment file

To modify the setenv.sh (for Windows setenv.bat) files to enable JMX monitoring for Stash

  1. Within the bin directory, locate the file setenv.sh (for Windows setenv.bat) and change these properties.

    JMX_REMOTE_AUTH=password
    JMX_REMOTE_PORT=3333
    RMI_SERVER_HOSTNAME=-Djava.rmi.server.hostname=<hostname>
    JMX_PASSWORD_FILE=<path>/jmx.access
  2. Restart Stash.

Expose JMX MBeans when Stash is run as a Windows service

To expose JMX MBeans when Stash is run as a Windows service.

  1. Stop the Stash service.
  2. Open the command line prompt and enter.

    cmd
  3. Navigate to the Stash bin directory.

    cd <Stash Installation dir>\bin
  4. Run this command.

    tomcat8w //ES//AtlassianStash
  5. In the window that appears, click on the Java tab to see the list of current startup options. Under "Java Options:" form, input the value

    -Dcom.sun.management.jmxremote.port=<JMX_REMOTE_PORT>   
    -Djava.rmi.server.hostname=<hostname>
    -Dcom.sun.management.jmxremote.ssl=false 
    -Dcom.sun.management.jmxremote.password.file=<JMX_PASSWORD_FILE> 

    Ensure the owner of this password file is the secure user Stash will run as. If the Stash user cannot read the file, Stash will fail to start.

  6. Replace the values within the < > characters.

    JMX_REMOTE_PORT=3333
    JMX_PASSWORD_FILE=<path>\jmx.access
  7. Restart Stash Service.


Verify JMX is configured correctly

These steps use JConsole to test that JMX has been configured correctly. JConsole is a utility that ships with the Oracle JDK.

  1. To start the jconsole utility, from a command line prompt enter

    jconsole

  2. Create a new JConsole connection with similar connection settings.

    stash the hostname of the instance of Stash to monitor

    3333

    the JMX port number previously configured.

    username, password values configured within the JMX password file jmx.access.
  3. Click Connect.

 

When configured correctly, you will see these properties.

com.atlassian.stash
  • Projects
  • Repositories
  • ScmStatistics
  • Tickets
com.atlassian.stash.thread-pools
  • EventThreadPool
  • IoPumpThreadPool
  • ScheduledThreadPool

Example performance dashboard

This dashboard was generated using Java Mission Control that ships with the Oracle JDK (since 1.7u40). See the documentation that comes with your JMX client of choice for more information. 

Configuring JMX to use SSL

You can find information about the options for configuring JMX to use SSL in the setenv files. Comprehensive documentation is available from Oracle.

Was this helpful?

Thanks for your feedback!

Why was this unhelpful?

Have a question about this article?

See questions about this article

Powered by Confluence and Scroll Viewport