Unable to access JMX monitoring via Jconsole when Bitbucket Server is hosted in AWS
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Problem
Unable to access JMX counters when Bitbucket Server is hosted in an Amazon EC2 instance which is part of a VPC.
Specifically jconsole says the connection to the remote JMX fails.
Diagnosis
Environment
- Amazon EC2
- VPC using a Security Group with restrictive rules
Diagnostic Steps
- Verify that the JMX port you specified is open and listening
- Run jconsole with debug logging enabled for more data on the issue:
jconsole -J -Djava.util.logging.config.file=path_to_logging.properties_for_jconsoleYou should see something similar to the following in the jconsole debug output:
May 02, 2017 10:09:51 AM RMIConnector connect
FINER: [javax.management.remote.rmi.RMIConnector: rmiServer=Proxy[RMIServer,RemoteObjectInvocationHandler[UnicastRef [liveRef: [endpoint:[52.XX.XXX.XXX:33920](remote),objID:[-6cdfb6b5:15bc6963763:-7fff, 2103495553046990789]]]]]] failed to connect: java.rmi.ConnectException: Connection refused to host: 52.XX.XXX.XXX; nested exception is:
java.net.ConnectException: Operation timed out (Connection timed out)Cause
JMX uses 2 ports, the one specified in the setenv.sh file and a port that is randomly assigned. In environments where access to ports is restricted, binding to the randomly assigned port may be prevented.
Resolution
Set RMI port to the same JMX port specified in the setenv file by adding the following to the JVM_SUPPORT_RECOMMENDED_ARG configuration of the setenv file:
JVM_SUPPORT_RECOMMENDED_ARG="-Dcom.sun.management.jmxremote.rmi.port=3333"setenv and environment variable changes in Bitbucket Server 5.0+
Starting with Bitbucket Server 5.0, setenv.sh and setenv.bat have been removed. The options that were set in this file can now be set via environment variables. Where to set the environment variable depends on which Operating System you're running on.
Linux
When using the atlbitbucket service on Linux, the environment variables are ignored. You must set the parameters in _start-webapp.sh (or start-bitbucket.sh). These values will be read when the service starts.
As an example, to set JVM_SUPPORT_RECOMMENDED_ARGS, you would add this line to the file:
JVM_SUPPORT_RECOMMENDED_ARGS=-XX:+HeapDumpOnOutOfMemoryErrorWindows
Set the parameter as an environment variable for the user running Bitbucket Server. For example, if you want to set JVM_SUPPORT_RECOMMENDED_ARGS, create it as an environment variable and assign the appropriate value to it. When Bitbucket Server starts using the startup scripts or service, it will pick up and apply this value.