Problem

Unable to access JMX counters when Confluence Server is hosted in an Amazon EC2 instance which is part of a VPC.

Specifically jConsole says the connection to the remote JMX fails. 

Diagnosis

Environment

• Amazon EC2
• VPC using a Security Group with restrictive rules

Diagnostic Steps

1. Verify that the JMX port you specified is open and listening
2. Verify that the AWS Security Group allows traffic through on the jmxremote port

3. Run jConsole with debug logging enabled for more data on the issue:

   jconsole -debug

4. You should see something similar to the following in the jconsole debug output:

   java.rmi.ConnectException: Connection refused to host 127.0.0.1; nested exception is
        java.net.ConnectException: Connection refused: connect

5. From your local machine (if Linux or Mac) run: netstat -anlp and check the results for the port you are trying to connect to.  You may see an ESTABLISHED result:

   tcp 0 0 <AWS IP ADDRESS>:<PORT NUMBER> <IP OF YOUR MACHINE>:51171 ESTABLISHED 11076/java

Cause

JMX uses 2 ports, the one specified in the setenv.sh file and a port that is randomly assigned. In environments where access to ports is restricted, binding to the randomly assigned port may be prevented. 

Resolution

• Specify an RMI port and host by adding the following to the CATALINA_OPTS configuration of the setenv file:
  

  -Dcom.sun.management.jmxremote.rmi.port=<SPECIFY DIFFERENT PORT THAN THE JMXREMOTE PORT>
  -Djava.rmi.server.hostname=<AWS IP ADDRESS FROM NETSTAT OUTPUT ABOVE>

• Make sure the new port above is allowed in your AWS Security Group
• Re-start your Confluence, and try to connect via jConsole again