Jira Service Management Data Center cluster authentication

The nodes in your Jira Data Center cluster exchange cache replication messages with each other via Java RMI over TCP. This communication is unencrypted and – until Jira 8.17 – it was also unauthenticated, as the RMI ports used by the cache replication process aren’t intended to be accessed by any external software.

To improve security of your Jira Data Center cluster, in Jira 8.17 we introduced the cluster authentication mechanism (also back-ported to Jira 8.13.8 and 8.5.16). It protects the RMI ports used for cache replication by using an authentication that is based on an automatically-generated key. This key is shared between the nodes in your cluster through the Jira database.

The authentication is implemented as a mutual symmetric challenge-response protocol that commences each RMI connection between the cluster nodes. Any subsequent messages exchanged within such a connection aren’t additionally secured. There is no extra encryption or signing of the connection payload.

Enabling the Jira cluster authentication

From Jira 8.17 (as well as 8.13.8 and 8.5.16), the cluster authentication is enabled by default, you don’t have to take any actions.

Jira will generate the authentication key and store it in your database after you start your Jira instance for the first time. If you’re using zero downtime upgrades, the key will be generated at the end of the upgrade process. Once the key is generated, any subsequent cache replication RMI connections are authenticated.

Regenerating the authentication key

It might happen that you’ll need to regenerate the authentication key – just in case it gets compromised or your security policy requires that you do it periodically. You can do it by deleting the existing key from your database.

To regenerate the authentication key:

1. Issue the following SQL statement in the Jira database, and commit it:

   delete from securityproperty where property_key = 'rmi.socket.cluster.auth.secret.key';

2. You don’t have to restart Jira. The new key will be generated within a minute. The communication will continue to use the old key for a short time, and then switch to using the new key, just to not allow for any unauthenticated communication.

Disabling the Jira cluster authentication

You can disable the Jira cluster authentication, for example for troubleshooting purposes, in two ways, and the biggest difference is in whether you can allow yourself to restart your Jira instance.

Restart required (recommended)

Set the com.atlassian.jira.cluster.distribution.localq.rmi.auth.disabled system property to true on each of your Data Center nodes.

(This is a Java system property and needs to be set with the JVM arguments to be picked up. It can not be set through the UI.)

Restart not required (temporary)

Another way is to edit the securityproperty table and set its value to an empty key, like in this example: 

update securityproperty set property_value = '' where property_key = 'rmi.socket.cluster.auth.secret.key';

This method, however, results in:

• Risk of decreasing performance

• Warnings being logged:

  WARN [c.a.j.c.d.l.rmi.auth.DefaultClusterAuthSharedKeySupplier] Cluster Authentithentication disabled due to missing security properties key rmi.socket.cluster.auth.secret.key, warning counter: 1. Note the frequency of this warning is controlled by com.atlassian.jira.cluster.distribution.localq.rmi.auth.warn.min system property

To re-enable the Jira cluster authentication after using this method (empty key), you’ll need to regenerate the authentication key as described in the section above.

Changing the server-side socket timeout

The server-side socket timeout used during the Jira cluster authentication phase of creating the RMI connection is 10 seconds by default.

To change the socket timeout:

• Change the value of the following system property on each node: 

  com.atlassian.jira.cluster.distribution.localq.rmi.server.auth.so.timeout.millis

Jira cluster authentication statistics

Jira stats logs are now extended with the metrics related to the Jira cluster authentication, described in the table bellow. For more info, see Jira stats logs.

How are these statistics collected?

These metric statistics are collected independently for each node and written to the primary Jira log file for a given node at the interval specified by the com.atlassian.jira.stats.logging.interval system property (per each node, defaults to 5 minutes).

Each node acts as both the server and client in the RMI communication, and so the statistics are collected separately for a client-side and a server-side. The statistics are accumulated in two manners: since node start (called total) and since last log entry (called snapshot). That’s why, you’ll find the following four categories of these statistics in the log file:

• Cluster authentication server snapshot stats
• Cluster authentication client snapshot stats
• Cluster authentication server total stats
• Cluster authentication client total stats

The cluster authentication statistics data is JSON-formatted and tagged with the [JIRA-STATS] [JIRA-RMI-AUTH] prefix in the log file, like in the following example:

INFO [c.a.j.c.d.l.rmi.auth.ClusterAuthStatsManager] [JIRA-STATS] [JIRA-RMI-AUTH] Cluster authentication client snapshot stats: {"startTimestampMillis":1618398658051,"authsTotal":37,"authsSkippedNoSecret":0,"authsFailed":0,"authsFailedByTimeout":0,"authsSuccess":37,"timeToAuthMillis":{"count":37,"min":0,"max":45,"sum":665,"avg":17,"distributionCounter":{"0":3,"1":2,"5":3,"10":8,"50":21,"100":0,"500":0,"1000":0,"5000":0}}}