Cache not replicating between Jira Nodes in Data Center

Summary

The cache is not replicating between Jira Nodes.

Environment

Jira Data Center using at least 2 nodes.

Any Jira version that supports Data Center.

Diagnosis

• The following failure can be found in the Jira Health Check file included in the support zip:

  1 2 3 4 Name: Cluster Cache Replication Is healthy: false Failure reason: The node nodeX is not replicating Severity: CRITICAL

• The following error can be found in the Jira application logs. Note the space character located between the host IP address "XX.XXX.X.X" and the character ";":

  1 2 3 4 5 6 7 8 9 10 11 12 2020-01-31 07:32:44,006-0400 localq-reader-7 ERROR [c.a.j.c.distribution.localq.LocalQCacheOpReader] [LOCALQ] [VIA-INVALIDATION] Abandoning sending: LocalQCacheOp{cacheName='com.atlassian.jira.propertyset.CachingOfBizPropertyEntryStore.cache', action=REMOVE, key=CacheKey[entityName=ApplicationUser,entityId=13871], value == null ? true, replicatePutsViaCopy=false, creationTimeInMillis=1622456749450} from cache replication queue: [queueId=queue_62b7f2e4a4c84b03a74d49c29a831172_7_4585b5a075e0343372457f1f073d6e92, queuePath=/var/atlassian/application-data/jira/localq/queue_62b7f2e4a4c84b03a74d49c29a831172_7_4585b5a075e0343372457f1f073d6e92], failuresCount: 10/10. Removing from queue. Error: java.rmi.UnknownHostException: Unknown host: XX.XXX.X.X ; nested exception is: java.net.UnknownHostException: XX.XXX.X.X com.atlassian.jira.cluster.distribution.localq.LocalQCacheOpSender$UnrecoverableFailure: java.rmi.UnknownHostException: Unknown host: XX.XXX.X.X ; nested exception is: java.net.UnknownHostException: XX.XXX.X.X at com.atlassian.jira.cluster.distribution.localq.rmi.LocalQCacheOpRMISender.send(LocalQCacheOpRMISender.java:90) at com.atlassian.jira.cluster.distribution.localq.LocalQCacheOpReader.run(LocalQCacheOpReader.java:96) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.rmi.UnknownHostException: Unknown host: XX.XXX.X.X ; nested exception is:

• When checking the cluster.properties file of the problematic node (located in the local node home folder), a trailing space can be found after the node hostname:

  1 2 3 4 5 6 7 8 9 # This ID must be unique across the cluster jira.node.id = nodeX # The location of the shared home directory for all Jira nodes jira.shared.home = /mounted_directory/jira_shared_home # IP Address used by this node for cache replication ehcache.listener.hostName = XX.XXX.X.X # Ports used by this node for cache replication ehcache.listener.port = 40001 ehcache.object.port = 40011

• In some cases, the configuration of the cluster.properties file is correct, but the HOSTNAME environment variable is set to the wrong name or IP address

Cause

The Jira nodes in the Data Center cluster are unable to reach the node that fails to replicate because its hostname is invalid in the cluster.properties file (it has a trailing space).

Solution

1. Shutdown all the Data Center nodes

2. Edit the ehcache.listener.hostName setting from the cluster.properties file of the problematic node by removing the empty space after the IP address:

   • This file is located at the local <JIRA_HOME> folder of this node

3. Make sure that the HOSTNAME environment variable is set to the appropriate hostname.

4. Update the entry in the clusternode table of the Jira Database to ensure that the empty space is also removed there:

   1 update clusternode set ip = '<node_IP_address>' where node_id = '<node_ID>';

5. Restart all the Jira nodes