Recovering from a Data Center cluster split-brain
This article applies to Confluence Data Center 5.8.5 or later.
Confluence Data Center node will not start up and you see the following message in the logs:
2014-08-15 15:23:00,023 ERROR [scheduler_Worker-6] [confluence.cluster.safety.ClusterPanicListener] onClusterPanicEvent Received a panic event, stopping processing on the node: Clustered Confluence: Database is being updated by an instance which is not part of the current cluster. You should check network connections between cluster nodes, especially multicast traffic. 2014-08-15 15:23:00,035 WARN [scheduler_Worker-6] [confluence.cluster.safety.ClusterPanicListener] onClusterPanicEvent com.atlassian.confluence.cluster.hazelcast.HazelcastClusterInformation@29f82619 2014-08-15 15:23:00,036 WARN [scheduler_Worker-6] [confluence.cluster.safety.ClusterPanicListener] onClusterPanicEvent Shutting down Quartz scheduler
This is known as cluster split-brain (sometimes known as cluster panic), and can happen on any node (for example if you restart a node you may see the cluster split-brain message above on the same node or on a different node).
The cluster safety mechanism is designed to ensure that Confluence cannot become inconsistent because updates by one user are not visible to another. A failure of this mechanism is a fatal error in Confluence and is called cluster split-brain. Because the cluster safety mechanism helps prevents data inconsistency whenever any two copies of Confluence running against the same database, it is enabled in all instances of Confluence, not just Confluence Data Center.
A scheduled task, ClusterSafetyJob, runs every 30 seconds. In a cluster, this job is run only on one of the nodes. The scheduled task operates on a safety number – a randomly generated number that is stored both in the database and in the distributed cache used across a cluster. It does the following:
- Generate a new random number
- Compare the existing safety numbers, if there is already a safety number in both the database and the cache.
- If the numbers differ, publish a ClusterPanicEvent. Currently in Confluence, this causes the following to happen on each node in the cluster:
- disable all access to the application
- disable all scheduled tasks
- In Confluence 5.5 and earlier, update the database safety number to a new value, which will cause all nodes accessing the database to fail. From Confluence 5.6 onwards, the database safety number is not updated, to allow the other Confluence node/s to continue processing requests.
- If the numbers are the same or aren't set yet, update the safety numbers:
- set the safety number in the database to the new random number
- set the safety number in the cache to the new random number.
Cluster split-brain can have a number of causes. You should:
- Check that the network connectivity for multicast traffic is working between the nodes.
Check that the same multicast address is being used by all the nodes.
To determine the multicast address being used by a node, look in the logs for the string
Configuring Hazelcast with. For example:
2014-08-15 15:20:08,140 INFO [RMI TCP Connection(4)-127.0.0.1] [confluence.cluster.hazelcast.HazelcastClusterManager] configure Configuring Hazelcast with instanceName [nutella-buster], multicast address 126.96.36.199:54327, multicast TTL , network interfaces [fe80:0:0:0:0:0:0:1%1, 0:0:0:0:0:0:0:1, 127.0.0.1] and local port 580
To recover from a cluster split-brain:
- Verify that the network connectivity is fine.
- Confirm that the same multicast address is being used by all the nodes.
- Restart the nodes that panicked one at a time, and ensure that each one rejoins the cluster (go to > General Configuration > Clustering) before starting the next node.