Data Center Troubleshooting

This page covers troubleshooting for a Data Center installation of Confluence.

If you're experiencing Cluster Panic messages in non-clustered installation of Confluence, visit the Knowledge Base article 'Database is being updated by an instance which is not part of the current cluster' Error Message.

(warning) You must ensure the clocks on your cluster nodes don't diverge, as it can result in a range of problems with your cluster.

Symptoms

Below is a list of potential problems with Confluence Data Center, and their likely solutions. 

On this page:

Related pages:

Problem

Likely solutions

Database is being updated by an instance which is not part of the current cluster errors on a stand-alone

'Database is being updated by an instance which is not part of the current cluster' Error Message

Database is being updated by an instance which is not part of the current cluster errors on a cluster

Add multicast route, Check firewall, Cluster Panic due to Multiple Deployments

Cannot assign requested address on startup, featuring an IPv6 address

Prefer IPv4

Error in log: The interface is not suitable for multicast communication

Change multicast interface, Add multicast route

Multicast being sent, but not received

Check firewall, Check intermediate routers, Increase multicast TTL

App is unlicensed on some nodes after updating the license on one node.Disable and re-enable the app in the Universal Plugin Manager.
After an app update, strings appear in the UI instead of buttons and icons on some nodes. Restart the affected node.
Hazelcast CANNOT start on this node. No matching network interface found.
See Hazelcast CANNOT start on this node. No matching network interface found KB article

Any issue not covered here

Contact support

Multicast

  • Which multicast address?

The multicast address and port used by Confluence can be found on the Cluster Configuration page, or in confluence.cfg.xml in the Confluence home directory.

  • Multicast address generation.

Confluence uses a hashing algorithm to take the inputted name during setup and it is then turned into a multicast address stored in the config file. Thus, once the initial setup is completed, Confluence will use the address this is the reason why user can change the address if needed, without actually changing the name. Consequently the additional nodes using the same multicast address specified in the config file are able to join the cluster.

Each node has a multicast address configured in the confluence.cfg.xml file

name="confluence.cluster.address">xxx.xx.xxx.xxx</property>

A warning message is displayed when an user changes the address from the one that Confluence has generated by the hashing of the name. There is no way of eliminating the message any other way other than by returning the address to the one that matches the cluster name. Purpose of the warning message is to remind the user that the address has been changed - as it is not the hashed version any longer - consequently the node can not join the cluster just by using the name. It is also necessary to provide the correct address as well.

Mapping interface to IP address.

To ensure that the interface name is mapped correctly, the following tool  can be used. It shows the mapping of the interface name to the IP address.

C:\>java -jar list-interfaces.jar
interfaces.size() = 4
networkInterface[0] = name:lo (MS TCP Loopback interface) index: 1 addresses:
/127.0.0.1;

networkInterface[1] = name:eth0 (VMware Virtual Ethernet Adapter for VMnet8) index: 2 addresses:
/192.168.133.1;

networkInterface[2] = name:eth1 (VMware Virtual Ethernet Adapter for VMnet1) index: 3 addresses:
/192.168.68.1;

networkInterface[3] = name:eth2 (Broadcom NetXtreme 57xx Gigabit Controller - Packet Scheduler Miniport) index: 4 addresses:
/192.168.0.101;

Debugging tools

Listed below are some debugging tools that help determine what the status of the multicast traffic is:

Tool

Information provided

netstat -gn

Lists multicast groups. Does not work on Mac OS X.

netstat -rn

Lists system routing table.

tcpdump -i interface

Captures network traffic on the given interface. Most useful on an interface that only receives cluster traffic.

Add multicast route

Multicast networking requirements vary across operating systems. Some operating systems require little configuration, while some require the multicast address to be explicitly added to a network interface before Confluence can use it. If multicast traffic can't be sent or received correctly, adding a route for multicast traffic on the correct interface will often fix the problem. The example below is for a Ubuntu Linux system:

route add -net 224.0.0.0 netmask 240.0.0.0 dev eth0

To support multiple applications using multicast on different interfaces, you may need to specify a route specific to the Confluence multicast address.

Check firewall

Ensure your firewall allows UDP traffic on the multicast address and port used by Confluence.

Prefer IPv4

There are known issues relating to IPv6.   You should configure your JVM to try binding to an IPv4 address first.  

Change multicast interface

Confluence might have selected the incorrect interface for multicast traffic, which means it cannot connect to other nodes in the cluster. To override the interface used for multicast traffic after initial setup, edit the confluence.cluster.interface property in  <local-home>/confluence.cfg.xml and specify the network interface. For example to tell Confluence to use eth1:

<property name="confluence.cluster.interface">eth1</property>

Overriding Hazelcast Configuration

If the solution to your problem involves changes to the Hazelcast configuration, these changes should not be made to the Confluence configuration files. Instead, to ensure your configuration survives upgrades, make your changes by creating a Hazelcast override file.

Increase multicast TTL

The multicast time-to-live (TTL) specifies how many hops a multicast packet should be allowed to travel before it is discarded by a router. It should be set to the number of routers in between your clustered nodes: 0 if both are on the same machine, 1 if on two different machines linked by a switch or cable, 2 if on two different machines with one intermediate router, and so on.

To increase the multicast TTL by edit the confluence.cluster.ttl property in the <local home>/confluence.cfg.xml file on each node.  For example to set the TTL to 3:

<property name="confluence.cluster.ttl">3</property>

Check intermediate routers

Advanced switches and routers have the ability to understand multicast traffic, and route it appropriately. Unfortunately sometimes this functionality doesn't work correctly with the multicast management information (IGMP) published by the operating system running Confluence.

If multicast traffic is problematic, try disabling advanced multicast features on switches and routers in between the clustered nodes. These features can prevent multicast traffic being transmitted by certain operating systems.

Didn't find a solution?

Check Related Articles from the Confluence Knowledge Base

Contact Atlassian support

We have dedicated staff on hand to support your installation of Confluence. Please follow the instructions for raising a support request and mention that you're having trouble setting up your Confluence cluster.

Last modified on Oct 11, 2021

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.