Summary

Bitbucket Data Center nodes may not form a cluster if one or more nodes has multiple network interfaces and IP addresses and if multicast is used for node discovery.

Bitbucket listens on all network interfaces by default. It does not choose the specific network interface to use and relies on the underlying Operating System to assign it an interface.

When a node has more than one IP address, the "wrong" IP address may get selected which would lead to the nodes not discovering each other. 

Environment

Bitbucket Data Center instances with:

• more than one node
• one or more nodes have multiple network interfaces and IP addresses

• using multicast as node discovery mechanism,
  i.e. the following property is set in the <BITBUCKET_HOME>/shared/bitbucket.properties  file:

  # Use multicast to discover cluster nodes
  hazelcast.network.multicast=true

Diagnosis

Enable debug logging for Bitbucket.

When a Data Center cluster is being formed, the $BITBUCKET_HOME/log/atlassian-bitbucket.log  will have logs that indicate that a new node was authenticated and added to the cluster.

Sample logs

These are sample logs for the existing node:

2023-04-21 08:30:25,654 DEBUG [hz.hazelcast.cached.thread-3]  c.a.s.i.c.DefaultClusterJoinManager ACCEPT(10.255.9.1:5701 <- 10.255.9.3:38113): Authenticating cluster node
2023-04-21 08:30:25,865 DEBUG [hz.hazelcast.cached.thread-3]  c.a.s.i.c.SharedSecretClusterAuthenticator Created: Response{proof=C1BC540D6526815590745657F64488C68051FD85ED786CE79BA53286A7F7A1B5}
2023-04-21 08:30:26,306 DEBUG [hz.hazelcast.cached.thread-3]  c.a.s.i.c.DefaultClusterJoinManager ACCEPT(10.255.9.1:5701 <- 10.255.9.3:38113): Node authenticated successfully
2023-04-21 08:30:26,557 INFO  [hz.hazelcast.IO.thread-in-0]  c.hazelcast.nio.tcp.TcpIpConnection [10.255.9.1]:5701 [bitbucket-docker] [3.12.13] Initialized new cluster connection between /10.255.9.1:5701 and /10.255.9.3:38113
2023-04-21 08:30:31,643 INFO  [hz.hazelcast.event-1]  c.a.s.i.c.HazelcastClusterService Node '/10.255.9.3:5701' was ADDED to the cluster. Updated cluster: 
	[/10.255.9.1:5701 master this uuid='31eeeedf-3c2c-4a37-947d-a983ad4399d4' vm-id='44ff2601-eb98-46b6-9578-e1e804c5a6ab'], 
	[/10.255.9.3:5701 uuid='3e10e740-313b-4e0f-974e-686a05785240' vm-id='457c6c42-b8f7-49b7-a194-54a549f48a1f']
2023-04-21 08:30:31,644 INFO  [hz.hazelcast.priority-generic-operation.thread-0]  c.h.internal.cluster.ClusterService [10.255.9.1]:5701 [bitbucket-docker] [3.12.13] 

Members {size:2, ver:3} [
	Member [10.255.9.1]:5701 - 31eeeedf-3c2c-4a37-947d-a983ad4399d4 this
	Member [10.255.9.3]:5701 - 3e10e740-313b-4e0f-974e-686a05785240 lite
]

These are sample logs on the new node that recently joined the cluster:

2023-04-21 08:31:29,725 INFO  [spring-startup]  c.a.s.i.hazelcast.HazelcastLifecycle Promoting Hazelcast instance at 10.255.9.3:5701 to full member
2023-04-21 08:31:29,728 INFO  [hz.hazelcast.priority-generic-operation.thread-0]  c.h.i.cluster.impl.MembershipManager [10.255.9.3]:5701 [bitbucket-docker] [3.12.13] Member [10.255.9.3]:5701 - 3e10e740-313b-4e0f-974e-686a05785240 this lite is promoted to normal member.
2023-04-21 08:31:29,729 INFO  [hz.hazelcast.priority-generic-operation.thread-0]  c.h.internal.cluster.ClusterService [10.255.9.3]:5701 [bitbucket-docker] [3.12.13] 

Members {size:2, ver:4} [
	Member [10.255.9.1]:5701 - 31eeeedf-3c2c-4a37-947d-a983ad4399d4
	Member [10.255.9.3]:5701 - 3e10e740-313b-4e0f-974e-686a05785240 this
]

If similar logs are not found, the nodes are not discovering each other.

Solution

Option 1

Use TCP/IP instead of Multicast for node discovery.

• Update the <BITBUCKET_HOME>/shared/bitbucket.properties  file and set the following:

  # Disable multicast
  hazelcast.network.multicast=false
   
  # Use TCP/IP
  hazelcast.network.tcpip=true
  
  # IP addresses of some or all of the cluster nodes. 
  # Not all of the cluster nodes have to be listed here but at least one of them has to be active when a new node joins
  hazelcast.network.tcpip.members=192.168.0.1:5701,192.168.0.2:5701,192.168.0.3:5701

Option 2

If multicast is still preferred,  the following workaround can be used, which involves creating a new bitbucket.properties  file that's local to the node

• Create a new bitbucket.properties  file located at <BITBUCKET_HOME> . This file is not shared and is local to the specific node

• Add the following property to the local <BITBUCKET_HOME>/bitbucket.properties  file:

  # Specify the public address to be used explicitly
  hazelcast.local.public.address=192.168.0.1:5701