Bitbucket server unable to start in cluster mode
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Bitbucket does not start in cluster mode, the Clustering section on the Bitbucket UI shows only one node.
Environment
Bitbucket version 7.13.0
Diagnosis
You would see the following in the atlassian-bitbucket.log file:
WARN [spring-startup] com.hazelcast.instance.Node [172.xx.xx.xx]:5701 [atlbitbucket] [3.12.9] No join method is enabled! Starting standalone.
WARN [spring-startup] com.hazelcast.instance.Node [172.xx.xx.xx]:5701 [atlbitbucket] [3.12.9] No join method is enabled! Starting standalone.The shared directory in use is CIFS. CIFS is not currently supported by us, we support NFS. If Bitbucket starts with CIFS, it may not run correctly.
- Check if the
fstabconfiguration looks correct and confirm it is being mounted to the correct directory. - After updating
fstab, you must remount the NFS drive to get the settings. - Your
bitbucket.propertiesmust have the following settings:
hazelcast.network.multicast=true
hazelcast.group.name=<your-bitbucket-cluster>
hazelcast.group.password=<your-bitbucket-cluster>Replace your-bitbucket-cluster with any value you wish.
- Ensure that port 5701 is open and not blocked by firewalls
There may also be some issues with forks. Forking relies on filesystems being compatible - CIFS could be the cause of fork and tag issues.
Take thread dumps of the Bitbucket process while it was stuck. In case they look something like this:
at sun.nio.ch.FileDispatcherImpl.lock0(Native Method) at sun.nio.ch.FileDispatcherImpl.lock(FileDispatcherImpl.java:94) at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1114) at com.atlassian.stash.internal.home.HomeLock.acquireLock(HomeLock.java:112) at com.atlassian.stash.internal.home.HomeLock.lock(HomeLock.java:98) at com.atlassian.stash.internal.home.HomeLockAcquirer.lock(HomeLockAcquirer.java:58)It means that the startup thread is stuck trying to perform a filesystem lock. Try to investigate filesystem locking issues with the NFS mount as the next step.
- Try running the commands
showmount -e server_nameandrpcinfo -u servername mountdand see if both work as expected. - Try creating a blank file in the NFS drive, that should also work as expected
- Try running the commands
It would be good to check what is running the NFS backend, and why the filesystem locking fails for this particular mount. It is possible that this could be on the NFS server side, rather the Bitbucket mounting side.
For example, some issues exists with certain NFS servers. NFS file lock operation hangs when NFS share is mounted using a certain NFS server IP addresses using FileStore is one example.
Cause
One of the possible causes of the issue could be firewall on the NFS server.
Solution
Turn off firewall on the NFS server, Bitbucket startup will be successful.