Summary

Bitbucket Data Center fails to start. The following appears in the atlassian-bitbucket.log:

Lock file .../shared/.lock cannot be created in home directory...

Environment

This article is meant for a clustered Bitbucket Data Center implementation that has a shared home mounted via NFS, but could also possibly impact a Bitbucket Server instance using a mounted network location for other reasons.

Diagnosis

Attempt to perform a locked file operation (such as a mv) against a file in the shared Bitbucket home using flock or another similar command. 

# Run this command first
flock <BITBUCKET_HOME>/shared/data/test.lock mv test.lock test

# Run this one after the first one finishes to confirm if the flock and move worked.
ls <BITBUCKET_HOME>/shared/data/test*

Cause

Cause 1:

A common reason for this error is that your NFS lock service is disabled or misconfigured. This can happen because the portmap and dbus services (at least, in Ubuntu Linux) are not always enabled by default.

Bitbucket Server and Bitbucket Server Data Center will create the following lock files:

• ${BITBUCKET_HOME}/.lock
• ${BITBUCKET_HOME}/shared/.lock

Bitbucket Server acquires an exclusive lock on both directories, preventing any other instances of Bitbucket Server (Server or Data Center) from using them.

Bitbucket Data Center acquires an exclusive lock on ${BITBUCKET_HOME} (preventing any other Bitbucket Server instances from using it) and a shared lock on${BITBUCKET_HOME}/shared (allowing it to be shared with other Bitbucket Data Center cluster nodes but not Bitbucket Server instances).

The two lock files exist so that two Bitbucket Server instances can not use the same Bitbucket Server home directory or shared folder at the same time.  This ensures that there is no data loss events from this type of misconfiguration.

Cause 2:

If you see the following as part of the stack trace, it is likely that the statd service is disabled on the NFS server, preventing it from creating lock files within the shared Bitbucket home:

Caused by: java.io.IOException: No locks available

Cause 3:

This can also be caused on the client side, when the services rpcbind and rpc-statd are not running. 

The stack trace would also report:

java.io.IOException: No locks available
	at java.base/sun.nio.ch.FileDispatcherImpl.lock0(Native Method)
	at java.base/sun.nio.ch.FileDispatcherImpl.lock(FileDispatcherImpl.java:96)
	at java.base/sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1161)
	at com.atlassian.stash.internal.home.HomeLock.acquireLock(HomeLock.java:112)
	at com.atlassian.stash.internal.home.HomeLock.lock(HomeLock.java:98)
	at com.atlassian.stash.internal.home.HomeLockAcquirer.lock(HomeLockAcquirer.java:58)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583)
	at javax.servlet.GenericServlet.init(GenericServlet.java:158)
	at java.base/java.lang.Thread.run(Thread.java:829)
	... 37 frames trimmed

Solution

Cause 1:

Make sure the lockd, portmap, and dbus services are all running on your NFS server, and restart Bitbucket Server. See Installing Bitbucket Data Center for more information.

If you still have one or more leftover .lock files after shutting down all your instances of Bitbucket Server, it's safe to delete them and start over.  

Cause 2:

Enable and start the rpc-statd  service on the NFS server in order to allow the NFS server to create locks within ${BITBUCKET_HOME}/shared.

Ex:

sudo systemctl enable rpc-statd 
sudo systemctl start rpc-statd  

Cause 3:

Enable and start rpcbind and rpc-statd. This should be done on the server hosting each Bitbucket node. 

sudo systemctl enable rpc-statd 
sudo systemctl start rpc-statd  

sudo systemctl enable rpcbind 
sudo systemctl start rpcbind