Troubleshooting NFS
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Problem
Customer Bitbucket is not working using NFSv3. NFSv4 is however.
Below are the troubleshooting steps taken to investigate.
Atlassian Engineer may need to have the customer perform these steps during live troubleshooting as Bitbucket is not starting.
Tail Bitbucket Log
We are able to see the HomeLockAcquirer, but not the Shared Lock acquired.
1
2
3
4
5
6
7
8
[spring-startup] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /app/bitbucket-home
There should be another message after this one indicating a successful lock acquired on the shared directory:
[spring-startup] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /app/bitbucket-home/sharedDiagnosis
Diagnostic Steps
Verify NFSv3 mount options per: Install Bitbucket Data Center
1
2
3
4
cat /etc/fstab
rw,nfsvers=3,lookupcache=pos,noatime,intr,rsize=32768,wsize=32768,_netdev
Place Bitbucket in DEBUG mode.
1
2
3
4
5
Turn on Debug manually: Add the following to bitbucket.properties and restart BB
logging.logger.ROOT=DEBUG
logging.logger.com.atlassian.bitbucket=DEBUG
Check if NFSv3 is allowed on the NFS server
From the NFS server run the following commands from to see if NFSv3 is allowed.
1
2
3
4
5
6
7
8
9
10
11
from the NFS server:
rpcinfo -p |grep nfs
100003 3 tcp 2049 nfs. ---------> The 3 signifies that NFSv3 is allowed
100003 4 tcp 2049 nfs ---------> The 4 signifies that NFSv4 is allowed
100227 3 tcp 2049 nfs_acl
cat /proc/fs/nfsd/versions
+3 +4 +4.1 +4.2 ----------> The +3 signifies that NFSv3 is allowed.Note: With NFSv3 multiple NFS services also have to be running that automatically are included with NFSv4.
1
2
3
4
5
6
7
8
These include, but are not limited to:
mountd: Manages mount requests from NFS clients.
nlockmgr Network Lock Manager): Handles file locking.
statd: Monitors the state of NFS clients and servers for the purpose of lock recovery.
rquotad: Manages disk quotas on the NFS server.
portmapper: Service that maps RPC program numbers to network port numbers.
Note: These commands may change depending on the specific distribution being used.
1
2
3
4
Verify distribution:
cat /etc/release or cat /etc/*-release - to verify distribution. Tweak commands based on response.
Ensure these services are running on the nfs server run. Whichever services is not running needs to be started "start".
Run the following commands on the NFS server.
1
2
3
4
5
systemctl status nfs-server - rquotad service
systemctl status rpcbind. - Portmapper service
systemctl status nfs-mountd - mountd (if it exists - may be integrated)
systemctl status nfs-lock.service -nlockmgr (on RHEL-based systems this command works) if not user rpc-statd
systemctl status rpc-statd - statd (NFSlocking and recovery)Perform the FLOCK Test:
Run the commands against the appropriate mount share location on PROD and UAT. Don't worry about creating the test.lock file yourself; the command will take care of that.
1
2
3
flock /app/bitbucket-home/shared/test.lock mv test.lock test
ls /app/bitbucket-home/shared/test*Ensure the test.lock file was successfully renamed to test.
Ensure no error messages occurred during the execution of the flock command.
In the event that the FLOCK test doesn't work properly you will need to test port connectivity and get the network team engaged to ensure the ports listed in the rpcinfo -p are accessible.
NOTE: NFSv3 uses random ports for its services, so depending on how restrictive their firewall rules and overall network setup are, it could cause issues with NFSv3 but not with NFSv4, as NFSv4 uses static ports for its services.
Verify specific ports being used for NFSv3 services.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
/usr/bin/rpcinfo -p [IP of NFSv3 server] or simply /usr/bin/rpcinfo -p
Example Output:
#-> /usr/bin/rpcinfo -p 10.59.1xx.xxx
program vers proto port service
100000 4 tcp 1111 portmapper
100000 3 tcp 1111 portmapper
100000 2 tcp 1111 portmapper
100000 4 udp 1111 portmapper
100000 3 udp 1111 portmapper
100000 2 udp 1111 portmapper
100024 1 udp 41831 status
100024 1 tcp 48570 status
100005 1 udp 20047 mountd
100005 1 tcp 20047 mountd
100005 2 udp 20047 mountd
100005 2 tcp 20047 mountd
100005 3 udp 20047 mountd
100005 3 tcp 20047 mountd
100003 3 tcp 2049 nfs
100003 4 tcp 2049 nfs
100227 3 tcp 2049 nfs_acl
100021 1 udp 58432 nlockmgr
100021 3 udp 58432 nlockmgr
100021 4 udp 58432 nlockmgr
100021 1 tcp 34745 nlockmgr
100021 3 tcp 34745 nlockmgr
100021 4 tcp 34745 nlockmgrTest Port connectivity:
Test the ports use the following NCAT command:
A successful connection shows:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
FOR TCP:
nc -vz hostname port
nc -vz hostname.com 34745
Ncat:Connected to 10.59.456.321:34745
Successfull response for TCP
FOR UDP:
Successfull UDP connection:
nc -vz -u hostname.com 34745
Ncat:Connected to 10.59.456.321:34745
Successfull response for UDPA unsuccessful connection shows:
1
2
3
4
5
6
7
nc -vz hostname port
nc -vz hostname.com 34745
Ncat:Connected to 10.59.456.321:34745
Connection - timeoutCause
An unsuccesful connection should be investigated by the network admin to ensure the port ranges are accessible etc.
Note: To ensure that these ports can be hardcoded for NFSv3, it is suggested to have the customers network team configure the /etc/nfs.conf file.
Once updated, reboot to verify with /usr/bin/rpcinfo -p, to ensure the same ports are being listed.
Once the Network team opened connectivity to the ports we were able to get the lock successfully on the shared directory.
1
2
3
2025-02-04 12:25:20,988 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /Users/.../AtlassianLocalMac/ApplicationData/bb850
2025-02-04 12:25:29,281 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /Users/.../AtlassianLocalMac/ApplicationData/bb850/shared
Solution
Resolution
The customer's network team opened port connectivity from Bitbucket to NFS services listed in /usr/bin/rpcinfo -p, that were previously not accessible.
Was this helpful?