Hipchat Data Center: CentOS NFS Mount Failure
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Problem
Mounting the NFS v4 to the Hipchat Data Center node fails with the following error in the kern.log.
2017-10-16T07:21:33.679317+00:00 hipchat kernel: [ 331.367484] IN= OUT=eth0 SRC=192.168.10.199 DST=192.168.10.198 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=11180 DF PROTO=TCP SPT=792 DPT=2049 WINDOW=29200 RES=0x00 SYN URGP=0
2017-10-16T07:21:33.679317+00:00 hipchat kernel: [ 331.372815] RPC: AUTH_GSS upcall failed. Please check user daemon is running.
Diagnosis
Environment
- Hipchat Data Center 3.0.1
Diagnostic Steps
Check through the /etc/exports file and the structure should look like this for example
/exports/data *(rw,no_subtree_check,no_root_squash)
Cause
It is a known issue when user mount the NFS volume using NFS4 when the /etc/exports should include the parent directory of your NFS folder.
This issue was discussed in the Since Upgrading NFS and Ubuntu to 14.04, All NFS Mounts Are Failing article.
Workaround
There are 2 workarounds that you can apply, either on the node(s) hosting Hipchat Data Center or the instance hosting the NFS:
Workaround #1 (Data Center node)
Obtain the root access to the node
sudo dont-blame-hipchat
Edit the /etc/modprobe.d/blacklist.conf file using the editor of your choice. For example:
nano /etc/modprobe.d/blacklist.conf
Include the following line in the file:
blacklist rpcsec_gss_krb5
Once done, reboot the server and mount the NFS again
Workaround #2 (NFS server)
Edit the /etc/exports file on the NFS instance
nano /etc/exports
With reference to the Setup NFSv4 on CentOS page, include the root parent directory of your NFS folder with a read-only permisison. For example:
/exports *(ro,no_subtree_check,fsid=0,crossmnt) /exports/data *(rw,no_subtree_check,no_root_squash)
Once done, mount the NFS again and verify that it works