Summary

Bitbucket Data Center node(s) do not start up. Startup stuck on the "Initializing spring context" step. Neither atlassian-bitbucket.log, nor catalina.out show any particular errors.

 As of Bitbucket Server 5.x, catalina.out will no longer exist. It will be written to atlassian-bitbucket.log instead. 

Environment

• Bitbucket Data Center using shared NFS storage.
• File creation/modification on the NFS storage works as expected.

Diagnosis

• While startup is stuck, take thread dumps to identify why the startup hangs.

• If the following thread is seen in long running threads, you may be encountering the problem(s) outlined below: 

  "spring-startup" #18 daemon prio=5 tid=0x00007f4511e75000 nid=0x5afe runnable [0x00007f45a13dd000]
     java.lang.Thread.State: RUNNABLE
  	at sun.nio.ch.FileDispatcherImpl.lock0(Native Method)
  	at sun.nio.ch.FileDispatcherImpl.lock(FileDispatcherImpl.java:90)
  	at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1115)
  	at com.atlassian.stash.internal.home.HomeLock.acquireLock(HomeLock.java:108)
  	at com.atlassian.stash.internal.home.HomeLock.lock(HomeLock.java:94)
  	at com.atlassian.stash.internal.home.HomeLockAcquirer.lock(HomeLockAcquirer.java:58)
  	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  	at java.lang.reflect.Method.invoke(Method.java:498)
  	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1706)
  	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1645)
  
  ...

• You can use a C program to validate NFS locking ability. You will simply need to put the outlined C code into a file then compile it. When done, you can move it to the NFS share on the client (application node) and run it. It should finish in less than a second and create testfile for testing file locking. If this fails, locking almost certainly doesn't work.

  • See the practical example

    Put the following contents into a file nfs_lock.c then execute either make nfs_lock or gcc -o nfs_lock nfs_lock.c, then copy and run the command nfs_lock from the NFS share on the client (application node):

    #include <stdio.h> 
    #include <fcntl.h> 
    main() { 
        int fd , ret; 
        struct flock strlock; 
        fd = open("testfile", O_CREAT | O_RDWR , 0666 ); 
        if (fd == -1){ 
            printf("FATAL ERROR: Could not open file\n"); 
            exit(-1); 
        } 
     
    /* Set a write lock on the first 1024 bytes of the file. 
       It doesn't matter that they don't actually exist yet. */ 
     
        strlock.l_type = F_WRLCK; 
        strlock.l_whence = 0; 
        strlock.l_start = 0L; 
        strlock.l_len = 1024L; 
        ret = fcntl (fd, F_SETLK , &strlock); 
        if (ret == -1){ 
            printf("FATAL ERROR: Could not lock file\n"); 
            exit(-1); 
        } 
        ret = close( fd ); 
        if (ret == -1){ 
            printf("FATAL ERROR: Could not close file\n"); 
            exit(-1); 
        } 
    } 

    Source: http://community.microfocus.com/microfocus/cobol/net_express__server_express/w/knowledge_base/6215.c-program-to-validate-nfs-locking-ability.aspx

Cause

The above thread indicates that Bitbucket fails to acqurie a lock on the shared storage (NFS). The causes may be - but not limited to the - following:

• NFS daemons (lockd, nfsd, mountd, etc) not running on either the client or server side
• NFS daemons are hung or have problems communicating
• Permission problems on the NFS share
• Firewall blocking connection for either of the above listed daemons. Check which ports the required daemons run by executing portmap -p on the NFS server and make sure that the relevant TCP and UDP ports are accepting connections.

Solution

Make sure that all the NFS daemons are running and accessible from the client perspective. As a troubleshooting step, try disabling firewall on both the NFS server and client(s), test NFS locking manually and ensure it works as expected. Please note that this error is very likely caused by an environmental issue, rather than Bitbucket Data Center not operating as expected.

References

• Running NFS behind a firewall (Redhat)
• Firewalling a Linux NFS server with iptables
• C program to validate NFS locking ability
• BSERV-8969 - Getting issue details... STATUS