Bitbucket Data Center shows "command timed out" error on creating a repository
Platform Notice: Data Center - This article applies to Atlassian products on the Data Center platform.
Note that this knowledge base article was created for the Data Center version of the product. Data Center knowledge base articles for non-Data Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary:
While creating a repository on the Bitbucket Server GUI, an error message "command timed out" pops up after some time.
Environment
Tested on Bitbucket Data Center 8.9.3
Diagnosis
The
atlassian-bitbucket.logwill see the following logs:
2024-01-23 11:57:43,210 DEBUG [http-nio-7990-exec-10 url: /projects/PRO1/repos, /mvc/projects/PRO1/repos; user: user1] user1 @14HOEJJx715x161x0 ckqa1o 10.211.XX.X "POST /projects/PRO1/repos HTTP/1.1" c.a.s.i.r.DefaultRepositoryService Could not create repository PRO1/repo6
com.atlassian.bitbucket.scm.CommandTimeoutException: command timed out
... 400 frames trimmed
Caused by: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 129.987831398s. [closed=[], open=[[remote_addr=localhost/127.0.0.1:7777]]]
at io.grpc.Status.asRuntimeException(Status.java:535) The atlassian-mesh.log will have the following events:
2024-01-23 11:55:33,084 DEBUG [grpc-server:thread-9] user1 3CK63RGEx715x34x2 @14HOEJJx715x161x0 127.0.0.1 "RepositoryService/Create" (>1 <0) c.a.b.m.r.DefaultRpcRepositoryAccessTracker all has 1 active requests
2024-01-23 11:55:33,086 DEBUG [grpc-server:thread-9] user1 3CK63RGEx715x34x2 @14HOEJJx715x161x0 127.0.0.1 "RepositoryService/Create" (>1 <0) c.a.b.m.t.DefaultTransactionManager Registering transaction n/sidecar/tx/99c9dde90000000000000003 to automatically roll back on cleanup of the current RPC
2024-01-23 11:55:33,087 DEBUG [grpc-server:thread-9] user1 3CK63RGEx715x34x2 @14HOEJJx715x161x0 127.0.0.1 "RepositoryService/Create" (>1 <0) c.a.b.m.r.DefaultRepositoryManager Creating repository for ds/0/h/26272df14de3e8b0f131/r/2
2024-01-23 11:55:33,164 DEBUG [grpc-server:thread-9] user1 3CK63RGEx715x34x2 @14HOEJJx715x161x0 127.0.0.1 "RepositoryService/Create" (>1 <0) c.a.bitbucket.mesh.process.OsProcess 8562: [git init --bare --quiet --shared=false --template=/opt/homedir/shared/config/git/templates /opt/homedir/shared/data/repositories/2] exited 0 in 49ms (stdin: 0, stdout: 0, stderr: 0)
2024-01-23 11:55:33,165 DEBUG [grpc-server:thread-9] user1 3CK63RGEx715x34x2 @14HOEJJx715x161x0 127.0.0.1 "RepositoryService/Create" (>1 <0) c.a.b.m.r.DefaultRepositoryManager 26272df14de3e8b0f131-2: Configuring refs/heads/master as the default branch
2024-01-23 11:55:33,168 DEBUG [grpc-server:thread-9] user1 3CK63RGEx715x34x2 @14HOEJJx715x161x0 127.0.0.1 "RepositoryService/Create" (>1 <0) c.a.b.m.r.DefaultRepositoryManager Writing metadata for 26272df14de3e8b0f131-2Cause
The problem happens when the procedure of creating the bare repository on the filesystem fails because the Bitbucket server node (NFS client) is unable to communicate with the NFS server. Bitbucket waits by default for 130 seconds for the operation to finish, and if it doesn't, the GUI reports an error saying "command timed out".
The following two factors could be the cause of this problem.
- The Bitbucket server and the NFS server do not have the necessary NFS ports whitelisted.
- The lock mechanism in NFS isn't operating correctly. Git generates a lock file called
repository-config.lockinside the repository directory when the repository is created, which it later renames torepository-config.
In this particular case, the repository directory is created but git couldn't rename the repository-config.lock in the path <Bitbucket-home-directory>/shared/data/repositories/<repo-ID> into a regular file because the statd and lockd ports on the NFS server aren't reachable from the Bitbucket server. Checking the contents of the <Bitbucket-home-directory>/shared/data/repositories/<repo-ID>, you will see the below:
ls -ltrh <Bitbucket-home-directory>/shared/data/repositories/<repo-ID>
total 12K
drwxr-xr-x 1 atlbitbucket atlbitbucket 100 Jan 23 11:55 hooks
drwxr-xr-x 1 atlbitbucket atlbitbucket 18 Jan 23 11:55 refs
-rw-r--r-- 1 atlbitbucket atlbitbucket 548 Jan 23 11:55 config
drwxr-xr-x 1 atlbitbucket atlbitbucket 16 Jan 23 11:55 objects
-rw-r--r-- 2 atlbitbucket atlbitbucket 0 Jan 23 12:27 repository-config.lock
-rw-r--r-- 1 atlbitbucket atlbitbucket 23 Jan 23 11:55 HEADSolution
Check all the ports listed on the NFS servers are accessible from the Bitbucket server node(s). The following command can be used to list the port.
rpcinfo -p | awk '{print $3" "$4}' | sort -k2n | uniqCheck the lock functionality of the NFS. It can be checked by running the below command inside the shared home folder on the Bitbucket server node. The command should successfully exit and the file
testshould be generated.cd <Bitbucket-home-directory>/shared flock <Bitbucket-home-directory>/shared/test.lock mv test.lock test- If any of the above-mentioned requirements are not met then Bitbucket can't create the repository. In this situation, please seek help from the system admin or OS vendor to resolve the related problem.