Summary

While accessing a Bitbucket repository via the command line, like doing a push, fetch or clone, you see the error message: "An error occurred while executing an external process: Could not read loose ref <branch name>"

Diagnosis

• When cloning a Bitbucket repo, an error akin to the one below is thrown: 

  git clone ssh://git@<REPOSITORY_URL> Cloning into '<REPOSITORY>'... fatal: git upload-pack: not our ref 0000000000000000000000000000000000000000 fatal: remote error: upload-pack: not our ref 0000000000000000000000000000000000000000

• The atlassian-bitbucket.log will have the following error stack trace: 

  YYYY-MM-DD 14:45:46,372 ERROR [http-nio-8443-exec-4] username @1B4B8Y0x885x142743x0 uk4dsd XX.XX.XX.XX,XX.XX.XX.XX "GET /projects/KD/repos/<REPO_NAME> HTTP/1.1" c.a.s.i.w.HandlerExceptionResolverImpl An error occurred while executing an external process: Could not read loose ref refs/heads/feature/IMU-403-logida-graylogi-http-status-500-vead-live
  com.atlassian.bitbucket.ServerException: An error occurred while executing an external process: Could not read loose ref refs/heads/<REF_NAME>
  	at com.atlassian.stash.internal.scm.git.command.refdb.AbstractGitRefDbCommand.call(AbstractGitRefDbCommand.java:54)
  	at com.atlassian.stash.internal.request.DefaultRequestManager.doAsRequest(DefaultRequestManager.java:84)
  	at com.atlassian.stash.internal.hazelcast.ConfigurableWebFilter.doFilter(ConfigurableWebFilter.java:38)
  	at java.lang.Thread.run(Thread.java:748)
  	... 342 frames trimmed
  Caused by: java.io.UncheckedIOException: Could not read loose ref refs/heads/<REF_NAME>
  	at com.atlassian.stash.internal.scm.git.ref.LooseRefDb.retryingRead(LooseRefDb.java:154)
  	at com.atlassian.stash.internal.scm.git.ref.LooseRefDb.readRef(LooseRefDb.java:406)
  	at com.atlassian.stash.internal.scm.git.command.refdb.AbstractGitRefDbCommand.call(AbstractGitRefDbCommand.java:51)
  	... 44 common frames omitted
  Caused by: java.io.IOException: <BITBUCKET_HOME>/shared/data/repositories/74/refs/heads/<REF_NAME> does not contain a hash or a symbolic ref
  	at com.atlassian.stash.internal.scm.git.ref.LooseRefDb.lambda$readRef$16(LooseRefDb.java:410)
  	at com.atlassian.stash.internal.scm.git.ref.LooseRefDb.retryingRead(LooseRefDb.java:149)
  	... 52 common frames omitted

• Navigate to the repository location in the server ($BB_HOME/shared/data/repositories/<REPO_ID> ) using the repository ID.
• Perform a 'git fsck' check of the repository.

• The affected REF file is highlighted in the 'git fsck' output as shown in the example below:

  git fsck Checking object directories: 100% (256/256), done. error: refs/heads/<REF_NAME>: invalid sha1 pointer 0000000000000000000000000000000000000000

Cause

• This occurs if any of the referenced files are empty and/or the referenced object cannot be found or read which causes corruption.

Solution

Option 1:

• Delete the ref file highlighted in the 'git fsck' output and push the branch again from an updated local repo backup. The absolute path of the ref in the server can be found in the location: <BITBUCKET_HOME>/shared/data/repositories/<REPOSITORY_ID>/refs/heads/<REF_NAME>

Option 2:

• Find the latest commit hash of the affected ref from the push logs and update the commit hash referenced in the ref file in the server (<BITBUCKET_HOME>/shared/data/repositories/<REPOSITORY_ID>/refs/heads/<REF_NAME>). You can also retrieve the correct commit hash from the git log output of an up-to-date local repository and update the file accordingly.

If still unresolved, consult other KBs such as: Recommended action plan if a repository becomes corrupted in Bitbucket data center