Problem

High CPU and performance issues in Bitbucket appear along with the following messages in the atlassian-bitbucket-yyyy-mm-dd.log:

c.a.s.i.c.HazelcastBucketedExecutor Attempt 2 of 5 at processing bucket 'git:gc@385d519558f5df598f79' for executor 'git:gc' failed: ''/srv/git/current/bin/git gc' exited with code 255 saying: fatal: bad object refs/pull-requests/1/v3

You may also see similar errors for the following fiile names:

• merge-base
• merge-conflicted
• merge-clean
• notes
• to

These could be accompanied by hundreds of Hazelcast errors in the logs as well.

Cause

Prior to Stash 2.9 the refs/pull-requests/<pr id>/v* files were used, and from 2.9 onwards we use them from the stash-refs directory instead. It's possible the upgrade tasks failed to remove the v* files or these files were restored from a backup. Garbage collection continuously failing garbage collection can put a server under a huge amount of load. 

 Bitbucket 5+ moves GC off the BucketedExecutor and Hazelcast. Garbage collection will only create load on HazelCast in Bitbucket 4.x versions. 

Resolution

Remove everything under refs/pull-requests/ that match:

• merge-base
• merge-conflicted
• merge-clean
• notes
• to