High load on Hazelcast and Bitbucket due to failing GC on refs/pull-requests/<pr id>/v* files
High CPU and performance issues in Bitbucket appear along with the following messages in the
c.a.s.i.c.HazelcastBucketedExecutor Attempt 2 of 5 at processing bucket 'git:gc@385d519558f5df598f79' for executor 'git:gc' failed: ''/srv/git/current/bin/git gc' exited with code 255 saying: fatal: bad object refs/pull-requests/1/v3
You may also see similar errors for the following fiile names:
These could be accompanied by hundreds of Hazelcast errors in the logs as well.
Prior to Stash 2.9 the
refs/pull-requests/<pr id>/v* files were used, and from 2.9 onwards we use them from the
stash-refs directory instead. It's possible the upgrade tasks failed to remove the v* files or these files were restored from a backup. Garbage collection continuously failing garbage collection can put a server under a huge amount of load.
Bitbucket 5+ moves GC off the BucketedExecutor and Hazelcast. Garbage collection will only create load on HazelCast in Bitbucket 4.x versions.
Remove everything under
refs/pull-requests/ that match:
- Files should only be removed from the
/refs/pull-requests/…directories, and not the
- As always, please take a backup of the $
BITBUCKET_HOME before making any changes.
Was this helpful?Yes Provide feedback about this article