Symptoms

When cloning a large number of refs (branches and tags), Bitbucket Data Center fails the clone with a "The hook request is too large to process" error.

$ git clone ssh://git@example.com/project/repository.git
Cloning into 'repository'...
error: git upload-pack: git-pack-objects died with error.
fatal: git upload-pack: aborting due to possible repository corruption on the remote side.
remote: The hook request is too large to process.
remote: aborting due to possible repository corruption on the remote side.
fatal: early EOF
fatal: fetch-pack: invalid index-pack output

Diagnose

To verify why you are encountering this error, you can get the number of refs in your system and apply it to the calculation formula and determine if you've exceeded the buffer size.

1. Get the number of refs by running the following in your repository

   git for-each-ref | wc -l

2. Use the following formula to calculate your approximate ref byte size: (number of refs * 41) + 6
   
   1. Important to know that 1 byte can hold 1 character
   2. 41 bytes = 40 character SHA-1 (Git hash) + newline \n
   3. 6 bytes is for additional characters from pack-objects

Example: A sample repository

terminal:~/shared/data/repositories/7192$ git for-each-ref | wc -l
105455

Now calculate: (105455 refs * 41) + 6 = 4323661 bytes

Cause

The number of refs exceeds the buffer size available for processing hooks. 

A hook buffer restriction has always existed within Bitbucket since 1.2. As part of the clone operation, Git sends the entire input to pack-objects in order to create the pack for the client. This is being read as part of the hook process. In 7.x of Bitbucket, the size is set to 5mb, which allows for ~127,875 refs, which far exceeds the size of how many refs a repository should have. 

Resolution

Recommended resolution

The preferred recommended resolution is to reduce your refs. Having a large number of refs can impact Git performance. Git will have to process hooks, which scan the refs, pack these refs in fetch, clone, or push operations, which all impact response time to even the most basic Git commands such as ref advertisements.

What are refs? Refs are files that contain a Git hash on your file system. These are generally referenced as refs/heads or refs/tags. This means that branches and tags are refs. Therefore, teams will need to review their current environment to identify refs that are no longer needed or adjust their workflow so that too many refs aren't created (i.e. tagging every branch).