Bitbucket Data Center does not accept push after using BFG tool
Platform Notice: Data Center - This article applies to Atlassian products on the Data Center platform.
Note that this knowledge base article was created for the Data Center version of the product. Data Center knowledge base articles for non-Data Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
After using the BFG tool to reduce the size of a repository or for cleaning up a repository, Bitbucket does not accept a push.
Diagnosis
On the client terminal where you attempted the push, you would see the following messages:
remote: You are trying to update credentials reserved for Bitbucket's pull request functionality. Bitbucket manages these references automatically and may not be updated by users.
remote: Rejected refs:
remote: refs/pull-requests/1965/from
remote: refs/pull-requests/1973/from
! [remote rejected] stable/223.103 -> stable/223.103 (pre-receive hook declined)
! [remote rejected] stable/224.101 -> stable/224.101 (pre-receive hook declined)
! [remote rejected] test -> test (pre-receive hook declined)
error: unable to push references to '<HTTPS-URL>'The error indicates that you are attempting to update refs reserved for Bitbucket's pull request functionality:
Cause
The BFG tool requires you to use the --mirror flag to clone a repository before you can run it. When you mirror clone a repository from Bitbucket, you bring the PR references as well. When you push back the changes, you are pushing the rewritten PR references to Bitbucket, which is not accepted by the application, as you will try to write on a path that is reserved by the system for its own use.
Solution
One way to use the tool and push the rewritten history back to Bitbucket is:
- Create a new empty repo under the same project in Bitbucket.
- Mirror clone the original repository using the "
git clone --mirror" command. - Remove all the PR references by running the below command:
git show-ref | cut -d' ' -f2 | grep 'pull-request' | xargs -r -L1 git update-ref -d- Run the BFG tool to clean up the repo.
- "
git push --force" into the new repository.
Kindly note that the new repo won't have PR data. Even if you push to the original repo, existing PRs will be disconnected from the rest of the rewritten history produced by BFG. Unfortunately, there is no way to run BFG and tighten up the loose ends it produces in terms of PRs.
Our recommendation for an empty clean repo is just so you can have a fresh start with the rewritten history and go back to the original repository if you need to find contextual information through PRs or the old history.
Here are some of the consequences of changing the git structure using the BFG repo cleaner.
- Bitbucket uses its database as one of its sources of data. The stored information will include user IDs, repository IDs, and a lot of metadata that helps Bitbucket match all the information stored on disk, more specifically in the repositories folder.
- On the other hand, this isn't the only source of information. As Bitbucket works as a UI for Git, the greater part of the information comes directly from the disk.
- Once a pull request is opened, Bitbucket will retrieve data present in the repository directory to show information about commits, such as hashes and messages. If a repository is changed using BFG, some of these references can be removed from the repository on disk, and this will cause problems in the Pull Request UI.
- For instance, these are some possible behaviors on pull requests on a BFGed repository:
- Bitbucket will not be able to find the required reference and will throw a 500 HTTP error for the end user.
- Bitbucket will show the page with the data it was able to draw from the database, but if for instance more details about a commit are required, an error will be thrown when Bitbucket tries to get the information from disk.
- The same kind of behavior can be expected for repository forks.
- That's why we do not recommend the usage of the BFG tool on Bitbucket repositories. If the repository cleaning is a requirement, due to cloning speeds constraints, we recommend creating a new copy of the repository and performing the operation there, with prior archival of the source repository. On the original copy, all the required forks/pull request data will be left untouched and will be available for future visits if needed.
- If you're planning to run BFG due to storage concerns, we would suggest that you perform a full backup of its data. This backup would require that you copy the repository on disk and store a snapshot of the database at that point in time. After BFGing the code, as I've mentioned, most pull request screens will be left in an unusable state.
- For Bitbucket Datacenter, this page explains how BFG works: How do I remove sensitive/unwanted content that was pushed to my Bitbucket Server instance?. Even though it approaches removing sensitive data, the principle is the same for removing large blobs from the git structure.
- It is also extremely important to highlight that BFG performs changes inside the repository structure that the final user cannot control, hence we unfortunately cannot provide support for this kind of operation.