How to recover a Bitbucket Mirror/Mirror Farm after accidental deletion from staging instance, affecting production Mirrors
Platform Notice: Data Center - This article applies to Atlassian products on the Data Center platform.
Note that this knowledge base article was created for the Data Center version of the product. Data Center knowledge base articles for non-Data Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
This guide explains how to recover a Bitbucket Smart Mirror/Mirror farm instance that was accidentally deleted from the Bitbucket Staging UI, which was cloned from the Bitbucket Production instance.
When creating a staging instance based on the production Instance, it is strongly recommended to follow the steps outlined in the KB article How to establish staging server environments for Bitbucket Server.
It is important to ensure that the Staging Instance is completely blocked from making any requests to production-related services, such as the production database , Production Mirrors and other Atlassian production applications(Jira, Fisheye/Crucible). The Staging instance must be fully isolated from the Primary Instance and its connected services.
The KB How to establish staging server environments for Bitbucket Server also provides instructions on how to remove Bitbucket Smart Mirror-related details from the staging instance before starting it up. The steps mentioned on the KB should be followed carefully when setting up a Staging instance cloned from Production to prevent issues like this from occurring.
If the staging instance is not isolated (i.e. the steps mentioned on KB How to establish staging server environments for Bitbucket Server are not followed on the staging instance), production smart mirrors appear in staging instance. Deletion of smart mirrors in the staging instance can cause synchronization to stop on the production instance affecting production smart mirrors. This KB explains how to recover the production Bitbucket Smart Mirror/Mirror Farm after deleting smart mirror from the staging instance.
- This issue happens when the production Bitbucket Smart Mirror/Mirror Farm is mistakenly deleted from the staging environment UI (which is cloned from the production instance). If the staging instance can access the production Smart Mirror/Mirror Farm — because it shares the same mirror configuration since that's copied from the production database — staging Instance will be able to execute the DELETE Mirror request on the production Mirror/Mirror Farm, setting its Mirror status to "REMOVED," thereby breaking the synchronization between the production server and the Mirror.
- After the deletion of the Mirror from the staging instance UI, the production Primary Instance will display the affected mirror in an "Authorized" state instead of "Synchronized" because the Mirror's status has been set to "REMOVED" on the mirror side.
- Ideally, the DELETED mirror would attempt to send a POST request back to the Primary/Upstream server to re-register itself in case it was mistakenly deleted from the production server.
- Deleting and re-authorizing the same Mirror on the Primary/Upstream is safe to do, and this action does not delete any repository data on the Mirror side.
- Only the Mirror status is set to "REMOVED," which can be re-authorized with the same upstream using r
econnect=1added to the Mirror URL (e.g.,https://MIRROR_HOST:<port>/mirror?reconnect=1), allowing everything to resume functioning as it was before the deletion after accepting the Mirror Authorization request on the Primary/Upstream. This process is very safe to perform.- The DELETED mirror can be re-added by accessing the Mirror URL on Bitbucket versions 8.9.20+, 8.19.10+, and 9.2.0+. In these versions, you'll find a new option to add the mirror back to the same upstream. This option is added as part of BSERV-13503 - Getting issue details... STATUS
- However, if the production Mirror was deleted from the staging environment, the production Primary Instance doesn't recognize the Mirror as deleted, and as a result, the POST request from the Mirror will fail to re-register with the production server.
There is already a bug request OPEN to prevent the staging instance from being able to connect to the production Mirror instance and break the Mirror synchronization.
BSERV-19668 - Getting issue details... STATUS
Environment
The solution has been validated in Bitbucket Datacenter 8.9.9 but may be applicable to other versions (Bitbucket 7.x, 8.x ).
Solution
Git Changes to Staging Instance won't get Synced to the production Mirror
- Even if any Git changes were pushed to the staging instance for the mirrored repositories before deleting the Mirror from the staging UI, those changes would not get synced to the production Mirror because the Mirror will always try to reach out to the Registered upstream, which is the production URL for the Git changes.
- Therefore, changes made on the staging instance cannot be pulled by the production Mirror. Hence the below workaround is safe to execute.
- If the mirror is already deleted from the Staging instance, no further actions are required on the staging instance. Just ensure that you block all communications to the production services and follow the steps mentioned on the KB article How to establish staging server environments for Bitbucket Server
Option 1:
Steps to be performed on the Production Primary Instance
- Delete the affected mirror that is showing as "Authorized" on the production Primary Instance.
- Visit the affected Mirror URL with
reconnect=1added to the URL (e.g.,https://MIRROR_HOST_URL:<port>/mirror?reconnect=1). This will trigger a new Mirror Authorization request sent to the Primary/Upstream server. Note: In Bitbucket Datacenter versions 8.9.20+, 8.19.10+, and 9.2.0+, the DELETED mirror can be re-added with a UI option to the same upstream. - Click the "Go to the primary server" button, which will redirect the user to the Authorization request on the Upstream instance.
- Accept the authorization request on the Primary/Upstream instance.
- The Mirror should now start synchronizing successfully with the Primary/Upstream.
- In case if the Mirror doesn't show as "Synchronized", try restarting the Bitbucket Mirror service and see if the status updates to "Synchronized" on the Primary/Upstream.
Option 2:
An alternative solution for restoring the affected mirror would be to restore the last known working backup of the mirror and see if that fixes the issue. The last known working backup of the mirror should have it in the proper INSTALLED state with the primary and share the same JWT credentials to connect to the upstream. Therefore, if a backup is available, restoring it should help recover the affected mirror instead of rebuilding it from scratch. ( Please test the approach in test instance before performing in production instance.)
If, for some reason, something other than the deletion of the production mirror from the staging instance occurred, and attempts were made to fix things by manipulating the mirror filesystem or H2 database, it would be impossible to return it to the expected working state. In that case, the best way forward would be to rebuild the mirror from scratch. The steps can be found in How to re-install/re-register a single Bitbucket Smart Mirror.