Migrate repositories to Bitbucket Mesh
Bitbucket Mesh is a distributed, replicated, and horizontally scalable Git repository storage system. It offers better resiliency when compared to a clustered Data Center deployment backed by Network File System (NFS). Additionally, as it moves the processing closer to the storage, it leads to decreased I/O latency and increased performance for repositories hosted on it. This page shows you how to migrate repositories from your existing Bitbucket Data Center instance to Mesh.
On this page
Before you start
Before you attempt to migrate your repositories to Bitbucket Mesh, make sure you’ve noted the considerations and completed all the prerequisites described in this section.
We strongly recommend that you test the migration process in a staging environment first. If you have any questions about the migration process, reach out to support.
Prerequisites
Make sure that you have:
- Connected at least 3 Mesh nodes to your Bitbucket Data Center instance – Bitbucket Mesh is a replicated system built to provide resiliency. It is required that you connect a minimum of 3 Mesh nodes to your Bitbucket DC instance before attempting to migrate any repositories to it. Learn more about setting up and configuring Mesh nodes.
- Configure all additional Mesh nodes before you migrate – If you’re planning to have more than 3 Mesh nodes, we suggest that you configure all the Mesh nodes before you start migrating repositories as migrated repositories are not replicated on the nodes you add later.
Other considerations
The preview REST endpoint
You can use the preview function to review the repositories included in the migration, before kicking it off. The preview takes into account fork hierarchies, which are always migrated fully.
Git LFS objects
For now, LFS objects in your Git repositories are not migrated over to Mesh – they will stay on the shared home of your instance even after you migrate your repositories to Mesh.
Disk space requirements and garbage collection
Always make sure you have enough disk space for the files you're migrating. The space required on disk during export is roughly the size of the Git data being exported. If you are unsure of how much space you have available, you can check the available disk space by referring to how to identify a repository ID in Bitbucket.
You should test migrations for large repositories in a staging environment before attempting them in production, and should closely monitor filesystem usage after migrating.
Note: When preparing to test a migration in a staging environment, ensure all of the objects and packs in the test repository have a recent last modified timestamp. Many staging environments are populated with demotions of production data, and may be stale. Git’s garbage collection is sensitive to modification times, which means performing garbage collection in a stale repository can result in very different behavior compared to performing it in an active repository – even if they contain the same data.
We recommend that you proceed with your migration in batches and give each batch some time to rest so that garbage collection can run and settle before the next migration.
Time needed for migration
There is no real way to predict the exact time required for the migration to complete, it is impacted by your data set, individual load, and traffic.
Some tests have shown that a migration of 10GB can take around 10-15 minutes, but each case is unique and times may vary.
Repository creation on Mesh
You can choose to create new repositories on Mesh automatically using the Create new repositories on Mesh toggle. When this toggle is off, repositories are created in the shared home.
Related repositories are migrated together
If you migrate a repository that has forks, note that all related repositories are migrated together. If any fork in the hierarchy fails to migrate, migration for the entire hierarchy fails.
Migrating a repository back to the shared file system
Once a repository has been migrated to Mesh, you can’t migrate it back to the shared file system. However, if you’ve accidentally migrated a repository and need to reverse a migration, contact support.
Migrating repositories to Mesh
Decide which project or repositories you want to migrate only after you’ve carefully reviewed the above perquisites. A good strategy may involve migrating smaller/less busy repositories first, followed by bigger/busier repositories. We strongly recommend that you test the migration process in a staging environment first.
You can perform a migration using the UI or the REST API.
From the user interface
Start and monitor the migration
After adding a minimum of 3 Mesh nodes, you’ll be able to migrate repositories to Bitbucket Mesh:
- From the System Settings cog, navigate to Git > Bitbucket Mesh and select Migrate repositories.
- From the Repository migrations page, select the repositories you wish to migrate and select Migrate. Repositories that can be migrated are in READY status.
A progress bar indicates progress of the migration:
Migration progress is indicated by the following statuses:
- QUEUED: The repository is queued for migration.
- MIGRATING: Mesh is migrating the repository.
- FAILED: The repository couldn’t be migrated to Mesh.
- COMPLETE: The repository is successfully migrated to Mesh.
To see which Mesh nodes a repository is migrated to, navigate to Repository Settings > Repository details.
Cancel the migration?
You can cancel the migration using the Cancel button while a migration is in progress but note that this only cancels the migration of repositories that haven’t been migrated yet – the repository hierarchies that have already been fully migrated to Mesh will stay on it and will not be reverted to the shared home.
Using the REST API
To perform the migration using the REST API, you'll need to either use your favorite tool to make REST calls or refer to the sample cURL commands provided for migration.
If you plan to use cURL commands, we recommend you set up a .netrc file (sample commands use the -n flag) and install jq on your machine.
You can migrate all repositories in a project, multiple projects, multiple repositories, or a combination of projects and repositories using project or repository IDs. Learn more about identifying a repository ID in Bitbucket. The project ID can be retrieved from the project’s key using the Project Resource REST API.
Preview the migration
Previewing allows you to experiment with the migration request and returns a list of repositories that would be included with a given request.
To preview, use the below command:
curl -u <adminuser> -s -n -X POST -H 'Content-type: application/json' -d '{"projectIds":[1, 2, 3], "repositoryIds":[1, 2, 3]}' http://localhost:7990/rest/api/latest/migration/mesh/preview | jq .The following request values are required for preview:
| Description | Value |
|---|---|
| URL | /rest/api/latest/migration/mesh/preview |
| HTTP verb | POST |
| HTTP header | Content-type: application/json |
Authentication | Basic |
You can specify multiple repositories in the body of the request. Here are some examples:
To preview all repositories:
{
"projectIds" : [],
"repositoryIds" : []
}To preview one full project and a specific repository:
{
"projectIds" : [1],
"repositoryIds" : [42]
}You can specify as many project repository Ids as necessary, and the selection would be additive. For instance, the above request includes all repositories in the project with Id 1, and also a repository with Id 42, unless the repository is already a part of the project with Id 1.
Start the migration
When you start the migration, all repositories included in the migration are queued in the database, and are migrated to the Mesh individually. At a given point, note that only one migration is active.
You can perform the migration using the below command:
curl -u <adminuser> -s -n -X POST -H 'Content-type: application/json' -d '{"projectIds":[1, 2, 3], "repositoryIds":[1, 2, 3]}' http://localhost:7990/rest/api/latest/migration/mesh | jq .The following values are required for migration:
| Description | Value |
|---|---|
| URL | /rest/api/latest/migration/mesh |
| HTTP verb | POST |
| HTTP header | Content-type: application/json |
Authentication | Basic |
Use the same request body as in the Previewing the migration section.
The response contains the job ID – you can use this to query the status of the job later.
Monitor the progress of the migration
You can query the following specific REST endpoint to check the progress:
watch -n30 'curl -u <adminuser> -s -n -X GET http://localhost:7990/rest/api/latest/migration/mesh/<jobId>/summary | jq .'Replace < with the Id value returned in the request from the migration job.jobId>
The following values are required to query the progress:
| Description | Value |
|---|---|
| URL | /rest/api/latest/migration/mesh/<jobId>/summary |
| HTTP verb | GET |
Authentication | Basic |
Cancel the migration?
When you place a cancellation request, you may notice a small delay before the job gets cancelled. In most cases, this delay is inconsequential. Note that this only cancels the migration of repositories that haven’t been migrated yet – the repository hierarchies that have already been fully migrated to Mesh will stay on it and will not be reverted to the shared home.
curl -u <adminuser> -s -n -X POST -H 'Content-type: application/json' http://localhost:7990/rest/api/latest/migration/mesh/<jobId>/cancelReplace < with the Id value returned in the request from the migration job.jobId>
The following values are required to cancel the migration (REST Documentation):
| Description | Value |
|---|---|
| URL | /rest/api/latest/migration/mesh/<jobId>/cancel |
| HTTP verb | POST |
Authentication | Basic |