Bitbucket zero downtime backup

Backup best practices in Bitbucket

On this page

Related content

  • No related content found

Still need help?

The Atlassian Community is here for you.

Ask the community

This page describes how to back up Bitbucket Data Center without downtime, and restoring the backups correctly.

About zero downtime backup

Zero downtime backup is a technique introduced in Bitbucket 4.8 which backs up a Bitbucket instance without requiring it be locked for maintenance. It requires:

  1. your shared home directory to be on a file system volume capable of atomic (block level) snapshots, and
  2. your database to be capable of either taking atomic snapshots or restoring a snapshot at the same point in time as the home directory snapshot was taken.

Use of these technologies allows you to take backups as often as you need (e.g., hourly), without inconveniencing your users and build agents with frequent downtimes.

On this page

Prerequisites

Block level file system snapshots

Your shared home directory must be on a file system volume capable of atomic (block level) snapshots, for example, Amazon EBS, LVM, NetApp, XFS, or ZFS. These technologies are becoming increasingly common in modern operating systems and storage solutions. If your shared home directory volume pre-dates these technologies, then you must first move your shared home directory onto one that supports block level snapshots before using zero downtime backup. You also need to script the steps to snapshot the volume in the backup process. The atlassian-bitbucket-diy-backup script does not include fully worked examples for every vendor technology. If you are unable to create such a snapshot, please consider another backup strategy.

Read more about this...

Block level snapshots ensure your repository data will have full internal consistency when restored, even if taken without a maintenance lock under heavy load. Which block level snapshot technology you use depends on your choice of infrastructure for your shared home directory volume:

  • File server which provides block level snapshots (for example, NetApp Snapshots): Refer to your vendor's documentation on how to script the snapshot and restore process. Atlassian does not provide examples or support for the use of these vendor tools.
  • Linux file system: LVM, XFS, and ZFS are all capable of taking block level snapshots. Refer to the documentation with your Linux distribution on how to script the snapshot and restore process. Atlassian does not provide examples or support for their use.
  • Amazon Web Services (AWS) Elastic Block Store (EBS) volume: You can use Amazon EBS Snapshots to take block level snapshots. The EBS volume may be formatted with any file system type, but note that Linux typically requires the filesystem to be "frozen" with fsfreeze while the snapshot is taken, for its own internal consistency. The atlassian-bitbucket-diy-backup script has a worked example of using EBS Snapshots.

Whichever file system snapshot technology you ultimately choose, you will need to refer to your vendor's documentation to script the snapshot and restore process. The only fully worked examples currently included in the atlassian-bitbucket-diy-backup script are for Amazon EBS and ZFS.

Database snapshot technology

Your database must be capable of restoring a snapshot close to the same point in time as the home directory snapshot. The easiest way to do this is by taking database snapshots close to the home directory backup time. Alternatively, some databases support a vendor-specific "point-in-time recovery" feature at restore time. All database vendors supported by Bitbucket provide tools for taking fast snapshots and point-in-time recovery. 

Read more about this...

Which technology you choose is a tradeoff between how much work you need to do at backup and restore time. 

  • Vendor dump and restore utilities: This option needs you to dump your database to a file at backup time, in parallel with the home directory snapshot. This can generally achieve a database snapshot that is within a few seconds of your home directory snapshot. The atlassian-bitbucket-diy-backup script includes a worked example of using PostgreSQL's pg_dump and pg_restore with zero downtime backup.
  • Point-in-time recovery: Alternatively if you enable the point-in-time recovery feature of your database vendor, then you don't have to dump the database explicitly at backup time. Instead you can recover a snapshot of the database state at the same time as the home directory snapshot, at restore time. This saves you from having to do database dumps during backup, at the cost of a slightly longer restore time. The atlassian-bitbucket-diy-backup script does not include a worked example of using point-in-time recovery, you need to refer to your vendor's documentation on how to script this process.
  • Amazon Web Services (AWS) Relational Database Service (RDS) instance: You can use the RDS Backup and Restore feature to take snapshots at backup time. The atlassian-bitbucket-diy-backup script includes a worked example of using RDS snapshots with zero downtime backup.
  • Block level snapshot of the database's data volume: If your database's data directory is located on a file system volume that is capable of block level snapshots, then most databases support backup and restore at the file system level. The atlassian-bitbucket-diy-backup script includes a worked example of using block level snapshots where the database is on the same volume as your shared home directory.

Whichever database backup technology you choose, you will need to refer to your vendor's documentation to script the snapshot and restore process.

Bitbucket Server and Data Center 4.8 or later

You must be running Bitbucket 4.8 or higher to use zero downtime backup. See the Bitbucket Data Center upgrade guide.

Configure the example DIY Backup script for zero downtime backup

With the above prerequisites, you can create backups with any method that performs home directory and database snapshots simultaneously. Atlassian provides an example atlassian-bitbucket-diy-backup script which can automate the process, and can be used as a starting point for you to configure and customize your own backup procedures.

Step 1: Get the script

Clone or pull the latest version of the sample atlassian-bitbucket-diy-backup script, for example:

git clone https://bitbucket.org/atlassianlabs/atlassian-bitbucket-diy-backup.git
cd atlassian-bitbucket-diy-backup


Step 2: Configure the script

Create the file bitbucket.diy-backup.vars.sh by copying the appropriate bitbucket.diy-backup.vars.sh.example, and edit it to match your environment. For example, to use Amazon EBS snapshots of the volume containing your shared home directory and Amazon RDS snapshots of your database, you might configure it as follows:

BACKUP_DATABASE_TYPE=amazon-rds
BACKUP_HOME_TYPE=amazon-ebs
BACKUP_ZERO_DOWNTIME=true
BITBUCKET_URL=https://your-bitbucket-url

If your shared home directory is not on Amazon EBS, then you must define a new BACKUP_HOME_TYPE for your chosen file system technology. Give it a name (e.g., myhome) that will identify the snapshot script to run in the next step.

BACKUP_HOME_TYPE=myhome

BACKUP_HOME_TYPE=rsync is not a valid option to use with zero downtime backup.

Refer to Bitbucket DIY Backup and Using Bitbucket DIY Backup in AWS for more information on the configurable options in the DIY Backup example script.

Step 3: (If necessary) Script the snapshot process

If you have chosen BACKUP_HOME_TYPE and BACKUP_DATABASE_TYPE values that both already have worked examples included in the atlassian-bitbucket-diy-backup repository (e.g., ebs-home and rds), then you can just configure the appropriate variables in bitbucket.diy-backup.vars.sh to match your environment, and skip this step.

If your shared home directory is not one of the worked examples provided in atlassian-bitbucket-diy-backup, then you must script the process of taking and restoring snapshots of your home directory volume. 

Read more about this...

Create a script called bitbucket.diy-backup.myhome.sh (where myhome is the value of BACKUP_HOME_TYPE that you set in the previous step), defining the following BASH functions:

#!/bin/bash

function bitbucket_prepare_home {
# you can optionally do any backup-time validation you need to do here
}

function bitbucket_backup_home {
# this is where you freeze, snapshot, and unfreeze your home directory volume
}

function bitbucket_restore_home {
# this is where you restore a snapshot of your home directory volume
}


Similarly if you wish to use the point-in-time recovery feature of your database with zero downtime backup (or some other functionality that does not have a fully worked example provided in atlassian-bitbucket-diy-backup), then you must script the process.

Read more about this...

Create or modify a script bitbucket.diy-backup.mydb.sh (where mydb is the value of BACKUP_DATABASE_TYPE that you set in the previous step), defining the following BASH functions:

#!/bin/bash

function bitbucket_prepare_db {
# you can optionally do any backup-time validation you need to do here
}

function bitbucket_backup_db {
# this is where you can snapshot your database, if necessary
}

function bitbucket_prepare_db_restore {
# this is where you can do any restore-time validation, if necessary
}

function bitbucket_restore_db {
# this is where you restore a snapshot of your database
}

Back up your instance

Once your bitbucket.diy-backup.vars.sh is correctly configured, SSH to the appropriate node in a Data Center instance (typically, the file server) and run the backup script from the atlassian-bitbucket-diy-backup directory.

./bitbucket.diy-backup.sh

This script can also be run on a regular schedule (e.g., hourly), from cron.

Restore from a backup

Step 1: Stop Bitbucket

The restore process only works while Bitbucket is stopped. Stop the Bitbucket services (on all nodes if your instance is a clustered Data Center instance).

sudo service atlbitbucket stop
sudo service atlbitbucket_search stop 

See Start and stop Bitbucket for more information.

Step 2: Run the restore script

To restore, SSH to the appropriate node in a Data Center instance (typically, the file server) and run the restore script from the atlassian-bitbucket-diy-backup directory.

./bitbucket.diy-restore.sh

Step 3: Start Bitbucket

Data Center customers at this point may enable integrity checking to scan for potential inconsistencies between the database and home directory, and resolve them if necessary so that your pull request and repository state are completely consistent with each other. See Running integrity checks in Bitbucket Data Center for further information.

Start the Bitbucket services (on all nodes if your instance is a clustered Data Center instance).

sudo service atlbitbucket start
sudo service atlbitbucket_search start

See Start and stop Bitbucket for more information.

Last modified on Jun 18, 2024

Was this helpful?

Yes
No
Provide feedback about this article

Related content

  • No related content found
Powered by Confluence and Scroll Viewport.