[Other doc versions]
[Doc downloads]
Stash DIY Backup was introduced in Stash 2.12 as an alternative strategy to using the Stash Backup Client. It allows you to:
pg_dump
if your back end database is PostgreSQL, orsqlcmd
with an appropriate command for differential backup, if your back end database is MS SQL Server,
rsync
, if available.The key to reducing Stash downtime to a minimum is the use of optimal, vendor-specific database and file system backup tools. Such tools are generally able to take snapshots (though sometimes in a vendor-specific format) in much less time than the generic, vendor-neutral format used by the Stash Backup Client.
Stash DIY Backup does require you to write some code in a language of your choice to perform the required backup steps, using the REST API available for Stash 2.12.
DIY Backup supports Windows and Linux platforms, and Stash versions 2.12 and higher. DIY Backup supports both Stash Server and Stash Data Center instances equally - any DIY Backup solution that works on one should work on the other without modification.
For information about other backup strategies for Stash, see Data recovery and backups. That page also discusses the tight coupling between the Stash file system on disk and the database that Stash uses.
Please note that the examples on this page are provided as guidance for developing a DIY Backup solution. As such, the third-party tools described are for example only – you will need to choose the tools that are appropriate to your own specific installation of Stash.
Consult the vendor documentation for the third-party tools you choose – unfortunately, Atlassian can not provide support for those tools.
On this page:
Download the worked example scripts from Bitbucket:
This page:
bash
shell scripts.You can use this solution directly if your Stash instance has the same or similar configuration, or use this as a starting point to develop your own DIY Backup solution tailored to your hardware configuration.
When you use DIY Backup instead of the Stash Backup Client, you have complete control over the backup steps, and can implement any custom processes you like in the language of your choice. For example, you can use your database's incremental or fast snapshot tools and/or your file server's specific tools as part of a DIY Backup.
The DIY Backup works in a similar way to the Stash Backup Client and does the following:
rsync
of the home folder to the backup folder.pg_dump
.rsync
.A user will get an error message if they try to access the Stash web interface, or use the Stash hosting services, when Stash is in maintenance mode.
As an indication of the unavailability time that can be expected, in Atlassian's internal use we have seen downtimes for Stash of 7–8 minutes with repositories totalling 6 GB in size when using the Stash Backup Client. For comparison, using Stash DIY Backup for the same repositories typically results in a downtime of less than a minute.
DIY Backup backs up all the same data as the Stash Backup Client:
This section presents a complete DIY Backup solution that uses the following tools:
bash
- for scriptingjq
- an open source command line JSON processor for parsing the REST responses from Stashpg_dump
(or sqlcmd
) - for backing up a PostgreSQL databasersync
- for backing up the filesystemtar
- for making a backup archiveThis approach (with small modifications) can be used for running DIY Backups on:
Download the scripts from Bitbucket.
The scripts below are for example only and are not kept in sync with the scripts found in the Bitbucket repository.
These scripts implement one particular solution for performing a DIY Backup:
#!/bin/bash ########################################################## # Configure the settings in this section to suit your Stash installation. SCRIPT_DIR=$(dirname $0) # Contains all variables used by the other scripts. source ${SCRIPT_DIR}/stash.diy-backup.vars.sh # Contains util functions (bail, info, print). source ${SCRIPT_DIR}/stash.diy-backup.utils.sh # Contains functions that perform lock/unlock and backup of a Stash instance. source ${SCRIPT_DIR}/stash.diy-backup.common.sh # The following scripts contain functions which are dependant on the configuration of this Stash instance. # Generally every each of them exports certain functions, which can be implemented in different ways. # Exports the following functions: # stash_prepare_db - for making a backup of the DB if differential backups a possible. Can be empty. # stash_backup_db - for making a backup of the Stash DB. source ${SCRIPT_DIR}/stash.diy-backup.postgresql.sh # Exports the following functions: # stash_prepare_home - for preparing the filesystem for the backup. # stash_backup_home - for making the actual filesystem backup. source ${SCRIPT_DIR}/stash.diy-backup.rsync.sh # Exports the following functions: # stash_backup_archive - for archiving the backup folder and puting the archive in archive folder. source ${SCRIPT_DIR}/stash.diy-backup.tar.sh ########################################################## # This section does NOT need any configuration. # The actual back up process. It has the following steps: # Prepare the database and the filesystem for taking a backup. stash_prepare_db stash_prepare_home # Locking the Stash instance, starting an external backup and waiting for instance readiness. stash_lock stash_backup_start stash_backup_wait # Backing up the database and reporting 50% progress. stash_backup_db stash_backup_progress 50 # Backing up the filesystem and reporting 100% progress. stash_backup_home stash_backup_progress 100 # Unlocking the Stash instance. stash_unlock # Making an archive for this backup. stash_backup_archive ##########################################################
The stash.diy-backup.sh
script above references the following scripts:
Once you have downloaded the Bash scripts, you need to customize two files:
stash.diy-backup.vars.sh
stash.diy-backup.sh
for your setup.
For example, if your Stash server is called stash.example.com,
uses port 7990, and has its home directory in /stash-home
, here's how you might configure stash.diy-backup.vars.sh
:
#!/bin/bash
# Used by the scripts for verbose logging. If not true only errors will be shown.
STASH_VERBOSE_BACKUP=TRUE
# The base url used to access this Stash instance.
STASH_URL= http://stash.example.com:7990
# The username and password for the account used to make backups (and has permission for this).
STASH_BACKUP_USER= admin
STASH_BACKUP_PASS= admin
# The name of the database used by this instance.
STASH_DB= stash
# The path to Stash home folder (with trailing /).
STASH_HOME= /stash-home
# The path to the working folder for the backup.
STASH_BACKUP_ROOT= /stash-backup
STASH_BACKUP_DB=${STASH_BACKUP_ROOT}/stash-db/
STASH_BACKUP_HOME=${STASH_BACKUP_ROOT}/stash-home/
# The path to where the backup archives are stored.
STASH_BACKUP_ARCHIVE_ROOT= /stash-backup-archives
The supplied stash.diy-backup.vars.sh
is written to use PostgreSQL, rsync, and tar by default. But if you want to use different tools, you can also customize the top section of this file to use different tools as follows:
# The following scripts contain functions which are dependant on the configuration of this Stash instance.
# Generally, each of them exports certain functions, which can be implemented in different ways.
# Exports the following functions:
# stash_prepare_db - for making a backup of the DB if differential backups are possible. Can be empty.
# stash_backup_db - for making a backup of the Stash DB.
source ${SCRIPT_DIR}/ stash.diy-backup.postgresql.sh
# Exports the following functions:
# stash_prepare_home - for preparing the filesystem for the backup.
# stash_backup_home - for making the actual filesystem backup.
source ${SCRIPT_DIR}/ stash.diy-backup.rsync.sh
# Exports the following functions:
# stash_backup_archive - for archiving the backup folder and moving the archive to the archive folder.
source ${SCRIPT_DIR}/ stash.diy-backup.tar.sh
You also need to create two directories for DIY Backup to work:
${STASH_BACKUP_ROOT}
is a working directory (/stash-backup
in our example) where copies of Stash's home directory and database dump are built during the DIY Backup process. ${STASH_BACKUP_ARCHIVE_ROOT}
is the directory (/stash-backup-archives
in our example) where the final backup archives are saved. The Bash scripts may be run on any host, provided it has:
${STASH_BACKUP_ROOT}
and ${STASH_BACKUP_ARCHIVE_ROOT}
directories,${STASH_HOME}
directory,curl
commands on the Stash server. This is true regardless of whether you have an instance of Stash Server or Stash Data Center. It doesn't matter whether the filesystem access is direct or over NFS, or whether the network access is direct to a Stash node or to a load balancer / reverse proxy.
Once your stash.diy-backup.vars.sh
is correctly configured, run the backup in a terminal window:
$ ./stash.diy-backup.sh
The first time you run the backup, rsync
will do most of the work since the /stash-backup
working directory is initially empty. This is normal. Fortunately, this script performs one rsync
before locking Stash, followed by a second rsync
while Stash is locked. This minimizes downtime.
On second and subsequent backup runs, /stash-backup
is already populated so the backup process should be faster. The output you can expect to see looks something like this:
When restoring Stash, you must run the
stash.diy-restore.sh
script on the machine that Stash should be restored to. In order to ensure accidental restores do not delete existing data, you should never restore into an existing home directory.
The new database should be configured following the instructions in Connecting Stash to an external database and its sub-page that corresponds to your database type.
To see the available backups in your ${STASH_BACKUP_ARCHIVE_ROOT}
directory, just type:
$ ./stash.diy-restore.sh
You should see output similar to this:
To restore a backup, run stash.diy-restore.sh
with the file name as the argument:
$ ./stash.diy-restore.sh stash-20140320-092743-063.tar.gz
You should see output like this:
You can cancel the running backup operation if necessary.
To cancel the backup:
This section is optional and provides background information about how you might use the Stash REST APIs if you need to rewrite the DIY Backup scripts described above in your preferred language or to customize them heavily.
Note that this discussion shows curl
commands in Bash, however you can use any language.
The following steps are involved:
Before you lock Stash you can perform any preparation you like. It makes sense to perform as much processing as possible before you lock Stash, to minimize downtime later. For example, you could perform an rsync
:
rsync -avh --delete --delete-excluded --exclude=/caches/ --exclude=/data/db.* --exclude=/export/ --exclude=/log/ --exclude=/plugins/.*/ --exclude=/tmp --exclude=/.lock ${STASH_HOME} ${STASH_BACKUP_HOME}
The next step in making a backup of a Stash instance is to lock the instance for maintenance. This can be done using a POST request to the /mvc/maintenance/lock
REST point (where STASH_URL
points to the Stash instance, STASH_BACKUP_USER
is a Stash user with backup permissions, and STASH_BACKUP_PASS
is this user's password).
curl -s \ -u ${STASH_BACKUP_USER}:${STASH_BACKUP_PASS} \ -X POST \ -H "Content-type: application/json" \ "${STASH_URL}/mvc/maintenance/lock"
{ "unlockToken":"0476adeb6cde3a41aa0cc19fb394779191f5d306", "owner": { "displayName":"admin", "name":"admin" } }
If successful, the Stash instance will respond with a 202 and will return a response JSON similar to the one above. The unlockToken
should be used in all subsequent requests where $STASH_LOCK_TOKEN
is required. This token can also be used to manually unlock this Stash instance.
Next, all connections to both the database and the filesystem must be drained and latched. Your code must handle backing up of both the filesystem and the database.
At this point, you should make a POST
request to /mvc/admin/backups
. Notice that the curl
call includes the ?external=true
parameter (requires Stash 2.12+):
curl -s \ -u ${STASH_BACKUP_USER}:${STASH_BACKUP_PASS} \ -X POST \ -H "X-Atlassian-Maintenance-Token: ${STASH_LOCK_TOKEN}" \ -H "Accept: application/json" \ -H "Content-type: application/json" \ "${STASH_URL}/mvc/admin/backups?external=true"
{ "id":"d2e15c3c2da282b0990e8efb30b4bffbcbf09e04", "progress": { "message":"Closing connections to the current database", "percentage":5 }, "state":"RUNNING", "type":"BACKUP", "cancelToken":"d2e15c3c2da282b0990e8efb30b4bffbcbf09e04" }
If successful the Stash instance will respond with 202 and a response JSON similar to the one above will be returned. The cancelToken
can be used to manually cancel the back up process.
Wait for the Stash instance to complete preparation
Part of the back up process includes draining and latching the connections to the database and the filesystem. Before continuing with the back up we have to wait for the Stash instance to report that this has been done. To get details on the current status we make a GET
request to the /mvc/maintenance
REST point.
curl -s \ -u ${STASH_BACKUP_USER}:${STASH_BACKUP_PASS} \ -X GET \ -H "X-Atlassian-Maintenance-Token: ${STASH_LOCK_TOKEN}" \ -H "Accept: application/json" \ -H "Content-type: application/json" \ "${STASH_URL}/mvc/maintenance"
{ "task":{ "id":"0bb6b2ed52a6a12322e515e88c5d515d6b6fa95e", "progress":{ "message":"Backing up Stash home", "percentage":10 }, "state":"RUNNING", "type":"BACKUP" }, "db-state":"DRAINED", "scm-state":"DRAINED" }
This causes the Stash instance to report its current state. We have to wait for both db-state
and scm-state
to have a status of DRAINED
before continuing with the back up.
At this point we are ready to create the actual backup of the Stash filesystem. For example, you could use rsync again:
rsync -avh --delete --delete-excluded --exclude=/caches/ --exclude=/data/db.* --exclude=/export/ --exclude=/log/ --exclude=/plugins/.*/ --exclude=/tmp --exclude=/.lock ${STASH_HOME} ${STASH_BACKUP_HOME}
The rsync options shown here are for example only, but indicate how you can include only the required files in the backup process and exclude others. Consult the documentation for rsync
, or the tool of your choice, for a more detailed description.
When creating the database backup you could use your vendor-specific database backup tool, for example pg_dump
if you use PostgreSQL:
pg_dump -Fd ${STASH_DB} -j 5 --no-synchronized-snapshots -f ${STASH_BACKUP_DB}
While performing these operations, good practice is to update the Stash instance with progress on the backup so that it's visible in the UI. This can be done by issuing a POST
request to /mvc/admin/backups/progress/client
with the token and the percentage completed as parameters:
curl -s \ -u ${STASH_BACKUP_USER}:${STASH_BACKUP_PASS} \ -X POST \ -H "Accept: application/json" \ -H "Content-type: application/json" \ "${STASH_URL}/mvc/admin/backups/progress/client?token=${STASH_LOCK_TOKEN}&percentage=${STASH_BACKUP_PERCENTAGE}"
The Stash instance will respond to this request with an empty 202 if everything is OK.
When displaying progress to users, Stash divides the 100 percent progress into 90 percent user DIY Backup, and 10 percent Stash preparation. This means, for example, if your script sends percentage=0
, Stash may display up to 10 percent progress for its own share of the backup work.
Once we've finished the backup process we must report to the Stash instance that progress has reached 100 percent. This is done using a similar request to the progress request. We issue a POST
request to /mvc/admin/backups/progress/client
with the token and 100 as the percentage:
curl -s \ -u ${STASH_BACKUP_USER}:${STASH_BACKUP_PASS} \ -X POST \ -H "Accept: application/json" \ -H "Content-type: application/json" \ "${STASH_URL}/mvc/admin/backups/progress/client?token=${STASH_LOCK_TOKEN}&percentage=100"
The Stash will respond with an empty 202 if everything is OK. The back up process is considered completed once the percentage is 100. This will unlatch the database and the filesystem for this Stash instance.
The final step we need to do in the back up process is to unlock the Stash instance. This is done with a DELETE
request to the /mvc/maintenance/lock
REST point:
curl -s \ -u ${STASH_BACKUP_USER}:${STASH_BACKUP_PASS} \ -X DELETE \ -H "Accept: application/json" \ -H "Content-type: application/json" \ "${STASH_URL}/mvc/maintenance/lock?token=${STASH_LOCK_TOKEN}"
The Stash instance will respond to this request with an empty 202 if everything is OK, and will unlock access.