Using Stash DIY Backup

Stash DIY Backup was introduced in Stash 2.12 as an alternative strategy to using the Stash Backup Client. It allows you to:

  • significantly reduce the downtime needed to create a consistent backup,
  • use the vendor-specific database backup tool appropriate to your back end database, for example:
    • pg_dump if your back end database is PostgreSQL, or
    • sqlcmd with an appropriate command for differential backup, if your back end database is MS SQL Server,

  • use the optimal file system backup tool for your Stash home directory, for example:
    • an LVM snapshot logical volume if your Stash home directory uses LVM,
    • a SAN-based backup if your Stash home directory uses a Storage Area Network, or
    • rsync, if available.
  • take backups of Stash Data Center instances without having to bring nodes down manually.

The key to reducing Stash downtime to a minimum is the use of optimal, vendor-specific database and file system backup tools. Such tools are generally able to take snapshots (though sometimes in a vendor-specific format) in much less time than the generic, vendor-neutral format used by the Stash Backup Client.

Stash DIY Backup does require you to write some code in a language of your choice to perform the required backup steps, using the REST API available for Stash 2.12.

DIY Backup supports Windows and Linux platforms, and Stash versions 2.12 and higher. DIY Backup supports both Stash Server and Stash Data Center instances equally - any DIY Backup solution that works on one should work on the other without modification.

For information about other backup strategies for Stash, see Data recovery and backups. That page also discusses the tight coupling between the Stash file system on disk and the database that Stash uses.

Please note that the examples on this page are provided as guidance for developing a DIY Backup solution. As such, the third-party tools described are for example only – you will need to choose the tools that are appropriate to your own specific installation of Stash.

Consult the vendor documentation for the third-party tools you choose – unfortunately, Atlassian can not provide support for those tools.


On this page:

 

Download the worked example scripts from Bitbucket:

This page:

  • Describes a complete DIY Backup solution for a PostgreSQL database and local filesystem, using bash shell scripts.
  • Provides background information about how the Stash REST API can be used for DIY Backups.

You can use this solution directly if your Stash instance has the same or similar configuration, or use this as a starting point to develop your own DIY Backup solution tailored to your hardware configuration. 

How it works

When you use DIY Backup instead of the Stash Backup Client, you have complete control over the backup steps, and can implement any custom processes you like in the language of your choice.  For example, you can use your database's incremental or fast snapshot tools and/or your file server's specific tools as part of a DIY Backup. 

The DIY Backup works in a similar way to the Stash Backup Client and does the following:

  1. Prepare the instance for backup. This happens before Stash is locked, so we want to do as much processing as possible here in order to minimize downtime later.  For example, we can take an initial snapshot using incremental database and filesystem utilities.  These do not have to be 100% consistent as Stash is still running and modifying the database and filesystem.  But taking the initial snapshot now may reduce the amount of work done later (while Stash is locked), especially if the amount of data modified between backups is large.  The steps include:
    • Taking an initial backup of the database (if it supports progressive/differential backups).
    • Doing an initial rsync of the home folder to the backup folder.
  2. Tell Stash to initiate the backup. Stash will:
    • Lock the instance.
    • Drain and latch the connections to the database and the filesystem.
    • Wait for the drain/latch step to complete.
  3. Once the Stash instance is ready for backup we can start with the actual DIY Backup. This will include steps to:
    • Make a fully consistent backup of the database, using pg_dump.
    • Make a fully consistent backup of the filesystem, using rsync.
  4. Notify the Stash instance once the backup process finishes and unlock it.
  5. Archive all files created during the backup into one big archive.

A user will get an error message if they try to access the Stash web interface, or use the Stash hosting services, when Stash is in maintenance mode.

As an indication of the unavailability time that can be expected, in Atlassian's internal use we have seen downtimes for Stash of 7–8 minutes with repositories totalling 6 GB in size when using the Stash Backup Client. For comparison, using Stash DIY Backup for the same repositories typically results in a downtime of less than a minute. 

What is backed up

DIY Backup backs up all the same data as the Stash Backup Client:

  • the database Stash is connected to (either the internal or external DB)
  • managed Git repositories
  • the Stash logs
  • installed plugins and their data

DIY Backups using Bash scripts

This section presents a complete DIY Backup solution that uses the following tools:

  • bash - for scripting
  • jq - an open source command line JSON processor for parsing the REST responses from Stash
  • pg_dump (or sqlcmd) - for backing up a PostgreSQL database
  • rsync - for backing up the filesystem
  • tar - for making a backup archive

This approach (with small modifications) can be used for running DIY Backups on:

  • Linux and Unix
  • OSX
  • Windows with cygwin (note that cygwin Git is not supported by Stash).

Download the scripts from Bitbucket.

The scripts below are for example only and are not kept in sync with the scripts found in the Bitbucket repository.

Bash scripts

These scripts implement one particular solution for performing a DIY Backup: 

 

stash.diy-backup.sh
#!/bin/bash
 
##########################################################
# Configure the settings in this section to suit your Stash installation.

SCRIPT_DIR=$(dirname $0)
# Contains all variables used by the other scripts.
source ${SCRIPT_DIR}/stash.diy-backup.vars.sh
# Contains util functions (bail, info, print).
source ${SCRIPT_DIR}/stash.diy-backup.utils.sh
# Contains functions that perform lock/unlock and backup of a Stash instance.
source ${SCRIPT_DIR}/stash.diy-backup.common.sh

# The following scripts contain functions which are dependant on the configuration of this Stash instance.
# Generally every each of them exports certain functions, which can be implemented in different ways.

# Exports the following functions:
#     stash_prepare_db     - for making a backup of the DB if differential backups a possible. Can be empty.
#     stash_backup_db      - for making a backup of the Stash DB.
source ${SCRIPT_DIR}/stash.diy-backup.postgresql.sh

# Exports the following functions:
#     stash_prepare_home   - for preparing the filesystem for the backup.
#     stash_backup_home    - for making the actual filesystem backup.
source ${SCRIPT_DIR}/stash.diy-backup.rsync.sh

# Exports the following functions:
#     stash_backup_archive - for archiving the backup folder and puting the archive in archive folder.
source ${SCRIPT_DIR}/stash.diy-backup.tar.sh

##########################################################
# This section does NOT need any configuration.
# The actual back up process. It has the following steps:

# Prepare the database and the filesystem for taking a backup.
stash_prepare_db
stash_prepare_home

# Locking the Stash instance, starting an external backup and waiting for instance readiness.
stash_lock
stash_backup_start
stash_backup_wait

# Backing up the database and reporting 50% progress.
stash_backup_db
stash_backup_progress 50

# Backing up the filesystem and reporting 100% progress.
stash_backup_home
stash_backup_progress 100

# Unlocking the Stash instance.
stash_unlock

# Making an archive for this backup.
stash_backup_archive
##########################################################


The stash.diy-backup.sh script above references the following scripts:

 

  stash.diy-backup.vars.sh
stash.diy-backup.vars.sh
#!/bin/bash

# Used by the scripts for verbose logging. If not true only errors will be shown.
STASH_VERBOSE_BACKUP=TRUE

# The base url used to access this Stash instance.
STASH_URL=

# The username and password for the user used to make backups (and have this permission).
STASH_BACKUP_USER=
STASH_BACKUP_PASS=

# The name of the database used by this instance.
STASH_DB=stash

# The path to Stash home folder (with trailing /).
STASH_HOME=

# The path to working folder for the backup.
STASH_BACKUP_ROOT=
STASH_BACKUP_DB=${STASH_BACKUP_ROOT}/stash-db/
STASH_BACKUP_HOME=${STASH_BACKUP_ROOT}/stash-home/

# The path to where the backup archives are stored.
STASH_BACKUP_ARCHIVE_ROOT=
  stash.diy-backup.utils.sh
stash.diy-backup.utils.sh
#!/bin/bash

function error {
    echo "[${STASH_URL}] ERROR:" $*
}

function bail {
    error $*
    exit 99
}

function info {
    if [ "${STASH_VERBOSE_BACKUP}" == "TRUE" ]; then
        echo "[${STASH_URL}]  INFO:" $*
    fi
}

function print {
    if [ "${STASH_VERBOSE_BACKUP}" == "TRUE" ]; then
        echo $*
    fi
}

 
function check_command {
    type -P $1 &>/dev/null || bail "Unable to find $1, please install it and run this script again"
}
  stash.diy-backup.common.sh
stash.diy-backup.common.sh
#!/bin/bash

check_command "curl"
check_command "jq"

STASH_HTTP_AUTH="-u ${STASH_BACKUP_USER}:${STASH_BACKUP_PASS}"

function stash_lock {
    STASH_LOCK_RESULT=`curl -s -f ${STASH_HTTP_AUTH} -X POST -H "Content-type: application/json" "${STASH_URL}/mvc/maintenance/lock"`
    if [ -z "${STASH_LOCK_RESULT}" ]; then
        bail "Locking this Stash instance failed"
    fi

    STASH_LOCK_TOKEN=`echo ${STASH_LOCK_RESULT} | jq -r ".unlockToken"`
    if [ -z "${STASH_LOCK_TOKEN}" ]; then
        bail "Unable to find lock token. Result was '$STASH_LOCK_RESULT'"
    fi

    info "locked with '$STASH_LOCK_TOKEN'"
}

function stash_backup_start {
    STASH_BACKUP_RESULT=`curl -s -f ${STASH_HTTP_AUTH} -X POST -H "X-Atlassian-Maintenance-Token: ${STASH_LOCK_TOKEN}" -H "Accept: application/json" -H "Content-type: application/json" "${STASH_URL}/mvc/admin/backups?external=true"`
    if [ -z "${STASH_BACKUP_RESULT}" ]; then
        bail "Entering backup mode failed"
    fi

    STASH_BACKUP_TOKEN=`echo ${STASH_BACKUP_RESULT} | jq -r ".cancelToken"`
    if [ -z "${STASH_BACKUP_TOKEN}" ]; then
        bail "Unable to find backup token. Result was '${STASH_BACKUP_RESULT}'"
    fi

    info "backup started with '${STASH_BACKUP_TOKEN}'"
}

function stash_backup_wait {
    STASH_PROGRESS_DB_STATE="AVAILABLE"
    STASH_PROGRESS_SCM_STATE="AVAILABLE"

    print -n "[${STASH_URL}] .INFO: Waiting for DRAINED state "
    while [ ${STASH_PROGRESS_DB_STATE} != "DRAINED" -a ${STASH_PROGRESS_SCM_STATE} != "DRAINED" ]; do
        print -n "."

        STASH_PROGRESS_RESULT=`curl -s -f ${STASH_HTTP_AUTH} -X GET -H "X-Atlassian-Maintenance-Token: ${STASH_LOCK_TOKEN}" -H "Accept: application/json" -H "Content-type: application/json" "${STASH_URL}/mvc/maintenance"`
        if [ -z "${STASH_PROGRESS_RESULT}" ]; then
            bail "[${STASH_URL}] ERROR: Unable to check for backup progress"
        fi

        STASH_PROGRESS_DB_STATE=`echo ${STASH_PROGRESS_RESULT} | jq -r '.["db-state"]'`
        STASH_PROGRESS_SCM_STATE=`echo ${STASH_PROGRESS_RESULT} | jq -r '.["scm-state"]'`
    done

    print "done"
    info "db state '${STASH_PROGRESS_DB_STATE}'"
    info "scm state '${STASH_PROGRESS_SCM_STATE}'"
}

function stash_backup_progress {
    STASH_REPORT_RESULT=`curl -s -f ${STASH_HTTP_AUTH} -X POST -H "Accept: application/json" -H "Content-type: application/json" "${STASH_URL}/mvc/admin/backups/progress/client?token=${STASH_LOCK_TOKEN}&percentage=$1"`
    if [ $? != 0 ]; then
        bail "Unable to update backup progress"
    fi

    info "Backup progress updated to $1"
}

function stash_unlock {
    STASH_UNLOCK_RESULT=`curl -s -f ${STASH_HTTP_AUTH} -X DELETE -H "Accept: application/json" -H "Content-type: application/json" "${STASH_URL}/mvc/maintenance/lock?token=${STASH_LOCK_TOKEN}"`
    if [ $? != 0 ]; then
        bail "Unable to unlock instance with lock ${STASH_LOCK_TOKEN}"
    fi

    info "Stash instance unlocked"
}
  stash.diy-backup.postgresql.sh
stash.diy-backup.postgresql.sh
#!/bin/bash

check_command "pg_dump"
check_command "psql"
check_command "pg_restore"

function stash_prepare_db {
    info "Prepared backup of DB ${STASH_DB} in ${STASH_BACKUP_DB}"
}

function stash_backup_db {
    rm -r ${STASH_BACKUP_DB}
    pg_dump -Fd ${STASH_DB} -j 5 --no-synchronized-snapshots -f ${STASH_BACKUP_DB}
    if [ $? != 0 ]; then
        bail "Unable to backup ${STASH_DB} to ${STASH_BACKUP_DB}"
    fi
    info "Performed backup of DB ${STASH_DB} in ${STASH_BACKUP_DB}"
}

function stash_bail_if_db_exists {
    psql -d ${STASH_DB} -c '' >/dev/null 2>&1
    if [ $? = 0 ]; then
        bail "Cannot restore over existing database ${STASH_DB}. Try dropdb ${STASH_DB} first."
    fi
}

function stash_restore_db {
    pg_restore -C -Fd -j 5 ${STASH_RESTORE_DB} | psql -q
    if [ $? != 0 ]; then
        bail "Unable to restore ${STASH_RESTORE_DB} to ${STASH_DB}"
    fi
    info "Performed restore of ${STASH_RESTORE_DB} to DB ${STASH_DB}"
}

An alternative stash.diy-backup.mssql.sh script is included in the downloadable scripts bundle.

  stash.diy-backup.rsync.sh
stash.diy-backup.rsync.sh
#!/bin/bash

check_command "rsync"
function stash_perform_rsync {
    mkdir -p ${STASH_BACKUP_HOME}
    rsync -avh --delete --delete-excluded --exclude=/caches/ --exclude=/data/db.* --exclude=/export/ --exclude=/log/ --exclude=/plugins/.*/ --exclude=/tmp --exclude=/.lock ${STASH_HOME} ${STASH_BACKUP_HOME}
    if [ $? != 0 ]; then
        bail "Unable to rsynch from ${STASH_HOME} to ${STASH_BACKUP_HOME}"
    fi
}

function stash_prepare_home {
    stash_perform_rsync
    info "Prepared backup of ${STASH_HOME} to ${STASH_BACKUP_HOME}"
}

function stash_backup_home {
    stash_perform_rsync
    info "Performed backup of ${STASH_HOME} to ${STASH_BACKUP_HOME}"
}

function stash_restore_home {
    cp -rf ${STASH_RESTORE_HOME}/`basename ${STASH_HOME}` `dirname ${STASH_HOME}`
    info "Performed restore of ${STASH_RESTORE_ROOT} to ${STASH_HOME}"
}
 
  stash.diy-backup.tar.sh
stash.diy-backup.tar.sh
#!/bin/bash

check_command "tar"
function stash_backup_archive {
    mkdir -p ${STASH_BACKUP_ARCHIVE_ROOT}
    STASH_BACKUP_ARCHIVE_NAME=`perl -we 'use Time::Piece; my $sydTime = localtime; print "stash-", $sydTime->strftime("%Y%m%d-%H%M%S-"), substr($sydTime->epoch, -3), ".tar.gz"'`
    tar -czf ${STASH_BACKUP_ARCHIVE_ROOT}/${STASH_BACKUP_ARCHIVE_NAME} -C ${STASH_BACKUP_ROOT} .
    
	info "Archived ${STASH_BACKUP_ROOT} into ${STASH_BACKUP_ARCHIVE_ROOT}/${STASH_BACKUP_ARCHIVE_NAME}"
}
 
function stash_restore_archive {
    if [ -f ${STASH_BACKUP_ARCHIVE_NAME} ]; then
        STASH_BACKUP_ARCHIVE_NAME=${STASH_BACKUP_ARCHIVE_NAME}
    else
        STASH_BACKUP_ARCHIVE_NAME=${STASH_BACKUP_ARCHIVE_ROOT}/${STASH_BACKUP_ARCHIVE_NAME}
    fi
    tar -xzf ${STASH_BACKUP_ARCHIVE_NAME} -C ${STASH_RESTORE_ROOT}
    info "Extracted ${STASH_BACKUP_ARCHIVE_ROOT}/${STASH_BACKUP_ARCHIVE_NAME} into ${STASH_RESTORE_ROOT}"
}
  stash.diy-restore.sh
stash.diy-restore.sh
#!/bin/bash
 
SCRIPT_DIR=$(dirname $0)
# Contains all variables used by the other scripts.
source ${SCRIPT_DIR}/stash.diy-backup.vars.sh
# Contains util functions (bail, info, print).
source ${SCRIPT_DIR}/stash.diy-backup.utils.sh
 
# The following scripts contain functions which are dependant on the configuration of this Stash instance.
# Generally every each of them exports certain functions, which can be implemented in different ways.
 
# Exports the following functions:
# stash_restore_db - for restoring the Stash DB.
source ${SCRIPT_DIR}/stash.diy-backup.postgresql.sh
 
# Exports the following functions:
# stash_restore_home - for restoring the filesystem backup.
source ${SCRIPT_DIR}/stash.diy-backup.rsync.sh
 
# Exports the following functions:
# stash_restore_archive - for un-archiving the archive folder.
source ${SCRIPT_DIR}/stash.diy-backup.tar.sh
 
##########################################################
# The actual restore process. It has the following steps:
 
function available_backups {
    echo "Available backups:"
    ls ${STASH_BACKUP_ARCHIVE_ROOT}
}

if [ $# -lt 1 ]; then
    echo "Usage: $0 <backup-file-name>.tar.gz"
    if [ ! -d ${STASH_BACKUP_ARCHIVE_ROOT} ]; then
        error "${STASH_BACKUP_ARCHIVE_ROOT} does not exist!"
    else
        available_backups
    fi
    exit 99
fi
STASH_BACKUP_ARCHIVE_NAME=$1
if [ ! -f ${STASH_BACKUP_ARCHIVE_ROOT}/${STASH_BACKUP_ARCHIVE_NAME} ]; then
        error "${STASH_BACKUP_ARCHIVE_ROOT}/${STASH_BACKUP_ARCHIVE_NAME} does not exist!"
        available_backups
        exit 99
fi
stash_bail_if_db_exists
if [ -e ${STASH_HOME} ]; then
        bail "Cannot restore over existing contents of ${STASH_HOME}. Please rename or delete this first."
fi
STASH_RESTORE_ROOT=`mktemp -d /tmp/stash.diy-restore.XXXXXX`
STASH_RESTORE_DB=${STASH_RESTORE_ROOT}/stash-db
STASH_RESTORE_HOME=${STASH_RESTORE_ROOT}/stash-home

# Extract the archive for this backup.
stash_restore_archive
 
# Restore the database.
stash_restore_db
 
# Restore the filesystem.
stash_restore_home
 
##########################################################

Running the Bash script

Once you have downloaded the Bash scripts, you need to customize two files:

  • stash.diy-backup.vars.sh 
  • stash.diy-backup.sh

for your setup.

For example, if your Stash server is called stash.example.com, uses port 7990, and has its home directory in /stash-home, here's how you might configure  stash.diy-backup.vars.sh:

Example usage:

#!/bin/bash

# Used by the scripts for verbose logging. If not true only errors will be shown.
STASH_VERBOSE_BACKUP=TRUE

# The base url used to access this Stash instance.
STASH_URL= http://stash.example.com:7990

# The username and password for the account used to make backups (and has permission for this).
STASH_BACKUP_USER= admin
STASH_BACKUP_PASS= admin

# The name of the database used by this instance.
STASH_DB= stash

# The path to Stash home folder (with trailing /).
STASH_HOME= /stash-home

# The path to the working folder for the backup.
STASH_BACKUP_ROOT= /stash-backup
STASH_BACKUP_DB=${STASH_BACKUP_ROOT}/stash-db/
STASH_BACKUP_HOME=${STASH_BACKUP_ROOT}/stash-home/

 

# The path to where the backup archives are stored.
STASH_BACKUP_ARCHIVE_ROOT= /stash-backup-archives

The supplied  stash.diy-backup.vars.sh is written to use PostgreSQL, rsync, and tar by default.  But if you want to use different tools, you can also customize the top section of this file to use different tools as follows:

Example usage:

# The following scripts contain functions which are dependant on the configuration of this Stash instance.
# Generally, each of them exports certain functions, which can be implemented in different ways.

 

# Exports the following functions:
# stash_prepare_db - for making a backup of the DB if differential backups are possible. Can be empty.
# stash_backup_db - for making a backup of the Stash DB.

source ${SCRIPT_DIR}/ stash.diy-backup.postgresql.sh

# Exports the following functions: # stash_prepare_home - for preparing the filesystem for the backup. # stash_backup_home - for making the actual filesystem backup.

source ${SCRIPT_DIR}/ stash.diy-backup.rsync.sh

# Exports the following functions:
# stash_backup_archive - for archiving the backup folder and moving the archive to the archive folder.

source ${SCRIPT_DIR}/ stash.diy-backup.tar.sh

You also need to create two directories for DIY Backup to work:

  1. ${STASH_BACKUP_ROOT} is a working directory (/stash-backup in our example) where copies of Stash's home directory and database dump are built during the DIY Backup process. 
  2. ${STASH_BACKUP_ARCHIVE_ROOT} is the directory (/stash-backup-archives in our example) where the final backup archives are saved. 

The Bash scripts may be run on any host, provided it has:

  • read/write access to the above ${STASH_BACKUP_ROOT} and ${STASH_BACKUP_ARCHIVE_ROOT} directories,
  • read access to the ${STASH_HOME} directory,
  • read access to the database, and
  • network access to run curl commands on the Stash server. 

This is true regardless of whether you have an instance of Stash Server or Stash Data Center.  It doesn't matter whether the filesystem access is direct or over NFS, or  whether the network access is direct to a Stash node or to a load balancer / reverse proxy. 

Once your stash.diy-backup.vars.sh is correctly configured, run the backup in a terminal window:

$ ./stash.diy-backup.sh 

 

The first time you run the backup, rsync will do most of the work since the /stash-backup working directory is initially empty. This is normal. Fortunately, this script performs one rsync before locking Stash, followed by a second rsync while Stash is locked. This minimizes downtime. 

On second and subsequent backup runs,  /stash-backup is already populated so the backup process should be faster.  The output you can expect to see looks something like this:

 

Restoring a DIY Backup

When restoring Stash, you must run the stash.diy-restore.sh script on the machine that Stash should be restored to. In order to ensure accidental restores do not delete existing data, you should never restore into an existing home directory.

The new database should be configured following the instructions in Connecting Stash to an external database and its sub-page that corresponds to your database type. 

To see the available backups in your ${STASH_BACKUP_ARCHIVE_ROOT} directory, just type:

$ ./stash.diy-restore.sh

 

You should see output similar to this:

To restore a backup, run stash.diy-restore.sh with the file name as the argument:

$ ./stash.diy-restore.sh stash-20140320-092743-063.tar.gz

 

You should see output like this:

Cancelling the backup

You can cancel the running backup operation if necessary.

To cancel the backup:

  1. Copy the cancel token echoed in the terminal (or the Command Prompt on Windows):

  2. Go to the Stash interface in your browser. Stash will display this screen:

  3. Click Cancel backup, and enter the cancel token:

  4. Click Cancel backup.

Advanced - writing your own DIY Backup using the REST APIs

This section is optional and provides background information about how you might use the Stash REST APIs if you need to rewrite the DIY Backup scripts described above in your preferred language or to customize them heavily.

Note that this discussion shows curl commands in Bash, however you can use any language.

The following steps are involved:

Preparation

Before you lock Stash you can perform any preparation you like. It makes sense to perform as much processing as possible before you lock Stash, to minimize downtime later. For example, you could perform an rsync:

rsync -avh --delete --delete-excluded --exclude=/caches/ --exclude=/data/db.* --exclude=/export/ --exclude=/log/ --exclude=/plugins/.*/ --exclude=/tmp --exclude=/.lock ${STASH_HOME} ${STASH_BACKUP_HOME}

Lock the Stash instance

The next step in making a backup of a Stash instance is to lock the instance for maintenance. This can be done using a POST request to the /mvc/maintenance/lock REST point (where STASH_URL points to the Stash instance, STASH_BACKUP_USER is a Stash user with backup permissions, and STASH_BACKUP_PASS is this user's password).

REQUEST
curl -s \
	 -u ${STASH_BACKUP_USER}:${STASH_BACKUP_PASS} \
	 -X POST \
	 -H "Content-type: application/json" \
     "${STASH_URL}/mvc/maintenance/lock"
RESPONSE
{
	"unlockToken":"0476adeb6cde3a41aa0cc19fb394779191f5d306",
	"owner": {
		"displayName":"admin",
		"name":"admin"
	}
}

If successful, the Stash instance will respond with a 202 and will return a response JSON similar to the one above. The  unlockToken  should be used in all subsequent requests where  $STASH_LOCK_TOKEN  is required. This token can also be used to manually unlock this Stash instance.

Start the backup process

Next, all connections to both the database and the filesystem must be drained and latched. Your code must handle backing up of both the filesystem and the database.

At this point, you should make a POST request to /mvc/admin/backups. Notice that the curl call includes the ?external=true 
parameter (requires Stash 2.12+):

REQUEST
curl -s \
	 -u ${STASH_BACKUP_USER}:${STASH_BACKUP_PASS} \
	 -X POST \
	 -H "X-Atlassian-Maintenance-Token: ${STASH_LOCK_TOKEN}" \
	 -H "Accept: application/json" \
	 -H "Content-type: application/json" \
	 "${STASH_URL}/mvc/admin/backups?external=true"
RESPONSE
{
	"id":"d2e15c3c2da282b0990e8efb30b4bffbcbf09e04",
	"progress": {
		"message":"Closing connections to the current database",
		"percentage":5
	},
	"state":"RUNNING",
	"type":"BACKUP",
	"cancelToken":"d2e15c3c2da282b0990e8efb30b4bffbcbf09e04"
}

If successful the Stash instance will respond with 202 and a response JSON similar to the one above will be returned. The  cancelToken  can be used to manually cancel the back up process.


Wait for the Stash instance to complete preparation

Part of the back up process includes draining and latching the connections to the database and the filesystem. Before continuing with the back up we have to wait for the Stash instance to report that this has been done. To get details on the current status we make a GET request to the /mvc/maintenance REST point.

REQUEST
curl -s \
	 -u ${STASH_BACKUP_USER}:${STASH_BACKUP_PASS} \
	 -X GET \
	 -H "X-Atlassian-Maintenance-Token: ${STASH_LOCK_TOKEN}" \
	 -H "Accept: application/json" \
	 -H "Content-type: application/json" \
	 "${STASH_URL}/mvc/maintenance" 
RESPONSE
 {
	"task":{
		"id":"0bb6b2ed52a6a12322e515e88c5d515d6b6fa95e",
		"progress":{
			"message":"Backing up Stash home",
			"percentage":10
		},
		"state":"RUNNING",
		"type":"BACKUP"
	},
	"db-state":"DRAINED",
	"scm-state":"DRAINED"
}

This causes the Stash instance to report its current state. We have to wait for both  db-state  and  scm-state  to have a status of  DRAINED  before continuing with the back up.  

Perform the actual backup

At this point we are ready to create the actual backup of the Stash filesystem. For example, you could use rsync again:

rsync -avh --delete --delete-excluded --exclude=/caches/ --exclude=/data/db.* --exclude=/export/ --exclude=/log/ --exclude=/plugins/.*/ --exclude=/tmp --exclude=/.lock ${STASH_HOME} ${STASH_BACKUP_HOME}

The rsync options shown here are for example only, but indicate how you can include only the required files in the backup process and exclude others. Consult the documentation for rsync, or the tool of your choice, for a more detailed description. 

When creating the database backup you could use your vendor-specific database backup tool, for example pg_dump if you use PostgreSQL:

pg_dump -Fd ${STASH_DB} -j 5 --no-synchronized-snapshots -f ${STASH_BACKUP_DB}

 

While performing these operations, good practice is to update the Stash instance with progress on the backup so that it's visible in the UI. This can be done by issuing a POST request to /mvc/admin/backups/progress/client with the token and the percentage completed as parameters:

REQUEST
curl -s \
	 -u ${STASH_BACKUP_USER}:${STASH_BACKUP_PASS} \
	 -X POST \
	 -H "Accept: application/json" \
	 -H "Content-type: application/json" \
	 "${STASH_URL}/mvc/admin/backups/progress/client?token=${STASH_LOCK_TOKEN}&percentage=${STASH_BACKUP_PERCENTAGE}"

The Stash instance will respond to this request with an empty 202 if everything is OK. 

When displaying progress to users, Stash divides the 100 percent progress into 90 percent user DIY Backup, and 10 percent Stash preparation.  This means, for example, if your script sends percentage=0, Stash may display up to 10 percent progress for its own share of the backup work. 

Inform the Stash instance that the backup is complete

Once we've finished the backup process we must report to the Stash instance that progress has reached 100 percent. This is done using a similar request to the progress request. We issue a POST request to /mvc/admin/backups/progress/client with the token and 100 as the percentage:

REQUEST
curl -s \
	 -u ${STASH_BACKUP_USER}:${STASH_BACKUP_PASS} \
	 -X POST \
	 -H "Accept: application/json" \
	 -H "Content-type: application/json" \
	 "${STASH_URL}/mvc/admin/backups/progress/client?token=${STASH_LOCK_TOKEN}&percentage=100"

The Stash will respond with an empty 202  if everything is OK. The back up process is considered completed once the percentage is 100. This will unlatch the database and the filesystem for this Stash instance.  

Unlock the Stash instance 

The final step we need to do in the back up process is to unlock the Stash instance. This is done with a DELETE request to the /mvc/maintenance/lock REST point:

REQUEST
curl -s \
	 -u ${STASH_BACKUP_USER}:${STASH_BACKUP_PASS} \
	 -X DELETE \
	 -H "Accept: application/json" \
	 -H "Content-type: application/json" \
	 "${STASH_URL}/mvc/maintenance/lock?token=${STASH_LOCK_TOKEN}"

The Stash instance will respond to this request with an empty 202 if everything is OK, and will unlock access.

Redirection notice

This page will redirect to /display/BitbucketServer/Using+Bitbucket+Server+DIY+Backup .

Stash DIY Backup was introduced in Stash 2.12 as an alternative strategy to using the Stash Backup Client. It allows you to:

  • significantly reduce the downtime needed to create a consistent backup,
  • use the vendor-specific database backup tool appropriate to your back end database, for example:
    • pg_dump if your back end database is PostgreSQL, or
    • sqlcmd with an appropriate command for differential backup, if your back end database is MS SQL Server,

  • use the optimal file system backup tool for your Stash home directory, for example:
    • an LVM snapshot logical volume if your Stash home directory uses LVM,
    • a SAN-based backup if your Stash home directory uses a Storage Area Network, or
    • rsync, if available.
  • take backups of Stash Data Center instances without having to bring nodes down manually.

The key to reducing Stash downtime to a minimum is the use of optimal, vendor-specific database and file system backup tools. Such tools are generally able to take snapshots (though sometimes in a vendor-specific format) in much less time than the generic, vendor-neutral format used by the Stash Backup Client.

Stash DIY Backup does require you to write some code in a language of your choice to perform the required backup steps, using the REST API available for Stash 2.12.

DIY Backup supports Windows and Linux platforms, and Stash versions 2.12 and higher. DIY Backup supports both Stash Server and Stash Data Center instances equally - any DIY Backup solution that works on one should work on the other without modification.

For information about other backup strategies for Stash, see Data recovery and backups. That page also discusses the tight coupling between the Stash file system on disk and the database that Stash uses.

Please note that the examples on this page are provided as guidance for developing a DIY Backup solution. As such, the third-party tools described are for example only – you will need to choose the tools that are appropriate to your own specific installation of Stash.

Consult the vendor documentation for the third-party tools you choose – unfortunately, Atlassian can not provide support for those tools.


On this page:

 

Download the worked example scripts from Bitbucket:

This page:

  • Describes a complete DIY Backup solution for a PostgreSQL database and local filesystem, using bash shell scripts.
  • Provides background information about how the Stash REST API can be used for DIY Backups.

You can use this solution directly if your Stash instance has the same or similar configuration, or use this as a starting point to develop your own DIY Backup solution tailored to your hardware configuration. 

How it works

When you use DIY Backup instead of the Stash Backup Client, you have complete control over the backup steps, and can implement any custom processes you like in the language of your choice.  For example, you can use your database's incremental or fast snapshot tools and/or your file server's specific tools as part of a DIY Backup. 

The DIY Backup works in a similar way to the Stash Backup Client and does the following:

  1. Prepare the instance for backup. This happens before Stash is locked, so we want to do as much processing as possible here in order to minimize downtime later.  For example, we can take an initial snapshot using incremental database and filesystem utilities.  These do not have to be 100% consistent as Stash is still running and modifying the database and filesystem.  But taking the initial snapshot now may reduce the amount of work done later (while Stash is locked), especially if the amount of data modified between backups is large.  The steps include:
    • Taking an initial backup of the database (if it supports progressive/differential backups).
    • Doing an initial rsync of the home folder to the backup folder.
  2. Tell Stash to initiate the backup. Stash will:
    • Lock the instance.
    • Drain and latch the connections to the database and the filesystem.
    • Wait for the drain/latch step to complete.
  3. Once the Stash instance is ready for backup we can start with the actual DIY Backup. This will include steps to:
    • Make a fully consistent backup of the database, using pg_dump.
    • Make a fully consistent backup of the filesystem, using rsync.
  4. Notify the Stash instance once the backup process finishes and unlock it.
  5. Archive all files created during the backup into one big archive.

A user will get an error message if they try to access the Stash web interface, or use the Stash hosting services, when Stash is in maintenance mode.

As an indication of the unavailability time that can be expected, in Atlassian's internal use we have seen downtimes for Stash of 7–8 minutes with repositories totalling 6 GB in size when using the Stash Backup Client. For comparison, using Stash DIY Backup for the same repositories typically results in a downtime of less than a minute. 

What is backed up

DIY Backup backs up all the same data as the Stash Backup Client:

  • the database Stash is connected to (either the internal or external DB)
  • managed Git repositories
  • the Stash logs
  • installed plugins and their data

DIY Backups using Bash scripts

This section presents a complete DIY Backup solution that uses the following tools:

  • bash - for scripting
  • jq - an open source command line JSON processor for parsing the REST responses from Stash
  • pg_dump (or sqlcmd) - for backing up a PostgreSQL database
  • rsync - for backing up the filesystem
  • tar - for making a backup archive

This approach (with small modifications) can be used for running DIY Backups on:

  • Linux and Unix
  • OSX
  • Windows with cygwin (note that cygwin Git is not supported by Stash).

Download the scripts from Bitbucket.

The scripts below are for example only and are not kept in sync with the scripts found in the Bitbucket repository.

Bash scripts

These scripts implement one particular solution for performing a DIY Backup: 

 

stash.diy-backup.sh
#!/bin/bash
 
##########################################################
# Configure the settings in this section to suit your Stash installation.

SCRIPT_DIR=$(dirname $0)
# Contains all variables used by the other scripts.
source ${SCRIPT_DIR}/stash.diy-backup.vars.sh
# Contains util functions (bail, info, print).
source ${SCRIPT_DIR}/stash.diy-backup.utils.sh
# Contains functions that perform lock/unlock and backup of a Stash instance.
source ${SCRIPT_DIR}/stash.diy-backup.common.sh

# The following scripts contain functions which are dependant on the configuration of this Stash instance.
# Generally every each of them exports certain functions, which can be implemented in different ways.

# Exports the following functions:
#     stash_prepare_db     - for making a backup of the DB if differential backups a possible. Can be empty.
#     stash_backup_db      - for making a backup of the Stash DB.
source ${SCRIPT_DIR}/stash.diy-backup.postgresql.sh

# Exports the following functions:
#     stash_prepare_home   - for preparing the filesystem for the backup.
#     stash_backup_home    - for making the actual filesystem backup.
source ${SCRIPT_DIR}/stash.diy-backup.rsync.sh

# Exports the following functions:
#     stash_backup_archive - for archiving the backup folder and puting the archive in archive folder.
source ${SCRIPT_DIR}/stash.diy-backup.tar.sh

##########################################################
# This section does NOT need any configuration.
# The actual back up process. It has the following steps:

# Prepare the database and the filesystem for taking a backup.
stash_prepare_db
stash_prepare_home

# Locking the Stash instance, starting an external backup and waiting for instance readiness.
stash_lock
stash_backup_start
stash_backup_wait

# Backing up the database and reporting 50% progress.
stash_backup_db
stash_backup_progress 50

# Backing up the filesystem and reporting 100% progress.
stash_backup_home
stash_backup_progress 100

# Unlocking the Stash instance.
stash_unlock

# Making an archive for this backup.
stash_backup_archive
##########################################################


The stash.diy-backup.sh script above references the following scripts:

 

  stash.diy-backup.vars.sh
stash.diy-backup.vars.sh
#!/bin/bash

# Used by the scripts for verbose logging. If not true only errors will be shown.
STASH_VERBOSE_BACKUP=TRUE

# The base url used to access this Stash instance.
STASH_URL=

# The username and password for the user used to make backups (and have this permission).
STASH_BACKUP_USER=
STASH_BACKUP_PASS=

# The name of the database used by this instance.
STASH_DB=stash

# The path to Stash home folder (with trailing /).
STASH_HOME=

# The path to working folder for the backup.
STASH_BACKUP_ROOT=
STASH_BACKUP_DB=${STASH_BACKUP_ROOT}/stash-db/
STASH_BACKUP_HOME=${STASH_BACKUP_ROOT}/stash-home/

# The path to where the backup archives are stored.
STASH_BACKUP_ARCHIVE_ROOT=
  stash.diy-backup.utils.sh
stash.diy-backup.utils.sh
#!/bin/bash

function error {
    echo "[${STASH_URL}] ERROR:" $*
}

function bail {
    error $*
    exit 99
}

function info {
    if [ "${STASH_VERBOSE_BACKUP}" == "TRUE" ]; then
        echo "[${STASH_URL}]  INFO:" $*
    fi
}

function print {
    if [ "${STASH_VERBOSE_BACKUP}" == "TRUE" ]; then
        echo $*
    fi
}

 
function check_command {
    type -P $1 &>/dev/null || bail "Unable to find $1, please install it and run this script again"
}
  stash.diy-backup.common.sh
stash.diy-backup.common.sh
#!/bin/bash

check_command "curl"
check_command "jq"

STASH_HTTP_AUTH="-u ${STASH_BACKUP_USER}:${STASH_BACKUP_PASS}"

function stash_lock {
    STASH_LOCK_RESULT=`curl -s -f ${STASH_HTTP_AUTH} -X POST -H "Content-type: application/json" "${STASH_URL}/mvc/maintenance/lock"`
    if [ -z "${STASH_LOCK_RESULT}" ]; then
        bail "Locking this Stash instance failed"
    fi

    STASH_LOCK_TOKEN=`echo ${STASH_LOCK_RESULT} | jq -r ".unlockToken"`
    if [ -z "${STASH_LOCK_TOKEN}" ]; then
        bail "Unable to find lock token. Result was '$STASH_LOCK_RESULT'"
    fi

    info "locked with '$STASH_LOCK_TOKEN'"
}

function stash_backup_start {
    STASH_BACKUP_RESULT=`curl -s -f ${STASH_HTTP_AUTH} -X POST -H "X-Atlassian-Maintenance-Token: ${STASH_LOCK_TOKEN}" -H "Accept: application/json" -H "Content-type: application/json" "${STASH_URL}/mvc/admin/backups?external=true"`
    if [ -z "${STASH_BACKUP_RESULT}" ]; then
        bail "Entering backup mode failed"
    fi

    STASH_BACKUP_TOKEN=`echo ${STASH_BACKUP_RESULT} | jq -r ".cancelToken"`
    if [ -z "${STASH_BACKUP_TOKEN}" ]; then
        bail "Unable to find backup token. Result was '${STASH_BACKUP_RESULT}'"
    fi

    info "backup started with '${STASH_BACKUP_TOKEN}'"
}

function stash_backup_wait {
    STASH_PROGRESS_DB_STATE="AVAILABLE"
    STASH_PROGRESS_SCM_STATE="AVAILABLE"

    print -n "[${STASH_URL}] .INFO: Waiting for DRAINED state "
    while [ ${STASH_PROGRESS_DB_STATE} != "DRAINED" -a ${STASH_PROGRESS_SCM_STATE} != "DRAINED" ]; do
        print -n "."

        STASH_PROGRESS_RESULT=`curl -s -f ${STASH_HTTP_AUTH} -X GET -H "X-Atlassian-Maintenance-Token: ${STASH_LOCK_TOKEN}" -H "Accept: application/json" -H "Content-type: application/json" "${STASH_URL}/mvc/maintenance"`
        if [ -z "${STASH_PROGRESS_RESULT}" ]; then
            bail "[${STASH_URL}] ERROR: Unable to check for backup progress"
        fi

        STASH_PROGRESS_DB_STATE=`echo ${STASH_PROGRESS_RESULT} | jq -r '.["db-state"]'`
        STASH_PROGRESS_SCM_STATE=`echo ${STASH_PROGRESS_RESULT} | jq -r '.["scm-state"]'`
    done

    print "done"
    info "db state '${STASH_PROGRESS_DB_STATE}'"
    info "scm state '${STASH_PROGRESS_SCM_STATE}'"
}

function stash_backup_progress {
    STASH_REPORT_RESULT=`curl -s -f ${STASH_HTTP_AUTH} -X POST -H "Accept: application/json" -H "Content-type: application/json" "${STASH_URL}/mvc/admin/backups/progress/client?token=${STASH_LOCK_TOKEN}&percentage=$1"`
    if [ $? != 0 ]; then
        bail "Unable to update backup progress"
    fi

    info "Backup progress updated to $1"
}

function stash_unlock {
    STASH_UNLOCK_RESULT=`curl -s -f ${STASH_HTTP_AUTH} -X DELETE -H "Accept: application/json" -H "Content-type: application/json" "${STASH_URL}/mvc/maintenance/lock?token=${STASH_LOCK_TOKEN}"`
    if [ $? != 0 ]; then
        bail "Unable to unlock instance with lock ${STASH_LOCK_TOKEN}"
    fi

    info "Stash instance unlocked"
}
  stash.diy-backup.postgresql.sh
stash.diy-backup.postgresql.sh
#!/bin/bash

check_command "pg_dump"
check_command "psql"
check_command "pg_restore"

function stash_prepare_db {
    info "Prepared backup of DB ${STASH_DB} in ${STASH_BACKUP_DB}"
}

function stash_backup_db {
    rm -r ${STASH_BACKUP_DB}
    pg_dump -Fd ${STASH_DB} -j 5 --no-synchronized-snapshots -f ${STASH_BACKUP_DB}
    if [ $? != 0 ]; then
        bail "Unable to backup ${STASH_DB} to ${STASH_BACKUP_DB}"
    fi
    info "Performed backup of DB ${STASH_DB} in ${STASH_BACKUP_DB}"
}

function stash_bail_if_db_exists {
    psql -d ${STASH_DB} -c '' >/dev/null 2>&1
    if [ $? = 0 ]; then
        bail "Cannot restore over existing database ${STASH_DB}. Try dropdb ${STASH_DB} first."
    fi
}

function stash_restore_db {
    pg_restore -C -Fd -j 5 ${STASH_RESTORE_DB} | psql -q
    if [ $? != 0 ]; then
        bail "Unable to restore ${STASH_RESTORE_DB} to ${STASH_DB}"
    fi
    info "Performed restore of ${STASH_RESTORE_DB} to DB ${STASH_DB}"
}

An alternative stash.diy-backup.mssql.sh script is included in the downloadable scripts bundle.

  stash.diy-backup.rsync.sh
stash.diy-backup.rsync.sh
#!/bin/bash

check_command "rsync"
function stash_perform_rsync {
    mkdir -p ${STASH_BACKUP_HOME}
    rsync -avh --delete --delete-excluded --exclude=/caches/ --exclude=/data/db.* --exclude=/export/ --exclude=/log/ --exclude=/plugins/.*/ --exclude=/tmp --exclude=/.lock ${STASH_HOME} ${STASH_BACKUP_HOME}
    if [ $? != 0 ]; then
        bail "Unable to rsynch from ${STASH_HOME} to ${STASH_BACKUP_HOME}"
    fi
}

function stash_prepare_home {
    stash_perform_rsync
    info "Prepared backup of ${STASH_HOME} to ${STASH_BACKUP_HOME}"
}

function stash_backup_home {
    stash_perform_rsync
    info "Performed backup of ${STASH_HOME} to ${STASH_BACKUP_HOME}"
}

function stash_restore_home {
    cp -rf ${STASH_RESTORE_HOME}/`basename ${STASH_HOME}` `dirname ${STASH_HOME}`
    info "Performed restore of ${STASH_RESTORE_ROOT} to ${STASH_HOME}"
}
 
  stash.diy-backup.tar.sh
stash.diy-backup.tar.sh
#!/bin/bash

check_command "tar"
function stash_backup_archive {
    mkdir -p ${STASH_BACKUP_ARCHIVE_ROOT}
    STASH_BACKUP_ARCHIVE_NAME=`perl -we 'use Time::Piece; my $sydTime = localtime; print "stash-", $sydTime->strftime("%Y%m%d-%H%M%S-"), substr($sydTime->epoch, -3), ".tar.gz"'`
    tar -czf ${STASH_BACKUP_ARCHIVE_ROOT}/${STASH_BACKUP_ARCHIVE_NAME} -C ${STASH_BACKUP_ROOT} .
    
	info "Archived ${STASH_BACKUP_ROOT} into ${STASH_BACKUP_ARCHIVE_ROOT}/${STASH_BACKUP_ARCHIVE_NAME}"
}
 
function stash_restore_archive {
    if [ -f ${STASH_BACKUP_ARCHIVE_NAME} ]; then
        STASH_BACKUP_ARCHIVE_NAME=${STASH_BACKUP_ARCHIVE_NAME}
    else
        STASH_BACKUP_ARCHIVE_NAME=${STASH_BACKUP_ARCHIVE_ROOT}/${STASH_BACKUP_ARCHIVE_NAME}
    fi
    tar -xzf ${STASH_BACKUP_ARCHIVE_NAME} -C ${STASH_RESTORE_ROOT}
    info "Extracted ${STASH_BACKUP_ARCHIVE_ROOT}/${STASH_BACKUP_ARCHIVE_NAME} into ${STASH_RESTORE_ROOT}"
}
  stash.diy-restore.sh
stash.diy-restore.sh
#!/bin/bash
 
SCRIPT_DIR=$(dirname $0)
# Contains all variables used by the other scripts.
source ${SCRIPT_DIR}/stash.diy-backup.vars.sh
# Contains util functions (bail, info, print).
source ${SCRIPT_DIR}/stash.diy-backup.utils.sh
 
# The following scripts contain functions which are dependant on the configuration of this Stash instance.
# Generally every each of them exports certain functions, which can be implemented in different ways.
 
# Exports the following functions:
# stash_restore_db - for restoring the Stash DB.
source ${SCRIPT_DIR}/stash.diy-backup.postgresql.sh
 
# Exports the following functions:
# stash_restore_home - for restoring the filesystem backup.
source ${SCRIPT_DIR}/stash.diy-backup.rsync.sh
 
# Exports the following functions:
# stash_restore_archive - for un-archiving the archive folder.
source ${SCRIPT_DIR}/stash.diy-backup.tar.sh
 
##########################################################
# The actual restore process. It has the following steps:
 
function available_backups {
    echo "Available backups:"
    ls ${STASH_BACKUP_ARCHIVE_ROOT}
}

if [ $# -lt 1 ]; then
    echo "Usage: $0 <backup-file-name>.tar.gz"
    if [ ! -d ${STASH_BACKUP_ARCHIVE_ROOT} ]; then
        error "${STASH_BACKUP_ARCHIVE_ROOT} does not exist!"
    else
        available_backups
    fi
    exit 99
fi
STASH_BACKUP_ARCHIVE_NAME=$1
if [ ! -f ${STASH_BACKUP_ARCHIVE_ROOT}/${STASH_BACKUP_ARCHIVE_NAME} ]; then
        error "${STASH_BACKUP_ARCHIVE_ROOT}/${STASH_BACKUP_ARCHIVE_NAME} does not exist!"
        available_backups
        exit 99
fi
stash_bail_if_db_exists
if [ -e ${STASH_HOME} ]; then
        bail "Cannot restore over existing contents of ${STASH_HOME}. Please rename or delete this first."
fi
STASH_RESTORE_ROOT=`mktemp -d /tmp/stash.diy-restore.XXXXXX`
STASH_RESTORE_DB=${STASH_RESTORE_ROOT}/stash-db
STASH_RESTORE_HOME=${STASH_RESTORE_ROOT}/stash-home

# Extract the archive for this backup.
stash_restore_archive
 
# Restore the database.
stash_restore_db
 
# Restore the filesystem.
stash_restore_home
 
##########################################################

Running the Bash script

Once you have downloaded the Bash scripts, you need to customize two files:

  • stash.diy-backup.vars.sh 
  • stash.diy-backup.sh

for your setup.

For example, if your Stash server is called stash.example.com, uses port 7990, and has its home directory in /stash-home, here's how you might configure  stash.diy-backup.vars.sh:

Example usage:

#!/bin/bash

# Used by the scripts for verbose logging. If not true only errors will be shown.
STASH_VERBOSE_BACKUP=TRUE

# The base url used to access this Stash instance.
STASH_URL= http://stash.example.com:7990

# The username and password for the account used to make backups (and has permission for this).
STASH_BACKUP_USER= admin
STASH_BACKUP_PASS= admin

# The name of the database used by this instance.
STASH_DB= stash

# The path to Stash home folder (with trailing /).
STASH_HOME= /stash-home

# The path to the working folder for the backup.
STASH_BACKUP_ROOT= /stash-backup
STASH_BACKUP_DB=${STASH_BACKUP_ROOT}/stash-db/
STASH_BACKUP_HOME=${STASH_BACKUP_ROOT}/stash-home/

 

# The path to where the backup archives are stored.
STASH_BACKUP_ARCHIVE_ROOT= /stash-backup-archives

The supplied  stash.diy-backup.vars.sh is written to use PostgreSQL, rsync, and tar by default.  But if you want to use different tools, you can also customize the top section of this file to use different tools as follows:

Example usage:

# The following scripts contain functions which are dependant on the configuration of this Stash instance.
# Generally, each of them exports certain functions, which can be implemented in different ways.

 

# Exports the following functions:
# stash_prepare_db - for making a backup of the DB if differential backups are possible. Can be empty.
# stash_backup_db - for making a backup of the Stash DB.

source ${SCRIPT_DIR}/ stash.diy-backup.postgresql.sh

# Exports the following functions: # stash_prepare_home - for preparing the filesystem for the backup. # stash_backup_home - for making the actual filesystem backup.

source ${SCRIPT_DIR}/ stash.diy-backup.rsync.sh

# Exports the following functions:
# stash_backup_archive - for archiving the backup folder and moving the archive to the archive folder.

source ${SCRIPT_DIR}/ stash.diy-backup.tar.sh

You also need to create two directories for DIY Backup to work:

  1. ${STASH_BACKUP_ROOT} is a working directory (/stash-backup in our example) where copies of Stash's home directory and database dump are built during the DIY Backup process. 
  2. ${STASH_BACKUP_ARCHIVE_ROOT} is the directory (/stash-backup-archives in our example) where the final backup archives are saved. 

The Bash scripts may be run on any host, provided it has:

  • read/write access to the above ${STASH_BACKUP_ROOT} and ${STASH_BACKUP_ARCHIVE_ROOT} directories,
  • read access to the ${STASH_HOME} directory,
  • read access to the database, and
  • network access to run curl commands on the Stash server. 

This is true regardless of whether you have an instance of Stash Server or Stash Data Center.  It doesn't matter whether the filesystem access is direct or over NFS, or  whether the network access is direct to a Stash node or to a load balancer / reverse proxy. 

Once your stash.diy-backup.vars.sh is correctly configured, run the backup in a terminal window:

$ ./stash.diy-backup.sh 

 

The first time you run the backup, rsync will do most of the work since the /stash-backup working directory is initially empty. This is normal. Fortunately, this script performs one rsync before locking Stash, followed by a second rsync while Stash is locked. This minimizes downtime. 

On second and subsequent backup runs,  /stash-backup is already populated so the backup process should be faster.  The output you can expect to see looks something like this:

 

Restoring a DIY Backup

When restoring Stash, you must run the stash.diy-restore.sh script on the machine that Stash should be restored to. In order to ensure accidental restores do not delete existing data, you should never restore into an existing home directory.

The new database should be configured following the instructions in Connecting Stash to an external database and its sub-page that corresponds to your database type. 

To see the available backups in your ${STASH_BACKUP_ARCHIVE_ROOT} directory, just type:

$ ./stash.diy-restore.sh

 

You should see output similar to this:

To restore a backup, run stash.diy-restore.sh with the file name as the argument:

$ ./stash.diy-restore.sh stash-20140320-092743-063.tar.gz

 

You should see output like this:

Cancelling the backup

You can cancel the running backup operation if necessary.

To cancel the backup:

  1. Copy the cancel token echoed in the terminal (or the Command Prompt on Windows):

  2. Go to the Stash interface in your browser. Stash will display this screen:

  3. Click Cancel backup, and enter the cancel token:

  4. Click Cancel backup.

Advanced - writing your own DIY Backup using the REST APIs

This section is optional and provides background information about how you might use the Stash REST APIs if you need to rewrite the DIY Backup scripts described above in your preferred language or to customize them heavily.

Note that this discussion shows curl commands in Bash, however you can use any language.

The following steps are involved:

Preparation

Before you lock Stash you can perform any preparation you like. It makes sense to perform as much processing as possible before you lock Stash, to minimize downtime later. For example, you could perform an rsync:

rsync -avh --delete --delete-excluded --exclude=/caches/ --exclude=/data/db.* --exclude=/export/ --exclude=/log/ --exclude=/plugins/.*/ --exclude=/tmp --exclude=/.lock ${STASH_HOME} ${STASH_BACKUP_HOME}

Lock the Stash instance

The next step in making a backup of a Stash instance is to lock the instance for maintenance. This can be done using a POST request to the /mvc/maintenance/lock REST point (where STASH_URL points to the Stash instance, STASH_BACKUP_USER is a Stash user with backup permissions, and STASH_BACKUP_PASS is this user's password).

REQUEST
curl -s \
	 -u ${STASH_BACKUP_USER}:${STASH_BACKUP_PASS} \
	 -X POST \
	 -H "Content-type: application/json" \
     "${STASH_URL}/mvc/maintenance/lock"
RESPONSE
{
	"unlockToken":"0476adeb6cde3a41aa0cc19fb394779191f5d306",
	"owner": {
		"displayName":"admin",
		"name":"admin"
	}
}

If successful, the Stash instance will respond with a 202 and will return a response JSON similar to the one above. The  unlockToken  should be used in all subsequent requests where  $STASH_LOCK_TOKEN  is required. This token can also be used to manually unlock this Stash instance.

Start the backup process

Next, all connections to both the database and the filesystem must be drained and latched. Your code must handle backing up of both the filesystem and the database.

At this point, you should make a POST request to /mvc/admin/backups. Notice that the curl call includes the ?external=true 
parameter (requires Stash 2.12+):

REQUEST
curl -s \
	 -u ${STASH_BACKUP_USER}:${STASH_BACKUP_PASS} \
	 -X POST \
	 -H "X-Atlassian-Maintenance-Token: ${STASH_LOCK_TOKEN}" \
	 -H "Accept: application/json" \
	 -H "Content-type: application/json" \
	 "${STASH_URL}/mvc/admin/backups?external=true"
RESPONSE
{
	"id":"d2e15c3c2da282b0990e8efb30b4bffbcbf09e04",
	"progress": {
		"message":"Closing connections to the current database",
		"percentage":5
	},
	"state":"RUNNING",
	"type":"BACKUP",
	"cancelToken":"d2e15c3c2da282b0990e8efb30b4bffbcbf09e04"
}

If successful the Stash instance will respond with 202 and a response JSON similar to the one above will be returned. The  cancelToken  can be used to manually cancel the back up process.


Wait for the Stash instance to complete preparation

Part of the back up process includes draining and latching the connections to the database and the filesystem. Before continuing with the back up we have to wait for the Stash instance to report that this has been done. To get details on the current status we make a GET request to the /mvc/maintenance REST point.

REQUEST
curl -s \
	 -u ${STASH_BACKUP_USER}:${STASH_BACKUP_PASS} \
	 -X GET \
	 -H "X-Atlassian-Maintenance-Token: ${STASH_LOCK_TOKEN}" \
	 -H "Accept: application/json" \
	 -H "Content-type: application/json" \
	 "${STASH_URL}/mvc/maintenance" 
RESPONSE
 {
	"task":{
		"id":"0bb6b2ed52a6a12322e515e88c5d515d6b6fa95e",
		"progress":{
			"message":"Backing up Stash home",
			"percentage":10
		},
		"state":"RUNNING",
		"type":"BACKUP"
	},
	"db-state":"DRAINED",
	"scm-state":"DRAINED"
}

This causes the Stash instance to report its current state. We have to wait for both  db-state  and  scm-state  to have a status of  DRAINED  before continuing with the back up.  

Perform the actual backup

At this point we are ready to create the actual backup of the Stash filesystem. For example, you could use rsync again:

rsync -avh --delete --delete-excluded --exclude=/caches/ --exclude=/data/db.* --exclude=/export/ --exclude=/log/ --exclude=/plugins/.*/ --exclude=/tmp --exclude=/.lock ${STASH_HOME} ${STASH_BACKUP_HOME}

The rsync options shown here are for example only, but indicate how you can include only the required files in the backup process and exclude others. Consult the documentation for rsync, or the tool of your choice, for a more detailed description. 

When creating the database backup you could use your vendor-specific database backup tool, for example pg_dump if you use PostgreSQL:

pg_dump -Fd ${STASH_DB} -j 5 --no-synchronized-snapshots -f ${STASH_BACKUP_DB}

 

While performing these operations, good practice is to update the Stash instance with progress on the backup so that it's visible in the UI. This can be done by issuing a POST request to /mvc/admin/backups/progress/client with the token and the percentage completed as parameters:

REQUEST
curl -s \
	 -u ${STASH_BACKUP_USER}:${STASH_BACKUP_PASS} \
	 -X POST \
	 -H "Accept: application/json" \
	 -H "Content-type: application/json" \
	 "${STASH_URL}/mvc/admin/backups/progress/client?token=${STASH_LOCK_TOKEN}&percentage=${STASH_BACKUP_PERCENTAGE}"

The Stash instance will respond to this request with an empty 202 if everything is OK. 

When displaying progress to users, Stash divides the 100 percent progress into 90 percent user DIY Backup, and 10 percent Stash preparation.  This means, for example, if your script sends percentage=0, Stash may display up to 10 percent progress for its own share of the backup work. 

Inform the Stash instance that the backup is complete

Once we've finished the backup process we must report to the Stash instance that progress has reached 100 percent. This is done using a similar request to the progress request. We issue a POST request to /mvc/admin/backups/progress/client with the token and 100 as the percentage:

REQUEST
curl -s \
	 -u ${STASH_BACKUP_USER}:${STASH_BACKUP_PASS} \
	 -X POST \
	 -H "Accept: application/json" \
	 -H "Content-type: application/json" \
	 "${STASH_URL}/mvc/admin/backups/progress/client?token=${STASH_LOCK_TOKEN}&percentage=100"

The Stash will respond with an empty 202  if everything is OK. The back up process is considered completed once the percentage is 100. This will unlatch the database and the filesystem for this Stash instance.  

Unlock the Stash instance 

The final step we need to do in the back up process is to unlock the Stash instance. This is done with a DELETE request to the /mvc/maintenance/lock REST point:

REQUEST
curl -s \
	 -u ${STASH_BACKUP_USER}:${STASH_BACKUP_PASS} \
	 -X DELETE \
	 -H "Accept: application/json" \
	 -H "Content-type: application/json" \
	 "${STASH_URL}/mvc/maintenance/lock?token=${STASH_LOCK_TOKEN}"

The Stash instance will respond to this request with an empty 202 if everything is OK, and will unlock access.

Was this helpful?

Thanks for your feedback!

Why was this unhelpful?

Have a question about this article?

See questions about this article

Powered by Confluence and Scroll Viewport