Using Stash DIY Backup

This page:

• Describes a complete DIY Backup solution for a PostgreSQL database and local filesystem, using bash shell scripts.
• Provides background information about how the Stash REST API can be used for DIY Backups.

You can use this solution directly if your Stash instance has the same or similar configuration, or use this as a starting point to develop your own DIY Backup solution tailored to your hardware configuration. 

How it works

When you use DIY Backup instead of the Stash Backup Client, you have complete control over the backup steps, and can implement any custom processes you like in the language of your choice.  For example, you can use your database's incremental or fast snapshot tools and/or your file server's specific tools as part of a DIY Backup. 

The DIY Backup works in a similar way to the Stash Backup Client and does the following:

1. Prepare the instance for backup. This happens before Stash is locked, so we want to do as much processing as possible here in order to minimize downtime later.  For example, we can take an initial snapshot using incremental database and filesystem utilities.  These do not have to be 100% consistent as Stash is still running and modifying the database and filesystem.  But taking the initial snapshot now may reduce the amount of work done later (while Stash is locked), especially if the amount of data modified between backups is large.  The steps include:
   • Taking an initial backup of the database (if it supports progressive/differential backups).
   • Doing an initial rsync of the home folder to the backup folder.
2. Tell Stash to initiate the backup. Stash will:
   • Lock the instance.
   • Drain and latch the connections to the database and the filesystem.
   • Wait for the drain/latch step to complete.
3. Once the Stash instance is ready for backup we can start with the actual DIY Backup. This will include steps to:
   • Make a fully consistent backup of the database, using pg_dump.
   • Make a fully consistent backup of the filesystem, using rsync.
4. Notify the Stash instance once the backup process finishes and unlock it.
5. Archive all files created during the backup into one big archive.

A user will get an error message if they try to access the Stash web interface, or use the Stash hosting services, when Stash is in maintenance mode.

As an indication of the unavailability time that can be expected, in our testing and internal use we have seen downtimes for Stash of 7–8 minutes with repositories totalling 6 GB in size when using the Stash Backup Client. For comparison, using Stash DIY Backup for the same repositories typically results in a downtime of less than a minute. 

What is backed up

DIY Backup backs up all the same data as the Stash Backup Client:

• the database Stash is connected to (either the internal or external DB)
• managed Git repositories
• the Stash logs
• installed plugins and their data

DIY Backups using Bash scripts

This section presents a complete DIY Backup solution that uses the following tools:

• bash - for scripting
• jq - an open source command line JSON processor for parsing the REST responses from Stash
• pg_dump (or sqlcmd) - for backing up a PostgreSQL database
• rsync - for backing up the filesystem
• tar - for making a backup archive

This approach (with small modifications) can be used for running DIY Backups on:

• Linux and Unix
• OSX
• Windows with cygwin (note that cygwin Git is not supported by Stash).

Download the scripts from Bitbucket.

Bash scripts

These scripts implement one particular solution for performing a DIY Backup: 

#!/bin/bash
 
##########################################################
# Configure the settings in this section to suit your Stash installation.

SCRIPT_DIR=$(dirname $0)
# Contains all variables used by the other scripts.
source ${SCRIPT_DIR}/stash.diy-backup.vars.sh
# Contains util functions (bail, info, print).
source ${SCRIPT_DIR}/stash.diy-backup.utils.sh
# Contains functions that perform lock/unlock and backup of a Stash instance.
source ${SCRIPT_DIR}/stash.diy-backup.common.sh

# The following scripts contain functions which are dependant on the configuration of this Stash instance.
# Generally every each of them exports certain functions, which can be implemented in different ways.

# Exports the following functions:
#     stash_prepare_db     - for making a backup of the DB if differential backups a possible. Can be empty.
#     stash_backup_db      - for making a backup of the Stash DB.
source ${SCRIPT_DIR}/stash.diy-backup.postgresql.sh

# Exports the following functions:
#     stash_prepare_home   - for preparing the filesystem for the backup.
#     stash_backup_home    - for making the actual filesystem backup.
source ${SCRIPT_DIR}/stash.diy-backup.rsync.sh

# Exports the following functions:
#     stash_backup_archive - for archiving the backup folder and puting the archive in archive folder.
source ${SCRIPT_DIR}/stash.diy-backup.tar.sh

##########################################################
# This section does NOT need any configuration.
# The actual back up process. It has the following steps:

# Prepare the database and the filesystem for taking a backup.
stash_prepare_db
stash_prepare_home

# Locking the Stash instance, starting an external backup and waiting for instance readiness.
stash_lock
stash_backup_start
stash_backup_wait

# Backing up the database and reporting 50% progress.
stash_backup_db
stash_backup_progress 50

# Backing up the filesystem and reporting 100% progress.
stash_backup_home
stash_backup_progress 100

# Unlocking the Stash instance.
stash_unlock

# Making an archive for this backup.
stash_backup_archive
##########################################################

The stash.diy-backup.sh script above references the following scripts:

#!/bin/bash

# Used by the scripts for verbose logging. If not true only errors will be shown.
STASH_VERBOSE_BACKUP=TRUE

# The base url used to access this Stash instance.
STASH_URL=

# The username and password for the user used to make backups (and have this permission).
STASH_BACKUP_USER=
STASH_BACKUP_PASS=

# The name of the database used by this instance.
STASH_DB=stash

# The path to Stash home folder (with trailing /).
STASH_HOME=

# The path to working folder for the backup.
STASH_BACKUP_ROOT=
STASH_BACKUP_DB=${STASH_BACKUP_ROOT}/stash-db/
STASH_BACKUP_HOME=${STASH_BACKUP_ROOT}/stash-home/

# The path to where the backup archives are stored.
STASH_BACKUP_ARCHIVE_ROOT=

#!/bin/bash

function error {
    echo "[${STASH_URL}] ERROR:" $*
}

function bail {
    error $*
    exit 99
}

function info {
    if [ "${STASH_VERBOSE_BACKUP}" == "TRUE" ]; then
        echo "[${STASH_URL}]  INFO:" $*
    fi
}

function print {
    if [ "${STASH_VERBOSE_BACKUP}" == "TRUE" ]; then
        echo $*
    fi
}

 
function check_command {
    type -P $1 &>/dev/null || bail "Unable to find $1, please install it and run this script again"
}

#!/bin/bash

check_command "curl"
check_command "jq"

STASH_HTTP_AUTH="-u ${STASH_BACKUP_USER}:${STASH_BACKUP_PASS}"

function stash_lock {
    STASH_LOCK_RESULT=`curl -s -f ${STASH_HTTP_AUTH} -X POST -H "Content-type: application/json" "${STASH_URL}/mvc/maintenance/lock"`
    if [ -z "${STASH_LOCK_RESULT}" ]; then
        bail "Locking this Stash instance failed"
    fi

    STASH_LOCK_TOKEN=`echo ${STASH_LOCK_RESULT} | jq -r ".unlockToken"`
    if [ -z "${STASH_LOCK_TOKEN}" ]; then
        bail "Unable to find lock token. Result was '$STASH_LOCK_RESULT'"
    fi

    info "locked with '$STASH_LOCK_TOKEN'"
}

function stash_backup_start {
    STASH_BACKUP_RESULT=`curl -s -f ${STASH_HTTP_AUTH} -X POST -H "X-Atlassian-Maintenance-Token: ${STASH_LOCK_TOKEN}" -H "Accept: application/json" -H "Content-type: application/json" "${STASH_URL}/mvc/admin/backups?external=true"`
    if [ -z "${STASH_BACKUP_RESULT}" ]; then
        bail "Entering backup mode failed"
    fi

    STASH_BACKUP_TOKEN=`echo ${STASH_BACKUP_RESULT} | jq -r ".cancelToken"`
    if [ -z "${STASH_BACKUP_TOKEN}" ]; then
        bail "Unable to find backup token. Result was '${STASH_BACKUP_RESULT}'"
    fi

    info "backup started with '${STASH_BACKUP_TOKEN}'"
}

function stash_backup_wait {
    STASH_PROGRESS_DB_STATE="AVAILABLE"
    STASH_PROGRESS_SCM_STATE="AVAILABLE"

    print -n "[${STASH_URL}] .INFO: Waiting for DRAINED state "
    while [ ${STASH_PROGRESS_DB_STATE} != "DRAINED" -a ${STASH_PROGRESS_SCM_STATE} != "DRAINED" ]; do
        print -n "."

        STASH_PROGRESS_RESULT=`curl -s -f ${STASH_HTTP_AUTH} -X GET -H "X-Atlassian-Maintenance-Token: ${STASH_LOCK_TOKEN}" -H "Accept: application/json" -H "Content-type: application/json" "${STASH_URL}/mvc/maintenance"`
        if [ -z "${STASH_PROGRESS_RESULT}" ]; then
            bail "[${STASH_URL}] ERROR: Unable to check for backup progress"
        fi

        STASH_PROGRESS_DB_STATE=`echo ${STASH_PROGRESS_RESULT} | jq -r '.["db-state"]'`
        STASH_PROGRESS_SCM_STATE=`echo ${STASH_PROGRESS_RESULT} | jq -r '.["scm-state"]'`
    done

    print "done"
    info "db state '${STASH_PROGRESS_DB_STATE}'"
    info "scm state '${STASH_PROGRESS_SCM_STATE}'"
}

function stash_backup_progress {
    STASH_REPORT_RESULT=`curl -s -f ${STASH_HTTP_AUTH} -X POST -H "Accept: application/json" -H "Content-type: application/json" "${STASH_URL}/mvc/admin/backups/progress/client?token=${STASH_LOCK_TOKEN}&percentage=$1"`
    if [ $? != 0 ]; then
        bail "Unable to update backup progress"
    fi

    info "Backup progress updated to $1"
}

function stash_unlock {
    STASH_UNLOCK_RESULT=`curl -s -f ${STASH_HTTP_AUTH} -X DELETE -H "Accept: application/json" -H "Content-type: application/json" "${STASH_URL}/mvc/maintenance/lock?token=${STASH_LOCK_TOKEN}"`
    if [ $? != 0 ]; then
        bail "Unable to unlock instance with lock ${STASH_LOCK_TOKEN}"
    fi

    info "Stash instance unlocked"
}

#!/bin/bash

check_command "pg_dump"
check_command "psql"
check_command "pg_restore"

function stash_prepare_db {
    info "Prepared backup of DB ${STASH_DB} in ${STASH_BACKUP_DB}"
}

function stash_backup_db {
    rm -r ${STASH_BACKUP_DB}
    pg_dump -Fd ${STASH_DB} -j 5 --no-synchronized-snapshots -f ${STASH_BACKUP_DB}
    if [ $? != 0 ]; then
        bail "Unable to backup ${STASH_DB} to ${STASH_BACKUP_DB}"
    fi
    info "Performed backup of DB ${STASH_DB} in ${STASH_BACKUP_DB}"
}

function stash_bail_if_db_exists {
    psql -d ${STASH_DB} -c '' >/dev/null 2>&1
    if [ $? = 0 ]; then
        bail "Cannot restore over existing database ${STASH_DB}. Try dropdb ${STASH_DB} first."
    fi
}

function stash_restore_db {
    pg_restore -C -Fd -j 5 ${STASH_RESTORE_DB} | psql -q
    if [ $? != 0 ]; then
        bail "Unable to restore ${STASH_RESTORE_DB} to ${STASH_DB}"
    fi
    info "Performed restore of ${STASH_RESTORE_DB} to DB ${STASH_DB}"
}

#!/bin/bash

check_command "rsync"
function stash_perform_rsync {
    mkdir -p ${STASH_BACKUP_HOME}
    rsync -avh --delete --delete-excluded --exclude=/caches/ --exclude=/data/db.* --exclude=/export/ --exclude=/log/ --exclude=/plugins/.*/ --exclude=/tmp --exclude=/.lock ${STASH_HOME} ${STASH_BACKUP_HOME}
    if [ $? != 0 ]; then
        bail "Unable to rsynch from ${STASH_HOME} to ${STASH_BACKUP_HOME}"
    fi
}

function stash_prepare_home {
    stash_perform_rsync
    info "Prepared backup of ${STASH_HOME} to ${STASH_BACKUP_HOME}"
}

function stash_backup_home {
    stash_perform_rsync
    info "Performed backup of ${STASH_HOME} to ${STASH_BACKUP_HOME}"
}

function stash_restore_home {
    cp -rf ${STASH_RESTORE_HOME}/`basename ${STASH_HOME}` `dirname ${STASH_HOME}`
    info "Performed restore of ${STASH_RESTORE_ROOT} to ${STASH_HOME}"
}
 

#!/bin/bash

check_command "tar"
function stash_backup_archive {
    mkdir -p ${STASH_BACKUP_ARCHIVE_ROOT}
    STASH_BACKUP_ARCHIVE_NAME=`perl -we 'use Time::Piece; my $sydTime = localtime; print "stash-", $sydTime->strftime("%Y%m%d-%H%M%S-"), substr($sydTime->epoch, -3), ".tar.gz"'`
    tar -czf ${STASH_BACKUP_ARCHIVE_ROOT}/${STASH_BACKUP_ARCHIVE_NAME} -C ${STASH_BACKUP_ROOT} .
    
	info "Archived ${STASH_BACKUP_ROOT} into ${STASH_BACKUP_ARCHIVE_ROOT}/${STASH_BACKUP_ARCHIVE_NAME}"
}
 
function stash_restore_archive {
    if [ -f ${STASH_BACKUP_ARCHIVE_NAME} ]; then
        STASH_BACKUP_ARCHIVE_NAME=${STASH_BACKUP_ARCHIVE_NAME}
    else
        STASH_BACKUP_ARCHIVE_NAME=${STASH_BACKUP_ARCHIVE_ROOT}/${STASH_BACKUP_ARCHIVE_NAME}
    fi
    tar -xzf ${STASH_BACKUP_ARCHIVE_NAME} -C ${STASH_RESTORE_ROOT}
    info "Extracted ${STASH_BACKUP_ARCHIVE_ROOT}/${STASH_BACKUP_ARCHIVE_NAME} into ${STASH_RESTORE_ROOT}"
}

#!/bin/bash
 
SCRIPT_DIR=$(dirname $0)
# Contains all variables used by the other scripts.
source ${SCRIPT_DIR}/stash.diy-backup.vars.sh
# Contains util functions (bail, info, print).
source ${SCRIPT_DIR}/stash.diy-backup.utils.sh
 
# The following scripts contain functions which are dependant on the configuration of this Stash instance.
# Generally every each of them exports certain functions, which can be implemented in different ways.
 
# Exports the following functions:
# stash_restore_db - for restoring the Stash DB.
source ${SCRIPT_DIR}/stash.diy-backup.postgresql.sh
 
# Exports the following functions:
# stash_restore_home - for restoring the filesystem backup.
source ${SCRIPT_DIR}/stash.diy-backup.rsync.sh
 
# Exports the following functions:
# stash_restore_archive - for un-archiving the archive folder.
source ${SCRIPT_DIR}/stash.diy-backup.tar.sh
 
##########################################################
# The actual restore process. It has the following steps:
 
function available_backups {
    echo "Available backups:"
    ls ${STASH_BACKUP_ARCHIVE_ROOT}
}

if [ $# -lt 1 ]; then
    echo "Usage: $0 <backup-file-name>.tar.gz"
    if [ ! -d ${STASH_BACKUP_ARCHIVE_ROOT} ]; then
        error "${STASH_BACKUP_ARCHIVE_ROOT} does not exist!"
    else
        available_backups
    fi
    exit 99
fi
STASH_BACKUP_ARCHIVE_NAME=$1
if [ ! -f ${STASH_BACKUP_ARCHIVE_ROOT}/${STASH_BACKUP_ARCHIVE_NAME} ]; then
        error "${STASH_BACKUP_ARCHIVE_ROOT}/${STASH_BACKUP_ARCHIVE_NAME} does not exist!"
        available_backups
        exit 99
fi
stash_bail_if_db_exists
if [ -e ${STASH_HOME} ]; then
        bail "Cannot restore over existing contents of ${STASH_HOME}. Please rename or delete this first."
fi
STASH_RESTORE_ROOT=`mktemp -d /tmp/stash.diy-restore.XXXXXX`
STASH_RESTORE_DB=${STASH_RESTORE_ROOT}/stash-db
STASH_RESTORE_HOME=${STASH_RESTORE_ROOT}/stash-home

# Extract the archive for this backup.
stash_restore_archive
 
# Restore the database.
stash_restore_db
 
# Restore the filesystem.
stash_restore_home
 
##########################################################

Running the Bash script

Once you have downloaded the Bash scripts, you need to customize two files:

• stash.diy-backup.vars.sh 
• stash.diy-backup.sh

for your setup.

For example, if your Stash server is called stash.example.com, uses port 7990, and has its home directory in /stash-home, here's how you might configure  stash.diy-backup.vars.sh:

The supplied  stash.diy-backup.vars.sh is written to use PostgreSQL, rsync, and tar by default.  But if you want to use different tools, you can also customize the top section of this file to use different tools as follows:

You also need to create two directories for DIY Backup to work:

1. ${STASH_BACKUP_ROOT} is a working directory (/stash-backup in our example) where copies of Stash's home directory and database dump are built during the DIY Backup process. 
2. ${STASH_BACKUP_ARCHIVE_ROOT} is the directory (/stash-backup-archives in our example) where the final backup archives are saved. 

Once your stash.diy-backup.vars.sh is correctly configured, run the backup in a terminal window:

$ ./stash.diy-backup.sh 

The first time you run the backup, rsync will do most of the work since the /stash-backup working directory is initially empty. This is normal. Fortunately, this script performs one rsync before locking Stash, followed by a second rsync while Stash is locked. This minimizes downtime. 

On second and subsequent backup runs,  /stash-backup is already populated so the backup process should be faster.  The output you can expect to see looks something like this:

Restoring a DIY Backup

When restoring Stash, you must run the stash.diy-restore.sh script on the machine that Stash should be restored to. In order to ensure accidental restores do not delete existing data, you should never restore into an existing home directory.

The new database should be configured following the instructions in Connecting Stash to an external database and its sub-page that corresponds to your database type. 

To see the available backups in your ${STASH_BACKUP_ARCHIVE_ROOT} directory, just type:

$ ./stash.diy-restore.sh

You should see output similar to this:

To restore a backup, run stash.diy-restore.sh with the file name as the argument:

$ ./stash.diy-restore.sh stash-20140320-092743-063.tar.gz

You should see output like this:

Cancelling the backup

You can cancel the running backup operation if necessary.

To cancel the backup:

1. Copy the cancel token echoed in the terminal (or the Command Prompt on Windows):
   
   
2. Go to the Stash interface in your browser. Stash will display this screen:
   
   
3. Click Cancel backup, and enter the cancel token:
   
   
4. Click Cancel backup.

Advanced - writing your own DIY Backup using the REST APIs

Note that this discussion shows curl commands in Bash, however you can use any language.

The following steps are involved:

Preparation

Before you lock Stash you can perform any preparation you like. It makes sense to perform as much processing as possible before you lock Stash, to minimize downtime later. For example, you could perform an rsync:

rsync -avh --delete --delete-excluded --exclude=/caches/ --exclude=/data/db.* --exclude=/export/ --exclude=/log/ --exclude=/plugins/.*/ --exclude=/tmp --exclude=/.lock ${STASH_HOME} ${STASH_BACKUP_HOME}

Lock the Stash instance

The next step in making a backup of a Stash instance is to lock the instance for maintenance. This can be done using a POST request to the /mvc/maintenance/lock REST point (where STASH_URL points to the Stash instance, STASH_BACKUP_USER is a Stash user with backup permissions, and STASH_BACKUP_PASS is this user's password).

If successful, the Stash instance will respond with a 202 and will return a response JSON similar to the one above. The  unlockToken  should be used in all subsequent requests where  $STASH_LOCK_TOKEN  is required. This token can also be used to manually unlock this Stash instance.

Start the backup process

Next, all connections to both the database and the filesystem must be drained and latched. When backing up the database, we have two options:

• Use the facilities provided by the Stash server – your code handles backing up of the filesystem and Stash backs up the database.
• Do a completely custom back up of the database – your code must handle backing up of both the filesystem and the database.

Both options make a POST request to /mvc/admin/backups but if we require a custom database backup (the second option) we need to add the ?external=true parameter (requires Stash 2.12+):

If successful the Stash instance will respond with 202 and a response JSON similar to the one above will be returned. The  cancelToken  can be used to manually cancel the back up process.

${STASH_HOME}/export/backup-admin-20140325-223629-897Z.zip

Wait for the Stash instance to complete preparation

Part of the back up process, even if Stash is doing a database backup, includes draining and latching the connections to the database and the filesystem. Before continuing with the back up we have to wait for the Stash instance to report that this has been done. To get details on the current status we make a GET request to the /mvc/maintenance REST point.

This causes the Stash instance to report its current state. We have to wait for both  db-state  and  scm-state  to have a status of  DRAINED  before continuing with the back up.  

Perform the actual backup

At this point we are ready to create the actual backup of the Stash filesystem. For example, you could use rsync again:

rsync -avh --delete --delete-excluded --exclude=/caches/ --exclude=/data/db.* --exclude=/export/ --exclude=/log/ --exclude=/plugins/.*/ --exclude=/tmp --exclude=/.lock ${STASH_HOME} ${STASH_BACKUP_HOME}

The rsync options shown here are for example only, but indicate how you can include only the required files in the backup process and exclude others. Consult the documentation for rsync, or the tool of your choice, for a more detailed description. 

When creating the database backup you could use your vendor-specific database backup tool, for example pg_dump if you use PostgreSQL:

pg_dump -Fd ${STASH_DB} -j 5 --no-synchronized-snapshots -f ${STASH_BACKUP_DB}

While performing these operations, good practice is to update the Stash instance with progress on the backup so that it's visible in the UI. This can be done by issuing a POST request to /mvc/admin/backups/progress/client with the token and the percentage completed as parameters:

curl -s \
	 -u ${STASH_BACKUP_USER}:${STASH_BACKUP_PASS} \
	 -X POST \
	 -H "Accept: application/json" \
	 -H "Content-type: application/json" 
	 "${STASH_URL}/mvc/admin/backups/progress/client?token=${STASH_LOCK_TOKEN}&percentage=${STASH_BACKUP_PERCENTAGE}"

The Stash instance will respond to this request with an empty 202 if everything is OK. 

The amount of progress that Stash displays to users is calculated differently, depending on whether you used the external=true parameter with the /mvc/admin/backups REST endpoint described in the section above.

• If you did not specify external=true, Stash backs up the database and divides the 100 per cent progress into 50% user DIY Backup, and 50% Stash (preparation + database).
• If you did specify external=true, your code must handle backing up of both the filesystem and the database, and Stash divides the 100 per cent progress into 90% user DIY Backup, and 10% Stash preparation.

Inform the Stash instance that the backup is complete

Once we've finished the backup process we must report to the Stash instance that progress has reached 100 percent. This is done using a similar request to the progress request. We issue a POST request to /mvc/admin/backups/progress/client with the token and 100 as the percentage:

curl -s \
	 -u ${STASH_BACKUP_USER}:${STASH_BACKUP_PASS} \
	 -X POST \
	 -H "Accept: application/json" \
	 -H "Content-type: application/json" 
	 "${STASH_URL}/mvc/admin/backups/progress/client?token=${STASH_LOCK_TOKEN}&percentage=100"

The Stash will respond with an empty 202  if everything is OK. The back up process is considered completed once the percentage is 100. This will unlatch the database and the filesystem for this Stash instance.  

Unlock the Stash instance 

The final step we need to do in the back up process is to unlock the Stash instance. This is done with a DELETE request to the /mvc/maintenance/lock REST point:

curl -s \
	 -u ${STASH_BACKUP_USER}:${STASH_BACKUP_PASS} \
	 -X DELETE \
	 -H "Accept: application/json" \
	 -H "Content-type: application/json" \
	 "${STASH_URL}/mvc/maintenance/lock?token=${STASH_LOCK_TOKEN}"

The Stash instance will respond to this request with an empty 202 if everything is OK, and will unlock access.