Overview and requirements

It's worth getting a clear understanding of what you're aiming to achieve, before starting to provision your Stash Data Center.

A Stash Data Center instance consists of a cluster of dedicated machines connected like this:

The URL of the Stash Data Center instance will be the URL of the load balancer, so this is the machine that you will need to assign the name of your Stash instance in the DNS. 

The remaining machines (Stash cluster nodes, shared database, and shared file system) do not need to be publicly accessible to your users. 

1. Upgrade your existing production instance of Stash

2. Back up your production instance

Now, take a backup of your production Stash instance's database and home directory. For this you can:

• use the Stash backup client,
• use your own DIY backup solution, or
• just stop Stash and manually dump your database, and zip up the home directory. 

3. Provision your shared database

Set up your shared database server. Note that clustered databases are not yet supported.

See Connecting Stash to an external database for more information.

4. Provision your shared file system

Set up your shared file server.

See Stash Data Center FAQ for performance guidelines when using NFS.

Create a Stash user account (recommended name atlstash) on the shared file system server to own everything in the Stash shared home directory.  This user account must have the same UID on all cluster nodes and the shared file system server.  In a fresh Linux install the UID of a newly created account is typically 1001, but in general there is no guarantee that this UID will be free on every Linux system.  Choose a UID for atlstash that's free on all your cluster nodes and the shared file system server, and substitute this for 1001 in the following command:

sudo useradd -c "Atlassian STASH" -u 1001 atlstash

Then restore the backup you have taken in step 2 into the new shared database and shared home directory.

Only the shared directory in the Stash home directory needs to be restored into the shared home directory. The remaining directories (bin, caches, export, lib, log, plugins, and tmp) contain only caches and temporary files, and do not need to be restored. 

5. Provision your cluster nodes

1. We highly recommend provisioning cluster nodes using an automated configuration management tool such as Chef, Puppet, or Vagrant, or by spinning up identical virtual machine snapshots. 

2. On each cluster node, mount the shared home directory as ${STASH_HOME}/shared. For example, suppose your Stash home directory is/var/atlassian/application-data/stash, and your shared home directory is available as an NFS export called stash-san:/stash-shared. Add the following line to /etc/fstab on each cluster node:

   stash-san:/stash-shared /var/atlassian/application-data/stash/shared nfs nfsvers=3,lookupcache=pos,noatime,intr,rsize=32768,wsize=32768 0 0

   Only the ${STASH_HOME}/shared directory should be shared between cluster nodes. All other directories, including ${STASH_HOME}, should be node-local (that is, private to each node). Stash Data Center checks during startup that ${STASH_HOME} is node local and ${STASH_HOME}/shared is shared, and will fail to form a cluster if this is not true.

   Your shared file system must provide sufficient consistency for Stash and Git. Linux NFS clients require the lookupcache=pos mount option to be specified for proper consistency.

   NFSv4 may have issues in Linux kernels from about version 3.2 to 3.8 inclusive. The issues may cause very high load average, processes hanging in "uninterruptible sleep", and in some cases may require rebooting the machine. We recommend using NFSv3 unless you are 100% sure that you know what you're doing and your operating system is free from such issues. Linux NFS clients should use the nfsvers=3 mount option to force NFSv3.

   Then mount it:

   mkdir -p /var/atlassian/application-data/stash/shared sudo mount -a

3. Ensure all your cluster nodes have synchronized clocks and identical timezone configuration. For example, in RedHat Enterprise Linux or CentOS:

   sudo yum install ntp sudo service ntpd start sudo tzselect

   In Ubuntu Linux:

   sudo apt-get install ntp sudo service ntp start sudo dpkg-reconfigure tzdata

   For other operating systems, consult your system documentation. 
   
   

   The system clocks on your cluster nodes must remain reasonably synchronized (say, to within a few seconds or less). If your system clocks drift excessively or undergo abrupt "jumps" of minutes or more, then cluster nodes may log warnings, become slow, or in extreme cases become unresponsive and require restarting. You should run the NTP service on all your cluster nodes with identical configuration, and never manually tamper with the system clock on a cluster node while Stash Data Center is running.

4. Download the latest Stash Data Center distribution from https://www.atlassian.com/software/stash/download, and install Stash as normal on all the cluster nodes. See Getting started.

6. Start the first cluster node

Edit the file ${STASH_HOME}/shared/stash-config.properties, and add the following lines:

# Use multicast to discover cluster nodes (recommended).
hazelcast.network.multicast=true

# If your network does not support multicast, you may uncomment the following lines and substitute
# the IP addresses of some or all of your cluster nodes. (Not all of the cluster nodes have to be
# listed here but at least one of them has to be active when a new node joins.) 
#hazelcast.network.tcpip=true
#hazelcast.network.tcpip.members=192.168.0.1:5701,192.168.0.2:5701,192.168.0.3:5701

# The following should uniquely identify your cluster on the LAN. 
hazelcast.group.name=your-stash-cluster
hazelcast.group.password=your-stash-cluster

Using multicast to discover cluster nodes (hazelcast.network.multicast=true) is recommended, but requires all your cluster nodes to be accessible to each other via a multicast-enabled network. If your network does not support multicast then you can set hazelcast.network.multicast=false, hazelcast.network.tcpip=true, and hazelcast.network.tcpip.members to a comma-separated list of cluster nodes instead. Only enable one of hazelcast.network.tcpip or hazelcast.network.multicast, not both!

Choose a name for hazelcast.group.name and hazelcast.group.password that uniquely identifies the cluster on your LAN. If you have more than one cluster on the same LAN (for example, other Stash Data Center instances or other products based on similar technology such as Confluence Data Center) then you must assign each cluster a distinct name, to prevent them from attempting to join together into a "super cluster". 

Then start Stash. See Starting and stopping Stash.

Then go to http://<stash>:7990/admin/license, and install the Stash Data Center license you were issued. Restart Stash for the change to take effect. If you need a Stash Data Center license, please contact us!

7. Install and configure your load balancer 

You can use the load balancer of your choice, either hardware or software. Stash Data Center does not bundle a load balancer. 

Your load balancer must proxy three protocols:

If your load balancer supports health checks of the cluster nodes, configure it to perform a periodic HTTP GET of http:// <stash>:7990/status, where <stash> is the cluster node's name or IP address. This returns one of two HTTP status codes:

• 200 OK
• 500 Internal Server Error

If a cluster node does not return 200 OK within a reasonable amount of time, the load balancer should not direct any traffic to it. 

You should then be able to navigate to http://<load-balancer>/, where <load-balancer> is your load balancer's name or IP address. This should take you to your Stash front page. 

Example: HAProxy load balancer

If you don't have a particular preference or policy for load balancers, you can use HAProxy which is a popular Open Source software load balancer.

Here is an example haproxy.cfg configuration file (typically found in the location /etc/haproxy/haproxy.cfg).  This assumes:

• Your Stash cluster node is at address 192.168.0.1, listening on the default ports 7990 (HTTP) and 7999 (SSH). 
• You have a valid SSL certificate at /etc/cert.pem.

global
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

defaults
    log                     global
    option                  dontlognull
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000
    tune.ssl.default-dh-param 1024
    errorfile               408 /dev/null	# Workaround for Chrome 35-36 bug.  See http://blog.haproxy.com/2014/05/26/haproxy-and-http-errors-408-in-chrome/

frontend stash_http_frontend
    bind *:80
    bind *:443 ssl crt /etc/cert.pem ciphers RC4-SHA:AES128-SHA:AES256-SHA
    default_backend stash_http_backend

backend stash_http_backend
    mode http
    option httplog
    option httpchk GET /status
    option forwardfor
    option http-server-close
    appsession JSESSIONID len 52 timeout 1h
    balance roundrobin
    cookie JSESSIONID prefix
    stick-table type string len 52 size 5M expire 30m
    stick store-response set-cookie(JSESSIONID)
    stick on cookie(JSESSIONID)
    server stash01 192.168.0.1:7990 check inter 10000 rise 2 fall 5
    #server stash02 192.168.0.2:7990 check inter 10000 rise 2 fall 5
    # The following "backup" servers are just here to show the startup page when all nodes are starting up
    server backup01 192.168.0.1:7990 backup
    #server backup02 192.168.0.2:7990 backup
 
frontend stash_ssh_frontend
    bind *:7999
    default_backend stash_ssh_backend
    timeout client 15m
    maxconn 50

backend stash_ssh_backend
    mode tcp
    balance roundrobin
    server stash01 192.168.0.1:7999 check port 7999
    #server stash02 192.168.0.2:7999 check port 7999
    timeout server 15m
 
listen admin
    mode http
    bind *:8090
    stats enable
    stats uri /

Once you have configured the haproxy.cfg file, start the haproxy service.

sudo service haproxy start

You can also monitor the health of your cluster by navigating to HAProxy's statistics page at http://<load-balancer>:8090/. You should see a page similar to this:

8. Add a new Stash cluster node to the cluster

Go to a new cluster node, and start Stash. See Starting and stopping Stash.

Once Stash has started, go to http://<load-balancer>/admin/clustering. You should see a page similar to this:

Verify that the new node you have started up has successfully joined the cluster. If it does not, please check your network configuration and the ${STASH_HOME}/log/atlassian-stash.log files on all nodes. If you are unable to find a reason for the node failing to join successfully, please contact Atlassian Support.

9. Connect the new Stash cluster node to the load balancer

If you are using your own hardware or software load balancer, consult your vendor's documentation on how to add the new Stash cluster node to the load balancer.

If you are using HAProxy, just uncomment the lines

server stash02 192.168.0.2:7990 check inter 10000 rise 2 fall 5

server stash02 192.168.0.2:7999 check port 7999

in your haproxy.cfg file and restart haproxy:

sudo service haproxy restart

Verify that the new node is in the cluster and receiving requests by checking the logs on each node to ensure both are receiving traffic and also check that updates done on one node are visible on the other. 

10. Repeat steps 8 and 9 for each additional cluster node

11. Congratulations!

You have now set up a clustered instance of Stash Data Center!  We are very interested in hearing your feedback on this process – please contact us!

Please see Using Stash in the enterprise for information about using Stash in a production environment.