Set up a Bamboo Data Center warm standby
Bamboo Data Center allows you to run a cluster of Bamboo nodes in a warm standby configuration, providing higher availability. This guide walks you through the process of configuring a Data Center cluster on your infrastructure.
Not sure if a warm standby is right for you? Check out Running Bamboo Data Center in a cluster for a detailed overview.
Before you begin
Things you should know about when setting up your Data Center:
Supported platforms
See our supported platforms for information on the database, Java, and operating systems you'll be able to use. These requirements are the same for Server and Data Center deployments.
Component requirements
You can see a component diagram of a typical Bamboo Data Center instance and read about the detailed requirements of each component on the Clustering with Bamboo Data Center page
A Bamboo Data Center instance consists of a cluster of components, each running on a dedicated machine:
A cluster of Bamboo application nodes all running the same version of Bamboo Data Center web application. These can be virtual or physical machines, have synchronized clocks (for example, using NTP) and be configured with the identical timezone.
A load balancer that supports both HTTP mode (for web traffic) and TCP mode (for ActiveMQ traffic), and supports session affinity ("sticky sessions"). If the load balancer doesn’t support both modes, then configure two separate load balancers for each traffic type, e.g. Amazon ALB and Amazon ELB.
A supported external database shared and available to all cluster nodes.
A shared file system that is physically located in the same data center, available to all cluster nodes, and accessible by NFS as a single mount point.
You’ll need to create a remote directory that is readable and writable by all nodes in the cluster. There are multiple ways to do this, but the simplest is to use an NFS share, which we use as an example.
Terminology
In this guide we'll use the following terminology:
Installation directory: The directory where you installed Bamboo.
Local home directory: The home or data directory stored locally on each cluster node (if Bamboo is not running in a cluster, this is simply known as the home directory).
Shared home directory: The directory you created that is accessible to all nodes in the cluster, preferably via the same path. If you are not running in a cluster, this directory will be present inside the home directory.
To set up and configure your cluster
We recommend completing this process in a staging environment, and testing your warm standby installation, before moving to production.
1. Install Bamboo Data Center on the first application node
First, you'll need to make a fresh installation of Bamboo Data Center in one node following the Bamboo installation instructions.
2. Provision the shared database and filesystem
Once you've installed the first Bamboo application node, you now need to provision the share database and shared filesystem to use with Bamboo Data Center.
Step 1. Provision your shared database
Set up your shared database server.
Ensure your database is configured to allow enough concurrent connections. For example, in PostgreSQL the default limit is usually 100 connections. If you use PostgreSQL, you may need to edit your postgresql.conf file, to increase the value of max_connections, and restart Postgres.
Note that, while warm standby nodes are mostly idle, they periodically connect to the database in order to update their status and, possibly, decide to take over the role of the primary node.
See Connecting Bamboo Server to an external database for more information, and note that clustered databases are not supported.
Step 2. Provision your shared file system
A properly resourced and configured NFS server can perform well even under very heavy load. We’ve created some recommendations for setting up and configuring your file server for optimal performance.
If your Bamboo server is running a version older than 8.0, you’ll need to upgrade to 8.0 first. This will rearrange Bamboo server home, creating the shared subdirectory.
Once on 8.0, the shared subdirectory of the Bamboo Server home directory will contain all the configuration data, build and deployment result files, among other important files. The migration will consist of moving this folder to the Bamboo Data Center’s NFS file system and alter the path of the shared home folder inside the <bamboo-install-dir>/atlassian-bamboo/WEB-INF/classes/bamboo-init.properties.
#bamboo-init.properties
bamboo.home=/your/local/home/path
bamboo.shared.home=/the/mounted/shared/filesystem/path+sharedThe remaining of the subdirectories (analytics-logs, caches, export, local-working-dir,lib, logs, plugins, and temp), inside the now local home path, contain only caches and temporary files. You don’t need to restore them.
Provision application cluster nodes
Provision cluster node infrastructure. You can automate this using a configuration management tool such as Chef, Puppet, or Vagrant, and/or by spinning up identical virtual machine snapshots.
Step 1. Configure file share mounts
On each cluster node, mount the shared home directory as any path. We recommend using the same filesystem path on all the nodes for the sake of simplicity. Configure that path as bamboo shared home, by exporting ${BAMBOO_SHARED_HOME} or setting bamboo.shared.home in the bamboo-init.properties file. Alternatively, if you can skip the last step by mounting the shared directory as ${BAMBOO_HOME}/shared , which is the default location of shared bamboo home in case it is not defined explicitly. Note that only the shared directory should be shared between cluster nodes. All other directories, including ${BAMBOO_HOME} , should be node-local (that is, private to each node).
Example
For example, suppose your Bamboo home directory is /var/atlassian/application-data/bamboo , and your shared home directory is available as an NFS export called bamboo-san:/bamboo-shared. To configure the mount on each cluster node:
Add the following line to
/etc/fstabon each cluster node.
/etc/fstabbamboo-san:/bamboo-shared /var/atlassian/application-data/bamboo/shared nfs rw,nfsvers=3,lookupcache=pos,noatime,intr,rsize=32768,wsize=32768,_netdev 0 0Mount the share on each node. Issue:
mkdir -p /var/atlassian/application-data/bamboo/shared sudo mount -a
Step 2. Synchronize system clocks
Ensure all your cluster nodes have synchronized clocks and identical timezone configuration. Here are some examples for how to do this:
Step 3. Install Bamboo Data Center on each node
On each cluster node, perform the same steps from Install Bamboo Data Center on the first application node but without making migrations or running a fresh installation. Before starting the new nodes in the latter steps, you need to make sure that the installation has the correct paths, and that the bamboo.cfg.xml from the initial node is copied inside the local home folder of this node.
If you automate the installation or clone the new cluster node, make sure that the cluster-node.properties file isn't a direct copy from the previous cluster node. Bamboo uses the cluster-node.properties file to identify nodes. Any duplicates of the node.id or node.name properties will cause multiple active nodes to start in parallel, leading to data corruption.
Ensure that the node.hostname is the node-specific hostname or IP address under which the node is reachable from other nodes in the cluster. You might also want to change the node.internal.communication.port properties (by default, 9090). The port must be reachable from the other nodes, but it doesn’t have to be reachable from the outside world.
Additionally, gRPC communication isn't proxied by default. If you want the underlying gRPC communication to use a proxy, set the bamboo.enable.grpc.via.proxy property to true and remove any references to your Bamboo nodes from the http.nonProxyHosts property.
To authenticate, gRPC uses the certificates and keys generated on the first run and stored in the `<bamboo-shared-home>/ssl` directory. To increase security, the directory should be manually restricted to be accessible only by the users/services running the application nodes, with read, write, and execute access.
Step 4. Start the first cluster node
If you haven’t configured the shared home folder path yet, edit ${BAMBOO_INSTALLATION_FOLDER}/atlassian-bamboo/WEB-INF/classes/bamboo-init.properties by altering the bamboo.shared.home property. Remember that every node will need access to this folder over the NFS.
#bamboo-init.properties
bamboo.home=/your/local/home/path
bamboo.shared.home=/the/mounted/shared/filesystem/path+sharedInstall and configure your load balancer
Step 1. Configure protocols and health checks on your load balancer
Your load balancer must proxy three protocols:
Protocol | Typical port on the load balancer | Default port on the Bamboo cluster nodes | Notes |
|---|---|---|---|
HTTP | 80 | 8085 | HTTP mode. Make sure to enable session affinity ("sticky sessions"). |
HTTPS | 443 | 8085 | HTTP mode. Terminating SSL at the load balancer and running plain HTTP to the Bamboo cluster nodes is highly recommended. |
TCP | 54663 | 54663 | TCP mode. For remote agents JMS connection. To prevent communication failure, the Load Balancer should be configured to SSL Passthrough the request from the remote agents to Bamboo server. |
Your load balancer must support session affinity ("sticky sessions"). Bamboo Data Center assumes that your load balancer always directs each user's requests to the same cluster node.
When choosing a load balancer, it must support the HTTP, HTTPS, and TCP protocols. Note that:
Apache doesn't support TCP mode load balancing.
HAProxy versions older than 1.5.0 do not support HTTPS.
If you don't have a particular preference or policy for load balancers, you can use HAProxy, which is a popular open-source load balancer. For more information, go to Configuring the HAProxy load balancer.
If your load balancer supports health checks of the cluster nodes, configure it to perform a periodic HTTP GET to http://<bamboo-url>:8085/rest/api/latest/status, where <bamboo-url> is the cluster node's name or IP address.
If a cluster node does not return 200 OK within a reasonable amount of time, the load balancer should not direct any traffic to it.
You should then be able to navigate to http://<load-balancer>/, where <load-balancer> is your load balancer's name or IP address. This should take you to your Bamboo front page.