Bitbucket Data Center in AWS
Bitbucket Data Center is an excellent fit for the Amazon Web Services (AWS) environment. Not only does AWS allow you to scale your deployment elastically by resizing and quickly launching additional nodes, it also provides a number of managed services that work "out of the box" with Bitbucket Data Center and that handle all their configuration and maintenance automatically.
AWS components
This diagram depicts a typical Bitbucket Data Center deployment in AWS.
On this page:
This deployment consists of these components:
- One or more Amazon Elastic Compute Cloud (EC2) instance(s) as cluster node(s), running Bitbucket.
- Typically these instance(s) are managed by an Amazon Auto Scaling Group, which can scale up and down the number of cluster node(s) either manually or automatically in response to load between prescribed minimum and maximum limits.
- An Amazon Elastic Load Balancer (ELB), both as load balancer and SSL-terminating reverse proxy.
- An EC2 instance as dedicated NFS server, with an attached Elastic Block Store (EBS) volume for the shared home directory.
- An Amazon Relational Database (RDS) instance as the shared database.
- An Amazon Elasticsearch Service domain for code and repository search.
| Bitbucket version | Bitbucket Server 4.10.0 or higher. | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Cluster nodes | One or more EC2 instances running Bitbucket, managed by an Auto Scaling Group. | ||||||||||||||||
| Load balancer & reverse proxy | Amazon Elastic Load Balancer (ELB), with Listeners configured as follows:
Note that Bitbucket is configured to immediately redirect all http:// requests received on port 7991 to https://, for security. | ||||||||||||||||
| NFS server | An EC2 instance configured as a dedicated NFS server for the cluster, with an EBS volume storing the Bitbucket shared home directory. This contains all of Bitbucket Server's repository, attachment, and other data. | ||||||||||||||||
| Database | Amazon RDS for PostgreSQL 9.5. | ||||||||||||||||
| Elasticsearch | Amazon Elasticsearch Service, version 2.3. |
CloudFormation template and Quick Start guide
An AWS CloudFormation template is a single JSON or YAML file which describes a collection of AWS resources, allowing you to create, update, and delete them all together as a single "stack". Using AWS CloudFormation to manage your resources is much quicker and more convenient that deploying and managing the separate resources individually.
Amazon also publishes Quick Starts as standard, automated reference deployments for a number of key applications in AWS. Each Quick Start launches, configures, and runs the AWS compute, network, storage, and other services required to deploy a specific workload on AWS, using AWS best practices for security and availability.
The easiest way to launch Bitbucket Data Center in the AWS environment is to follow the Quick Start guide and associated CloudFormation template. The complete Bitbucket Data Center Quick Start guide and template are available here:
AWS CloudFormation and Quick Starts do not incur any additional AWS charges, though of course you must pay for all AWS resources launched via CloudFormation normally with your usual AWS account billing process.
Note: The Bitbucket Data Center Quick Start is not available in some regions, such as govcloud-us (AWS GovCloud (US)) and cn-north-1 (China (Beijing)).
Scaling up and down
To increase or decrease the number of cluster nodes
- Go to Services > CloudFormation in the AWS console, select the stack, and click Update Stack.
- Change the Minimum number of cluster nodes and Maximum number of cluster nodes parameters as desired.
It may take several minutes for the Auto Scaling Group to detect and apply changes to these parameters.
Unless you specify the same number for Minimum and Maximum number of cluster nodes, the Auto Scaling Group is allowed to launch new cluster nodes and terminate existing ones automatically to achieve the optimal desired number of nodes between these two limits. By default, this target number is determined by the following CloudWatch metrics:
- If the average CPU utilization across the Auto Scaling Group exceeds 60% for 5 minutes, the target number of nodes increases by one (up to the Maximum).
- If the average CPU utilization across the Auto Scaling Group is lower than 40% for 30 minutes, the target number of nodes decreases by one (down to the Minimum).
A default "cooldown" period of 10 minutes between scaling events is also applied. See Scaling Based on Metrics for more information.
Note: Adding new cluster nodes, especially automatically in response to load spikes, is a great way to increase capacity of a cluster temporarily. Beyond a certain point, adding very large numbers of cluster nodes will bring diminishing returns. In general, increasing the size of each node (i.e., "vertical" scaling) will be able to handle a greater sustained capacity than increasing the number of nodes (i.e., "horizontal" scaling), especially if the nodes themselves are small. See Recommendations for running Bitbucket in AWS for more information.
Connecting to your nodes over SSH
Sooner or later you will need to SSH to your Bitbucket cluster node(s) and file server to perform configuration or maintenance tasks. Note that you must keep your SSH private key file (the PEM file you downloaded from Amazon and specified as the Key Name parameter) in a safe place. This is the key to all the nodes in your instance, and if you lose it you may find yourself "locked out." See Administering Bitbucket Server in AWS for more information.
Note: BitbucketDataCenter.template deploys all EC2 instances in the Subnets specified by the Internal subnets parameter. If you have specified Internal subnets that are completely unreachable from outside, then you may need to launch an EC2 instance with SSH running and accessible in one of the the External subnets, and use this as a "jump box" to SSH to any instances in your Internal subnets. That is, you SSH first to your "jump box", and from there to any instance deployed in the Internal subnets.
When connecting to your instance over SSH, use ec2-user as the user name, for example:
ssh -i keyfile.pem ec2-user@ec2-xx-xxx-xxx-xxx.compute-1.amazonaws.com
The ec2-user has sudo access. SSH access is by root is not allowed.
Installing an SSL certificate
Before launching BitbucketDataCenter.template for the first time, it is recommended to upload your SSL certificate into Amazon, as described in SSL Certificates for Classic Load Balancers. Then, when launching Bitbucket Data Center, you can specify the SSL certificate's name in the SSL Certificate Name parameter, and it will be deployed automatically on your ELB, with listeners set up as follows:
| Load Balancer Protocol | Load Balancer Port | Instance Protocol | Instance Port |
|---|---|---|---|
| HTTP | 80 | HTTP | 7991 |
| HTTPS | 443 | HTTP | 7990 |
But if you didn't have an SSL certificate set up at initial launch time, or omit the SSL Certificate Name parameter, then your ELB will be configured with only one HTTP listener as follows:
| Load Balancer Protocol | Load Balancer Port | Instance Protocol | Instance Port |
|---|---|---|---|
| HTTP | 80 | HTTP | 7990 |
To install or change your SSL certificate after initial launch:
- In the AWS console, go to Services >> CloudFormation, select the stack, and click Update Stack.
- Change the SSL Certificate Name parameter to the name of your new certificate.
- Configure the HTTP to HTTPS redirect manually in the
bitbucket.propertiesfile, located in the<Bitbucket home directory>/shareddirectory, as described in Redirect HTTP Requests to HTTPS. Restart the Bitbucket service by running the following command on all application nodes
sudo service atlbitbucket restart- If the hostname of your Bitbucket instance has changed you will need to update Bitbucket's base URL as described in the page Specifying the base URL for Bitbucket Server.
Upgrading
Review the steps in the Bitbucket Server upgrade guide and check the upgrade notes for the version of Bitbucket you are upgrading to, as additional steps to the ones described below may be required!
Note: it is recommended to update the AMI to the latest version before upgrading Bitbucket. For some versions, such as Bitbucket Server 5.0, the latest AMI is required. The latest AMI IDs are referenced in the BitbucketDataCenter.template in the bitbucket.org repository.
To upgrade a Bitbucket Data Center instance launched from BitbucketDataCenter.template:
- Go to Services > CloudFormation in the AWS console, select the stack, and click Update Stack.
- Change the Version parameter to the new version (this does not actually upgrade any existing cluster node(s), but ensures that any new nodes launched in the Auto Scaling Group will be on the new version).
- Terminate your existing cluster node(s), and allow the Auto Scaling Group to launch new node(s) running the new version.
Bitbucket Data Center currently does not allow upgrading an instance without some downtime in between the last cluster node of the old version shutting down and the first cluster node on the new version starting up.
Backing up
Every Bitbucket Data Center instance launched from BitbucketDataCenter.template comes with the Bitbucket DIY Backup utilities pre-installed and pre-configured to use Amazon EBS and RDS snapshots for Zero Downtime Backup. Just SSH to your file server instance and run the following commands:
cd /opt/atlassian/bitbucket-diy-backup
git pull
./bitbucket.diy-backup.sh
This script can also be run on a regular schedule (e.g., hourly), from cron. See Using Bitbucket zero downtime backup for more information.
Deploying mirrors
To deploy a mirror of an existing Bitbucket Data Center instance in AWS
Deploy BitbucketServer.template, with the Bitbucket properties parameter set to
application.mode=mirror.SSH to your new mirror instance and install your SSL certificate as described in Securing Bitbucket Server behind nginx using SSL.
- Proceed through the new mirror's Setup Wizard as usual.
See Set up a mirror for more information.
Deploying a Disaster recovery standby instance
To deploy a Disaster recovery standby of an existing Bitbucket Data Center instance in AWS
- Find out the Amazon Resource Name (ARN) of your primary instance's RDS database. This is given by the DBMaster output of your primary's CloudFormation stack, and will be a string of the form
arn:aws:rds:eu-west-1:123456789012:db:qdc37u5sysye14. Deploy BitbucketDataCenter.template in your desired AWS Region (typically not the same Region as your primary instance) with the Bitbucket properties parameter set to this ARN.
At this point, you should have a standby instance with no cluster nodes running and an Amazon RDS read replica database.
Follow the Disaster recovery guide for Bitbucket Data Center, with atlassian-bitbucket-diy-backup variables configured as follows:
BACKUP_HOME_TYPE=zfs
BACKUP_DATABASE_TYPE=amazon-rds
DR_RDS_READ_REPLICA=<identifier of your RDS read replica>
STANDBY_JDBC_URL=jdbc:postgresql://<endpoint of your RDS read replica>.rds.amazonaws.com:5432/bitbucket
If you need to fail over from your primary Bitbucket instance to the Disaster recovery standby, you can launch cluster node(s) by updating the Minimum number of cluster nodes parameter in the standby stack, as described in the section Scaling up and down above.
See Disaster recovery guide for Bitbucket Data Center for more information.
Migrating your existing instance into AWS
To migrate an existing instance into AWS
- Migrate it's database to PostgreSQL (if not already).
- Take a backup of your existing home directory and database, as described in Bitbucket Server DIY Backup.
- Copy the backup file to your file server EC2 instance.
- Unpack the backup file under
/media/atl/bitbucket/sharedof your file server. - Restore the PostgreSQL database dump contained in the backup file to your RDS instance with
pg_restore