Upgrade a Bitbucket cluster on AWS without downtime
Step 1: Enable upgrade mode
- Go to
> Administration > Rolling upgrades. Select the Upgrade mode toggle (1).
Enabling upgrade mode allows your cluster to accept nodes running a later bug fix version. This lets you upgrade one node and let it rejoin the cluster (along with the other non-upgraded nodes). Both upgraded and non-upgraded active nodes work together in keeping Bitbucket available to all users.
You can disable Upgrade mode as long as you haven’t upgraded any nodes yet.
Step 2: Find all the current application nodes in your stack
In the AWS console, go to Services > CloudFormation. Select your deployment’s stack to view its Stack Details.
- Expand the Resources drop-down. Look for the ClusterNodeGroup and click its Physical ID. This will take you to a page showing the Auto Scaling Group details of your application nodes.
- In the Auto Scaling Group details, click on the Instances tab. Note all of the Instance IDs listed there; you'll be terminating them at a later step.
Step 3: Update your CloudFormation template
Your deployment uses a CloudFormation template that defines each component of your environment. In this case, upgrading Bitbucket means updating the version used in the template. During the upgrade, we highly recommend that you add a node temporarily to your cluster as well.
In the AWS console, go to Services > CloudFormation. Select your deployment’s stack to view its Stack Details.
In the Stack Details screen, click Update Stack.
From the Select Template screen, select Use current template and click Next.
- Set the Version parameter to the version you’re updating to. Since this is a rolling upgrade, you can only set this to a later bug fix version (for example, from Bitbucket 7.9.0 to 7.9.4).
- Add an extra node to your cluster. This will help ensure that your cluster won't have a shortage of nodes for user traffic. To do this, increase the value of the following parameters by 1:
- Maximum number of cluster nodes
- Minimum number of cluster nodes
Click Next. Click through the next pages, and then to apply the change using the Update button.
After updating the stack, you will have one extra node already running the new Bitbucket version. With Upgrade mode enabled, that node will be allowed to join the cluster and start work. Your other nodes won't be upgraded yet.
Mixed status
As soon as the first upgraded node joins the cluster, your cluster status will transition to Mixed. This means that you won’t be able to disable Upgrade mode until all nodes are running the same version.
Once the new upgraded node is running an in an Active state, you can start upgrading another node. To do that, shut down and terminate the Bitbucket node – AWS will then replace the node with a new one running the updated Bitbucket version.
Step 4: Upgrade another node
Start with the least busy node
We recommend that you start upgrading the node with the least number of running tasks and active users. This is typically the node with the lowest CPU utilization. If you deployed Bitbucket with Amazon CloudWatch, you can view each node's CPU utilization through your CloudWatch dashboard. Learn more about listing available CloudWatch metrics
In Step 2, you noted the instance ID of each node in your cluster. Choose a node to upgrade first, and terminate it:
- In the AWS console, go to Services > EC2. From there, click Running Instances.
- Check the instance matching your chosen node.
- From the Actions drop-down, select Instance State > Terminate.
- Click through to terminate the instance.
Each time you terminate a Bitbucket node, AWS will automatically replace it. The replacement will be running the new version of Bitbucket. Once the new node's status is Active, you can move on to upgrading another node.
Only terminate the Bitbucket nodes. Do not terminate the NFS Server EC2 instance.
Step 5: Upgrade all other nodes individually
At this point, your cluster should have two nodes running the new version of Bitbucket. You can now upgrade other nodes. To do so, simply repeat the previous step on another node. As always, we recommend that you upgrade the node with the lowest CPU utilization each time.
Step 6: Finalize the upgrade
After completing the rolling upgrade, you should:
- Update your apps accordingly
- Perform UAT and other tests as needed
Scale down cluster
In Step 3, we added a node temporarily to the cluster as a replacement for each one we terminated. This was to help ensure we'd have enough nodes to handle normal user traffic. After finalizing the upgrade, you can remove that node:
In the AWS console, go to Services > CloudFormation. Select your deployment’s stack to view its Stack Details.
In the Stack Details screen, click Update Stack.
From the Select Template screen, select Use current template and click Next.
- Decrease the value of the following parameters by 1:
- Maximum number of cluster nodes
- Minimum number of cluster nodes
Select Next. Click through the next pages, and then to apply the change using the Update button.
You can now remove one node from your cluster without AWS replacing it. Find the least busy Bitbucket node and terminate it (refer to Step 4 for detailed instructions).
Troubleshooting
Disconnect a node from the cluster through the load balancer
If an error prevents you from terminating a node, try disconnecting the node from the cluster through the load balancer. In the AWS Classic Load Balancer, each node is registered as a target – so to disconnect a node, you'll have to de-register it. See Register or deregister instances and Configure connection draining for more information.
Node errors during rolling upgrade
There are several ways to address this:
Shut down Bitbucket gracefully on the node. This should disconnect the node from the cluster, allowing the node to transition to an Offline status.
If you can’t shut down Bitbucket gracefully, shut down the node altogether.
Once all active nodes are upgraded with no nodes in Error, you can finalize the rolling upgrade. You can investigate any problems with the problematic node afterwards and re-connect it to the cluster once you address the error.
Rolling back to the original version
To roll back the upgraded nodes to the original version:
In the AWS console, go to Services > CloudFormation. Select your deployment’s stack to view its Stack Details.
In the Stack Details screen, click Update Stack.
From the Select Template screen, select Use current template and click Next.
- Set the Version parameter to your original version.
- Click Next. Click through the next pages, and then to apply the change using the Update button.
Afterwards, terminate each Bitbucket node running the new version of Bitbucket. Each time you do, AWS will replace the node with one running the original version. Wait until the AWS replacement node is Active before terminating another node.
Once all nodes are running the same version, the cluster’s status will revert back to Ready to upgrade. This will also allow you to disable Upgrade mode.
Disabling upgrade mode
- haven't been upgraded yet
- aren't in an Error state
The cluster's status will transition to Mixed if an upgraded node joins the cluster or a node enters an error state.
Mixed status with Upgrade mode disabled
If a node is in an Error state with Upgrade mode disabled, you can't enable Upgrade mode. Fix the problem or remove the node from the cluster to enable Upgrade mode.