This document provides step-by-step instructions on performing a rolling upgrade on an AWS deployment orchestrated through CloudFormation. In particular, these instructions are suitable for Confluence Data Center deployments based on our AWS Quick Starts.

For an overview of rolling upgrades (including planning and preparation information), see Upgrade Confluence without downtime. 

Step 1: Enable upgrade mode

Enabling upgrade mode allows your cluster to accept nodes running a later bug fix version. This lets you upgrade one node and let it rejoin the cluster (along with the other non-upgraded nodes). Both upgraded and non-upgraded active nodes work together in keeping Confluence available to all users.

You can disable upgrade mode as long as you haven’t upgraded any nodes yet.

Step 2: Find all the current application nodes in your stack

In AWS, note the Instance IDs of all running application nodes in your stack. These are all the application nodes running your current version. You'll need these IDs for a later step.

1. In the AWS console, go to Services > CloudFormation. Select your deployment’s stack to view its Stack Details.

2. Expand the Resources drop-down. Look for the ClusterNodeGroup and click its Physical ID. This will take you to a page showing the Auto Scaling Group details of your application nodes.
3. In the Auto Scaling Group details, click on the Instances tab. Note all of the Instance IDs listed there; you'll be terminating them at a later step.

Step 3: Update your CloudFormation template

Your deployment uses a CloudFormation template that defines each component of your environment. In this case, upgrading Confluence means updating the version of Confluence used in the template. During the upgrade, we highly recommend that you add a node temporarily to your cluster as well.

1. In the AWS console, go to Services > CloudFormation. Select your deployment’s stack to view its Stack Details.
2. In the Stack Details screen, click Update Stack.
3. From the Select Template screen, select Use current template and click Next.
4. Set the Version parameter to the version you’re updating to. Since this is a rolling upgrade, you can only set this to a later bug fix version. 
5. Add an extra node to your cluster. This will help ensure that your cluster won't have a shortage of nodes for user traffic. To do this, increase the value of the following parameters by 1:
   • Maximum number of cluster nodes
   • Minimum number of cluster nodes
     
6. Select Next. Click through the next pages, and then to apply the change using the Update button.

After updating the stack, you will have one extra node already running the new Confluence version. With Upgrade mode enabled, that node will be allowed to join the cluster and start work. Your other nodes won't be upgraded yet.

As soon as the first upgraded node joins the cluster, your cluster status will transition to Mixed. This means that you won’t be able to disable Upgrade mode until all nodes are running the same version.

Once the new upgraded node is running an in an Active state, you can start upgrading another node. To do that, shut down and terminate the node – AWS will then replace the node with a new one running the updated Confluence version.

Step 4: Upgrade another node

In Step 2, you noted the instance ID of each node in your cluster. Terminate the node where you gracefully shut down Confluence. To do this:

1. In the AWS console, go to Services > EC2. From there, click Running Instances.
2. Check the instance of matching the node where you gracefully shut down Confluence. 
3. From the Actions drop-down, select Instance State > Terminate. 
4. Click through to terminate the instance. 

Each time you terminate a node, AWS will automatically replace it. The replacement will be running the new version of Confluence. Once the new node's status is Active, you can move on to upgrading another node.

Step 5: Upgrade all other nodes individually

At this point, your cluster should have two nodes running the new version of Confluence. You can now upgrade other nodes. To do so, simply repeat the previous step on another node. As always, we recommend that you upgrade the node with the least number of running tasks each time.

Step 6: Finalize the upgrade

Step 7: Scale down your cluster

In Step 3, we added a node temporarily to the cluster as a replacement for each one we terminated. This was to help ensure we'd have enough nodes to handle normal user traffic. After finalizing the upgrade, you can remove that node:

1. In the AWS console, go to Services > CloudFormation. Select your deployment’s stack to view its Stack Details.
2. In the Stack Details screen, click Update Stack.
3. From the Select Template screen, select Use current template and select Next.
4. Decrease the value of the following parameters by 1:
   • Maximum number of cluster nodes
   • Minimum number of cluster nodes
5. Select Next. Click through the next pages, and then to apply the change using the Update button.

You can now remove one node from your cluster without AWS replacing it. To do this:

• Choose the node with the least number of running tasks. 
• Shut down Confluence gracefully on the node.  
• Terminate the node. 

Refer to Step 4 for detailed instructions.

Troubleshooting

Disconnect a node from the cluster through the load balancer

If an error prevents you from terminating a node, try disconnecting the node from the cluster through the load balancer. In the AWS Application Load Balancer, each node is registered as a target – so to disconnect a node, you'll have to de-register it. For more information on how to do this, see Target groups for your Application Load Balancers and Registered targets.

Traffic is disproportionately distributed during or after upgrade

Some load balancers might use strategies that send a disproportionate amount of active users to a newly-upgraded node. When this happens, the node might become overloaded, slowing down Confluence for all users logged in to the node.

To address this, you can also temporarily disconnect the node from the cluster. This will force the load balancer to re-distribute active users between all other available nodes. Afterwards, you can add the node again to the cluster.

Node errors during rolling upgrade

Roll back to the original version

To roll back the upgraded nodes to the original version:

1. In the AWS console, go to Services > CloudFormation. Select your deployment’s stack to view its Stack Details.

2. In the Stack Details screen, select Update Stack.

3. From the Select Template screen, select Use current template and select Next.

4. Set the Version parameter to your original version.
5. Select Next. Click through the next pages, and then to apply the change using the Update button.

Afterwards, terminate all nodes running the new version of Confluence. AWS will replace each with a node running the original version. Once all nodes are running the same version, the cluster’s status will revert back to Ready to upgrade. This will also allow you to disable Upgrade mode.

Node won't start up