Upgrade task troubleshooting

When introducing a new feature, or making a significant change to your application, we sometimes need to transform existing data in your database or index, or change the way some data is stored.  

Here's a simple example. If we stored "Sydney Australia" in a location column in the database, but later decide to store city and country information separately, we might use an upgrade task to take the existing data in the location column, and split it into a city and country column, containing "Sydney" and "Australia" respectively.  Actual upgrade tasks are rarely this simplistic, but you get the idea.  

On this page:

We don't include changes that require upgrade tasks in bug fix releases, but they can be quite common in feature and platform releases. You can tell if a version has an upgrade task if the build number is different to your current version. 

When are upgrade tasks run? 

This depends on the type of upgrade task, and whether you are upgrading with or without downtime. 

Rolling upgrade without downtime

If you're performing a rolling upgrade: 

  • node-specific upgrade tasks happen just prior to the application starting up on that node. 
  • cluster-wide upgrade tasks happen when all nodes are running the new version, and you select Run upgrade tasks and finalize upgrade in the rolling upgrades screen.

During a rolling upgrade, there's a short time where data might exist in both old and new formats. Cluster-wide upgrade tasks include tasks that transform data in the database, and may also include changes to the shared home and local home directories on each node. These tasks require all nodes to have been upgraded before they can be run. 

Using the example above, a node that has not yet been upgraded would continue to write to the location column, while an upgraded node would write to the new city and country columns, as well as the old location column (to prevent data loss if you need to roll back). Once all nodes are upgraded, it's safe for us to split the existing data in the location field into the new city and country fields.  This is often the part of the upgrade task that can take some time, depending on how much data you have. 

Upgrade with downtime

When upgrading a non-clustered deployment, upgrade tasks are usually run just prior to the application starting up after the upgrade.

When upgrading a cluster with downtime (not a rolling upgrade), cluster-wide upgrade tasks are run when the first node starts up.

Troubleshooting failed upgrade tasks

If an upgrade task fails, there are a number of things you'll need to do to resolve the issue. 

Check the application logs

The first step is to check the application logs. If you're running Confluence in a cluster, you may need to check the logs on more than one node. 

Sometimes the cause will be obvious, such as a network timeout, not enough disk space in the local or shared home directory, or the database user / confluence user has inadequate permissions to complete the action.  

Check your database configuration 

The most common reason upgrade tasks fail include:

  • database user does not have adequate permissions to perform the required action 
  • database configuration is incorrect (for example the character set or encoding) 
  • database version or edition not supported.

This usually results in a database error being written to the application logs. Check for KB articles about the specific problem, and confirm your setup matches the database setup specified in our documentation. 

Re-run upgrade tasks

Once you've resolved any issues, you'll need to re-run the upgrade tasks. How you do this depends on whether you are upgrading with or without downtime. 

  • If you're performing a rolling upgrade, re-start the application on any failed nodes, then select Re-run upgrade tasks and finalize upgrade.
  • If you're upgrading with downtime, re-start the application.  Upgrade tasks will run prior to the application starting up.  

Don't leave your application in upgrade mode

If you're performing a rolling upgrade, it's important you don't leave your cluster in an upgrade mode longer than is necessary. This is because there may be data that needs to be handled in multiple ways until the final upgrade tasks can be run. 

 Known issues

  • Some non-enterprise editions of Microsoft SQL Server don't support online index creation. If an upgrade task needs to acquire an exclusive table lock, you may experience some performance degradation or downtime. We'll warn you if we detect that you database edition may be affected. 
Last modified on Jun 10, 2021

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.