Upgrade task troubleshooting
We don't include changes that require upgrade tasks in bug fix releases, but they can be quite common in feature and platform releases. You can tell if a version has an upgrade task if the build number is different to your current version.
When are upgrade tasks run?
This depends on the type of upgrade task, and whether you are upgrading with or without downtime.
Rolling upgrade without downtime
If you're performing a rolling upgrade:
- node-specific upgrade tasks happen just prior to the application starting up on that node.
- cluster-wide upgrade tasks happen when all nodes are running the new version, and you select Run upgrade tasks and finalize upgrade in the rolling upgrades screen.
During a rolling upgrade, there's a short time where data might exist in both old and new formats. Cluster-wide upgrade tasks include tasks that transform data in the database, and may also include changes to the shared home and local home directories on each node. These tasks require all nodes to have been upgraded before they can be run.
Using the example above, a node that has not yet been upgraded would continue to write to the location column, while an upgraded node would write to the new city and country columns, as well as the old location column (to prevent data loss if you need to roll back). Once all nodes are upgraded, it's safe for us to split the existing data in the location field into the new city and country fields. This is often the part of the upgrade task that can take some time, depending on how much data you have.
Upgrade with downtime
When upgrading a non-clustered deployment, upgrade tasks are usually run just prior to the application starting up after the upgrade.
When upgrading a cluster with downtime (not a rolling upgrade), cluster-wide upgrade tasks are run when the first node starts up.
Troubleshooting failed upgrade tasks
If an upgrade task fails, there are a number of things you'll need to do to resolve the issue.
Check the application logs
The first step is to check the application logs. If you're running Confluence in a cluster, you may need to check the logs on more than one node.
Sometimes the cause will be obvious, such as a network timeout, not enough disk space in the local or shared home directory, or the database user / confluence user has inadequate permissions to complete the action.
Check your database configuration
The most common reason upgrade tasks fail include:
- database user does not have adequate permissions to perform the required action
- database configuration is incorrect (for example the character set or encoding)
- database version or edition not supported.
This usually results in a database error being written to the application logs. Check for KB articles about the specific problem, and confirm your setup matches the database setup specified in our documentation.
Re-run upgrade tasks
Once you've resolved any issues, you'll need to re-run the upgrade tasks. How you do this depends on whether you are upgrading with or without downtime.
- If you're performing a rolling upgrade, re-start the application on any failed nodes, then select Re-run upgrade tasks and finalize upgrade.
- If you're upgrading with downtime, re-start the application. Upgrade tasks will run prior to the application starting up.
Don't leave your application in upgrade mode
If you're performing a rolling upgrade, it's important you don't leave your cluster in an upgrade mode longer than is necessary. This is because there may be data that needs to be handled in multiple ways until the final upgrade tasks can be run.
Known issues
- Some non-enterprise editions of Microsoft SQL Server don't support online index creation. If an upgrade task needs to acquire an exclusive table lock, you may experience some performance degradation or downtime. We'll warn you if we detect that you database edition may be affected.