Summary

If by mistake a Jira administrator starts a Jira node in a version different than the others we might observe different behaviors, outputs, and errors:

• If the first started node is in a higher version than seen in the database, it will try to upgrade your Jira Database and will update the Jira shared home. With the upgrade, the remaining nodes will not be able to start, for the reason shown below. 
• If the first started node is in a lower version than seen in the database, the node will become unavailable and will not receive connections (the process is kept alive). The remaining nodes, if in the correct version, will start as expected.
• If the first started node is in the same version seen in the database, but the remaining nodes are in a higher version, those will not get available. 

Details below.

Scenarios and errors

The first node is in a higher version

When the first node to be started is in a higher version than seen in Jira database, it will upgrade your Jira Database and Jira shared home. With the upgrade, the remaining nodes (considering in lower versions) will not connect to the cluster or be available for the users.

Example from lab: The Jira database is on version 940006 (9.4.6) - you can check the Jira database version using the query below - and the first started node is on version 9.6.8.

select propertyvalue from propertystring where id = (select id from propertyentry where property_key = 'jira.version.patched');

This first node started will upgrade the Jira database, the remaining nodes will not get available for a similar reason seen in the next scenario. 

The first node is in a lower version

When the first node started is in a lower version than seen in the Jira database, although the process will not stop, this node will not get available for the cluster. 

Example from lab: The Jira database is on version 940008 (9.4.8), the first started node is on version 9.4.6.

• From the UI, connecting direct to the lower version node, we observe a message as below:

• From the node in version 9.4.6 logs we have:

main ERROR      [c.a.jira.health.HealthChecks] Failed to start Jira due to a build number inconsistency.
...
main ERROR      [c.a.jira.health.HealthChecks] The version of the data present in your database is higher than the version of Jira you are trying to start.
        - Database build number: 940008
        - Jira app build number: 940006

    The best solution is to upgrade Jira to a version that is equal or higher than the version of your data.
    You could also try to downgrade your current database build version to match the version of Jira you are trying to start.
    Review our documentation and try what works for you.
...
main ERROR      [c.a.jira.startup.DefaultJiraLauncher] JIRA has failed to start because of the following errors: [(Event: Level = (EventLevel: fatal) , Key = (EventType: database) , Desc = Failed to start Jira due to a build number inconsistency. , Exception = The version of the data present in your database is higher than the version of Jira you are trying to start.<br/><ul><li>Database build number: <strong>940008</strong></li><li>Jira app build number: <strong>940006</strong></li></ul><br/>The best solution is to upgrade Jira to a version that is equal or higher than the version of your data.<br/>You could also try to downgrade your current database build version to match the version of Jira you are trying to start.<br/>Review our documentation and try what works for you.<br/>)]

The first node in correct version but the others in different version

If the first node started is in the same version as seen in the database it will proceed with a clean start, however if the remaining node is in a different version this last will not become available:  the process will not stop, and when the last node is in higher version it will not attempt to upgrade the Jira database. 

Example from lab: The Jira database is on version 940006 (9.4.6), the first started node is on version 9.6.4, and the second on version 9.4.8.

• From the UI we have for the first node that node 2 (9.4.8) attempted to connect, but is without a heartbeat.

• From the node 2 (9.4.8) logs we have:

main WARN      [c.a.jira.startup.ClusterNodeVersionCheckLauncher] Node with different build number ("node1" with version 9.4.6/940006) from this node (9.4.8/940008) detected, delaying start-up for up to 300 seconds.
...
main ERROR      [c.a.jira.startup.LauncherContextListener] Unable to start JIRA.
java.lang.RuntimeException: Refusing to start up this node. Node with different build number ("node1" with version 9.4.6/940006) from this node (9.4.8/940008) detected

    The following nodes run a different version of JIRA and are
    preventing startup of this node:

    Node ID                           Build Number              Version
    -------------------------------------------------------------------
    node1                                   940006                9.4.6
    These nodes must be stopped or removed from the cluster before
    this node will start.

What about the ZDU?

The Zero Downtime Upgrade allows Jira administrators to run minor different Jira versions for a short period of time when upgrading the environment. The ZDU is not expected to be used out of a change period or for long operation time. 

In some rare cases, the cluster may enter an inconsitent state during a ZDU. You'll see an error like the one noted above, where an upgraded node is blocked from entering the cluster. The issue can normally be dealt with by following these steps:

1. Shut down service on the upgraded node (affected node)
2. Select the "Cancel upgrade" option
3. Put Jira into upgrade mode again
4. Start service on the upgraded node