Summary

This KB article focuses on some specific and more practical aspects of the upgrade to Jira 9 and is intended as additional material to the official upgrade guide:

• Jira 9.4 Long Term Support release upgrade guide
• Preparing for Jira 9.12

These other articles are also considered required reading material to plan the upgrade:

• Preparing Jira for the 9.4 upgrade
• Upgrading Jira Data Center to 9.4
• Upgrading Jira Server to 9.4

You should first run the upgrade in lower environments to familiarize yourself with the upgrade steps, validate if your specific setup requires any additional steps, and validate if the new version and compatible apps behave as expected and meet your requirements.

Please contact Atlassian Support if you have questions after reading the materials or have problems upgrading the lower environments.

Environment

Jira Software (JSW) Data Center 8 to 9

Jira Service Management (JSM) Data Center 4 to 5

Solution

Main takeaways

The key points when compared to the Jira 7 to 8 upgrade are:

• Incompatible apps should be turned off as they may prevent a successful startup.
• New flags are required to prevent the automatic full reindex at startup.
• Full reindex time may be considerably longer than in Jira 8 (possibly extending the maintenance window).

Other points worth mentioning:

• As a Platform Release upgrade, there's no ZDU (Zero Downtime Upgrade): the whole cluster needs to be down.
• The Jira 9 Index is incompatible with Jira 8's, so a full (foreground) reindex must be performed after Jira 9 is up and running before it's released to the users.

Last, some Platform Release upgrades best-practices:

• Set up an Internal Directory System Admin account to mitigate the risk of being unable to log in if the User Directory fails. You may need to temporarily disable or allow bypassing of SAML/SSO, too.
• Configure an alternate connector in Jira's Tomcat (through the server.xml) to allow bypassing the reverse proxy. Jira may respond "MAINTENANCE" on the /status endpoint (used by Load Balancers) until it has a healthy index and only a proxy-bypassing connector would allow the admin to login (if the automatic full reindex has been turned off through the new flags).

General example upgrade steps (without ZDU)

1. Check the supported platforms
2. Run the bundled App Update Check
3. Copy over (merge) customizations
4. Add the JVM flags to bypass the automatic full reindex
5. Bypass Load Balancer checks
6. Disable all incompatible apps
7. Stop Jira (on all nodes)
8. Backup the DB and the Jira home folder
9. Start Jira 9 up in only one node 
10. log in as an admin after the upgrade on that node, and enable the apps disabled on step 4
11. Kick off a full (foreground) reindex
12. Validation
13. Start the Jira 9 on other nodes one by one
14. Migrate log4j configs (if needed)

1. Check the supported platforms

Check the Supported Platforms of the version of Jira you're upgrading to and confirm you meet the requirements (version dropdown selector on the top of the article):

• Supported platforms

If you don't meet any requirements (like Java version, DB version, etc.), you should first upgrade the platforms and then upgrade Jira.

If you need to skip too many versions, you may need to upgrade the platforms first, then Jira to an intermediate version, then the platforms again, and then Jira again to the final version.

2. Apps Update Check

The App Update Check is a powerful built-in feature available through the Manage Apps screen (more on Checking app compatibility with application updates). It checks the compatibility of all apps against a selected destination Jira version.

No changes are performed during these checks, so running it at any time in Production is safe.

Example screenshot of a Jira Update Check:

This App Update Check is also listed in our article on Preparing for the upgrade. Please read that article, too.

Alternatively, you may consider updating the apps listed under "Compatible if updated" in a maintenance window (at least some days) before the Jira upgrade. You should do these updates in a lower environment first to ensure the new apps' versions work as expected.

3. Copy over (merge) customizations

Also, as part of regular upgrades, you should manually edit the Jira 9 files customized in Jira 8.

From Migrating Jira applications to another server:

Some of the files we usually modify:

• server.xml
• dbconfig.xml
• jira-config.properties
• setenv.sh / setenv.bat (memory allocation and other JVM arguments)
• for more, see Important files in Jira
  

See Important files in Jira for a complete list of commonly modified files and their locations within your Jira Installation Directory.

The DB drivers (usually for Oracle or MySQL DB) should also be copied.

4. Add the JVM flags to bypass the automatic full reindex

Jira 8 allowed startup without an index with the jira-config.properties entry of "upgrade.reindex.allowed=false".

In Jira 9, this config only works for the first-ever startup — the one running (or had run) the DB update tasks. If the node's restarted (for whatever reason), Jira 9 will understand it's no longer an "upgrade startup" and enforce a full reindex upon startup. This has been raised through:

JRASERVER-74882 - Getting issue details... STATUS

To guarantee Jira 9 won't kick off a full reindex, not even in subsequent restarts during the upgrade maintenance window, add these two Opts to the JVM startup params before starting Jira 9 for the first time:

-Dcom.atlassian.jira.startup.allow.full.reindex=false 
-Dcom.atlassian.jira.startup.rebuild.local.index=false

You may add them to the JVM_SUPPORT_RECOMMENDED_ARGS in the setenv.sh/bat file (appended to other parameters you may have there, separated by space characters):

JVM_SUPPORT_RECOMMENDED_ARGS="-Dcom.atlassian.jira.startup.allow.full.reindex=false -Dcom.atlassian.jira.startup.rebuild.local.index=false"

And add this to the jira-config.properties file (you may need to create the file if it doesn't exist — see Advanced Jira application configuration):

upgrade.reindex.allowed=false

Please have a look at the article on Setting properties and options on startup for an overview of setting JVM flags.

These two flags will trigger this dismissible startup screen (Ignore all warnings and continue):

Once the upgrade's over, you should remove those flags so Jira 9 behaves as expected on all subsequent restarts, optimizing Index acquisition and delta catch-up with multiple Threads. Nodes will rely on their local indexes and fall back to the shared-home snapshots when needed.

For more details on the Index startup strategy introduced in Jira 9.4, please look at the Index startup procedure in Jira 9.4 and later.

5. Bypass Load Balancer checks

Since we're forcing Jira not to perform the Full Reindex on startup, Admins should be able to log in to Jira to update required apps and kick off the Full Reindex manually.

If you don't have a Proxy/Load-Balancer bypassing connector on Tomcat, you may not be able to access Jira through the Browser because, since 8.19, Jira responds "Maintenance" to the /status URL if its Indexes aren't healthy.

To work around this, you need to either:

a. Configure a bypass connector in the server.xml and access Jira through its IP:port on the browser

• Refer to Bypass a proxy or SSL to test network connectivity for Jira server

Regardless of the upgrade, this alternate connector is good to have, as it facilitates troubleshooting several performance and UI glitches in Jira. (by testing direct Browser connection to Jira)

You should add this connector before the upgrade and validate it's working.

or

b. Add a temporary JVM Opt to bypass the Index Consistency Checks

You may add this parameter together with the others from step 4:

-Dcom.atlassian.jira.status.index.check=false

If you already had this option (to workaround JRASERVER-73281, for example), you can keep it.

If not, you should remove it as soon as the upgrade is completed — this is a critical feature in Jira and should only be applied if you can't setup an alternate connector in time for the upgrade and Jira's behind a Load-Balancer using the /status endpoint to validate if the node's online.

6. Disable all incompatible apps

Some Jira 8 apps are incompatible with Jira 9, so they may compromise Jira startup.

Before bringing all nodes down for the upgrade, we should turn off all apps listed under any of these categories in the Update Check:

• Incompatible
• Compatible if updated
• Compatible once both Jira and the app are updated

We'll update and enable them once Jira 9 comes up and only then kick off the full reindex.

Given this impact, this step may be considered the start of Jira unavailability during the maintenance window.

7. Stop Jira (on all nodes)

Jira doesn't support ZDU (Zero-Downtime Upgrades) between "Platform Releases" (7.x to 8.x, 8.x to 9.x, etc), so all nodes must be brought down before the upgrade can be started in any of them.

Also, don't put the cluster in "upgrade mode" — this feature is only for ZDU, and enabling it may compromise the upgrade procedure. It should only be enabled when upgrading Jira within the same Platform Release, like from 8.0.0 to any version up to 8.22.99* or 9.0.0 to 9.99.99*

* Hypothetical versions to illustrate the version range.

8. Backup the DB and the Jira home folder

With Jira down, performing a backup of the local home folder and the database is safe now.

If Jira is not configured with the cluster.properties (requires Jira Data Center license), the attachments may be located within the local home folder, and its backup may extend the maintenance window considerably. If so, you may consider backing up the attachments folder before bringing the nodes down and excluding the folder from the backup on this "step 6".

Please refer to Set up a Jira Data Center cluster to learn how to set up a Jira cluster. You may even set up a "single node cluster, " but you will still benefit from the shared-home design.

If the attachments were backup up previously, you may exclude them from this step to save time.

9. Start Jira 9 up in only one node

Jira 9 should be started up only in one node. It'll perform the upgrade tasks, including altering the DB tables to add more columns and creating missing version records for entities like Issues, Comments, and Worklogs.

Because of these upgrade tasks, this startup is expected to take longer than the subsequent Jira 9 startups.

Wait to start other nodes until the first node's up and the manual full reindex has been completed.

10. Login as an admin and upgrade, then enable the apps disabled on step 4

Once the first node is up and running, you may access it through the browser as an admin, upgrade each app disabled in step 4, and then enable them.

It's essential to run this through UAT sometime in advance as some apps may have become paid when upgrading them to Jira 9.4 compatible versions.

To reduce login issues, you should set up a local admin account (in the Internal Directory) and temporarily configure SAML/SSO bypassing (if you have them) to avoid being locked out of the instance.

Also, it's advised to configure an alternate Tomcat connector to bypass the reverse proxy and access the node's Tomcat directly, as Jira 9 will respond MAINTENANCE on the /status endpoint to the Load Balancer until it has a healthy index of its own.

Refer to Bypass a proxy or SSL to test network connectivity for Jira server for more on this.

11. Kick off a full (foreground) reindex

Once all disabled apps have been updated to compatible versions and re-enabled, you may kick off the full reindex.

Please note some customers have observed a considerable increase in the full reindex duration from Jira 8 to Jira 9:

JRASERVER-74787 - Getting issue details... STATUS

Depending on how many entity versions and non-archived Issues Jira has, some customers reported a full reindex duration up to 3 times what was observed in Jira 8.20. This issue was fixed in 9.7.0 (JSM 5.7.0) and backported to Jira 9.4.6.

The good news is that the Jira 9 index is more reliable than Jira 8's, and admins shouldn't need to perform full reindex as regularly as they had on Jira 8 — and even then, it shouldn't have been required so much since Jira 8.13!

See our article on Periodic full reindex in Jira Data Center is not required for more. If you still have to perform regular full reindexes to fix unexpected behavior in Jira 8.13 – 8.22, please contact Atlassian Support so we can troubleshoot this!

12. Validation

Once the full reindex is completed in the first node, admins may perform the usual post-upgrade validations they're used to.

This may include creating and changing the status of some test Issues, validating critical apps and features, and confirming the bundled Health Check's reporting is all good.

13. Start the other nodes one by one

Now, the other nodes may be upgraded and started up individually.

The process is the same as performed for the first node, either through the installer or manual upgrade (tar.gz/zip):

• Upgrading Jira (installer)
• Upgrading Jira (manual)

The customizations from step 2 should also be ported over on each node — though the two new JVM flags from step 3 are technically dismissible (they're only relevant to the single node being worked on until the first Jira 9 full reindex happens).

When the nodes come up, they'll fetch the recently generated index snapshot from the shared-home (created by the first node manual full reindex) and be available as quickly as possible.

-Dcom.atlassian.jira.startup.allow.full.reindex=false
-Dcom.atlassian.jira.startup.rebuild.local.index=false

14. Migrate log4j configs (if needed)

If upgrading to JSW 9.5 / JSM 5.5 or above, you should port over custom changes you may have made to the log4j.properties.

Jira SW 9.5 / SM 5.5 bundles Log4j 2, and its configuration file is an XML, not the .properties file from Log4j 1.2.

You can refer to this article on Migrating custom logging configurations to Log4j 2 to port over custom loggers if needed. We advise doing this post-upgrade once Jira's stable to isolate any impacts from log4j2 misconfiguration from the upgrade.