Jira 8 to Jira 9 minimal downtime upgrade

Still need help?

The Atlassian Community is here for you.

Ask the community

This KB will guide you from upgrading your Jira 8 cluster to Jira 9 cluster with minimal downtime by eliminating the 2 longest downtime elements: upgrade task adding index versions and full re-index.

The idea is to perform those tasks on the side, while keeping Jira 8 production cluster operational.

  • First we will add index versions to our production database to eliminate need of running upgrade task during downtime.
  • Secondly we will perform full re-index using Jira 9 Single Node DC helper instance and database copy, and produce an index snapshot. We will use this index snapshot when starting first Jira 9 production node.

Try to keep the time between db clone, snapshot creation and cluster upgrade as short as possible.
Do not carry on mass modifications between index snapshot creation time and first Jira 9 production node start.

  1. On production cluster database run sql queries that are adding versions to indexed entities:

    SQL creating index versions
    INSERT INTO atldb.public.worklog_version
    SELECT id, issueid, updated, 1, 'N'
    from atldb.public.worklog
             LEFT JOIN atldb.public.worklog_version on atldb.public.worklog_version.worklog_id = worklog.id
    where atldb.public.worklog_version.worklog_id is null
    LIMIT 300000;
    
    INSERT INTO atldb.public.issue_version
    SELECT id, null, updated, 1, 'N'
    FROM atldb.public.jiraissue
             LEFT JOIN atldb.public.issue_version ON atldb.public.issue_version.issue_id = atldb.public.jiraissue.id
    WHERE atldb.public.issue_version.issue_id is null
    LIMIT 300000;
    
    INSERT INTO atldb.public.comment_version
    SELECT id, issueid, updated, 1, 'N'
    FROM atldb.public.jiraaction
             LEFT JOIN atldb.public.comment_version
                       ON atldb.public.comment_version.comment_id = atldb.public.jiraaction.id
    WHERE actiontype = 'comment'
      AND atldb.public.comment_version.comment_id is null
    LIMIT 300000;

    These queries will continue to insert rows as long as there are issues, worklogs or comments without associated version. These queries must be run until they return no modified rows. It takes around 5 minutes to insert 10_000_000 rows.
    Even though production cluster will still run the upgrade task that serves the same purpose it will have no work to perform and will conclude immediately.


  2. Create a copy of Jira database. How to do that depends on the engine used. Please follow the engine specific documentation to complete this step. Do not use original database instance for the needs of minimal downtime upgrade as it may cause huge load on database and unwanted data modification.

  3. Prepare Jira 9 Single Node DC helper instance to be started. The purpose of this node will be solely to create index snapshot ready to use for the production cluster.
    To do that follow instructions from here: Installing Jira applications. Do not start the node yet, just prepare it.

  4. Before starting Jira 9 Single Node DC helper instance copy dbconfig.xml from one of the running nodes to newly created Jira 9 Single Node DC helper instance home. In the file replace the DB to the clone we created in point 2. dbconfig.xml should be the only file present in helper instance home. Do not copy cluster.properties file.

  5. Start Jira 9 Single Node DC helper instance. During the start full reindex will be performed.

  6. Verify re-index is completed. Look for below log with timestamp after the time you started.

    [c.a.j.index.request.DefaultReindexRequestManager] Re-indexing finished
  7. Since Jira Single Node DC does not create snapshots after full re-index we need to manually create an index snapshot. To do this go to System/Indexing and set the Index Recovery setting daily to near time in the future. Index snapshot will be created as scheduled.

  8. Verify that snapshot has been created. It should appear in <jirahome>/caches/indexesV2/snapshots catalogue and have a name analogical to this example: IndexSnapshot_10300_220225-080500.tar.sz. Don't worry if there are more snapshots created (it might have happened if custom snapshot schedule was set). Just pick whichever - they should be identical.

  9. Copy snapshot to production cluster <sharedhome>/indexesv2/snapshots catalogue. Make sure your jira user has permissions to access it. To be sure execute chown jira:jira <filename> command.

  10. Prepare production cluster to upgrade. Install Jira 9 on each node and point it to the right local home directories.

  11. Set property to not perform a full re-index after the upgrade. For each node edit or create the following file: <jirahome>/jira-config.properties
    Add the following line, and save the file:

    upgrade.reindex.allowed=false 
  12. Modify how old snapshot picked up can be if it is older than 24h. Set property: com.atlassian.jira.startup.max.age.of.usable.index.snapshot.in.hours to value that would include your snapshot. To check how old your snapshot is you can use stat <filename> command. Look for Modify property. Remember to delete the entry after you are done with upgrade. To set the property modify same file as in step 11.

  13. Shut down the production cluster and start one node of the Jira 9 production cluster. The below logs should give you a hint that everything went ok.

    [c.a.jira.index.DefaultIndexFetcher] [INDEX_FETCHER]  Fresh enough snapshot: IndexSnapshot_10300_220225-080500.tar.sz exists. Jira will try to use it during startup.
    
    [c.a.jira.cluster.DefaultClusterManager] Current node: mycluster1 Done recovering indexes from snapshot found in shared home.
  14. After you first Jira 9 production cluster node has successfully started you can continue adding nodes.
Last modified on May 13, 2022

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.