Reindexing outside of the cluster

Upgrading Jira Data Center with faster reindexing

On this page

Related content

  • No related content found

Still need help?

The Atlassian Community is here for you.

Ask the community

To minimize downtime for your users, you’ll use a copy of your database to create the index outside of the production cluster. With most of the data already indexed, your upgraded nodes will only add the missing changes instead of creating the whole index from scratch.

Overview

Read more about this approach...

Every time a node goes offline and then rejoins the cluster, it updates its index with recent operations read directly from the database, only reindexing in the background. This applies if a node has an index, the missing changes are still in the database (there’s a configurable retention period, which we’ll mention later), and they constitute fewer than 10% of the whole index.

We’ll use the same mechanism here. You’ll create an index by connecting a copy of your database to a fresh installation of Jira 8.5, to make the index compatible with the new version. This index will contain most of your data, and the nodes will just need to apply the missing changes that were added to your original database since you’ve started the upgrade.

After you upgrade the whole cluster, your nodes will detect the new index and add the missing changes. From the node’s perspective, it will look like it was offline for a short while.

The only downtime for your users is the upgrade itself, as Jira will be functional shortly after the upgrade.

Before you begin

You can complete these steps without shutting down your Jira Data Center.

Creating the index outside of the cluster

You’ll make a copy of your database and connect Jira 8.5 to it. This will allow you to create an index that is compatible with Jira 8 outside of your production cluster.

1. Copying the NodeIndexCounter table

  • Make a copy of the NodeIndexCounter table from your database.

Why do I need this?

The NodeIndexCounter table records the changes that were applied by individual nodes to their copies of the index. By copying it now, and restoring it later to your original database (during the upgrade), you’ll make sure that the upgraded nodes include all possible changes made by your users from this point on.

2. Copying the database

  • Make a copy of your live database. Don’t worry about users adding new data, because you’ll switch back to the original database later on.

These steps should be completed as close to the actual upgrade as possible so the difference between the original and cloned database is not too big, and doesn’t exceed the retention period.

Tell me more about the retention period...

The max. time is 2 days by default. That’s how long information about recent operations is stored in the database (table: ReplicatedIndexOperations). If you exceed this limit, the nodes won’t be able to add all missing changes from your original database, as changes older than 2 days will be deleted. You can extend the retention period if you need more time, but be careful here — if the changes exceed 10% of the whole index, the nodes will need a full reindex (it’s hard to exceed 10%, but it really depends on how many changes your users make).


3. Connecting the database copy to Jira 8.5

You’ll need a separate Jira 8.5 environment that you’ll connect your database copy to. Here are requirements for this environment:

  • Must be Jira 8.5.

  • Must have the same plugins as your production instance (plugins can provide data coming from custom fields, and add data to the index during reindexing).

  • Must have a Data Center license, but you can use a single node.


The easiest way to set up this environment is to complete these steps:

By following these steps, you will install a separate 8.5 instance, disable automatic reindex, giving yourself time to install the plugins (or copy them from your production environment), and then perform a manual reindex whenever you’re ready.

  1. Follow Upgrade Jira on the first node from Upgrading Jira Data Center. When configuring Jira 8.5, make sure to connect it to the copy of your database that you created in the previous steps.

  2. Follow Post-upgrade steps on the first node from Upgrading Jira Data Center. 


Result

After completing the manual reindex, you should have a Jira 8 index that’s based on the information from the copy of your database. It’s still missing all the information that was added to the original database since you’ve made the copy, but it will be easier for your nodes to add missing entries for these changes than create the whole index from scratch.

Copying the index to your nodes

The Jira 8 index is stored in a different directory than the old one, so it won’t affect your existing cluster. You can already copy the new index to each of the nodes, so they don’t have to download it.

  1. Copy the Jira 8 index to the following directory on each node: 

    <local-home-directory>/caches/indexesV1

Upgrading Jira Data Center

This is where you’ll need to shut down your Jira Data Center and start the upgrade.

Upgrading the first node

We recommend that you upgrade Jira Data Center by choosing a single node, upgrading it, and then copying the installation directory to other nodes, just to save time. You can choose any node here.

  1. Stop the cluster.

  2. Restore the NodeIndexCounter table, which you copied at the beginning of this procedure, to your original database. This will make sure that the nodes create entries for all the information added to your original database since you’ve started the upgrade.

  3. Upgrade the first node. Follow Upgrade Jira on the first node from Upgrading Jira Data Center. Before you start, make sure to read the note below. 

    Disabling automatic reindex

    The last step of the linked procedure will tell you to disable automatic reindex (Step 5: Disabling automatic reindex). Make sure to complete it, as otherwise Jira will initiate automatic reindex, something you’re trying to avoid.

  4. Complete post-upgrade steps on the first node. Follow Post-upgrade steps on the first node from Upgrading Jira Data Center. 

    One of the steps will tell you to rebuild index. Make sure to omit this step, as you have already created the index outside of the cluster, so there’s no need to create it again.

What happens here?

After you start the first node, the database will be upgraded. As the next step, the node would check the state of the index, and reindex if needed. But, since you’ve already copied a compatible index, the node will detect it and only add entries for the missing changes — the difference between you original and cloned database that you used to create the Jira 8 index. From the node’s perspective, it looks like it was offline for a short while.

Optional: Requesting a snapshot of the index

This optional step can make your nodes download a snapshot of the index from the first node instead of replaying the updates from the database.

Tell me more...

In general, we recommend that you omit this step and make the nodes replay the updates. However, getting a snapshot might be faster if your index is not too large and the number of updates since you’ve started the upgrade is very high.

To make the nodes download the snapshot, you need to delete the index files on the remaining nodes (not the first node). With no index in their home directories, the nodes will request a snapshot from the first node.


Upgrading remaining nodes

You can now copy the installation directory to remaining nodes to upgrade them.

  1. Upgrade remaining nodes. Follow Upgrade remaining nodes from Upgrading Jira Data Center.

Reindexing again

Your nodes should already have a complete index, but we recommend that you reindex again, just to be sure that not a single change has been omitted. You don’t have to shut down your Jira Data Center, as reindexing will be performed on a single node.

  1. Go to  > System > Indexing (you can search for it by pressing '.')

  2. Choose Full reindex.

This reindexing will take more time to complete, but it doesn’t block your users from using Jira. Once it’s complete, the new index will be automatically distributed to the remaining nodes.

Last modified on Sep 16, 2020

Was this helpful?

Yes
No
Provide feedback about this article

Related content

  • No related content found
Powered by Confluence and Scroll Viewport.