Jira Data Center search indexing

To provide fast searching, Jira creates indexes of the text entered into issue fields. These indexes are stored on the file system, and updated whenever a piece of text is added or modified. They're called Lucene indexes, because they are provided by a third-party library with that name. This page explains how indexes are managed and kept in sync in Jira Data Center.

Where are the indexes stored?

The indexes are stored in a number of directories in the local Jira home directory under caches. Each node in the cluster has its own set of indexes.

`-- indexes

    |-- changes

    |-- comments

    |-- entities

    |   |-- portalpage

    |   `-- searchrequest

    `-- issues

Synchronizing the indexes

Jira keeps all the copies of the indexes up to date automatically. The synchronization is not fully synchronous but aims for eventual consistency, which means that there is some delay before the index changes are seen on other nodes in the cluster.

The indexes are synchronized continuously – each node polls for the changes once per 5 seconds. But where are these changes recorded?

Indexes and database

• Database table: replicatedindexoperation
  Each index operation writes a row to this database table. All nodes then look for entries that were written by other nodes in the cluster. After finding such changes, the nodes apply them to their local Lucene index.
• Database table: nodeindexcounter
  To avoid the nodes checking for all possible operations all the time, they always record the latest processed operation in this table, so that during the next check, they only need to read new operations.

Document Based Replication

In Jira 8.12 we introduced Document Based Replication which also serves as index synchronization mechanism for Jira Data center.

Learn more about DBR

Retention period

With the number of indexing changes, the database tables might get very big. To avoid that, we've introduced a service that runs on each node and removes messages that have been there longer than a set period of time. The default retention period is set to 2800 minutes (2 days), which works well with indexing, but you can customize it in Jira.

1. In Jira, go to  > System.
2. In the Advanced section, select Services.
3. Edit the com.atlassian.jira.service.services.index.ReplicatedIndexCleaningService service, and enter a new retention period.
   

   

Replicating the indexes

When a node joins the cluster for the first time, or if it has been offline for an extended period of time, it will receive a copy of an up-to-date index from another active node instead of applying all these changes from the database. It's just simpler and more effective. The indexes are replicated in the following way:

1. A node sends a "backup index request" to the cluster.
2. One of the active nodes receives the request, removes it from the message queue, and creates a backup of the index in the shared home directory.
3. The node that created the backup sends an "index backed up" message to the node that requested the backup.
4. The requesting node replaces its current index with the backup.
5. The requesting node also applies any changes that have occurred since the backup was created.

Checking the health of the indexes

Health checks

Jira Data Center provides a health check that helps you make sure the indexes are replicated without any issues. The knowledge base article describing it also contains some troubleshooting information, as well as links to specific issues related to indexes. For more info, see HealthCheck: Cluster Index Replication.

APIs

You can also use the API to check the condition of the index on a particular node. For more info, see Get index summary.

Q&A 

We've also gathered here some basic questions that we often get about the indexes.