Problem

In a clustered setup of JIRA, the nodes rely on database tables specifically designed for JIRA Data Center to keep search indexes in sync at all times. This makes it possible for an issue update reflect instantly across all nodes in the cluster.

However, sometimes, usually due to a bug such as  JRA-42126 - Getting issue details... STATUS  or the use of a plugin not fully compatible with JIRA Data Center, the indexes will fall out of sync constantly and permanently, requiring for indexes to be manually copied between nodes.

Diagnosis

One or more of these symptoms are associated with this problem:

• An issue is created on one node and doesn't appear when running a search in the others, even after waiting several minutes or hours;
• The number of issues that appears in JIRA's Issue Navigator heavily differs from the database;
• Nodes have different count of issues;

You may also be warned by JIRA's Lucene Health Check about inconsistencies in search indexes.

Cause

The main database tables used by JIRA on replicating indexes across Data Center nodes are replicatedindexoperation and nodeindexcounter. The first is updated by any node that registers an update in the indexes, and queried by all nodes every second. The latter stores the ID of the last index operation registered by each node and the node that registered it.

On JIRA instances affected by  JRA-42126 - Getting issue details... STATUS , a DELETE operation performed in a node would cause it to insert multiple entries to the replicatedindexoperation table until the ID of the entry reaches the limit for Integer numeric values. However, in some cases it stopped looping at some point, but the indexes would still fail to be replicated.

The reason it causes index synchronization problems is because when a node reads this table, it will constantly update its indexes and register the last index operation on nodeindexcounter to an unusually high ID. JIRA will not, however, update its table sequences to match the high ID, so future index updates will be registered following the last known sequence. When nodes read that table, it will not update its indexes because the last index operation registered on nodeindexcounter will have the higher ID.

Resolution

Upgrade your JIRA Data Center installation to version 6.3.16, 6.4.1 or newer, which contains a fix to the looping register of DELETE operations.

If your environment is already on JIRA 6.3.16, 6.4.1 or later, make sure all plugins installed on your instance are certified to be compatible with JIRA Data Center. There are a number of plugins that rely on index operations, and these plugins have to be adapted for changes to the indexing engine used in a clustered environment.

Please contact Atlassian Support for any further clarification. We do not recommend manually modifying database tables.