ReIndexing reporting wrong numbers on Jira Datacenter with Archived projects
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Problem
Jira Datacenter is reporting wrong numbers after finishing foreground re-Indexing.
The re-Indexing finishes unexpectedly before it reaches 100% and no errors are being logged.
The following appears in the atlassian-jira.log:
2019-01-01 00:00:23,624 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing started
2019-01-01 00:00:23,624 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.util.index.CompositeIndexLifecycleManager] Reindex All starting...
2019-01-01 00:00:23,624 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.index.ha.DefaultNodeReindexService] Pausing node re-index service
2019-01-01 00:00:23,647 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing is 0% complete. Current index: Issue
2019-01-01 00:00:23,647 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.issue.index.DefaultIndexManager] ReindexAll in foreground: {indexIssues=true, indexChangeHistory=true, indexComments=true, indexWorklogs=true, forceReloadFromDatabase=false}
.....
.....
2019-01-01 01:04:26,624 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing is 68% complete. Current index:
2019-01-01 01:04:26,624 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.util.index.CompositeIndexLifecycleManager] Reindex All complete. Total time: 3840094ms. Reindex run: 101
2019-01-01 01:04:26,624 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing finishedAlso the below could be seen in the logs:
2019-01-1 12:35:00,166 http-nio-8080-exec-1 DEBUG user2 111x11111x2 1111aab xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/jira/IndexAdmin.jspa [c.a.j.issue.index.IndexConsistencyUtils] isIndexConsistent: Issue: expectedCount=235000; actualCount=159000
2019-01-1 12:35:00,166 http-nio-8080-exec-1 WARN user2 111x11111x2 1111aab xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/jira/IndexAdmin.jspa [c.a.j.issue.index.IndexConsistencyUtils] Index consistency check failed for index 'Issue': expectedCount=235000; actualCount=159000As can be seen the Index is having a big discrepancy from the number of issues got from the database (expectedCount).
Diagnosis
Environment
Jira Datacenter 7.10.x or later
Having 1 or more projects in Archive.
Diagnostic Steps
Enable Debug logs on the below packages:
com.atlassian.jira.index.AccumulatingResultBuilder com.atlassian.jira.issue.index
This should allow Jira to list all the individual issues being indexed and we could verify how many issues are being indexed exactly:
$ egrep "111x11111x1.*Indexing issue:" atlassian-jira.log* |wc -l
159000
$This would then needs to be compared with total number of issues in the Jira database minus the amount of issues in the Archived projects.
Another diagnostic step is to check the output of the below rest API call:
GET <JIRA_BASE_URL>/rest/api/2/index/summary
This call will report the number of issues in Index, in the database and in Archive, Jira stores those number in a special cache. Those number are then used to drive the reporting of the re-Indexing operation.
This cache is being repopulated if a project is marked for Archive or if it is brought back from Archive.
The healthy output should look like this:
{"nodeId":"jira_node1","reportTime":"2019-01-1T15:50:00.356+0100","issueIndex":
{"indexReadable":true,"countInDatabase":235000,"countInIndex":159000,"countInArchive":76000,"lastUpdatedInDatabase":"2019-01-1T00:40:00.357+0100","lastUpdatedInIndex":"2019-01-1T12:35:00.000+0100"}Cause
For some currently unknown reason, the Jira issue counts cache might get corrupted and reports a wrong count for the issue count in Archive, this results in wrong reporting for the re-Indexing job and causes confusion.
The summary rest API call will look like this:
{"indexReadable":true,"countInDatabase":235000,"countInIndex":159000,"countInArchive":0,"lastUpdatedInDatabase":"2019-01-1T00:40:00.357+0100","lastUpdatedInIndex":"2019-01-1T12:35:00.000+0100"}The count of issues in Archive can be seen as 0, even though Jira does have issues in the archived projects.
This issue is reported in the following bug report: JRASERVER-69296 - Getting issue details... STATUS
Workaround
The workaround for this issue is to correct the values of the cache by flushing it. This should rebuild the number and correct the indexing calculations. This can be done by adding or removing a project from Archive.
Please follow the below steps:
Create new empty project or recover an existing project from Archive.
The size of project doesn't matter since Deindexing issues for archived project is done by a single query to index (shouldn't take more than 100ms)
After restoring a project you don't need to reindex this project.Archive this project again (either archive or restoring of any project from Archive is clearing the issue counts cache)
Verify if countInArchive has changed in /rest/api/2/index/summary response.
Note
If you use the AWS QuickStart templates, scaling a new node may show the same warning after a new node is provisioned. If this is the case, you may need to repeat this for the new nodes by deleting the dummy project we created as part of the workaround and re-creating/archiving. Alternatively, you can create another new project or unarchive/archive an existing project.