Troubleshoot a reindex failure in Jira server

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.

Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible


Problem

After attempting a reindex, the following appears in the atlassian-jira.log and on the JIRA index page:

2013-11-28 20:45:16,765 JiraTaskExectionThread-3 ERROR admin 1243x1446x1 12q9trp 127.0.0.1 /secure/admin/jira/IndexReIndex.jspa [jira.util.index.CompositeIndexLifecycleManager] Reindex All FAILED.  Indexer: DefaultIndexManager: paths: [/home/XX/XX/XXX/XXX/jirahome/caches/indexes/comments, /XX/XX/XXX/XXX/jirahome/caches/indexes/issues, /XX/XX/XXX/XXX/jirahome/caches/indexes/changes]
com.atlassian.jira.index.IndexingFailureException: Indexing completed with 1 errors
	at com.atlassian.jira.index.AccumulatingResultBuilder$CompositeResult.await(AccumulatingResultBuilder.java:147)
	at com.atlassian.jira.issue.index.DefaultIndexManager.doIndexIssuesInBatchMode(DefaultIndexManager.java:877)
	at com.atlassian.jira.issue.index.DefaultIndexManager.doStopTheWorldReindex(DefaultIndexManager.java:844)
	at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexAll(DefaultIndexManager.java:307)
	at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexAll(DefaultIndexManager.java:272)
	at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexAll(DefaultIndexManager.java:267)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)

When JIRA applications perform a reindex, errors can happen throughout that process. If an error occurs, exceptions will typically be thrown for individual issues. This means the index can fail for a single issue; however, it does not fail for the entire instance because one issue is causing problems. If the above exception and stack trace is displayed after indexing has finished, it means one or more of the issues failed during the process; however, the entire process did not. It's highly likely that part of the instance has been successfully indexed. Tracking back through the logs prior to the stack trace being thrown will often show exceptions being thrown as issues are indexed.

Some of the known causes for indexing failure are below:

  1. There is a NULL value under the issuenum column of jiraissue table  JSWSERVER-16473 - Getting issue details... STATUS
  2. There is an invalid issuelinktype in the database. 
  3. Entries in the changegroup table in the database with null values in the created column.
  4. Entries in the changeitem table in the database with null values in the field column.
  5. There is a corruption in the workflow post-function that came from third-party plugin.
  6. If you are using the Charting plugin, you might be running into  Unable to locate Jira server for this macro. It may be due to Application Link configuration.
  7. There is a failure while restoring an XML backup or as part of a CSV import which kicks of an index during import. A bug report for this was raised here JRA-36857 - Getting issue details... STATUS
  8. Sometimes Windows Defender or Anti-Virus softwares throw a false positive quarantining the file created during the indexing process, causing it to fail. The $JIRA_HOME and $JIRA_INSTALL directories should be removed from the antivirus scanning as described in Configure Anti-virus software for use with Jira server KB.
  9. There are orphaned sub-tasks in the database.
  10. An unsupported database.

Diagnosis

Identifying the specific issue that is identified in the error can be determined by searching the following keyword in the atlassian-jira.log which contains full traces of the re-indexing process:

Indexing failed for Issue

java.lang.NullPointerException exception thrown as a warning message with no other error message in the indexing process.

IssueIndexer:thread-6 WARN mr80639 466x686x1 1jdr0h0 10.56.14.1,10.237.54.143 /secure/admin/IndexReIndex!reindex.jspa [c.a.jira.index.AccumulatingResultBuilder] java.lang.NullPointerException
java.lang.RuntimeException: java.lang.NullPointerException
	at com.atlassian.jira.index.DefaultIndex$Failure.<init>(DefaultIndex.java:100)
......
Caused by: java.lang.NullPointerException
com.atlassian.rm.teams.customfields.team.IssueTeamService.getTeamForNonSubTask(IssueTeamService.java:56)


Cause 1 

  • NULL value under the issuenum column of jiraissue table
  • Delete an issue while it is being processed by a bulk edit, creates a null issuenum entry in jiraissue table  JSWSERVER-16473 - Getting issue details... STATUS

Run the following SQL query against JIRA's database to check if there are any issues with a NULL in the issuenum column:

SELECT id,issuenum,project FROM jiraissue WHERE issuenum IS NULL;

Cause 2

  •  invalid issuelinktype in the database

Run the following SQL query against JIRA's database to check if there are any invalid issuelink type associations in the database:

SELECT * FROM issuelink WHERE linktype NOT IN (SELECT id FROM issuelinktype);

Cause 3

  • the changegroup table contains NULL values in the created column

Run the following SQL query against JIRA's database to check for NULL entries in the created column of the changegroup table:

SELECT * FROM changegroup WHERE created IS NULL ORDER BY issueid;

Cause 4

  • the changeitem table contains NULL values in the field column

Run the following SQL query against JIRA's database to check for NULL entries in the field column of the changeitem table:

(info) All these entries with changeitem.field=null are related to custom fields. However, since the field is empty, it is not possible to determine if they are related to a plugin or not.

SELECT ji.id, ji.issuenum, ji.project, cg.*, ci.* 
      FROM jiraissue ji
INNER JOIN changegroup cg ON ji.id = cg.issueid 
 LEFT JOIN changeitem ci ON cg.id = ci.groupid
       where ci.field IS NULL;

There is also a use case where there will be NULL ID values in the changegroup table rather than NULL values in the field column in the changeitem table if the above query returns records. You can verify if this is the case if the following query returns no records for the field column of the changeitem table:


select ci.* from changeitem as ci where ci.field is null;

Cause 5

  • A corrupted workflow due to a post-function provided by third-party Add-on

Check for third-party Add-ons used in workflows and possible incompatible versions installed. See Resolution for more details.

Cause 6

  • There is a bug in the JIRA Charting Plugin, tracked as part of  Unable to locate Jira server for this macro. It may be due to Application Link configuration.

Cause 7

  • There is a bug in JIRA tracked as part of  JRA-36857 - Getting issue details... STATUS

Cause 9

  • There is a bug in Jira tracked as part of  JPOSERVER-2015 - Getting issue details... STATUS

Cause 10

  • An unsupported database; for example MySQL Maria DB or latest PostgreSQL versions that are not yet supported by the JIRA version you're running. Please always check the Jira supported platforms article for your version of JIRA (you can choose the version in the top right corner) before upgrading the database

Getting more data

If none of the solutions below fit your scenario, then you need to enable extra logging. For that, change the Logging Level of the package com.atlassian.jira.index.AccumulatingResultBuilder from WARN to DEBUG on Logging & Profiling (Administration > System > Logging & Profiling). This will enable JIRA to write more information about the errors found on the logs. If you are not sure how to analyse or interpret the extra logging, get in touch with Atlassian Support and attach a Support Zip so our Support Engineers can help you.

First thing to try: 

  1. Restart the JIRA application.

  2. Do a full Re-index, by choosing the option to 'Lock and Re-Index'.
  3. Go into Safe Mode and retest

(info) Specifically for Data center environment, ensure that the Data Center nodes are linked to an NTP server and that time is identical on those nodes.

If this does not help, please contact Atlassian Support

Resolution

Solution 1

  • If the query from the diagnosis returns any results, run the following to remove the NULL entries:

    DELETE FROM jiraissue WHERE issuenum IS NULL;

Solution 2

  • If the query from the diagnosis returns any results, run the following to remove any invalid entries:

    DELETE FROM issuelink WHERE linktype NOT IN (SELECT id FROM issuelinktype);

Solution 3

  • If the query from the diagnosis returns any results, run the following to fix it by inserting a date where null occurs:

    UPDATE changegroup SET CREATED='2015-01-01 00:00:00' WHERE CREATED IS NULL;

Solution 4

  • If the query from the diagnosis returns any results, run the following to fix it by deleting the entries where column field=null :

    DELETE FROM changeitem
    WHERE id IN ( 
           SELECT ci.id FROM changeitem ci
       INNER JOIN changegroup cg on cg.id = ci.groupid
            WHERE ci.field IS NULL
    );

    (info) Since the column field is empty in database table changeitem, the reference to determine the history of the change was lost. We can only see an entry of change, it's not possible to determine the field it is related to.

  • (warning) If the query above fails with the error ERROR 1093 (HY000): You can't specify target table 'changeitem' for update in FROM clause in MySQL Database, you can run the query below:

    DELETE FROM changeitem  WHERE id IN (
    select * from (
    SELECT ci.id FROM changeitem ci INNER JOIN changegroup cg on cg.id = ci.groupid WHERE ci.field IS NULL
    )
    as t);
  • If the second query from the diagnosis of Cause 4 returns no result but the first query does, run the following SQL query on DBs other than MySQL to delete the entries from the changegroup table rather than the changeitem table where there is a NULL ID instead:

    DELETE FROM changegroup
    WHERE id IN (
    SELECT cg.id
    FROM jiraissue ji
    INNER JOIN changegroup cg ON ji.id = cg.issueid
    LEFT JOIN changeitem ci ON cg.id = ci.groupid
    where ci.id IS NULL
    );
  • For MySQL DB, we'll need to wrap the select in another select. Run the below query instead:

    DELETE FROM changegroup
    WHERE id IN ( select * from (
    SELECT cg.id
    FROM jiraissue ji
    INNER JOIN changegroup cg ON ji.id = cg.issueid
    LEFT JOIN changeitem ci ON cg.id = ci.groupid
    where ci.id IS NULL) as t
    );

Solution 5

Solution 6

  • Upgrade to a later version of the JIRA Charting Plugin that includes a fix for  Unable to locate Jira server for this macro. It may be due to Application Link configuration.

Solution 7

  • Upgrade to a later version of JIRA that includes a fix for  JRA-36857 - Getting issue details... STATUS

Solution 8

  • Create an exception in Windows Defender for the indexes directory and all its content. 

Solution 9

  • Follow the steps documented in the workaround session of bug  JPOSERVER-2015 - Getting issue details... STATUS

Solution 10

  • If the unsupported database is a result of the DB server upgrade, you may need to revert the DB upgrade back to the supported version: Supported platforms


DescriptionWhen JIRA applications perform a reindex, errors can happen throughout that process. If an error occurs, exceptions will typically be thrown for individual issues. This means the index can fail for a single issue; however, it does not fail for the entire instance because one issue is causing problems. 
ProductJira
PlatformServer
Last modified on Apr 24, 2023

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.