Lucene index corruption

Corrupted indexes might appear after an unexpected shutdown of JIRA, and can lead to problems with searching for issues, and displaying data on JIRA dashboards.

Symptoms

atlassian-jira.log contains:

Caused by: java.io.FileNotFoundException: /usr/local/test/user/atlassian/jira/data/caches/indexes/issues/_skx8_l.del (No such file or directory)
        at java.io.RandomAccessFile.open0(Native Method)
        at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
        at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:214)
        at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:345)
        at org.apache.lucene.index.SegmentReader.loadDeletedDocs(SegmentReader.java:158)
        at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:118)
        at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:702)
        at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:677)
        at org.apache.lucene.index.BufferedDeletesStream.applyDeletes(BufferedDeletesStream.java:249)
        at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3571)
        at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3508)
        at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:3371)
        at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3444)
        at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3426)
        at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3410)
        at com.atlassian.jira.index.WriterWrapper.commit(WriterWrapper.java:143)

JIRA Data Center

In JIRA Data Center, you can copy the indexes from one node to another to replace the corrupted ones. Go to  > Indexing, and fill out the needed info in the Copy the Search Index from another node section.

JIRA Server

Automatic index recovery

Corrupted indexes are fixed automatically for the following JIRA versions:

  • JIRA 7.2.10, or later (7.2 line)
  • JIRA 7.3.9, or later (7.3 line)
  • JIRA 7.4.2, or later

See  JRASERVER-65655 - Fix corrupted index during startup Resolved .

You can recognize it by the following line in the log file:

2017-07-18 23:51:49,990 JIRA-Bootstrap ERROR      [c.a.jira.index.LuceneCorruptionChecker] Index '/jirahome/caches/indexes/issues' is broken: 2 bad segments.
	Performing healing the index. 2 documents will be lost

It means, however, that some documents were deleted from the index, which might result in e.g. JQL not being able to find some issues. If the number of lost documents is significant, you should run a reindex in JIRA to recreate them.

Performance concerns

  • In case of healthy index the check is fast: ~300msec to 20Gb index
  • In case of recovery it may take some time: the speed of check is ~5Gb/min. 

Manual index recovery

If your JIRA version is not mentioned in the Automatic index recovery section, you need to fix the corrupted indexes manually. Before you apply the fix, read the following restrictions:

  • Temporary data loss. Corrupted documents will be deleted from the index, which might result in e.g. JQL not being able to find some issues. All documents, however, can be recreated by running a reindex in JIRA.
  • The fix might take some time to complete, depending on the size of the index.
  1. Stop JIRA.
  2. Download lucene-core-3.3.0-atlassian-2.jar.

  3. Choose whether you want to fix all indexes, or only the corrupted ones. 

    • To fix all indexes, use the following command. It's easier to choose all of them, but the fix will take much longer.

      Fix all indexes
      for index in $JIRA_HOME/caches/indexes/*;
          java -cp lucene-core-3.3.0-atlassian-2.jar org.apache.lucene.index.CheckIndex -fix $index;
      done
    • To fix only specific indexes, first read the exception messages to know which of them are corrupted. Then, specify them in the path. The following command fixes the issues index, which was shown as corrupted in the log:

      Fix the issues index
      java -cp lucene-core-3.3.0-atlassian-2.jar org.apache.lucene.index.CheckIndex -fix $JIRA_HOME/caches/indexes/issues

      The example of an exception message in this KB points to the /usr/local/test/user/atlassian/jira/data/caches/indexes/issues index.

       

  4. After running the command, you'll see a warning about a number of documents to be deleted. It helps you understand whether you really need that reindex or not.

    WARNING: 4 documents will be lost
  5. Start JIRA.

  6. If you want to recreate the deleted documents, run a reindex.

Last modified on Oct 3, 2017

Was this helpful?

Yes
No
Provide feedback about this article

Not finding the help you need?

Ask the community

Powered by Confluence and Scroll Viewport.