Troubleshoot Lucene index corruption in Jira server
Platform Notice: Server and Data Center Only - This article only applies to Atlassian products on the server and data center platforms.
Corrupted indexes might appear after an unexpected shutdown of JIRA, and can lead to problems with searching for issues, and displaying data on JIRA dashboards.
Caused by: java.io.FileNotFoundException: /usr/local/test/user/atlassian/jira/data/caches/indexes/issues/_skx8_l.del (No such file or directory) at java.io.RandomAccessFile.open0(Native Method) at java.io.RandomAccessFile.open(RandomAccessFile.java:316) at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243) at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:214) at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:345) at org.apache.lucene.index.SegmentReader.loadDeletedDocs(SegmentReader.java:158) at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:118) at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:702) at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:677) at org.apache.lucene.index.BufferedDeletesStream.applyDeletes(BufferedDeletesStream.java:249) at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3571) at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3508) at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:3371) at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3444) at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3426) at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3410) at com.atlassian.jira.index.WriterWrapper.commit(WriterWrapper.java:143)
JIRA Data Center
In JIRA Data Center, you can copy the indexes from one node to another to replace the corrupted ones. Go to> Indexing, and fill out the needed info in the Copy the Search Index from another node section.
Automatic index recovery
Corrupted indexes are fixed automatically for the following JIRA versions:
- JIRA 7.2.10, or later (7.2 line)
- JIRA 7.3.9, or later (7.3 line)
- JIRA 7.4.2, or later
You can recognize it by the following line in the log file:
2017-07-18 23:51:49,990 JIRA-Bootstrap ERROR [c.a.jira.index.LuceneCorruptionChecker] Index '/jirahome/caches/indexes/issues' is broken: 2 bad segments. Performing healing the index. 2 documents will be lost
It means, however, that some documents were deleted from the index, which might result in e.g. JQL not being able to find some issues. If the number of lost documents is significant, you should run a reindex in JIRA to recreate them.
- In case of healthy index the check is fast: ~300msec to 20Gb index
- In case of recovery it may take some time: the speed of check is ~5Gb/min.
Manual index recovery
If your JIRA version is not mentioned in the Automatic index recovery section, you need to fix the corrupted indexes manually. Before you apply the fix, read the following restrictions:
- Temporary data loss. Corrupted documents will be deleted from the index, which might result in e.g. JQL not being able to find some issues. All documents, however, can be recreated by running a reindex in JIRA.
- The fix might take some time to complete, depending on the size of the index.
- Stop JIRA.
Choose whether you want to fix all indexes, or only the corrupted ones.
To fix all indexes, use the following command. It's easier to choose all of them, but the fix will take much longer.Fix all indexes
for index in $JIRA_HOME/caches/indexes/*; java -cp lucene-core-3.3.0-atlassian-2.jar org.apache.lucene.index.CheckIndex -fix $index; done
To fix only specific indexes, first read the exception messages to know which of them are corrupted. Then, specify them in the path. The following command fixes the
issuesindex, which was shown as corrupted in the log:Fix the issues index
java -cp lucene-core-3.3.0-atlassian-2.jar org.apache.lucene.index.CheckIndex -fix $JIRA_HOME/caches/indexes/issues
The example of an exception message in this KB points to the
After running the command, you'll see a warning about a number of documents to be deleted. It helps you understand whether you really need that reindex or not.
WARNING: 4 documents will be lost
If you want to recreate the deleted documents, run a reindex.