Unable to search issues and the log shows read past EOF messages in Jira

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform Notice: Server and Data Center Only. This article only applies to Atlassian products on the server and data center platforms.

 

Summary

Several operations in Jira may fail or take too much time to perform, like loading dashboards, gadgets, filters creating or updating issues.

In the atlassian-jira.log  we see a read past EOF error message.


Environment

All versions of Jira 7.x and 8 up to 8.11.


Diagnosis

Jira's behavior becomes erratic and it may take a long time to load some screens, but the easiest way to diagnose this scenario is:

  1. Accessing an issue's direct URL loads the issue successfully: JIRA-BASE-URL/browse/JIRA-123, yet trying to perform a search doesn't bring any result — if the screens loads at all.
  2. On the atlassian-jira.log we can see several "read past EOF" messages. Some of the scenarios include the ones below.

During Jira startup

2021-07-12 14:25:25,064 JIRA-Bootstrap ERROR      [c.a.jira.index.LuceneCorruptionChecker] Can't open index '/opr/atlassian/jira/caches/indexes/issues'. Reason:
java.io.IOException: read past EOF

Accessing certain JIRA pages

SEVERE: Internal server error
com.atlassian.jira.issue.index.SearchUnavailableException: com.atlassian.util.concurrent.LazyReference$InitializationException: com.atlassian.jira.util.RuntimeIOException: org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _ow2: fieldsReader shows 1 but segmentInfo shows 50
 at com.atlassian.jira.issue.index.DefaultIndexManager$1.get(DefaultIndexManager.java:85)
 at com.atlassian.jira.issue.index.DefaultIndexManager$1.get(DefaultIndexManager.java:76)
 at com.atlassian.jira.issue.index.SearcherCache.retrieveIssueSearcher(SearcherCache.java:38)

Searching for issues or accessing a Jira filter

Caused by: java.io.FileNotFoundException: /local/jira/jira444/home/caches/indexes/entities/searchrequest/_o5.fnm (No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.<init>(SimpleFSDirectory.java:78)
at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.<init>(SimpleFSDirectory.java:108)

Trying to reindex Jira

java.io.IOException: read past EOF
 Task completed in 2 seconds with unexpected error.
 Started Today 8:29 PM.
 Finished Today 8:29 PM.
 com.atlassian.jira.util.RuntimeIOException: java.io.IOException: read past EOF at com.atlassian.jira.index.DefaultIndexEngine.clean(DefaultIndexEngine.java:145) at com.atlassian.jira.index.DefaultManager.deleteIndexDirectory(DefaultManager.java:44) at com.atlassian.jira.issue.index.DefaultIssueIndexer.deleteIndexes(DefaultIssueIndexer.java:170) at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexAll(DefaultIndexManager.java:219) ... at java.util.concurrent.FutureTask.run(FutureTask.java:138) at com.atlassian.jira.task.ForkedThreadExecutor$ForkedRunnableDecorator.run(ForkedThreadExecutor.java:249) at java.lang.Thread.run(Thread.java:619)
 Caused by: java.io.IOException: read past EOF

or

2012-11-25 16:21:36,897 JiraTaskExectionThread-1 ERROR admin 981x203x1 6slf04 10.1.250.116 /secure/admin/IndexReIndex.jspa [jira.util.index.CompositeIndexLifecycleManager] Reindex All FAILED.  Indexer: DefaultIndexManager: paths: [/var/atlassian/application-data/jira/caches/indexes/comments, /var/atlassian/application-data/jira/caches/indexes/issues, /var/atlassian/application-data/jira/caches/indexes/changes]
com.atlassian.jira.util.RuntimeIOException: java.io.IOException: directory '/var/atlassian/application-data/jira/caches/indexes/comments' exists and is a directory, but cannot be listed: list() returned null
	at com.atlassian.jira.index.DefaultIndexEngine.clean(DefaultIndexEngine.java:146)
	at com.atlassian.jira.index.DefaultManager.deleteIndexDirectory(DefaultManager.java:43)
	at com.atlassian.jira.issue.index.DefaultIssueIndexer.deleteIndexes(DefaultIssueIndexer.java:205)
	at com.atlassian.jira.issue.index.DefaultIndexManager.doStopTheWorldReindex(DefaultIndexManager.java:824)
	at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexAll(DefaultIndexManager.java:335)
	at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexAll(DefaultIndexManager.java:301)
Caused by: java.io.IOException: directory '/var/atlassian/application-data/jira/caches/indexes/comments' exists and is a directory, but cannot be listed: list() returned null


Cause

This EOF error is a consequence of corrupted Lucene index files. Jira's unable to recover on it's own from this corruption and any operations fail therefor.

This is likely to happen if Jira's process was killed abruptly while writing to the index files or if the machine or VM itself has been interrupted. Storage failures may also corrupt the index files if they were being written at the time.


Solution

The solution is to restore the Indexes or rebuild them from scratch.

If you're running a Jira Data Center with two or more nodes, you may copy the indexes from a healthy node to the corrupted one through the Admin interface, instead.

Restoring indexes

If you have a backup from the index folder (JIRA-HOME/caches/indexes), you may follow this procedure:

  1. Stop Jira
  2. Move all the contents from JIRA-HOME/caches/indexes to a backup folder just for the sake of it
  3. Restore the contents of a indexes  backup folder into JIRA-HOME/caches/indexes
  4. Start Jira
  5. Confirm it's operational and dashboards, gadgets and filters are working fine — albeit without recent data (only the data up to when the index backup was taken)
  6. Start a background reindex so the more recent data is gradually updated in the index

Rebuilding index from scratch

If there's no recent indexes folder backup, the only solution is to lock and reindex Jira. Unfortunately this will cause a downtime for Jira users until the reindex is complete.

  1. Stop JIRA
  2. Make a backup of JIRA_HOME/caches/indexes and it's sub-directories

  3. Remove all the contents from JIRA_HOME/caches/indexes 

  4. Start JIRA
  5. Perform a locked/full reindex


Last modified on Jul 13, 2021

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.