Symptoms

Search results or filters are not displaying the recently updated issues.

JIRA applications are timing out when users try to create a new issue

The following error appears in the log:

ERROR [jira.issue.index.DefaultIndexManager] Wait attempt timed out - waited 30000 milliseconds

OR:

ERROR [jira.issue.index.DefaultIndexManager] Lock attempt timed out - waited 30000 milliseconds

You could also see a long running index thread waiting on a read of the index:

"ISSUE-indexQueue:thread-1" #260 prio=5 os_prio=0 tid=0x nid=0x4cd runnable [0x000]
   java.lang.Thread.State: RUNNABLE
	at java.io.RandomAccessFile.open0(Native Method)
	at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
	at java.io.RandomAccessFile. (RandomAccessFile.java:243)
	at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:214)
	at org.apache.lucene.index.CompoundFileReader. (CompoundFileReader.java:64)
	at org.apache.lucene.index.SegmentCoreReaders. (SegmentCoreReaders.java:67)
	at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:114)
	at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:702)
	- eliminated 0x00007f70e2000000> (a org.apache.lucene.index.IndexWriter$ReaderPool)
	at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:677)
	- locked <0x0000> (a org.apache.lucene.index.IndexWriter$ReaderPool)
	at org.apache.lucene.index.BufferedDeletesStream.applyDeletes(BufferedDeletesStream.java:249)
	- locked <0x000000020> (a org.apache.lucene.index.BufferedDeletesStream)
	at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3571)
	- locked <0x01f00> (a org.apache.lucene.index.IndexWriter)
	at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3508)
	at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:3371)
	at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3444)
	- locked <0x0088> (a java.lang.Object)
	at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3426)
	at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3410)
	at com.atlassian.jira.index.WriterWrapper.commit(WriterWrapper.java:115)

Diagnosis

The text of this error message has changed. Search the atlassian-jira.log for 'attempt timed out - waited', which appears in both versions of the error message.

The DVCS job may report the error while Jira is running a re-index. If this is the case, there's nothing to worry about. The job will go back to normal once the re-index task is done. 

Cause

When an issue is created or updated, JIRA applications acquire a lock to the Lucene index. Should another user perform an update at the same time, and the lock isn't released within 30 seconds, the error gets generated. This can be caused by any of the following:

• The indexes are mounted on remote storage over NFS, which is not supported as per Supported platforms.
• Using a JIRA application in a system with slow disk speed.
• Antivirus can slow disk access and cause this to occur.
• The database may be overloaded - when Lucene creates an index it attempts to access the database. If it's waiting for a response this can increase the time taken for the index, subsequently causing timeouts.
• The CPU may be overloaded due to garbage collection or expensive threads which will slow down operations in JIRA applications as threads are contending for the CPU.
• The network speed between JIRA applications and the database may be slow - this would cause the reads from the database to create the indexes in JIRA to be slow.
• Add-on calculated custom fields take very long to re-index. 

Workaround

It is possible to increase the wait lock time for the indexes, however this is not an ideal way of addressing the problem - it will increase the wait time for the threads and will increase response times from JIRA application. It is recommended to identify why the threads are timing out in the first place rather than just increasing the timeout.

1. Ensure the indexes are not mounted over NFS as this is not supported - they should be using a local disk.
2. Shut down the application.
3. If running antivirus, set an exclusion for $JIRA_HOME/caches, which is where the indexes are stored.

4. Alleviate the problem by editing the jira-config.properties file (for JIRA 4.4.x and later) or jira-application.properties file (for JIRA 4.3.x and earlier) with these entries:

   jira.index.lock.waittime 
   

   Try increasing the waittime from 30000 ms to 90000 ms.

5. Re-index a JIRA application.
   
   

   Note that re-index jobs may take significantly longer than normal as it will take more time to wait for the lock to be release.

Resolution

1. Ensure the indexes are not mounted over NFS as this is not supported - they should be using a local disk.

2. If running antivirus, disable it from scanning the JIRA application index by setting an exclusion rule for $JIRA_HOME/caches.

   If Symantec is run on the JIRA application server, there have been reports where it will not respect the exclusion. Please verify this by testing disk speed before and after setting the exclusion, and if it continues it may need to be replaced.

3. If the server is slow, you may consider upgrading it. See Testing Disk Access Speed and JIRA Requirements for tips on assessment.

4. (Only applies to JIRA 5.1.5 and earlier) Disable the Index Optimisation Job, as in our (Archived) How to Disable the Index Optimization Job in JIRA KB.

5. Additionally, change the permissions scheme to reduce the frequency of bulk issue changes. See Managing Project Permissions.
6. Increase your instance's memory settings as per Increasing JIRA Memory.
7. Audit your add-on calculated custom fields and either: 
   1. Change the type to non-calculated if not needed
   2. Remove indexes on them if you don't require them to be user-searchable