Indexer stops with ERROR java.io.IOException: No space left on device

Still need help?

The Atlassian Community is here for you.

Ask the community


Platform Notice: Server and Data Center Only - This article only applies to Atlassian products on the server and data center platforms.

Summary

Newly added content is unsearchable because Indexer stopped working after system ran out of memory.

Bellow are some of the exceptions that can be found in atlassian-confluence.log

ERROR [Indexer: 1] [internal.index.lucene.LuceneBatchIndexer] doIndex Exception indexing document with id: 12345678
│java.io.IOException: No space left on device                                 
at java.io.RandomAccessFile.writeBytes(Native Method)                 
at java.io.RandomAccessFile.write(RandomAccessFile.java:525)          
at org.apache.lucene.store.FSDirectory$FSIndexOutput.flushBuffer(FSDirectory.java:474)                   
at org.apache.lucene.store.BufferedIndexOutput.flushBuffer(BufferedIndexOutput.java:113)
ERROR [scheduler_Worker-10] [org.quartz.core.ErrorLogger] schedulerError Job (DEFAULT.IndexQueueFlusher threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: com.atlassian.bonnie.LuceneException: com.atlassian.confluence.api.service.exceptions.ServiceException: Failed to process entries]
...
Caused by: com.atlassian.confluence.api.service.exceptions.ServiceException: Failed to process entries
...
Caused by: java.io.IOException: No space left on device
WARN [scheduler_Worker-10] [search.lucene.tasks.AddDocumentIndexTask] perform Error getting document from searchable
java.io.IOException: No space left on device
WARN [scheduler_Worker-1] [search.lucene.tasks.AddDocumentIndexTask] perform Error getting document from searchable
java.io.FileNotFoundException: /opt/confluence/confluence-home/index/_z8ey.fdx (No space left on device)
    at java.io.RandomAccessFile.open0(Native Method)
    at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
    at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:466)

Diagnosis

On Linux

  • Check free disk space and inode count for <confluence-home>/index directory
# df -h
# df -i

On Windows

  • Check free disk space from properties of the drive
  • Check free disk space from PowerShell
Get-WmiObject -Class Win32_logicaldisk
  • Increase space on disk partition or free up space by removing unnecessary and large files

Cause

As documented in 'java.io.IOException No space left on device' due to disk space KB, "No space left on device" exception indicates that there is not enough space left for Confluence to write inside <confluence-home> directory.

Besides failures to generate backups, or save attachments, indexing tasks will also fail .

Solution

Confluence stores it's index locally, inside home directory, and it needs to have enough free space allocated to continue indexing content.

After enough space is freed, rebuild index from scratch by following one of the bellow procedures:

Lucene index must be stored locally on the server and can't be stored on NFS mount as outlined in Health Check: Lucene index files location

Last modified on Jan 4, 2021

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.