Indexer stops with ERROR java.io.IOException: No space left on device
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Newly added content is unsearchable because Indexer stopped working after system ran out of memory.
Bellow are some of the exceptions that can be found in atlassian-confluence.log
ERROR [Indexer: 1] [internal.index.lucene.LuceneBatchIndexer] doIndex Exception indexing document with id: 12345678
│java.io.IOException: No space left on device
at java.io.RandomAccessFile.writeBytes(Native Method)
at java.io.RandomAccessFile.write(RandomAccessFile.java:525)
at org.apache.lucene.store.FSDirectory$FSIndexOutput.flushBuffer(FSDirectory.java:474)
at org.apache.lucene.store.BufferedIndexOutput.flushBuffer(BufferedIndexOutput.java:113)ERROR [scheduler_Worker-10] [org.quartz.core.ErrorLogger] schedulerError Job (DEFAULT.IndexQueueFlusher threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: com.atlassian.bonnie.LuceneException: com.atlassian.confluence.api.service.exceptions.ServiceException: Failed to process entries]
...
Caused by: com.atlassian.confluence.api.service.exceptions.ServiceException: Failed to process entries
...
Caused by: java.io.IOException: No space left on deviceWARN [scheduler_Worker-10] [search.lucene.tasks.AddDocumentIndexTask] perform Error getting document from searchable
java.io.IOException: No space left on device
WARN [scheduler_Worker-1] [search.lucene.tasks.AddDocumentIndexTask] perform Error getting document from searchable
java.io.FileNotFoundException: /opt/confluence/confluence-home/index/_z8ey.fdx (No space left on device)
at java.io.RandomAccessFile.open0(Native Method)
at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:466)Diagnosis
On Linux
- Check free disk space and inode count for <confluence-home>/index directory
# df -h
# df -i- Increase disk capacity or analyze which directories / files are large and can be removed
- If you ran out of inodes, you might need to migrate to another file system or partition, see 'Not enough space on disk' errors in Confluence 6.5 or later when disk space is available
On Windows
- Check free disk space from properties of the drive
- Check free disk space from PowerShell
Get-WmiObject -Class Win32_logicaldisk- Increase space on disk partition or free up space by removing unnecessary and large files
Cause
As documented in 'java.io.IOException No space left on device' due to disk space KB, "No space left on device" exception indicates that there is not enough space left for Confluence to write inside <confluence-home> directory.
Besides failures to generate backups, or save attachments, indexing tasks will also fail .
Solution
Confluence stores it's index locally, inside home directory, and it needs to have enough free space allocated to continue indexing content.
After enough space is freed, rebuild index from scratch by following one of the bellow procedures:
- How to Rebuild the Content Indexes From Scratch on Confluence Data Center
- How to Rebuild the Content Indexes From Scratch on Confluence Server
Lucene index must be stored locally on the server and can't be stored on NFS mount as outlined in Health Check: Lucene index files location