Index snapshot failed due to insufficient space on the storage
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Problem
Copying index from node to node is failing with message No space left on device. This is also applicatable for creation of index recovery snapshot.
The following appears in the log
2016-06-10 08:47:00,311 ClusterMessageHandlerServiceThread:thread-1 ERROR [atlassian.jira.cluster.OfBizMessageHandlerService] There was a problem handling a cluster message
java.lang.RuntimeException: java.io.IOException: No space left on device
at com.atlassian.jira.index.ha.IndexUtils.copySpecificIndex(IndexUtils.java:256)
at com.atlassian.jira.index.ha.IndexUtils.copyIndexes(IndexUtils.java:173)
at com.atlassian.jira.index.ha.IndexUtils.takeIndexSnapshot(IndexUtils.java:127)
at com.atlassian.jira.index.ha.DefaultIndexCopyService$MessageConsumer.copyIndex(DefaultIndexCopyService.java:164)
at com.atlassian.jira.index.ha.DefaultIndexCopyService$MessageConsumer.backupIndex(DefaultIndexCopyService.java:142)
at com.atlassian.jira.index.ha.DefaultIndexCopyService$MessageConsumer.backupIndex(DefaultIndexCopyService.java:124)
...
Caused by: java.io.IOException: No space left on device
at java.io.RandomAccessFile.writeBytes(Native Method)
at java.io.RandomAccessFile.write(RandomAccessFile.java:525)
at org.apache.lucene.store.FSDirectory$FSIndexOutput.flushBuffer(FSDirectory.java:448)
at org.apache.lucene.store.BufferedIndexOutput.flushBuffer(BufferedIndexOutput.java:99)
at org.apache.lucene.store.BufferedIndexOutput.flush(BufferedIndexOutput.java:88)
at org.apache.lucene.store.BufferedIndexOutput.close(BufferedIndexOutput.java:113)
at org.apache.lucene.store.FSDirectory$FSIndexOutput.close(FSDirectory.java:457)
at org.apache.lucene.util.IOUtils.closeSafely(IOUtils.java:118)
at org.apache.lucene.index.FieldsWriter.close(FieldsWriter.java:127)
at org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:251)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:107)
at org.apache.lucene.index.IndexWriter.addIndexes(IndexWriter.java:3080)
at com.atlassian.jira.index.ha.IndexUtils.copySpecificIndex(IndexUtils.java:247)
... 17 moreDiagnosis
Environment
- Different partitions for <JIRA_INSTALL> and <JIRA_HOME> are used.
Cause
During creating the snapshot of the index it is created in java.io.tmpdir directory, by default it set to CATALINA_BASE/temp (<JIRA_INSTALL>/temp).
Workaround
- Set in bin/setenv.sh CATALINA_TMPDIR variable and point to directory at <JIRA_HOME> partition.
- You can use a Symbolic Link in a *nix environment.
Resolution
Please check that partition for <JIRA_INSTALL> has enough space.