Jira not able to recover index snapshot due to "invalid stored block lengths" error
Platform Notice: Data Center - This article applies to Atlassian products on the Data Center platform.
Note that this knowledge base article was created for the Data Center version of the product. Data Center knowledge base articles for non-Data Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
In some very specific cases, Jira is not able to recover an index snapshot and the nodes can go into MAINTENANCE mode and become unusable.
Environment
9.x
Diagnosis
The following error is found in the atlassian-jira.log:
2024-11-14 03:07:25,780-0800 main ERROR [c.a.jira.cluster.DefaultClusterManager] Current node: node1. Couldn't recover index even though it had been found in shared. Current list of other nodes: [node2, node3, node4]
com.atlassian.jira.issue.index.IndexException: java.util.zip.ZipException: invalid stored block lengths
at com.atlassian.jira.index.ha.DefaultIndexRecoveryManager.recoverIndexFromBackup(DefaultIndexRecoveryManager.java:181)
at com.atlassian.jira.index.ha.DefaultIndexRecoveryManager.safeRecoverIndexFromBackup(DefaultIndexRecoveryManager.java:138)
at com.atlassian.jira.index.DefaultIndexFetcher.recoverIndexFromMostRecentSnapshot(DefaultIndexFetcher.java:108)
at com.atlassian.jira.cluster.DefaultClusterManager.pickIndexSnapshotFromSharedHome(DefaultClusterManager.java:442)
at com.atlassian.jira.cluster.DefaultClusterManager.checkIndexOnStart(DefaultClusterManager.java:224)
at com.atlassian.jira.startup.ClusteringLauncher.clusterSynchronizedCheckIndex(ClusteringLauncher.java:99)
at com.atlassian.jira.startup.ClusteringLauncher.start(ClusteringLauncher.java:130)
at com.atlassian.jira.startup.DefaultJiraLauncher.postDBActivated(DefaultJiraLauncher.java:173)
at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDbLaunch$2(DefaultJiraLauncher.java:150)Cause
The exact cause for this error is unknown. However, based on research done, one of the following could be the cause:
- The zip file containing the index snapshot is corrupted,
- The target node location path (local index path) does not have proper permission to restore the index, or
- Jira cache is corrupted and needs to be cleared
Solution
- Make sure that the node local home has proper permissions for the Jira user to write into the directory (compare the permissions with the other nodes and make sure they are the same)
- Try to clear the affected node's plugin cache, then restart the node and see if the error is gone.
- If that does not help:
- Back up the snapshot compressed files from the shared home, then delete the old snapshots from the snapshot directory in the Shared home.
- In Jira, go to System > Indexing > Create index snapshot. This will create a fresh, new snapshot from a working node
- Now restart the affected node, check if the index gets picked up and restored, and if Jira starts up properly.
If the Solution steps do not help, please raise a new case with Atlassian Support.