Jira Disaster Recovery Fail to Recover Index on Start up
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Problem
When performing Disaster Recovery for Jira, the Disaster Recovery node did not automatically restore an existing index using available Index Snapshot on Jira Startup.
Disaster Recovery Mode
There are three modes of Disaster Recovery
- PRIMARY - the main Jira instance, which has no special DR configuration set
- SECONDARY - the recovery instance as indicated by the
disaster.recoveryproperty being true - COLD - a secondary instance on initialization. This indicates an instance that is currently doing work to recover after which it will report itself as SECONDARY.
Diagnosis
Environment
Jira Data Center
Disaster Recovery enabled with
disaster.recovery=true
Diagnostic Steps
Clarify and understand if the correct steps have been performed. To copy the Index snapshot to the respective location.
- Verify that the Index snapshot file has a naming syntax of
IndexSnapshot_* - Review the Startup logs for traces of cold or secondary instance
Problem 1
The following appears in the atlassian-jira.log
2019-05-15 08:31:35,294 localhost-startStop-1 ERROR [c.a.j.index.ha.DisasterRecoveryLauncher] No valid index backup found in '/nfs/sharedhome/import/indexsnapshots'Problem 2
The following appears in the atlassian-jira.log
2019-05-16 11:37:56,297 localhost-startStop-1 INFO [c.a.j.index.ha.DisasterRecoveryLauncher] Starting secondary instanceCause
The following are the list of causes based on the problems above.
Cause 1
The Index snapshot files are missing from the location of /<sharedhome>/import/indexsnapshots/ directory
Cause 2
The startup is not the first startup and is already in a Secondary mode. The restore of Index snapshot from /<sharedhome>/import/indexsnapshots/ only triggered on the first startup of Disaster Recovery mode, which is as a Cold instance.
Resolution
Resolution 1
Refer the steps of Performing the disaster recovery testing and follow the following
- Ensure that the new database is ready, with the latest snapshot and no replication.
- Ensure that you have a copy of Jira on a clean server with the proper
dbconfig.xmlconnection. - Ensure that you have
Jira_HOMEmapped as it was in the standby instance, but in the test server. It is important to have the latest index snapshot inJira_HOME/export/folder.indexsnapshots - Ensure there is no folder of
/<sharedhome>/indexarchive/exist, if yes move out of the shared directory. - Disable email.
- Start Jira in Disaster Recovery mode, by starting it with the following parameter: disaster.recovery=true.
Resolution 2
- Ensure there is no folder of
/<sharedhome>/indexarchive/exist, if yes move out of the shared directory. - Start Jira in Disaster Recovery mode, by starting it with the following parameter: disaster.recovery=true.