Problem

Some scheduled backups are randomly corrupted, but the log indicates that the backup service was successfully run and completed.

Diagnosis & Symptoms

• Corruption happens randomly

• The backup archive fails to extract with any tools

  root@localhost:/# unzip 2016-Apr-18--0900.zip 
  Archive:  2016-Apr-18--0900.zip
    inflating: entities.xml            
    error:  invalid compressed data to inflate
  file #2:  bad zipfile offset (local header sig):  603570562

• Generating the XML backup manually in an attempt to replicate the issue never gets corrupted
• Server performance is healthy
• No other external backups are running at the same time which might potentially contribute to this issue, including but not limited to database backup or VM image replication

• Enabling DEBUG logging did not return any useful information, in fact when the corrupted backup file was generated, the log is showing that the backup was successfully completed.

  2016-04-24 21:00:02,451 atlassian-scheduler-quartz1.clustered_Worker-4 DEBUG anonymous    Backup Service [jira.bc.dataimport.DefaultExportService] numberOfEntities = 194
  2016-04-24 21:00:02,455 atlassian-scheduler-quartz1.clustered_Worker-3 DEBUG anonymous    ?????? ???? [jira.bc.dataimport.DefaultExportService] numberOfEntities = 194
  2016-04-24 21:04:50,788 atlassian-scheduler-quartz1.clustered_Worker-3 INFO anonymous    ?????? ???? [jira.bc.dataimport.DefaultExportService] Data export completed in 288333ms. Wrote 12101045 entities to export in memory.
  2016-04-24 21:04:50,791 atlassian-scheduler-quartz1.clustered_Worker-3 INFO anonymous    ?????? ???? [jira.bc.dataimport.DefaultExportService] Attempting to save the Active Objects Backup
  2016-04-24 21:04:57,153 atlassian-scheduler-quartz1.clustered_Worker-3 INFO anonymous    ?????? ???? [jira.bc.dataimport.DefaultExportService] Finished saving the Active Objects Backup
  2016-04-24 21:05:15,590 atlassian-scheduler-quartz1.clustered_Worker-4 INFO anonymous    Backup Service [jira.bc.dataimport.DefaultExportService] Data export completed in 313139ms. Wrote 12101045 entities to export in memory.
  2016-04-24 21:05:15,595 atlassian-scheduler-quartz1.clustered_Worker-4 INFO anonymous    Backup Service [jira.bc.dataimport.DefaultExportService] Attempting to save the Active Objects Backup
  2016-04-24 21:05:21,350 atlassian-scheduler-quartz1.clustered_Worker-4 INFO anonymous    Backup Service [jira.bc.dataimport.DefaultExportService] Finished saving the Active Objects Backup

Cause

There are two backup service scheduled at the same time, as the backup file naming scheme is YYYY-MMM-DD–HHMM.zip, it is possible that both services are attempting to write to the same file thus corrupting the file. The logs above are showing two different worker threads are attempting to generate the backup at the same time.

Resolution

Remove the duplicate backup service which is scheduled at the same time.  Please refer to Automating Jira Application Backups for configuration details