Scheduled Backup Corrupted Randomly

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform Notice: Server and Data Center Only - This article only applies to Atlassian products on the server and data center platforms.

Problem

Some scheduled backups are randomly corrupted, but the log indicates that the backup service was successfully run and completed.

Diagnosis & Symptoms

  • Corruption happens randomly
  • The backup archive fails to extract with any tools

    root@localhost:/# unzip 2016-Apr-18--0900.zip 
    Archive:  2016-Apr-18--0900.zip
      inflating: entities.xml            
      error:  invalid compressed data to inflate
    file #2:  bad zipfile offset (local header sig):  603570562
  • Generating the XML backup manually in an attempt to replicate the issue never gets corrupted
  • Server performance is healthy
  • No other external backups are running at the same time which might potentially contribute to this issue, including but not limited to database backup or VM image replication
  • Enabling DEBUG logging did not return any useful information, in fact when the corrupted backup file was generated, the log is showing that the backup was successfully completed.

    2016-04-24 21:00:02,451 atlassian-scheduler-quartz1.clustered_Worker-4 DEBUG anonymous    Backup Service [jira.bc.dataimport.DefaultExportService] numberOfEntities = 194
    2016-04-24 21:00:02,455 atlassian-scheduler-quartz1.clustered_Worker-3 DEBUG anonymous    ?????? ???? [jira.bc.dataimport.DefaultExportService] numberOfEntities = 194
    2016-04-24 21:04:50,788 atlassian-scheduler-quartz1.clustered_Worker-3 INFO anonymous    ?????? ???? [jira.bc.dataimport.DefaultExportService] Data export completed in 288333ms. Wrote 12101045 entities to export in memory.
    2016-04-24 21:04:50,791 atlassian-scheduler-quartz1.clustered_Worker-3 INFO anonymous    ?????? ???? [jira.bc.dataimport.DefaultExportService] Attempting to save the Active Objects Backup
    2016-04-24 21:04:57,153 atlassian-scheduler-quartz1.clustered_Worker-3 INFO anonymous    ?????? ???? [jira.bc.dataimport.DefaultExportService] Finished saving the Active Objects Backup
    2016-04-24 21:05:15,590 atlassian-scheduler-quartz1.clustered_Worker-4 INFO anonymous    Backup Service [jira.bc.dataimport.DefaultExportService] Data export completed in 313139ms. Wrote 12101045 entities to export in memory.
    2016-04-24 21:05:15,595 atlassian-scheduler-quartz1.clustered_Worker-4 INFO anonymous    Backup Service [jira.bc.dataimport.DefaultExportService] Attempting to save the Active Objects Backup
    2016-04-24 21:05:21,350 atlassian-scheduler-quartz1.clustered_Worker-4 INFO anonymous    Backup Service [jira.bc.dataimport.DefaultExportService] Finished saving the Active Objects Backup

Cause

There are two backup service scheduled at the same time, as the backup file naming scheme is YYYY-MMM-DD–HHMM.zip, it is possible that both services are attempting to write to the same file thus corrupting the file. The logs above are showing two different worker threads are attempting to generate the backup at the same time.

Resolution

Remove the duplicate backup service which is scheduled at the same time.  Please refer to Automating Jira Application Backups for configuration details

   

 

Last modified on Nov 2, 2018

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.