Backup client or DIY backup fails due to the error: One or more database connections were not closed within the allotted timeout on Bitbucket Data Center

Related content

  • No related content found

Still need help?

The Atlassian Community is here for you.

Ask the community

Problem

After executing the backup client, it fails with the exception below:

2014-10-04 18:45:26,343 ERROR  (10%) A backup could not be created. Reason: The backup process on the server failed. Check the server logs for more details
com.atlassian.bitbucket.internal.backup.client.BackupFailedException: The backup process on the server failed. Check the server logs for more details
        at com.atlassian.bitbucket.internal.backup.client.DefaultBackupClient.throwIfCancelledOrFailed(DefaultBackupClient.groovy:148) ~[bitbucket-backup-client.jar:1.1.0]
        at com.atlassian.bitbucket.internal.backup.client.layout.DefaultBitbucket ServerHome.archive(DefaultBitbucket ServerHome.groovy:69) ~[bitbucket-backup-core-1.1.0.jar:na]
        ... 7 more frames available in the log file


You can see the following entries in the atlassian-bitbucket.log:

2014-10-04 18:43:28,175 WARN  [threadpool:thread-2565052] ADMIN_SVC 1122x7007747x8 1lsjwd8 127.0.0.1 "POST /mvc/admin/backups HTTP/1.1" c.a.s.i.m.DefaultMaintenanceTaskMonitor BACKUP maintenance has failed (Cause: BackupException: A backup file could not be created.)
com.atlassian.bitbucket.internal.backup.BackupException: A backup file could not be created.
        at com.atlassian.bitbucket.internal.maintenance.backup.BackupPhase.run(BackupPhase.java:78) ~[bitbucket-service-impl-2.10.1.jar:na]
        at com.atlassian.bitbucket.internal.maintenance.CompositeMaintenanceTask$Step.run(CompositeMaintenanceTask.java:130) ~[bitbucket-service-impl-2.10.1.jar:na]
        at com.atlassian.bitbucket.internal.maintenance.CompositeMaintenanceTask.run(CompositeMaintenanceTask.java:69) ~[bitbucket-service-impl-2.10.1.jar:na]
        at com.atlassian.bitbucket.internal.maintenance.MaintenanceModePhase.run(MaintenanceModePhase.java:43) ~[bitbucket-service-impl-2.10.1.jar:na]
        at com.atlassian.bitbucket.internal.maintenance.backup.BackupTask.run(BackupTask.java:93) ~[bitbucket-service-impl-2.10.1.jar:na]
        at com.atlassian.bitbucket.internal.maintenance.DefaultMaintenanceTaskMonitor.run(DefaultMaintenanceTaskMonitor.java:172) ~[bitbucket-service-impl-2.10.1.jar:na]
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_25]
        at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) ~[na:1.7.0_25]
        at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_25]
        at com.atlassian.bitbucket.internal.concurrent.StateTransferringExecutor$StateTransferringRunnable.run(StateTransferringExecutor.java:69) ~[bitbucket-platform-2.10.1.jar:na]
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_25]
        at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) ~[na:1.7.0_25]
        at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_25]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source) ~[na:1.7.0_25]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[na:1.7.0_25]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:1.7.0_25]
        at java.lang.Thread.run(Unknown Source) ~[na:1.7.0_25]
        ... 1 frame trimmed
Caused by: com.atlassian.bitbucket.internal.migration.MigrationException: One or more database connections were not closed within the allotted timeout. To prevent corruption due to inconsistent state, the migration has been aborted. Please try your migration again when the system is under less load.
        at com.atlassian.bitbucket.internal.maintenance.LatchAndDrainDatabaseStep.run(LatchAndDrainDatabaseStep.java:74) ~[bitbucket-service-impl-2.10.1.jar:na]
        at com.atlassian.bitbucket.internal.maintenance.CompositeMaintenanceTask$Step.run(CompositeMaintenanceTask.java:130) ~[bitbucket-service-impl-2.10.1.jar:na]
        at com.atlassian.bitbucket.internal.maintenance.CompositeMaintenanceTask.run(CompositeMaintenanceTask.java:69) ~[bitbucket-service-impl-2.10.1.jar:na]
        at com.atlassian.bitbucket.internal.maintenance.backup.BackupPhase.run(BackupPhase.java:74) ~[bitbucket-service-impl-2.10.1.jar:na]
        ... 17 common frames omitted

Cause

A plugin is holding on to the database connections, not allowing the backup to be completed by the client.

We are also aware of an issue with the Database Force Drain mechanism, which causes the Force draining of database connections to fail, resulting in the backup process failing with the above mentioned error. BSERV-14374 - Getting issue details... STATUS

Diagnosis

Do you recall installing any plugin (3rd party or custom) at the time you started getting backup failures? If you do, you could try disabling it and retrying the backup.

I don't remember installing a 3rd party plugin. Maybe someone else did it. How can I test for that? Use UPM Safe mode (link below):

  • UPM safe mode: running your application in 'safe mode' disables all user installed add-ons at once. All add-ons that were disabled when you entered Safe Mode will be re-enabled when you exit Safe Mode. You can try this option and, while your application is on safe mode, you can trigger a backup to prove an add-on (plugin) is causing the issue. If that works, we should work out which plugin is causing the issue by reenabling each one of them back one by one until we find it out. There is no way we can see that in the log. I'd recommend identifying that as it'd lead us to the root cause. See also how to programmatically perform this action:

Resolution

 There are 2 approaches to solve this:

  • You could uninstall the plugin causing this issue.
  • Or it could also be that the plugin is only taking too long to let go of the database connections and it needs more time to do so. You could configure the Bitbucket Server instance to wait longer before it times out the backup process and see if that works. As described on Bitbucket Server config properties, you can configure the following parameter: backup.drain.db.timeout=300 from the 60 default value. To configure that:
    • Stop Bitbucket Server
    • Edit the bitbucket.properties file contained in the shared folder of your $BITBUCKET_HOME. Create the file if it doesn't exist.
    • Add backup.drain.db.timeout=300
    • Start Bitbucket Server
    • Retry the backup

If the resolution does not work, please get in touch with Atlassian Support.

Last modified on Oct 5, 2024

Was this helpful?

Yes
No
Provide feedback about this article

Related content

  • No related content found
Powered by Confluence and Scroll Viewport.