Backup client fails due to the error: One or more database connections were not closed within the allotted timeout
Symptoms
After executing the backup client, it fails with the exception below:
2014-10-04 18:45:26,343 ERROR (10%) A backup could not be created. Reason: The backup process on the server failed. Check the server logs for more details
com.atlassian.stash.internal.backup.client.BackupFailedException: The backup process on the server failed. Check the server logs for more details
at com.atlassian.stash.internal.backup.client.DefaultBackupClient.throwIfCancelledOrFailed(DefaultBackupClient.groovy:148) ~[stash-backup-client.jar:1.1.0]
at com.atlassian.stash.internal.backup.client.layout.DefaultStashHome.archive(DefaultStashHome.groovy:69) ~[stash-backup-core-1.1.0.jar:na]
... 7 more frames available in the log file
You can see the following entries in the atlassian-stash.log:
2014-10-04 18:43:28,175 WARN [threadpool:thread-2565052] ADMIN_SVC 1122x7007747x8 1lsjwd8 127.0.0.1 "POST /mvc/admin/backups HTTP/1.1" c.a.s.i.m.DefaultMaintenanceTaskMonitor BACKUP maintenance has failed (Cause: BackupException: A backup file could not be created.)
com.atlassian.stash.internal.backup.BackupException: A backup file could not be created.
at com.atlassian.stash.internal.maintenance.backup.BackupPhase.run(BackupPhase.java:78) ~[stash-service-impl-2.10.1.jar:na]
at com.atlassian.stash.internal.maintenance.CompositeMaintenanceTask$Step.run(CompositeMaintenanceTask.java:130) ~[stash-service-impl-2.10.1.jar:na]
at com.atlassian.stash.internal.maintenance.CompositeMaintenanceTask.run(CompositeMaintenanceTask.java:69) ~[stash-service-impl-2.10.1.jar:na]
at com.atlassian.stash.internal.maintenance.MaintenanceModePhase.run(MaintenanceModePhase.java:43) ~[stash-service-impl-2.10.1.jar:na]
at com.atlassian.stash.internal.maintenance.backup.BackupTask.run(BackupTask.java:93) ~[stash-service-impl-2.10.1.jar:na]
at com.atlassian.stash.internal.maintenance.DefaultMaintenanceTaskMonitor.run(DefaultMaintenanceTaskMonitor.java:172) ~[stash-service-impl-2.10.1.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_25]
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) ~[na:1.7.0_25]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_25]
at com.atlassian.stash.internal.concurrent.StateTransferringExecutor$StateTransferringRunnable.run(StateTransferringExecutor.java:69) ~[stash-platform-2.10.1.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_25]
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) ~[na:1.7.0_25]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_25]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source) ~[na:1.7.0_25]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[na:1.7.0_25]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:1.7.0_25]
at java.lang.Thread.run(Unknown Source) ~[na:1.7.0_25]
... 1 frame trimmed
Caused by: com.atlassian.stash.internal.migration.MigrationException: One or more database connections were not closed within the allotted timeout. To prevent corruption due to inconsistent state, the migration has been aborted. Please try your migration again when the system is under less load.
at com.atlassian.stash.internal.maintenance.LatchAndDrainDatabaseStep.run(LatchAndDrainDatabaseStep.java:74) ~[stash-service-impl-2.10.1.jar:na]
at com.atlassian.stash.internal.maintenance.CompositeMaintenanceTask$Step.run(CompositeMaintenanceTask.java:130) ~[stash-service-impl-2.10.1.jar:na]
at com.atlassian.stash.internal.maintenance.CompositeMaintenanceTask.run(CompositeMaintenanceTask.java:69) ~[stash-service-impl-2.10.1.jar:na]
at com.atlassian.stash.internal.maintenance.backup.BackupPhase.run(BackupPhase.java:74) ~[stash-service-impl-2.10.1.jar:na]
... 17 common frames omittedCause
A plugin is holding on to the database connections, not allowing the backup to be completed by the client.
Diagnosis
Do you recall installing any plugin (3rd party or custom) at the time you started getting backup failures? If you do, you could try disabling it and retrying the backup.
I don't remember installing a 3rd party plugin. Maybe someone else did it. How can I test for that? Use UPM Safe mode (link below):
- UPM safe mode: running your application in 'safe mode' disables all user installed add-ons at once. All add-ons that were disabled when you entered Safe Mode will be re-enabled when you exit Safe Mode. You can try this option and, while your application is on safe mode, you can trigger a backup to prove an add-on (plugin) is causing the issue. If that works, we should work out which plugin is causing the issue by reenabling each one of them back one by one until we find it out. There is no way we can see that in the log. I'd recommend identifying that as it'd lead us to the root cause. See also how to programmatically perform this action:
Resolution
There are 2 approaches to solve this:
- You could uninstall the plugin causing this issue.
- Or it could also be that the plugin is only taking too long to let go of the database connections and it needs more time to do so. You could configure the Stash server to wait longer before it times out the backup process and see if that works. As described on Stash config properties, you can configure the following parameter:
backup.drain.database.timeout=300from the 60 default value. To configure that:- Stop Stash
- Edit the
stash-config.propertiesfile contained in the shared folder of your$STASH_HOME. Create the file if it doesn't exist. - Add
backup.drain.database.timeout=300 - Start Stash
- Retry the backup
Last modified on Feb 26, 2016
Powered by Confluence and Scroll Viewport.