Bamboo Database Corruption(HSQL) - "APPARENT DEADLOCK!!!"
Symptoms
This KB only applies if you're using Embedded HSQL Database.
Bamboo Service suddenly gets shut down, and after starting the service it shows the following in the logs:
2012-04-27 21:04:20,865 FATAL [WrapperSimpleAppMain] [UpgradeLauncher] Failed to initialise Bamboo container
java.lang.IllegalStateException: Could not transition server from 'SETUP' to 'STARTING'
.....
Caused by: org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is net.sf.hibernate.exception.GenericJDBCException: Cannot open connection
.....
Caused by: com.mchange.v2.resourcepool.CannotAcquireResourceException: A ResourcePool could not acquire a resource from its primary factory or source.
.....
2012-04-27 21:19:35,474 WARN [Timer-0] [ThreadPoolAsynchronousRunner] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@ca1666 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
2012-04-27 21:19:35,474 WARN [Timer-0] [ThreadPoolAsynchronousRunner] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@ca1666 -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 0
Active Tasks:
Pending Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1e9c5a7
Pool thread stack traces:
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,]
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,]
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,]Cause
Cause1: The lck file wasn't cleaned out during shutdown
The embedded HSQL Database places a *.lck file in BAMBOO_HOME/database folder when the database is being used. Its possible that the file wasn't deleted.
Cause2: The embedded HSQL database in BAMBOO_HOME/database/ folder is corrupted.
HSQL database is not recommended for production Bamboo instances as its susceptible for failures. It is mentioned on the Supported platforms page.
Resolution
Resolution1: Clear out the lck file
Shutdown Bamboo and make sure there are no orphaned java/Bamboo processes running and delete the BAMBOO_HOME/database/*.lck file
Resolution1: Start with a new database
Install a fresh Bamboo instance and migrate to an external database
Last modified on Aug 25, 2022
Powered by Confluence and Scroll Viewport.