Database Corruption - Bamboo server fails to start - error 503 - clearcase plugin
Symptoms
Bamboo server fails to start and the logs show this:
2012-06-29 17:47:25,904 DEBUG [main] [DefaultPluginManager] Plugin [com.atlassian.bamboo.plugins.clearcase] is disabled.
2012-06-29 17:47:25,906 DEBUG [AtlassianEvent::0-BAM::EVENTS:pool-1-thread-14] [BasicResourcePool] trace com.mchange.v2.resourcepool.BasicResourcePool@23f9e6e5 [managed: 8, unused: 6, excluded: 0] (e.g. com.mchange.v2.c3p0.impl.NewPooledConnection@60f00e0f)
2012-06-29 17:47:25,906 DEBUG [AtlassianEvent::0-BAM::EVENTS:pool-1-thread-14] [BasicResourcePool] trace com.mchange.v2.resourcepool.BasicResourcePool@23f9e6e5 [managed: 8, unused: 6, excluded: 0] (e.g. com.mchange.v2.c3p0.impl.NewPooledConnection@60f00e0f)
2012-06-29 17:47:25,907 ERROR [main] [DefaultRepositoryManager] Could not find repository plugin with key 'com.atlassian.bamboo.plugin.system.repository:nullRepository'. This build will be unable to build.
2012-06-29 17:47:25,909 DEBUG [main] [BasicResourcePool] trace com.mchange.v2.resourcepool.BasicResourcePool@23f9e6e5 [managed: 8, unused: 6, excluded: 0] (e.g. com.mchange.v2.c3p0.impl.NewPooledConnection@60f00e0f)
2012-06-29 17:47:25,911 FATAL [main] [BambooContainer] Cannot start bamboo
com.google.common.collect.ComputationException: com.atlassian.util.concurrent.LazyReference$InitializationException: java.lang.IllegalStateException: Cannot load repository. Plugins may have been disabled.
at com.google.common.collect.ComputingConcurrentHashMap$ComputingSegment.compute(ComputingConcurrentHashMap.java:167)
at com.google.common.collect.ComputingConcurrentHashMap$ComputingSegment.compute(ComputingConcurrentHashMap.java:116)
at com.google.common.collect.ComputingConcurrentHashMap.apply(ComputingConcurrentHashMap.java:67)
at com.google.common.collect.MapMaker$ComputingMapAdapter.get(MapMaker.java:623)
at com.atlassian.bamboo.build.CachedBuildDefinitionManagerImpl.getBuildDefinition(CachedBuildDefinitionManagerImpl.java:59)
...
Caused by: com.atlassian.util.concurrent.LazyReference$InitializationException: java.lang.IllegalStateException: Cannot load repository. Plugins may have been disabled.
at com.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:152)
at com.atlassian.util.concurrent.LazyReference.get(LazyReference.java:115)
at com.atlassian.bamboo.repository.RepositoryDefinitionImpl.getRepository(RepositoryDefinitionImpl.java:153)
at com.atlassian.bamboo.build.DefaultBuildDefinitionManager.getBuildDefinition(DefaultBuildDefinitionManager.java:79)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
...
Caused by: java.lang.IllegalStateException: Cannot load repository. Plugins may have been disabled.
at com.atlassian.bamboo.repository.RepositoryReference.create(RepositoryReference.java:31)
at com.atlassian.bamboo.repository.RepositoryReference.create(RepositoryReference.java:8)
at com.atlassian.util.concurrent.LazyReference$Sync.run(LazyReference.java:326)
at com.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:146)
... 36 more
Cause
The cause of this type of corruption is hard to predict as we see that clearcase plugin is involved (which is not supported by Atlassian). This problem might be a result of customization (changing the source code) of the clearcase plugin, or when you configure your plans to use clearcase, but the plugin gets removed or for some other reason Bamboo cannot find it to use clearcase repository. Somehow "nullRepository" gets to the database and that's what we see in the logs - com.atlassian.bamboo.plugin.system.repository:nullRepository.
Resolution
The resolution of this type of database corruptions and clearcase plugin issue can be achieved in 2 steps:
- moving the clearcase plugin BAMBOO_INSTALL/atlassian-bamboo/WEB-INF/lib directory.
- running SQL queries to fix the corrupted records in the database
For the first step is easy to test. By the way, please note that moving the clearcase plugin to to BAMBOO_HOME/plugins directory might mot help. Make sure that you have only one copy of that plugin in your Bamboo instance.
The second step includes some analyses and investigation.
Try to recall the names of the plans that might use clearcase repositories, and try to find those rows from VCS_LOCATION table:
SELECT * from vcs_location WHERE XML_DEFINITION_DATA like '%_name_of_a_plan_%'After finding out the corrupted rows (containing 'com.atlassian.bamboo.plugin.system.repository:nullRepository'), query your database and try to find an ordinary value for the same field. For instance, it might be 'com.atlassian.bamboo.plugins.atlassian-bamboo-plugin-git:git'. Next, you need to create a SQL query that will update the database and change the corrupted data. You might need to use a SQL query like the one below:
UPDATE vcs_location SET
PLUGIN_KEY='com.atlassian.bamboo.plugins.atlassian-bamboo-plugin-git:git'
WHERE PLUGIN_KEY like '%nullRepository%' In addition, here are some SQL queries that might be useful in the investigation process before running the update query mentioned above:
SELECT * FROM BUILD WHERE FULL_KEY like '%PLAN1-PROJ1%'
SELECT * FROM BUILD WHERE MARKED_FOR_DELETION=1;
select * from vcs_location where PLUGIN_KEY like '%nullRepository%'
SELECT * FROM PLAN_VCS_LOCATION, VCS_LOCATION, BUILD WHERE PLAN_VCS_LOCATION.VCS_LOCATION_ID=VCS_LOCATION.VCS_LOCATION_ID and PLAN_VCS_LOCATION.PLAN_ID in (SELECT BUILD_ID FROM BUILD WHERE FULL_KEY like '%PLAN1-PROJ1%') order by PLUGIN_KEY DESC
SELECT distinct BUILD.FULL_KEY FROM PLAN_VCS_LOCATION, VCS_LOCATION, BUILD WHERE PLAN_VCS_LOCATION.VCS_LOCATION_ID=VCS_LOCATION.VCS_LOCATION_ID and vcs_location.XML_DEFINITION_DATA like '%_name_of_a_plan_%'Please note that database corruption fixes are not standard and might be specific for each case. That's why make sure to backup your Bamboo instance before running any changes. After, shut down your Bamboo server, run the updates and start your instance to see if the changes helped to fix the problem.