Problem

Bitbucket Server fails to startup.

The following message is displayed on the UI:

The following problem occurred which prevents Atlassian Bitbucket Server from starting correctly:


SpringMVC dispatcher [springMvc] could not be started

Cause

There are different root causes for this problem.

1. Cause #1:
   The following exception is reported in the atlassian-bitbucket.log file:

   catalina.out

   15:08:16,767 |-ERROR in ch.qos.logback.core.rolling.RollingFileAppender[bitbucket.application] - openFile(/var/atlassian/application-data/bitbucket/log/atlassian-bitbucket.log,true) call failed. java.io.FileNotFoundException: /var/atlassian/application-data/bitbucket/log/atlassian-bitbucket.log (Permission denied)
   	at java.io.FileNotFoundException: /var/atlassian/application-data/bitbucket/log/atlassian-bitbucket.log (Permission denied)
   	at 	at java.io.FileOutputStream.open(Native Method)
   	at 	at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
   	at 	at ch.qos.logback.core.recovery.ResilientFileOutputStream.<init>(ResilientFileOutputStream.java:28)
   	at 	at ch.qos.logback.core.FileAppender.openFile(FileAppender.java:149)
   	at 	at ch.qos.logback.core.FileAppender.start(FileAppender.java:108)
   	at 	at ch.qos.logback.core.rolling.RollingFileAppender.start(RollingFileAppender.java:86)
   	at 	at ch.qos.logback.core.joran.action.AppenderAction.end(AppenderAction.java:96)
   	at 	at ch.qos.logback.core.joran.spi.Interpreter.callEndAction(Interpreter.java:317)
   	at 	at ch.qos.logback.core.joran.spi.Interpreter.endElement(Interpreter.java:196)
   	at 	at ch.qos.logback.core.joran.spi.Interpreter.endElement(Interpreter.java:182)
   	at 	at ch.qos.logback.core.joran.spi.EventPlayer.play(EventPlayer.java:62)
   	at 	at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:149)
   	at 	at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:135)
   	at 	at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:99)
   	at 	at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:49)
   	at 	at ch.qos.logback.classic.util.ContextInitializer.configureByResource(ContextInitializer.java:75)
   	at 	at ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:148)
   	at 	at org.slf4j.impl.StaticLoggerBinder.init(StaticLoggerBinder.java:85)
   	at 	at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:55)
   	at 	at org.slf4j.LoggerFactory.bind(LoggerFactory.java:129)
   	at 	at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:108)
   	at 	at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:302)
   	at 	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:276)
   	at 	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:288)
   	at 	at com.atlassian.bitbucket.internal.lifecycle.DefaultStartupManager.<clinit>(DefaultStartupManager.java:18)
   	at 	at com.atlassian.bitbucket.internal.spring.Bitbucket ServerWebApplicationInitializer.onStartup(Bitbucket ServerWebApplicationInitializer.java:65)
   	at 	at org.springframework.web.SpringServletContainerInitializer.onStartup(SpringServletContainerInitializer.java:175)
   	at 	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5423)
   	at 	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
   	at 	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1559)
   	at 	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1549)
   	at 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   	at 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   	at 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   	at 	at java.lang.Thread.run(Thread.java:744)

   In this case, /var/atlassian/application-data/bitbucket/, which is being used as Bitbucket Server Home Directory, is not accessible to the user running Bitbucket Server. The permissions on the directories contained within <Bitbucket home directory> might be mixed where the user running Bitbucket Server is not the same.

   For further steps on how to diagnose this please see - How to View Detailed permissions for Bitbucket Server directory
   
   

2. Cause #2:
   The following exception is reported in the atlassian-bitbucket.log file:

   2018-07-08 01:01:11,128 ERROR [spring-startup]  c.a.j.s.w.s.JohnsonDispatcherServlet SpringMVC dispatcher [springMvc] could not be started
   org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'crowdAliasDao': Unsatisfied dependency expressed through method 'setSessionFactory' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in class path resource [stash-context.xml]: Invocation of init method failed; nested exception is liquibase.exception.DatabaseException: java.sql.SQLTransientConnectionException: bitbucket - Connection is not available, request timed out after 15000ms.
   	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:667)
   	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
   	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)
   	at javax.servlet.GenericServlet.init(GenericServlet.java:158)
   	at java.lang.Thread.run(Thread.java:748)
   	... 18 frames trimmed
   Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in class path resource [stash-context.xml]: Invocation of init method failed; nested exception is liquibase.exception.DatabaseException: java.sql.SQLTransientConnectionException: bitbucket - Connection is not available, request timed out after 15000ms.
   	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1628)
   	... 5 common frames omitted
   Caused by: liquibase.exception.DatabaseException: java.sql.SQLTransientConnectionException: bitbucket - Connection is not available, request timed out after 15000ms.
   	at liquibase.database.jvm.JdbcConnection.getDatabaseProductName(JdbcConnection.java:43)
   	at liquibase.database.core.OracleDatabase.isCorrectDatabaseImplementation(OracleDatabase.java:210)
   	at liquibase.database.DatabaseFactory.findCorrectDatabaseImplementation(DatabaseFactory.java:111)
   	at liquibase.integration.spring.SpringLiquibase.createDatabase(SpringLiquibase.java:471)
   	at liquibase.database.ExtendedSpringLiquibase.createDatabase(ExtendedSpringLiquibase.java:40)
   	at liquibase.integration.spring.SpringLiquibase.createLiquibase(SpringLiquibase.java:437)
   	at liquibase.database.ExtendedSpringLiquibase.afterPropertiesSet(ExtendedSpringLiquibase.java:32)
   	... 5 common frames omitted
   Caused by: java.sql.SQLTransientConnectionException: bitbucket - Connection is not available, request timed out after 15000ms.
   	at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:602)
   	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:195)
   	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:145)
   	at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:85)
   	at com.atlassian.stash.internal.hikari.ExtendedHikariDataSource.getConnection(ExtendedHikariDataSource.java:125)
   	at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.getTargetConnection(LazyConnectionDataSourceProxy.java:403)
   	at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.invoke(LazyConnectionDataSourceProxy.java:376)
   	at liquibase.database.jvm.JdbcConnection.getDatabaseProductName(JdbcConnection.java:41)
   	... 11 common frames omitted

   In this case, the database used by Bitbucket Server is unavailable.
   
   
   

3. Cause #3

   The following exception is reported in atlassian-bitbucket.log

   
   

    WARN  [spring-startup]  o.s.w.c.s.XmlWebApplicationContext Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'crowdAliasDao': Unsatisfied dependency expressed through method 'setSessionFactory' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sharedHomeLockAcquirer' defined in class path resource [stash-context.xml]: Cannot resolve reference to bean 'licenseHelper' while setting constructor argument; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'licenseHelper' defined in URL [jar:file:/opt/atlassian/bitbucket/7.6.8/app/WEB-INF/lib/bitbucket-service-impl-7.6.8.jar!/com/atlassian/stash/internal/license/DefaultLicenseHelper.class]: Unsatisfied dependency expressed through constructor parameter 2; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'bootstrapLiquibase' defined in class path resource [stash-context.xml]: Invocation of init method failed; nested exception is java.lang.ClassCastException: java.time.LocalDateTime cannot be cast to java.lang.String
   2021-07-30 14:52:40,809 INFO  [spring-startup]  c.a.s.internal.home.HomeLockAcquirer Releasing lock on /var/atlassian/application-data/bitbucket
   2021-07-30 14:52:40,818 ERROR [spring-startup]  c.a.j.s.w.s.JohnsonDispatcherServlet SpringMVC dispatcher [springMvc] could not be started
   org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'crowdAliasDao': Unsatisfied dependency expressed through method 'setSessionFactory' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sharedHomeLockAcquirer' defined in class path resource [stash-context.xml]: Cannot resolve reference to bean 'licenseHelper' while setting constructor argument; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'licenseHelper' defined in URL [jar:file:/opt/atlassian/bitbucket/7.6.8/app/WEB-INF/lib/bitbucket-service-impl-7.6.8.jar!/com/atlassian/stash/internal/license/DefaultLicenseHelper.class]: Unsatisfied dependency expressed through constructor parameter 2; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'bootstrapLiquibase' defined in class path resource [stash-context.xml]: Invocation of init method failed; nested exception is java.lang.ClassCastException: java.time.LocalDateTime cannot be cast to java.lang.String
   	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:723)
   	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:878)
   	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
   	at javax.servlet.GenericServlet.init(GenericServlet.java:158)
   	at java.lang.Thread.run(Thread.java:748)
   	... 18 frames trimmed
   Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sharedHomeLockAcquirer' defined in class path resource [stash-context.xml]: Cannot resolve reference to bean 'licenseHelper' while setting constructor argument; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'licenseHelper' defined in URL [jar:file:/opt/atlassian/bitbucket/7.6.8/app/WEB-INF/lib/bitbucket-service-impl-7.6.8.jar!/com/atlassian/stash/internal/license/DefaultLicenseHelper.class]: Unsatisfied dependency expressed through constructor parameter 2; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'bootstrapLiquibase' defined in class path resource [stash-context.xml]: Invocation of init method failed; nested exception is java.lang.ClassCastException: java.time.LocalDateTime cannot be cast to java.lang.String
   	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:342)
   	... 5 common frames omitted
   Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'licenseHelper' defined in URL [jar:file:/opt/atlassian/bitbucket/7.6.8/app/WEB-INF/lib/bitbucket-service-impl-7.6.8.jar!/com/atlassian/stash/internal/license/DefaultLicenseHelper.class]: Unsatisfied dependency expressed through constructor parameter 2; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'bootstrapLiquibase' defined in class path resource [stash-context.xml]: Invocation of init method failed; nested exception is java.lang.ClassCastException: java.time.LocalDateTime cannot be cast to java.lang.String
   	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:798)
   	... 5 common frames omitted
   Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'bootstrapLiquibase' defined in class path resource [stash-context.xml]: Invocation of init method failed; nested exception is java.lang.ClassCastException: java.time.LocalDateTime cannot be cast to java.lang.String
   	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1796)
   	... 5 common frames omitted
   Caused by: java.lang.ClassCastException: java.time.LocalDateTime cannot be cast to java.lang.String
   	at liquibase.changelog.StandardChangeLogHistoryService.getRanChangeSets(StandardChangeLogHistoryService.java:324)
   	at liquibase.changelog.AbstractChangeLogHistoryService.upgradeChecksums(AbstractChangeLogHistoryService.java:66)
   	at liquibase.changelog.StandardChangeLogHistoryService.upgradeChecksums(StandardChangeLogHistoryService.java:293)
   	at liquibase.Liquibase.checkLiquibaseTables(Liquibase.java:1221)
   	at liquibase.Liquibase.update(Liquibase.java:193)
   	at liquibase.Liquibase.update(Liquibase.java:179)
   	at liquibase.integration.spring.SpringLiquibase.performUpdate(SpringLiquibase.java:317)
   	at liquibase.database.ExtendedSpringLiquibase.afterPropertiesSet(ExtendedSpringLiquibase.java:39)
   	... 5 common frames omitted

   
   

   This is caused by using an incompatible MySQL connector 8, whereas Bitbucket accepts only 5.1x

4. Cause #4
   The following exceptions are seen in atlassian-bitbucket.log

   2021-08-30 12:23:43,101 ERROR [spring-startup]  c.a.j.s.w.s.JohnsonDispatcherServlet SpringMVC dispatcher [springMvc] could not be started
   org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'scheduledJobLifecycle' defined in URL [jar:file:/opt/atlassian/atlassian-bitbucket-6.5.1/app/WEB-INF/lib/bitbucket-platform-6.5.1.jar!/com/atlassian/stash/internal/scheduling/ScheduledJobLifecycle.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'instanceAnalyticsJob' defined in URL [jar:file:/opt/atlassian/atlassian-bitbucket-6.5.1/app/WEB-INF/lib/bitbucket-service-impl-6.5.1.jar!/com/atlassian/stash/internal/server/analytics/InstanceAnalyticsJob.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'awsAnalyticsFactory' defined in URL [jar:file:/opt/atlassian/atlassian-bitbucket-6.5.1/app/WEB-INF/lib/bitbucket-service-impl-6.5.1.jar!/com/atlassian/stash/internal/server/analytics/DefaultAwsAnalyticsFactory.class]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.atlassian.stash.internal.server.analytics.DefaultAwsAnalyticsFactory]: Constructor threw exception; nested exception is java.lang.ExceptionInInitializerError
   Caused by: java.lang.IllegalArgumentException: Error in security property. Constraint unknown: jdk.disabled.namedCurves
   	at sun.security.util.DisabledAlgorithmConstraints$Constraints.<init>(DisabledAlgorithmConstraints.java:329)
   	at sun.security.util.DisabledAlgorithmConstraints.<init>(DisabledAlgorithmConstraints.java:100)
   	at sun.security.ssl.SSLAlgorithmConstraints.<clinit>(SSLAlgorithmConstraints.java:50)
   	... 18 common frames omitted

   In this case, underlying Java JVM couldn't understand the parameter jdk.disabled.namedCurves and fails to start the service.

Resolution

These are the resolutions for the root causes above:

1. Resolution for cause #1:
   

   Ensure the user has sufficient read/write/execute permissions on the above folder and restart the application.

   For the example above, the following commands were run:

   $ cd /var/atlassian/application-data/bitbucket
   $ chown -R atlbitbucket:atlbitbucket *

2. Resolution for cause #2:
   Make sure that the database service is up and running.
3. Resolution for cause #3:
   Download and install a compatible driver as described in Connecting Bitbucket to MySQL .
   Also, make sure that the driver in bitbucket.properties  is of the form 

    jdbc.driver=com.mysql.jdbc.Driver

   which is the syntax compatible with a 5.1x driver.
4. Resolution for cause #4:
   
   1. Verify if there is a problem with the version of ca-certificates java package.
   2. Use a different JDK version to start the Bitbucket server.