Bitbucket Data Center NodePassivationException when adding node to the cluster
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
When adding an additional node to the Data Center cluster, the node does not join the cluster and the following appears in the atlassian-bitbucket.log for that node:
2015-10-01 10:31:04,322 WARN [spring-startup] o.s.w.c.s.XmlWebApplicationContext Exception encountered during context initialization - cancelling refresh attempt
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'crowdUserDao' defined in class path resource [crowd-spi-context.xml]: Unsatisfied dependency expressed through constructor argument with index 1 of type [com.atlassian.bitbucket.internal.project.ProjectDao]: : Error creating bean with name 'projectDao' defined in URL [jar:file:/opt/atlassian/bitbucket/4.0.0/atlassian-bitbucket/WEB-INF/lib/bitbucket-dao-impl-4.0.0.jar!/com/atlassian/bitbucket/internal/project/HibernateProjectDao.class]: Unsatisfied dependency expressed through constructor argument with index 0 of type [org.hibernate.SessionFactory]: : Error creating bean with name 'hazelcast' defined in class path resource [bitbucket-context.xml]: Invocation of init method failed; nested exception is com.atlassian.bitbucket.internal.cluster.NodePassivationException: The current node has been passivated; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hazelcast' defined in class path resource [bitbucket-context.xml]: Invocation of init method failed; nested exception is com.atlassian.bitbucket.internal.cluster.NodePassivationException: The current node has been passivated; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'projectDao' defined in URL [jar:file:/opt/atlassian/bitbucket/4.0.0/atlassian-bitbucket/WEB-INF/lib/bitbucket-dao-impl-4.0.0.jar!/com/atlassian/bitbucket/internal/project/HibernateProjectDao.class]: Unsatisfied dependency expressed through constructor argument with index 0 of type [org.hibernate.SessionFactory]: : Error creating bean with name 'hazelcast' defined in class path resource [bitbucket-context.xml]: Invocation of init method failed; nested exception is com.atlassian.bitbucket.internal.cluster.NodePassivationException: The current node has been passivated; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hazelcast' defined in class path resource [bitbucket-context.xml]: Invocation of init method failed; nested exception is com.atlassian.bitbucket.internal.cluster.NodePassivationException: The current node has been passivated
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:749) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:757) ~[spring-context-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480) ~[spring-context-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at javax.servlet.GenericServlet.init(GenericServlet.java:158) [servlet-api.jar:3.1.FR]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
... 18 frames trimmed
...
Caused by: com.atlassian.bitbucket.internal.cluster.NodePassivationException: The current node has been passivatedEnvironment
Bitbucket Data Center
Cause
The NodePassivationException is thrown when a new node attempts to join the cluster and detects an issue, the node will passivate itself for safety. Potential reasons for this include:
- The failed node was using a different path for $
BITBUCKET_HOME/shared - The failed node is configured for a different time zone than the running nodes
Bitbucket Data Center requires that the shared home path match across all nodes due to the absolute paths required for some git operations.
Solution
Shared home mismatch
Ensure that all nodes in the cluster are using the same path to the shared directory inside of BITBUCKET_HOME
The path that is being used can be checked during startup in the atlassian-bitbucket.log:
2015-10-01 10:46:40,408 INFO [spring-startup] c.a.b.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /var/atlassian/application-data/bitbucket
2015-10-01 10:46:45,180 INFO [spring-startup] c.a.b.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /mnt/nfs/bitbucket-sharedThe first path is for BITBUCKET_HOME and the second is for shared directory. The second path must match across all nodes.
Time zone mismatch
2016-11-07 11:33:41,179 WARN [hz.hazelcast.IO.thread-Acceptor] com.hazelcast.nio.tcp.SocketAcceptor [10.137.158.47]:5701 [testbitbucket] [3.5.2-atlassian-37] com.atlassian.stash.internal.cluster.NodeConnectionException: Required property 'timeZone' should be 'America/Chicago' but is 'US/Central'
com.atlassian.stash.internal.cluster.NodeConnectionException: Required property 'timeZone' should be 'America/Chicago' but is 'US/Central'Note: The above log will appear in the node that is being connected to, rather than the node that is being rejected.
You can enforce a specific timezone for the JVM itself by editing the <BITBUCKET_INSTALL>/bin/_start-webapp.sh file and appending the "-Duser.timezone=<TIMEZONE>" argument to the JVM_SUPPORT_RECOMMENDED_ARGS variable. You have to do this for each one of the nodes:
The following example sets the time zone for Chicago:
JVM_SUPPORT_RECOMMENDED_ARGS="-Duser.timezone=America/Chicago"