Bitbucket fails to start up with Hazelcast instance is not active error
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Problem
When starting Bitbucket Server, the application fails to start and the following appears in the atlassian-bitbucket.log
2016-05-01 11:05:22,082 ERROR [localhost-startStop-2] c.a.e.i.AsynchronousAbleEventDispatcher There was an exception thrown trying to dispatch event [com.atlassian.config.lifecycle.events.ApplicationStoppingEvent@74c57907] from the invoker [SingleParameterMethodListenerInvoker{method=public void com.atlassian.analytics.client.listener.BitbucketEventListener.onEvent(java.lang.Object), listener=com.atlassian.analytics.client.listener.BitbucketEventListener@605b0ae1}]
java.lang.RuntimeException: Hazelcast instance is not active!
at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:54) ~[atlassian-event-3.0.0.jar:na]
at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$1$1.run(AsynchronousAbleEventDispatcher.java:46) ~[atlassian-event-3.0.0.jar:na]
at com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299) [guava-18.0.jar:na]
at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.dispatch(AsynchronousAbleEventDispatcher.java:105) [atlassian-event-3.0.0.jar:na]
at com.atlassian.event.internal.LockFreeEventPublisher$Publisher.dispatch(LockFreeEventPublisher.java:260) [atlassian-event-3.0.0.jar:na]
at com.atlassian.event.internal.LockFreeEventPublisher.publish(LockFreeEventPublisher.java:102) [atlassian-event-3.0.0.jar:na]
at com.atlassian.stash.internal.event.TransactionAwareEventPublisher.publish(TransactionAwareEventPublisher.java:84) [bitbucket-platform-4.3.2.jar:na]
at com.atlassian.stash.internal.config.ConfigLifecycle.stop(ConfigLifecycle.java:45) [bitbucket-service-impl-4.3.2.jar:na]
at com.atlassian.stash.internal.spring.AbstractSmartLifecycle.stop(AbstractSmartLifecycle.java:36) [bitbucket-platform-4.3.2.jar:na]
at org.springframework.context.support.DefaultLifecycleProcessor.doStop(DefaultLifecycleProcessor.java:229) [spring-context-4.2.2.RELEASE.jar:na]
at org.springframework.context.support.DefaultLifecycleProcessor.access$300(DefaultLifecycleProcessor.java:51) [spring-context-4.2.2.RELEASE.jar:na]
at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.stop(DefaultLifecycleProcessor.java:363) [spring-context-4.2.2.RELEASE.jar:na]
at org.springframework.context.support.DefaultLifecycleProcessor.stopBeans(DefaultLifecycleProcessor.java:202) [spring-context-4.2.2.RELEASE.jar:na]
at org.springframework.context.support.DefaultLifecycleProcessor.onClose(DefaultLifecycleProcessor.java:118) [spring-context-4.2.2.RELEASE.jar:na]
at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:969) [spring-context-4.2.2.RELEASE.jar:na]
at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:928) [spring-context-4.2.2.RELEASE.jar:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_51]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_51]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
... 11 frames trimmed
Caused by: com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at com.hazelcast.spi.AbstractDistributedObject.getService(AbstractDistributedObject.java:93) ~[hazelcast-3.5.2-atlassian-24.jar:na]
at com.hazelcast.map.impl.proxy.MapProxySupport.toData(MapProxySupport.java:1143) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24]
at com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:81) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24]
at com.atlassian.hazelcast.quartz2.HazelcastJobStore.retrieveTrigger(HazelcastJobStore.java:498) ~[atlassian-hazelcast-extras-quartz2-1.3.3.jar:na]
at org.quartz.core.QuartzScheduler.getTrigger(QuartzScheduler.java:1530) ~[quartz-2.2.1.jar:na]
at org.quartz.impl.StdScheduler.getTrigger(StdScheduler.java:508) ~[quartz-2.2.1.jar:na]
at com.atlassian.scheduler.quartz2.Quartz2SchedulerFacade.getTrigger(Quartz2SchedulerFacade.java:191) ~[atlassian-scheduler-quartz2-1.6.0.jar:na]
at com.atlassian.scheduler.quartz2.Quartz2SchedulerFacade.unscheduleJob(Quartz2SchedulerFacade.java:313) ~[atlassian-scheduler-quartz2-1.6.0.jar:na]
at com.atlassian.scheduler.quartz2.Quartz2SchedulerService.unscheduleJob(Quartz2SchedulerService.java:122) ~[atlassian-scheduler-quartz2-1.6.0.jar:na]
at com.atlassian.stash.internal.sal.scheduling.PluginSchedulerAdapter.unscheduleJob(PluginSchedulerAdapter.java:75) ~[bitbucket-platform-4.3.2.jar:na]
at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:30) ~[atlassian-plugins-core-4.0.4.jar:na]
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56) ~[na:na]
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60) ~[na:na]
at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70) ~[na:na]
at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53) ~[na:na]
at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57) ~[na:na]
at com.atlassian.scheduler.compat.local.LocalOnlyCompatibilityPluginScheduler.unscheduleClusteredJobImpl(LocalOnlyCompatibilityPluginScheduler.java:126) ~[na:na]
at com.atlassian.scheduler.compat.local.LocalOnlyCompatibilityPluginScheduler.unscheduleClusteredJob(LocalOnlyCompatibilityPluginScheduler.java:118) ~[na:na]
at com.atlassian.analytics.client.upload.PeriodicEventUploaderScheduler.unscheduleJob(PeriodicEventUploaderScheduler.java:113) ~[na:na]
at com.atlassian.analytics.client.upload.PeriodicEventUploaderScheduler.unscheduleJobs(PeriodicEventUploaderScheduler.java:156) ~[na:na]
at com.atlassian.analytics.client.listener.BitbucketEventListener.destroy(BitbucketEventListener.java:64) ~[na:na]
at com.atlassian.analytics.client.listener.BitbucketEventListener.onEvent(BitbucketEventListener.java:47) ~[na:na]
at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:36) ~[atlassian-event-3.0.0.jar:na]
... 19 common frames omittedDiagnosis
Environment
- This issue may have been preceded by a restart, OS or Java upgrade.
Cause
JAVA_HOME is not set correctly, and points to an old or incorrect JRE location.
Note: if you are seeing this error in a different scenario, where Bitbucket does start up but later crashes with an OutOfMemory error, you may be running into a different issue: Bitbucket Server become unresponsive with "Hazelcast instance is not active"
Resolution
Set JAVA_HOME to the correct value.
JAVA_HOME may be set as an environment variable for the user that runs Bitbucket (e.g. atlbitbucket) or it may be set inside the $BITBUCKET_INSTALL/bin/setenv.sh file. Check both places and ensure that this is pointing to a valid JRE location.
setenv and environment variable changes in Bitbucket Server 5.0+
Starting with Bitbucket Server 5.0, setenv.sh and setenv.bat have been removed. The options that were set in this file can now be set via environment variables. Where to set the environment variable depends on which Operating System you're running on.
Linux
When using the atlbitbucket service on Linux, the environment variables are ignored. You must set the parameters in _start-webapp.sh (or start-bitbucket.sh). These values will be read when the service starts.
As an example, to set JVM_SUPPORT_RECOMMENDED_ARGS, you would add this line to the file:
JVM_SUPPORT_RECOMMENDED_ARGS=-XX:+HeapDumpOnOutOfMemoryErrorWindows
Set the parameter as an environment variable for the user running Bitbucket Server. For example, if you want to set JVM_SUPPORT_RECOMMENDED_ARGS, create it as an environment variable and assign the appropriate value to it. When Bitbucket Server starts using the startup scripts or service, it will pick up and apply this value.