Starting Bitbucket Server/DC after upgrade to 8.x fails due to sidecar issue without any error
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
In addition to the case described in KB Starting Bitbucket Server/Data Center after upgrade to 8.x fails with the error UnsatisfiedDependencyException Mesh process may fail to start on time due to long hostname resolution.
Environment
Any Bitbucket version 8.x
Diagnosis
If your Bitbucket instance is failing to start due to Mesh Sidecar is currently unavailable but not throwing UnsatisfiedDependencyException or any other clearly defined exception like in the example below:
xxxx-xx-xx xx:xx:xx WARN [spring-startup] o.s.w.c.s.XmlWebApplicationContext Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Failed to start bean 'meshController'; nested exception is com.atlassian.stash.internal.mesh.SidecarUnavailableException: Sidecar is currently unavailable
xxxx-xx-xx xx:xx:xx,848 ERROR [hz.hazelcast.event-1] c.h.s.i.e.impl.EventServiceImpl [192.168.2.2]:5701 [atlbitbucket] [3.12.13] hz.hazelcast.event-1 caught an exception while processing:com.hazelcast.spi.impl.eventservice.impl.LocalEventDispatcher@3c0f46ab
com.atlassian.stash.internal.mesh.SidecarUnavailableException: Sidecar is currently unavailable
at com.atlassian.stash.internal.mesh.DefaultMeshController.maybeConfigureSidecar(DefaultMeshController.java:259)
at com.atlassian.stash.internal.mesh.DefaultMeshController.access$700(DefaultMeshController.java:71)
at com.atlassian.stash.internal.mesh.DefaultMeshController$ControlPlaneStateListener.lambda$onUpdated$0(DefaultMeshController.java:292)
at java.base/java.util.Optional.ifPresent(Unknown Source)
at com.atlassian.stash.internal.mesh.DefaultMeshController$ControlPlaneStateListener.onUpdated(DefaultMeshController.java:291)
at com.atlassian.stash.internal.mesh.DefaultMeshController$ControlPlaneStateListener.entryAdded(DefaultMeshController.java:278)
at com.hazelcast.map.impl.MapListenerAdaptors$1$1.onEvent(MapListenerAdaptors.java:67)
at com.hazelcast.map.impl.MapListenerAdaptors$1$1.onEvent(MapListenerAdaptors.java:64)
at com.hazelcast.map.impl.InternalMapListenerAdapter.onEvent(InternalMapListenerAdapter.java:56)
at com.hazelcast.map.impl.InternalMapListenerAdapter.onEvent(InternalMapListenerAdapter.java:35)
at com.hazelcast.map.impl.event.MapEventPublishingService.callListener(MapEventPublishingService.java:172)
at com.hazelcast.map.impl.event.MapEventPublishingService.dispatchEntryEventData(MapEventPublishingService.java:184)
at com.hazelcast.map.impl.event.MapEventPublishingService.dispatchEvent(MapEventPublishingService.java:83)
at com.hazelcast.map.impl.event.MapEventPublishingService.dispatchEvent(MapEventPublishingService.java:47)
at com.hazelcast.map.impl.MapService.dispatchEvent(MapService.java:105)
at com.hazelcast.map.impl.MapService.dispatchEvent(MapService.java:78)
at com.hazelcast.spi.impl.eventservice.impl.LocalEventDispatcher.run(LocalEventDispatcher.java:64)
at com.hazelcast.util.executor.StripedExecutor$Worker.process(StripedExecutor.java:244)
at com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:227)
xxxx-xx-xx xx:xx:xx,300 INFO [spring-startup] c.a.s.i.s.g.m.DefaultSidecarManager Sidecar has stopped (Exit code: 0)
xxxx-xx-xx xx:xx:xx,350 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Releasing lock on /var/atlassian/application-data/bitbucket/shared
xxxx-xx-xx xx:xx:xx,363 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Releasing lock on /var/atlassian/application-data/bitbucket
xxxx-xx-xx xx:xx:xx,366 ERROR [spring-startup] c.a.j.s.w.s.JohnsonDispatcherServlet SpringMVC dispatcher [springMvc] could not be started
org.springframework.context.ApplicationContextException: Failed to start bean 'meshController'; nested exception is com.atlassian.stash.internal.mesh.SidecarUnavailableException: Sidecar is currently unavailable
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:181)
at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:54)
at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:356)
at java.base/java.lang.Iterable.forEach(Unknown Source)
at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:155)
at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123)
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:935)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:586)
at javax.servlet.GenericServlet.init(GenericServlet.java:158)
at java.base/java.lang.Thread.run(Unknown Source)
... 8 frames trimmed
Caused by: com.atlassian.stash.internal.mesh.SidecarUnavailableException: Sidecar is currently unavailable
at com.atlassian.stash.internal.mesh.DefaultMeshController.maybeConfigureSidecar(DefaultMeshController.java:259)
at com.atlassian.stash.internal.mesh.DefaultMeshController.lambda$maybeConfigureSidecar$0(DefaultMeshController.java:250)
at java.base/java.util.Optional.ifPresent(Unknown Source)
at com.atlassian.stash.internal.mesh.DefaultMeshController.maybeConfigureSidecar(DefaultMeshController.java:246)
at com.atlassian.stash.internal.mesh.DefaultMeshController.start(DefaultMeshController.java:233)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178)
... 10 common frames omitted
xxxx-xx-xx xx:xx:xx,625 WARN [http-nio-7990-exec-10] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [hazelcastFilter]; no Spring WebApplicationContext is available
When the issue has happened, please check atlassian-mesh.log. You will see the following warnings:
xxxx-xx-xx xx:xx:xx,654 WARN [main] - o.s.boot.StartupInfoLogger InetAddress.getLocalHost().getHostName() took 5016 milliseconds to respond. Please verify your network configuration.
xxxx-xx-xx xx:xx:xx,675 WARN [main] - o.s.boot.system.ApplicationPid ManagementFactory.getRuntimeMXBean().getName() took 5017 milliseconds to respond. This may be due to slow host name resolution. Please verify your network configuration.Please note that Mesh should startup without any warnings. If you see any please create a ticket to Atlassian Support
Cause
The WARN you can observe above is caused by getHostName() function failed to obtain the hostname value from the OS within the timeout. The function getHostName() is called during the startup of both Bitbucket application and mesh nodes. If mesh node fails to obtain the hostname within timeout Bitbucket considers that sidecar has failed to start and initiate the termination sequence.
Solution
There are two solutions available:
- Fix the network configuration to make hostname resolution faster - check resolv.conf or speak to DNS admin of your network.
In /etc/hosts set the hostname to both 127.0.0.1 and the IP address of the server.
# Hosts file example ~ cat /etc/hosts 127.0.0.1 localhost ::1 localhost# Comment the line with ip6-loopback (::1) or change to 127.0.0.1 127.0.0.1 localhost #::1 localhost