Bitbucket Datacenter unable to start as Mesh sidecar isn't available
Platform Notice: Data Center - This article applies to Atlassian products on the Data Center platform.
Note that this knowledge base article was created for the Data Center version of the product. Data Center knowledge base articles for non-Data Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Unable to start Bitbucket because Mesh doesn't start with the log message "Sidecar is currently unavailable"
Environment
Bitbucket 8.x and above.
Mesh sidecar
Diagnosis
In the Mesh logs you would see that the Mesh sidecar is unable to start:
2024-01-24 00:02:33,453 INFO [main] - c.a.b.mesh.boot.MeshApplication Starting MeshApplication using Java 1.8.0_66 on eumucgit01drp with PID 9075 (/srv/git01/atlassian-bitbucket-8.14.4/app/WEB-INF/mesh/mesh-app.jar started by atlbitbucket in /srv/git01/bitbucket-home/mesh)
2024-01-24 00:02:33,456 INFO [main] - c.a.b.mesh.boot.MeshApplication The following 1 profile is active: "sidecar"
2024-01-24 00:02:34,326 INFO [main] - c.a.b.m.l.DefaultHomeSubdirectoryResolver Using '/srv/git01/bitbucket-home/caches' as the 'caches' directory
2024-01-24 00:07:35,279 WARN [main] - liquibase.database.core.H2Database Failed to restore the auto commit to true
2024-01-24 00:07:35,281 WARN [main] - o.s.c.a.AnnotationConfigApplicationContext Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in com.atlassian.bitbucket.mesh.MeshDatabaseWiring: Invocation of init method failed; nested exception is liquibase.exception.DatabaseException: org.h2.jdbc.JdbcSQLNonTransientConnectionException: Database is already closed (to disable automatic closing at VM shutdown, add ";DB_CLOSE_ON_EXIT=FALSE" to the db URL) [90121-220]which is why Bitbucket was unable to start successfully:
2024-01-23 12:42:07,737 INFO [main] c.a.b.i.b.BitbucketServerApplication Starting BitbucketServerApplication v8.14.4 using Java 1.8.0_66 on eumucgit01drp with PID 18859 (/srv/git01/atlassian-bitbucket-8.14.4/app/WEB-INF/classes started by atlbitbucket in /)
2024-01-23 12:42:08,430 INFO [main] c.a.b.i.boot.log.BuildInfoLogger JVM: Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 1.8.0_66-b17
2024-01-23 12:42:09,572 INFO [main] c.a.b.i.b.BitbucketServerApplication Started BitbucketServerApplication in 2.816 seconds (JVM running for 3.484)
2024-01-23 12:42:12,549 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /srv/git01/bitbucket-home
2024-01-23 12:42:15,570 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /srv/git01/bitbucket-home/shared
2024-01-23 12:42:17,256 WARN [spring-startup] com.hazelcast.instance.Node [10.80.101.46]:5701 [atlbitbucket] [3.12.13] No join method is enabled! Starting standalone.
2024-01-23 12:42:44,914 WARN [spring-startup] c.a.s.i.s.g.m.DefaultSidecarManager Interrupted after 15179ms while waiting for sidecar to start
2024-01-23 12:42:49,324 INFO [spring-startup] c.a.s.i.hook.DefaultHookService Hook callback socket listening on 127.0.0.1:33051
2024-01-23 12:42:53,365 INFO [hz.hazelcast.event-1] c.a.s.i.s.g.m.DefaultMeshSidebandRegistry Sidecar#0 (http://localhost:7777): Opening sideband channel
2024-01-23 12:42:53,608 INFO [mesh-grpc-request:thread-4] c.a.s.i.mesh.DefaultMeshNodeRegistry Node Sidecar (http://localhost:7777) went offline
2024-01-23 12:42:53,610 ERROR [spring-startup] c.a.s.i.mesh.DefaultMeshController Failed to configure sidecar on initialization
com.atlassian.stash.internal.mesh.SidecarUnavailableException: Sidecar is currently unavailable
at com.atlassian.stash.internal.mesh.DefaultMeshController.maybeConfigureSidecar(DefaultMeshController.java:289)
at com.atlassian.stash.internal.mesh.DefaultMeshController.lambda$maybeConfigureSidecar$0(DefaultMeshController.java:280)Cause
The Mesh sidecar Database needs to be upgraded.
Solution
- As the Mesh sidecar DB stores temporary information about the local git processing, you can delete the file from your system or rename it as a backup file and restart Bitbucket. A new database should be created during Bitbucket restart.
Please run the below command in your Linux terminal to rename these files:
cd <Bitbucket-home>/mesh mv mesh.mv.db mesh-backup.mv.db; mv mesh.trace.db mesh-backup.trace.dbThen restart Bitbucket. After a restart of Bitbucket, the Mesh Sidecar should try to recreate the database files with fresh copies.