Bitbucket Mesh Sidecar throws UnsatisfiedLinkError when creating a new repository
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Environment
Bitbucket 8+
- Startup: Successful
- Creating Projects: Successful
- Creating Repositories: Failure
Summary
In Bitbucket 8+, attempting to create a new repository will result in an HTTP 500 Server error. Checking the logs atlassian-bitbucket.log file, we may see an error such as:
"POST /projects/PROJ/repos HTTP/1.1"
c.a.s.i.w.HandlerExceptionResolverImpl UNKNOWN
io.grpc.StatusRuntimeException: UNKNOWN
at io.grpc.Status.asRuntimeException(Status.java:535)
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:479)
at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at com.atlassian.stash.internal.scm.git.mesh.LastSeenClientInterceptor$LastSeenClientListener.onClose(LastSeenClientInterceptor.java:40)
at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at com.atlassian.stash.internal.scm.git.mesh.StatefulClientCallListener.onClose(StatefulClientCallListener.java:34)
at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at com.atlassian.stash.internal.scm.git.mesh.DeadlinePropagatingClientInterceptor$DeadlinePropagatingListener.onClose(DeadlinePropagatingClientInterceptor.java:156)
at com.atlassian.stash.internal.scm.git.mesh.ErrorHandlingClientInterceptor$ErrorHandlingCall$1.onClose(ErrorHandlingClientInterceptor.java:149)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562)
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)Since Mesh Sidecar handles SCM tasks in Bitbucket 8+, the atlassian-mesh.log files will show the following failure:
WARN [grpc-server:thread-817] [...] 127.0.0.1 "RepositoryService/Create" (>1 <0) c.a.b.m.g.LoggingServerInterceptor RPC failed with an UNKNOWN error: null
java.lang.UnsatisfiedLinkError: /home/atlbitbucket/.cache/JNA/temp/jna152121760652375836092.tmp: /home/atlbitbucket/.cache/JNA/temp/jna152121760652375836092.tmp: cannot open shared object file: Operation not permittedDiagnosis
Java's JNA library allows code to call native functions and is effective in allowing Java code to run on multiple platforms. At runtime, these libraries are extracted to the jna.tmpdir and for the application to load. By default on Linux systems, Java will set this tmpdir to '/tmp' or the home dir of the user running the process.
Cause
If the value of jna.tmpdir is a mount where the noexec flag is set, the JVM will not be able to initialize the JNA libraries resulting in an UnsatisfiedLinkError.
Solution
We should first check if the mount point has the noexec flag set. We can do this via a command such as:
mount | grep -i noexecIf the output includes the process user's home folder or /tmp , we need to tell Sidecar to use a different location to store the JNA library files.
To change the location of the jna.tmpdir we need to pass the JVM property and value to Sidecar via the bitbucket.properties file. To ensure permissions, we suggest using the $BITBUCKET_HOME/tmp directory:
plugin.bitbucket-git.mesh.sidecar.jvm-args[n]=-Djna.tmpdir=<BITBUCKET_HOME>/tmpNote:
When passing JVM arguments, we should ensure that the value in the brackets is a number that increments from 0. It must be contiguous or Sidecar will fail to start with a configuration error.