Summary

Bitbucket application freezes after upgrading to version 8.9.x from 7.x.x very frequently. Users are unable to access Bitbucket UI and GIT operations start failing. 

Environment

Bitbucket Server and Data Center 8.9.x (Recently upgraded from 7.x.x) 

Diagnosis

• Check if third party plugins are on latest version supported by Bitbucket Server or Data Center

• A lot of http threads with similar stack traces can be seen in the atlassian-bitbucket.log file 

  2023-06-01 10:31:52,968 ERROR [http-nio-7990-exec-3145] admin @P0QVQUx638x34136x38 12f49fh x.x.x.x,127.0.0.1 "POST /rest/tagdetails/1.0/projects/API/repos/testrepo/notes HTTP/1.1" c.a.p.r.c.e.j.ThrowableExceptionMapper Uncaught exception thrown by REST service: TODO: Handle interrupts better
  java.lang.RuntimeException: TODO: Handle interrupts better
  	at com.atlassian.stash.internal.scm.git.mesh.AbstractFutureResponseObserver.asResult(AbstractFutureResponseObserver.java:167)
  	at com.atlassian.stash.internal.scm.git.mesh.AbstractFutureResponseObserver.asResult(AbstractFutureResponseObserver.java:139)
  	at com.atlassian.stash.internal.scm.git.mesh.AbstractFragmentResponseObserver.asResult(AbstractFragmentResponseObserver.java:147)
  	at com.atlassian.stash.internal.scm.git.mesh.BidirectionalFragmentResponseObserver.asResult(BidirectionalFragmentResponseObserver.java:66)
  	at com.atlassian.stash.internal.scm.git.mesh.GrpcPlumbingClient.call(GrpcPlumbingClient.java:68)
  	at jdk.internal.reflect.GeneratedMethodAccessor1228.invoke(Unknown Source)
  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
  	at jdk.internal.reflect.GeneratedMethodAccessor184.invoke(Unknown Source)
  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
  	at com.atlassian.stash.internal.scm.git.mesh.MeshGitFreeFormCommandBuilder$1.call(MeshGitFreeFormCommandBuilder.java:102)
  	at jh.atlassian.bitbucket.tagdetails.RestService.getNotesByCommitIds(RestService.java:78)
  	at jdk.internal.reflect.GeneratedMethodAccessor1811.invoke(Unknown Source)
  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
  	at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
  	at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
  	at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
  	at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
  	at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
  	at com.atlassian.analytics.client.filter.UniversalAnalyticsFilter.doFilter(UniversalAnalyticsFilter.java:75)
  	at com.atlassian.analytics.client.filter.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:33)
  	at com.atlassian.bitbucket.internal.xcode.web.XcodeUserAgentFilter.doFilter(XcodeUserAgentFilter.java:36)
  	at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
  	at com.atlassian.bitbucket.internal.ratelimit.servlet.filter.RateLimitFilter.doFilter(RateLimitFilter.java:75)
  	at com.atlassian.plugin.connect.plugin.auth.scope.ApiScopingFilter.doFilter(ApiScopingFilter.java:81)
  	at com.atlassian.troubleshooting.thready.filter.AbstractThreadNamingFilter.doFilter(AbstractThreadNamingFilter.java:46)
  	at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
  	at com.atlassian.stash.internal.web.auth.AuthorizationFailureInterceptor.doFilterInternal(AuthorizationFailureInterceptor.java:39)
  	at com.atlassian.stash.internal.spring.security.StashAuthenticationFilter.doFilter(StashAuthenticationFilter.java:85)
  	at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doInsideSpringSecurityChain(BeforeLoginPluginAuthenticationFilter.java:112)
  	at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doFilter(BeforeLoginPluginAuthenticationFilter.java:75)
  	at com.atlassian.security.auth.trustedapps.filter.TrustedApplicationsFilter.doFilter(TrustedApplicationsFilter.java:94)
  	at com.atlassian.oauth.serviceprovider.internal.servlet.OAuthFilter.doFilter(OAuthFilter.java:67)
  	at com.atlassian.oauth2.provider.core.web.AccessTokenFilter.doFilter(AccessTokenFilter.java:81)
  	at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
  	at com.atlassian.plugin.connect.plugin.auth.oauth2.DefaultSalAuthenticationFilter.doFilter(DefaultSalAuthenticationFilter.java:69)
  	at com.atlassian.plugin.connect.plugin.auth.user.ThreeLeggedAuthFilter.doFilter(ThreeLeggedAuthFilter.java:109)
  	at com.atlassian.plugins.authentication.basicauth.filter.DisableBasicAuthFilter.doFilter(DisableBasicAuthFilter.java:70)
  	at com.atlassian.jwt.internal.servlet.JwtAuthFilter.doFilter(JwtAuthFilter.java:37)
  	at com.atlassian.analytics.client.filter.DefaultAnalyticsFilter.doFilter(DefaultAnalyticsFilter.java:26)
  	at com.atlassian.analytics.client.filter.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:33)
  	at com.atlassian.troubleshooting.thready.filter.AbstractThreadNamingFilter.doFilter(AbstractThreadNamingFilter.java:46)
  	at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
  	at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doBeforeBeforeLoginFilters(BeforeLoginPluginAuthenticationFilter.java:90)
  	at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doFilter(BeforeLoginPluginAuthenticationFilter.java:73)
  	at com.atlassian.stash.internal.request.DefaultRequestManager.doAsRequest(DefaultRequestManager.java:84)
  	at com.atlassian.stash.internal.hazelcast.ConfigurableWebFilter.doFilter(ConfigurableWebFilter.java:38)
  	at java.base/java.lang.Thread.run(Unknown Source)
  	... 288 frames trimmed
  Caused by: java.lang.InterruptedException: null
  	at java.base/java.util.concurrent.CompletableFuture.reportGet(Unknown Source)
  	at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source)
  	at com.atlassian.stash.internal.scm.git.mesh.AbstractFutureResponseObserver$ResponseFuture.get(AbstractFutureResponseObserver.java:233)
  	at com.atlassian.stash.internal.scm.git.mesh.AbstractFutureResponseObserver.asResult(AbstractFutureResponseObserver.java:151)
  	... 47 common frames omitted

• From the above stack trace notice that jh.atlassian.bitbucket.tagdetails.RestService.getNotesByCommitIds is seen and causing the similar http threads to wait forever and eventually saturating all http threads which causes Bitbucket application to freeze. 

Cause

The third-party plugins may not be compatible or supported with the latest Bitbucket version. For example Tag Details And Git Notes plugin version 5.0.x is not supported on Bitbucket 8.x and causes application to freeze. 

Solution

• Disable the offending plugin and monitor the application 
• Update the 3rd party plugin such as Tag Details And Git Notes while upgrading Bitbucket server/ Data Center from old version to 8.x. 
• If the issue still persist, contact Atlassian Support.