Intermittently, pull request merges fail to delete branch with error replacing .packed-refs: permission denied

Still need help?

The Atlassian Community is here for you.

Ask the community


Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.

Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible

  

Summary

On attempting to merge a pull request, users intermittently receive an error message including "error replacing ./packed-refs: permission denied"

On receiving the above error, users are able to click 'OK' and will find that the merge has still completed without issue.

Environment

  • Bitbucket Versions: 
    • 7.6.0+
  • Operating Systems: 
    • Windows Server 

Diagnosis

First, debug logging must be enabled in order for the stack trace to show up within the logs. After the issue has been reproduced with debug logging enabled, the following stack trace can be seen within the application logs (atlassian-bitbucket.log) at the time when the issue was reproduced:


Click for full stack trace...
2021-02-19 11:16:09,440 DEBUG [http-nio-7990-exec-66] redacted @QZJEMNx676x29444x1 redacted redacted,redacted "GET /rest/api/latest/projects/redacted/repos/redacted/pull-requests/redacted/activities HTTP/1.0" c.a.s.i.r.e.ServiceExceptionMapper Mapping ServiceException to REST response 500
com.atlassian.bitbucket.scm.CommandFailedException: 'C:\Program Files\Git\cmd\git.exe rev-list --format=%H%x02%P%x02%aN%x02%aE%x02%at%x02%cN%x02%cE%x02%ct%n%B%n%x03END%x04 --no-walk=unsorted 4630420166redacted6f7dbc1483redactedd1c8 4630420166redacted6f7dbc1483redactedd1c8^@ --' exited with code 128 saying: fatal: couldn't read ./packed-refs: Permission denied
	at com.atlassian.bitbucket.scm.DefaultCommandExitHandler.onError(DefaultCommandExitHandler.java:47)
	at com.atlassian.bitbucket.scm.git.command.GitCommandExitHandler.evaluateThrowable(GitCommandExitHandler.java:111)
	at com.atlassian.bitbucket.scm.git.command.GitCommandExitHandler.onError(GitCommandExitHandler.java:208)
	at com.atlassian.bitbucket.scm.DefaultCommandExitHandler.onExit(DefaultCommandExitHandler.java:32)
	at com.atlassian.bitbucket.internal.process.nu.NioNuProcessHandler.callExitHandler(NioNuProcessHandler.java:285)
	at com.atlassian.bitbucket.internal.process.nu.NioNuProcessHandler.finish(NioNuProcessHandler.java:326)
	at com.atlassian.bitbucket.internal.process.nu.NioNuProcessHandler.onExit(NioNuProcessHandler.java:123)
	at com.zaxxer.nuprocess.windows.WindowsProcess.onExit(WindowsProcess.java:485)
	at com.zaxxer.nuprocess.windows.ProcessCompletions.cleanupProcess(ProcessCompletions.java:361)
	at com.zaxxer.nuprocess.windows.ProcessCompletions.process(ProcessCompletions.java:187)
	at com.zaxxer.nuprocess.windows.ProcessCompletions.run(ProcessCompletions.java:120)
	at com.zaxxer.nuprocess.windows.WindowsProcess.run(WindowsProcess.java:279)
	at com.zaxxer.nuprocess.windows.WinProcessFactory.runProcess(WinProcessFactory.java:51)
	at com.zaxxer.nuprocess.NuProcessBuilder.run(NuProcessBuilder.java:273)
	at com.atlassian.bitbucket.internal.process.nu.NuNioProcessHelper.run(NuNioProcessHelper.java:75)
	at com.atlassian.bitbucket.internal.process.NioCommand.call(NioCommand.java:52)
	at com.atlassian.stash.internal.commit.DefaultCommitService.getCommit(DefaultCommitService.java:160)
	at com.atlassian.stash.internal.commit.DefaultCommitService.findCommit(DefaultCommitService.java:89)
	at com.atlassian.stash.internal.pull.MergePullRequestActivityEnricher$MergeActivityVisitor.visit(MergePullRequestActivityEnricher.java:53)
	at com.atlassian.stash.internal.pull.InternalPullRequestMergeActivity.accept(InternalPullRequestMergeActivity.java:47)
	at com.atlassian.stash.internal.pull.MergePullRequestActivityEnricher.enrich(MergePullRequestActivityEnricher.java:35)
	at com.atlassian.stash.internal.pull.DefaultPullRequestService.enrichActivities(DefaultPullRequestService.java:1662)
	at com.atlassian.stash.internal.pull.DefaultPullRequestService.enrichActivities(DefaultPullRequestService.java:1655)
	at com.atlassian.stash.internal.pull.DefaultPullRequestService.getActivities(DefaultPullRequestService.java:579)
	at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:26)
	at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
	at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
	at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70)
	at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53)
	at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
	at com.atlassian.stash.internal.rest.pull.PullRequestResource.getActivities(PullRequestResource.java:349)
	at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
	at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
	at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
	at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
	at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
	at com.atlassian.analytics.client.filter.UniversalAnalyticsFilter.doFilter(UniversalAnalyticsFilter.java:75)
	at com.atlassian.analytics.client.filter.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:48)
	at com.atlassian.bitbucket.internal.xcode.web.XcodeUserAgentFilter.doFilter(XcodeUserAgentFilter.java:36)
	at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
	at com.atlassian.bitbucket.internal.ratelimit.servlet.filter.RateLimitFilter.doFilter(RateLimitFilter.java:75)
	at com.atlassian.plugin.connect.plugin.auth.scope.ApiScopingFilter.doFilter(ApiScopingFilter.java:81)
	at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
	at com.atlassian.stash.internal.web.auth.AuthorizationFailureInterceptor.doFilterInternal(AuthorizationFailureInterceptor.java:39)
	at com.atlassian.stash.internal.spring.security.StashAuthenticationFilter.doFilter(StashAuthenticationFilter.java:85)
	at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doInsideSpringSecurityChain(BeforeLoginPluginAuthenticationFilter.java:112)
	at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doFilter(BeforeLoginPluginAuthenticationFilter.java:75)
	at com.atlassian.security.auth.trustedapps.filter.TrustedApplicationsFilter.doFilter(TrustedApplicationsFilter.java:94)
	at com.atlassian.oauth.serviceprovider.internal.servlet.OAuthFilter.doFilter(OAuthFilter.java:67)
	at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
	at com.atlassian.plugin.connect.plugin.auth.oauth2.DefaultSalAuthenticationFilter.doFilter(DefaultSalAuthenticationFilter.java:69)
	at com.atlassian.plugin.connect.plugin.auth.user.ThreeLeggedAuthFilter.doFilter(ThreeLeggedAuthFilter.java:109)
	at com.atlassian.jwt.internal.servlet.JwtAuthFilter.doFilter(JwtAuthFilter.java:37)
	at com.atlassian.analytics.client.filter.DefaultAnalyticsFilter.doFilter(DefaultAnalyticsFilter.java:26)
	at com.atlassian.analytics.client.filter.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:48)
	at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
	at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doBeforeBeforeLoginFilters(BeforeLoginPluginAuthenticationFilter.java:90)
	at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doFilter(BeforeLoginPluginAuthenticationFilter.java:73)
	at com.atlassian.stash.internal.request.DefaultRequestManager.doAsRequest(DefaultRequestManager.java:81)
	at com.atlassian.stash.internal.hazelcast.ConfigurableWebFilter.doFilter(ConfigurableWebFilter.java:38)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
	... 336 frames trimmed
Caused by: com.atlassian.utils.process.ProcessException: Non-zero exit code: 128
	at com.atlassian.bitbucket.internal.process.nu.NioNuProcessHandler.callExitHandler(NioNuProcessHandler.java:277)
	... 58 common frames omitted

Cause

The root cause for this error is not currently known. The error has so far only been seen on Bitbucket environments hosted on Windows Server (varying versions), where the Bitbucket version is greater than 7.6.0.

  • (info) From our analysis, when monitoring Windows with Microsoft Sysinternals Live there seems to have some latency, or lag, in the file system releasing the file to be open for write.  The issue then seen when git will try to access the packed-refs file several times and ultimately fail as a result of the file system failing to release the write lock quickly.  
    • In our investigation into other usages of the packed-refs file at around the same time as the issue, we have also seen that another git process has "FILE LOCKED WITH ONLY READERS" which means that anyone can read it, but you cannot write to the file until the lock is released.  This aligns with the theory that Windows does not seem to release the lock quickly - causing a race between the number of times git will try and the file system releasing the lock. This race condition results in the issue being inconsistent and difficult to readily reproduce.

Solution

The only known solution at this time is to migrate the Bitbucket instance to be hosted on a Linux operating system instead of Windows, given that the issue appears related to git's interaction with the Windows filesystem.  

Migrating to Linux will also prepare your team for any planned change to our Data Center licensing option, as Data Center instances are only supported on Linux operating systems.

Workaround

It's possible that this race condition can be avoided by changing the timing on when Bitbucket performs the needed garbage collection operations.  

By default, the system waits ~1 minute after a push to try to repack the refs.  The following settings change forces Bitbucket to wait for ~5 minutes instead, which could make it less likely that Bitbucket would hit this race condition.  

plugin.bitbucket-git.gc.delay=300

Last modified on Mar 24, 2021

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.