Intermittently, pull request merges fail to delete branch with error replacing .packed-refs: permission denied
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
On attempting to merge a pull request, users intermittently receive an error message including "error replacing ./packed-refs: permission denied"
On receiving the above error, users are able to click 'OK' and will find that the merge has still completed without issue.
Environment
Bitbucket Versions:
7.6.0+
Operating Systems:
Windows Server
Diagnosis
First, debug logging must be enabled in order for the stack trace to show up within the logs. After the issue has been reproduced with debug logging enabled, the following stack trace can be seen within the application logs (atlassian-bitbucket.log) at the time when the issue was reproduced:
Click for full stack trace...
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
2021-02-19 11:16:09,440 DEBUG [http-nio-7990-exec-66] redacted @QZJEMNx676x29444x1 redacted redacted,redacted "GET /rest/api/latest/projects/redacted/repos/redacted/pull-requests/redacted/activities HTTP/1.0" c.a.s.i.r.e.ServiceExceptionMapper Mapping ServiceException to REST response 500
com.atlassian.bitbucket.scm.CommandFailedException: 'C:\Program Files\Git\cmd\git.exe rev-list --format=%H%x02%P%x02%aN%x02%aE%x02%at%x02%cN%x02%cE%x02%ct%n%B%n%x03END%x04 --no-walk=unsorted 4630420166redacted6f7dbc1483redactedd1c8 4630420166redacted6f7dbc1483redactedd1c8^@ --' exited with code 128 saying: fatal: couldn't read ./packed-refs: Permission denied
at com.atlassian.bitbucket.scm.DefaultCommandExitHandler.onError(DefaultCommandExitHandler.java:47)
at com.atlassian.bitbucket.scm.git.command.GitCommandExitHandler.evaluateThrowable(GitCommandExitHandler.java:111)
at com.atlassian.bitbucket.scm.git.command.GitCommandExitHandler.onError(GitCommandExitHandler.java:208)
at com.atlassian.bitbucket.scm.DefaultCommandExitHandler.onExit(DefaultCommandExitHandler.java:32)
at com.atlassian.bitbucket.internal.process.nu.NioNuProcessHandler.callExitHandler(NioNuProcessHandler.java:285)
at com.atlassian.bitbucket.internal.process.nu.NioNuProcessHandler.finish(NioNuProcessHandler.java:326)
at com.atlassian.bitbucket.internal.process.nu.NioNuProcessHandler.onExit(NioNuProcessHandler.java:123)
at com.zaxxer.nuprocess.windows.WindowsProcess.onExit(WindowsProcess.java:485)
at com.zaxxer.nuprocess.windows.ProcessCompletions.cleanupProcess(ProcessCompletions.java:361)
at com.zaxxer.nuprocess.windows.ProcessCompletions.process(ProcessCompletions.java:187)
at com.zaxxer.nuprocess.windows.ProcessCompletions.run(ProcessCompletions.java:120)
at com.zaxxer.nuprocess.windows.WindowsProcess.run(WindowsProcess.java:279)
at com.zaxxer.nuprocess.windows.WinProcessFactory.runProcess(WinProcessFactory.java:51)
at com.zaxxer.nuprocess.NuProcessBuilder.run(NuProcessBuilder.java:273)
at com.atlassian.bitbucket.internal.process.nu.NuNioProcessHelper.run(NuNioProcessHelper.java:75)
at com.atlassian.bitbucket.internal.process.NioCommand.call(NioCommand.java:52)
at com.atlassian.stash.internal.commit.DefaultCommitService.getCommit(DefaultCommitService.java:160)
at com.atlassian.stash.internal.commit.DefaultCommitService.findCommit(DefaultCommitService.java:89)
at com.atlassian.stash.internal.pull.MergePullRequestActivityEnricher$MergeActivityVisitor.visit(MergePullRequestActivityEnricher.java:53)
at com.atlassian.stash.internal.pull.InternalPullRequestMergeActivity.accept(InternalPullRequestMergeActivity.java:47)
at com.atlassian.stash.internal.pull.MergePullRequestActivityEnricher.enrich(MergePullRequestActivityEnricher.java:35)
at com.atlassian.stash.internal.pull.DefaultPullRequestService.enrichActivities(DefaultPullRequestService.java:1662)
at com.atlassian.stash.internal.pull.DefaultPullRequestService.enrichActivities(DefaultPullRequestService.java:1655)
at com.atlassian.stash.internal.pull.DefaultPullRequestService.getActivities(DefaultPullRequestService.java:579)
at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:26)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70)
at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53)
at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
at com.atlassian.stash.internal.rest.pull.PullRequestResource.getActivities(PullRequestResource.java:349)
at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
at com.atlassian.analytics.client.filter.UniversalAnalyticsFilter.doFilter(UniversalAnalyticsFilter.java:75)
at com.atlassian.analytics.client.filter.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:48)
at com.atlassian.bitbucket.internal.xcode.web.XcodeUserAgentFilter.doFilter(XcodeUserAgentFilter.java:36)
at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
at com.atlassian.bitbucket.internal.ratelimit.servlet.filter.RateLimitFilter.doFilter(RateLimitFilter.java:75)
at com.atlassian.plugin.connect.plugin.auth.scope.ApiScopingFilter.doFilter(ApiScopingFilter.java:81)
at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
at com.atlassian.stash.internal.web.auth.AuthorizationFailureInterceptor.doFilterInternal(AuthorizationFailureInterceptor.java:39)
at com.atlassian.stash.internal.spring.security.StashAuthenticationFilter.doFilter(StashAuthenticationFilter.java:85)
at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doInsideSpringSecurityChain(BeforeLoginPluginAuthenticationFilter.java:112)
at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doFilter(BeforeLoginPluginAuthenticationFilter.java:75)
at com.atlassian.security.auth.trustedapps.filter.TrustedApplicationsFilter.doFilter(TrustedApplicationsFilter.java:94)
at com.atlassian.oauth.serviceprovider.internal.servlet.OAuthFilter.doFilter(OAuthFilter.java:67)
at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
at com.atlassian.plugin.connect.plugin.auth.oauth2.DefaultSalAuthenticationFilter.doFilter(DefaultSalAuthenticationFilter.java:69)
at com.atlassian.plugin.connect.plugin.auth.user.ThreeLeggedAuthFilter.doFilter(ThreeLeggedAuthFilter.java:109)
at com.atlassian.jwt.internal.servlet.JwtAuthFilter.doFilter(JwtAuthFilter.java:37)
at com.atlassian.analytics.client.filter.DefaultAnalyticsFilter.doFilter(DefaultAnalyticsFilter.java:26)
at com.atlassian.analytics.client.filter.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:48)
at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doBeforeBeforeLoginFilters(BeforeLoginPluginAuthenticationFilter.java:90)
at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doFilter(BeforeLoginPluginAuthenticationFilter.java:73)
at com.atlassian.stash.internal.request.DefaultRequestManager.doAsRequest(DefaultRequestManager.java:81)
at com.atlassian.stash.internal.hazelcast.ConfigurableWebFilter.doFilter(ConfigurableWebFilter.java:38)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.lang.Thread.run(Unknown Source)
... 336 frames trimmed
Caused by: com.atlassian.utils.process.ProcessException: Non-zero exit code: 128
at com.atlassian.bitbucket.internal.process.nu.NioNuProcessHandler.callExitHandler(NioNuProcessHandler.java:277)
... 58 common frames omittedCause
The root cause for this error is not currently known. The error has so far only been seen on Bitbucket environments hosted on Windows Server (varying versions), where the Bitbucket version is greater than 7.6.0.
ℹ️ From our analysis, when monitoring Windows with Microsoft Sysinternals Live there seems to have some latency, or lag, in the file system releasing the file to be open for write. The issue then seen when git will try to access the
packed-refsfile several times and ultimately fail as a result of the file system failing to release the write lock quickly.In our investigation into other usages of the
packed-refsfile at around the same time as the issue, we have also seen that another git process has "FILE LOCKED WITH ONLY READERS" which means that anyone can read it, but you cannot write to the file until the lock is released. This aligns with the theory that Windows does not seem to release the lock quickly - causing a race between the number of times git will try and the file system releasing the lock. This race condition results in the issue being inconsistent and difficult to readily reproduce.
Solution
The only known solution at this time is to migrate the Bitbucket instance to be hosted on a Linux operating system instead of Windows, given that the issue appears related to git's interaction with the Windows filesystem.
Migrating to Linux will also prepare your team for any planned change to our Data Center licensing option, as
Data Center instances
are only supported on Linux operating systems
.
Workaround
It's possible that this race condition can be avoided by changing the timing on when Bitbucket performs the needed garbage collection operations.
By default, the system waits ~1 minute after a push to try to repack the refs. The following settings change forces Bitbucket to wait for ~5 minutes instead, which could make it less likely that Bitbucket would hit this race condition.
1
plugin.bitbucket-git.gc.delay=300ℹ️ This setting should be set in the Bitbucket Configuration properties file.
Was this helpful?