Problem

One or more of the following symptoms is evident:

• When trying to merge a pull request via a pull request, the user receives an error about merge conflicts even though there are no conflicts and automatic merging works in a local clone
• The latest commits to the source branch are not visible in the pull request. The pull request is only showing commits made up to when the PR was first opened, or else the pull request is simply not showing the most recent commits

Diagnosis

Symptom#1: Enabling debug logging and retrying the merge shows the following exception:

2017-03-08 15:37:48,871 DEBUG [http-nio-7990-exec-212] user1 *16PQXR9x937x12398006x1 1tpssec 192.168.X.XXX "POST /rest/api/latest/projects/ATLAS/repos/my-repo/pull-requests/123/merge HTTP/1.1" c.a.s.i.r.e.ServiceExceptionMapper Mapping ServiceException to REST response 409
com.atlassian.bitbucket.scm.MergeException: The merge could not be completed automatically. Please clone 'ATLAS/my-repo', checkout 'master' and merge 'feature/my-cool-feature' (or commit '4507d8c9900988afc2834cbbc6936d38e45a18bc') manually, resolving any conflicts, and push the result.
	at com.atlassian.stash.internal.scm.git.merge.MergeCommand.newMergeException(MergeCommand.java:170) ~[na:na]
	... 279 frames trimmed
Caused by: com.atlassian.bitbucket.scm.CommandFailedException: '/usr/local/bin/git update-ref refs/heads/master 72c4c311fe49a8af0d0dc2acb4bad7147e812114 35489f96511b968e7d44d2de48d0aeb75642e0eb' exited with code 128 saying: fatal: update_ref failed for ref 'refs/heads/master': cannot lock ref 'refs/heads/master': is at 06c585c775c1f1b79d2b1ad18842778875da2ffa but expected 35489f96511b968e7d44d2de48d0aeb75642e0eb
	at com.atlassian.bitbucket.scm.DefaultCommandExitHandler.onError(DefaultCommandExitHandler.java:46) ~[bitbucket-spi-4.11.1.jar:na]
	... 37 common frames omitted
Caused by: com.atlassian.utils.process.ProcessException: Non-zero exit code: 128
	at com.atlassian.bitbucket.scm.SummarizingProcessHandler.complete(SummarizingProcessHandler.java:47) ~[bitbucket-spi-4.11.1.jar:na]
	... 44 common frames omitted

Specifically, the following line in the above merge exception is indicative of this issue:

update_ref failed for ref 'refs/heads/master': cannot lock ref 'refs/heads/master': is at 06c585c775c1f1b79d2b1ad18842778875da2ffa but expected 35489f96511b968e7d44d2de48d0aeb75642e0eb

Symptom#2: Enabling debug logging and retrying is giving you such behavior when the rescoping of the commits finishes successfully and references updated to a new HEAD commit but a moment after the HEAD moved to an old position again.

Cause

Cause#1: The pull request is not up to date with the tip of the source branch, or the tip of the target branch.

When new commits are made to the source or the target, the pull request should be rescoped - in other words, recalculated to test whether the PR will be successful against the current state of both the source and the target. Under some rare circumstances, this rescoping might not have happened, meaning that the pull request is out of sync with the state of the source or target branches, and so attempting to merge from the PR will fail.

There are a couple of reasons for this to occur:

• New commits being received to branches, and the rescoping of PR's to include the latest commits, is inherently subject to race conditions. Whilst this is a rare occurrence it is possible on very busy repositories for this to happen. Normally this should resolve itself.
• On a clustered instance, where all nodes use Hazelcast to share objects in memory, there are some edge case scenarios where Hazelcast objects may fail to be kept up to date.

Cause#2: In the repository installed Post-Update hook changing the HEAD refs (some kind of backup mirroring). Most probably the hook is not registered in Bitbucket UI as an addon. Bitbucket rescope functionality and post update hook have a racing conditions.

Workaround

Workaround#1: The below steps can be followed, and these should be tried in order:

1. Wait a minute to allow the PR to be rescoped, then refresh the page and retry merging the PR

2. Decline the PR and reopen it to force the PR to be rescoped manually, then merge it
3. Restart Bitbucket Server. On a clustered instance, a full cluster shutdown needs to be performed (so all nodes are shut down before restarting any). You should only try this if the issue occurs repeatedly and the above workarounds are not working.

Workaround#2: The custom post-update hook should be disabled or fixed by adding some waiting time in order to prevent a racing condition with Bitbucket rescoping functionality.