Problem

When a user tries to merge a pull request when Bitbucket Server is hosted on Windows, they see either of these symptoms:

• The Merge button is deactivated and the warning "This pull request can't be merged" is visible, but the pull request's diff page does not show conflicts.
• The Merge button is deactivated, the warning "This pull request can't be merged" is visible and the pull request's diff page does not show files with long paths.

In some cases, Bitbucket will show a message saying:

Bitbucket Server could not create the merge diff for this pull request.
To resolve this warning you could try to push a new commit to update the pull request, attempt to merge the pull request anyway, or ask a system administrator to review the logs. Read more about this warning.

When this happens, the user can attempt to merge the changes locally and push them to Bitbucket later.

Diagnosis

In order to diagnose this issue you need to:

• Enable Bitbucket server DEBUG logging via the UI
• Enable Bitbucket server profiling logging via the UI

Once that's done, you should:

• Check your atlassian-bitbucket-profiler.log for the following string "create new merge <PRID>". For example, you can see below a profiling log for a merge that went wrong:

  * "create new merge 2254"

  * that is followed by a "git: resolve merge-base" which only happens on a failed merge

  2016-10-17 20:35:07,731 | AtlassianEvent::thread-2 | - | - | -^M
  [28782ms] - PullRequestMergeability com.atlassian.stash.pull.PullRequestService.canMerge(int,long)
    [31ms] - InternalPullRequest com.atlassian.stash.internal.pull.PullRequestDao.findByRepositoryScopedId(int,long)
    [28751ms] - PullRequestMergeability com.atlassian.stash.internal.pull.InternalMergeRequestCheckService.checkMergeability(InternalMergeRequest)
      [1ms] - Calling merge request check com.izymes.workzone:mdt-reviewers-merge-check of type com.izymes.workzone.hook.ApprovalsMergeCheck
      [1ms] - Calling merge request check com.atlassian.stash.stash-repository-hooks:configuredMergeCheck of type com.atlassian.stash.internal.hook.repository.RepositoryMergeRequestCheckAdapter
        [1ms] - boolean com.atlassian.stash.internal.hook.repository.InternalRepositoryHookService.visitEnabledHooks(Repository,Class,RepositoryHookSettingsCallback)
          [1ms] - List com.atlassian.stash.internal.repository.RepositoryHookDao.findEnabledByKey(InternalRepository,Collection)
      [4042ms] - Calling merge request check com.atlassian.stash.stash-branch-permissions:vetoingMergeCheck of type com.atlassian.stash.internal.repository.ref.restriction.RestrictionEnforcer
        [7ms] - Ticket com.atlassian.stash.throttle.ThrottleService.acquireTicket(String)
        [2154ms] - Ref com.atlassian.stash.repository.RefService.resolveRef(Repository,String)
          [2154ms] - C:\Program Files\Git\cmd\git.exe for-each-ref --sort=-objecttype --format=%(objecttype)%02%(refname)%02%(objectname)%02%(*objectname) refs/heads/ refs/tags/
        [1172ms] - Ref com.atlassian.stash.repository.RefService.resolveRef(Repository,String)
          [1172ms] - C:\Program Files\Git\cmd\git.exe for-each-ref --sort=-objecttype --format=%(objecttype)%02%(refname)%02%(objectname)%02%(*objectname) refs/heads/ refs/tags/
        [688ms] - Ref com.atlassian.stash.repository.RefService.resolveRef(Repository,String)
          [688ms] - C:\Program Files\Git\cmd\git.exe for-each-ref --sort=-objecttype --format=%(objecttype)%02%(refname)%02%(objectname)%02%(*objectname) refs/heads/ refs/tags/
      [22438ms] - git: with pull request lock 12:2254
        [22422ms] - git: resolve effective diff 2254@2
          [21125ms] - git: create new merge 2254@2
            [21078ms] - git: merge MFS/odyssey refs/heads/ODSY-10155 -> refs/heads/develop
              [1672ms] - git: clone 12
                [1656ms] - C:\Program Files\Git\cmd\git.exe clone --no-checkout --shared -b develop D:\Atlassian\Stash\shared\data\repositories\12 D:\Atlassian\Stash\tmp\git\odyssey-work2717632738003455963.git
              [6218ms] - git: reset 49705a41f3b78e2414c9c002ed8eee6701bfb46c
                [6218ms] - C:\Program Files\Git\cmd\git.exe reset --quiet 49705a41f3b78e2414c9c002ed8eee6701bfb46c --
              [13172ms] - git: merge 1d8556528170c16305e74b22ff31027a541f24f2
                [13172ms] - C:\Program Files\Git\cmd\git.exe merge -m Automatic merge --no-ff --log 1d8556528170c16305e74b22ff31027a541f24f2
          [641ms] - git: resolve merge-base 2254@2
            [641ms] - C:\Program Files\Git\cmd\git.exe merge-base 1d8556528170c16305e74b22ff31027a541f24f2 49705a41f3b78e2414c9c002ed8eee6701bfb46c
          [609ms] - git: update refs 2254@2
            [484ms] - C:\Program Files\Git\cmd\git.exe update-ref stash-refs/pull-requests/2254/to 49705a41f3b78e2414c9c002ed8eee6701bfb46c 2c7d7f41ab9fdf30233e0fa9e382ffc2ae7de7b9
            [125ms] - C:\Program Files\Git\cmd\git.exe update-ref stash-refs/pull-requests/2254/merge 2c7d7f41ab9fdf30233e0fa9e382ffc2ae7de7b9
      [1297ms] - Ref com.atlassian.stash.repository.RefService.resolveRef(Repository,String)
        [1297ms] - C:\Program Files\Git\cmd\git.exe for-each-ref --sort=-objecttype --format=%(objecttype)%02%(refname)%02%(objectname)%02%(*objectname) refs/heads/ refs/tags/
      [953ms] - Ref com.atlassian.stash.repository.RefService.resolveRef(Repository,String)
        [953ms] - C:\Program Files\Git\cmd\git.exe for-each-ref --sort=-objecttype --format=%(objecttype)%02%(refname)%02%(objectname)%02%(*objectname) refs/heads/ refs/tags/

• Surely enough you can crosscheck that information with your atlassian-bitbucket.log. In this specific case, that happened on AtlassianEvent::thread-2 so you can look for that thread at around the same time in your application logs. Sure enough you can find the stack trace:

  2016-10-17 20:35:04,122 DEBUG [AtlassianEvent::thread-2]  com.atlassian.stash.scm.BaseCommand Executed C:\Program Files\Git\cmd\git.exe merge -m Automatic merge --no-ff --log 1d8556528170c16305e74b22ff31027a541f24f2
  2016-10-17 20:35:04,200 INFO  [AtlassianEvent::thread-2]  c.a.s.i.s.g.p.DefaultPullRequestAutoMergeStrategy MFS/odyssey[12]: Pull request 2254@2 failed automatic merging
  com.atlassian.stash.scm.MergeException: ODSY-10155 could not be merged to develop automatically. Please clone MFS/odyssey, checkout develop and merge ODSY-10155 (or revision 1d8556528170c16305e74b22ff31027a541f24f2) manually, resolving any conflicts, and push the result.
  ...
  ...
  ...
  Caused by: com.atlassian.stash.exception.CommandFailedException: 'C:\Program Files\Git\cmd\git.exe merge -m Automatic merge --no-ff --log 1d8556528170c16305e74b22ff31027a541f24f2' exited with code 128 saying: error: Your local changes to the following files would be overwritten by merge:
  	Code/Your.File.Name/node_modules/gulp-eslint/node_modules/gulp-util/node_modules/dateformat/node_modules/meow/node_modules/indent-string/node_modules/repeating/node_modules/is-finite/package.json
  	Code/Your.File.Name/node_modules/gulp/node_modules/liftoff/node_modules/findup-sync/node_modules/glob/node_modules/minimatch/node_modules/brace-expansion/node_modules/balanced-match/.travis.yml
  ...
  ...
  ...
  Aborting
  	at com.atlassian.stash.scm.DefaultCommandExitHandler.onError(DefaultCommandExitHandler.java:47) ~[stash-spi-3.11.2.jar:na]
  	at com.atlassian.stash.scm.git.common.command.GitCommandExitHandler.evaluateThrowable(GitCommandExitHandler.java:105) ~[stash-scm-git-common-3.11.2.jar:na]
  	at com.atlassian.stash.scm.git.common.command.GitCommandExitHandler.onError(GitCommandExitHandler.java:159) ~[stash-scm-git-common-3.11.2.jar:na]
  	at com.atlassian.stash.scm.DefaultCommandExitHandler.onExit(DefaultCommandExitHandler.java:33) ~[stash-spi-3.11.2.jar:na]
  	at com.atlassian.stash.scm.BaseCommand.callExitHandler(BaseCommand.java:138) ~[stash-spi-3.11.2.jar:na]
  ...
  ...
  ...
  Caused by: com.atlassian.utils.process.ProcessException: Non-zero exit code: 128
  ...
  ...
  ...

• The issue above, thus was caused by Cause #2. In this specific case it was caused by the fact that Node was enforcing a long path.

Cause

Cause #1:

We've seen this issue happening because of a specific add-on: All Pull Requests Add-on release 1.3.2 installed on Stash release 3.7.2.

Cause #2:

Since Bitbucket Server uses Git to check out files when performing the merge in a pull request, they can can fail on Windows because Windows applies an absolute limit to the length (260 characters) of a directory's path.  Contributing factors include files in the pull request that:

• Are located in deeply nested directories (or  BITBUCKET_HOME is deeply nested).
• Have long file names.

Cause #3:

The disk where BITBUCKET_HOME is mounted is full. Bitbucket Server needs to check for merge conflicts, and to do that it will clone the repository. If it is unable to clone the repository (due to disk space) then this would cause an error, and the merge conflict to display.

Cause #4:

BITBUCKET_HOME  is a defined as a  UNC (\\network\path\here) path.

Cause #5:

Git processes are being killed and were unable to remove .lock files on the server side repository. Some background Git operations were attempted, but could not be performed because of the locks. If encountered this issue you would see the following error in your atlassian-bitbucket.log: 

2015-09-25 00:47:13,869 INFO  [drift:thread-2]  c.a.s.i.s.g.p.DefaultPullRequestAutoMergeStrategy <PROJECT_SLUG>/<REPO_SLUG>[<REPO_NUMBER]: Pull request 1666@173 failed automatic merging: '/usr/bin/git fetch /stash/home/tmp/git/repo691390098980685340.git +HEAD:' exited with code 0 saying: From /bitbucket/home/tmp/git/aim-work691390098980685340
 * branch            HEAD       -> FETCH_HEAD
Auto packing the repository for optimum performance. You may also
run "git gc" manually. See "git help gc" for more information.
fatal: Unable to create '/stash/home/shared/data/repositories/<REPO_NUMBER>/refs/heads/test.lock': File exists.
 
If no other git process is currently running, this probably means a
git process crashed in this repository earlier. Make sure no other git
process is running and remove the file manually to continue.
error: failed to run reflog

Cause #6:

In order to show the diff, Bitbucket Server will perform a merge in the background and then show the results. If that merge times out, the application will show conflicts (even if those were resolved). A timeout is shown in the atlassian-bitbucket.log:

2015-12-07 09:12:43,157 INFO  [AtlassianEvent::thread-2]  c.a.s.i.s.g.p.DefaultPullRequestAutoMergeStrategy PROJ/repo[12]: Pull request 2022@2 failed automatic merging
com.atlassian.stash.scm.MergeException: feature-branch1 could not be merged to develop automatically. Please clone PROJ/repo, checkout develop and merge feature-branch1 (or revision ae7dc44fb6d296373b0ad698e10e201ea38cefac) manually, resolving any conflicts, and push the result.
	at com.atlassian.stash.internal.scm.git.merge.MergeCommand.newMergeException(MergeCommand.java:130) ~[stash-scm-git-3.11.2.jar:na]
	at com.atlassian.stash.internal.scm.git.merge.MergeCommand.doWithWorkTree(MergeCommand.java:120) ~[stash-scm-git-3.11.2.jar:na]
	...
Caused by: com.atlassian.stash.exception.ServerException: An error occurred while executing an external process: process timed out
	at com.atlassian.stash.scm.git.common.command.GitCommandExitHandler.evaluateThrowable(GitCommandExitHandler.java:113) ~[stash-scm-git-common-3.11.2.jar:na]
	at com.atlassian.stash.scm.git.common.command.GitCommandExitHandler.onError(GitCommandExitHandler.java:159) ~[stash-scm-git-common-3.11.2.jar:na]
    ...
Caused by: com.atlassian.utils.process.ProcessTimeoutException: process timed out
	at com.atlassian.utils.process.ExternalProcessImpl.wrapUpProcess(ExternalProcessImpl.java:662) ~[atlassian-processutils-1.7.2.jar:na]
	at com.atlassian.utils.process.ExternalProcessImpl.finish(ExternalProcessImpl.java:161) ~[atlassian-processutils-1.7.2.jar:na]
	...

If you look for these operations in the atlassian-bitbucket-profiler.log, you will find:

          [124095ms] - git: merge MFS/odyssey refs/heads/ODSY-9107_decomission_are_2 -> refs/heads/develop
            [859ms] - git: clone 12
              [859ms] - C:\Program Files\Git\cmd\git.exe clone --no-checkout --shared -b develop D:\Atlassian\Stash\shared\data\repositories\12 D:\Atlassian\Stash\tmp\git\odyssey-work522816173142781568.git
            [3188ms] - git: reset b1f652d2b7f7f4833ac1304f11f1f4769834d0b2
              [3188ms] - C:\Program Files\Git\cmd\git.exe reset --quiet b1f652d2b7f7f4833ac1304f11f1f4769834d0b2 --
            [120032ms] - git: merge ae7dc44fb6d296373b0ad698e10e201ea38cefac
              [120032ms] - C:\Program Files\Git\cmd\git.exe merge -m Automatic merge --no-ff --log ae7dc44fb6d296373b0ad698e10e201ea38cefac

Line 7 from logs suggests you're hitting the default timeout value of the pull request merge timeout property, which defines the maximum amount of time (in seconds) any command used to perform a merge to support the "merge" diff mode is allowed to execute or idle.

In the Bitbucket Server config properties file, look for this property to determine what the value is set at:

plugin.bitbucket-git.pullrequest.merge.auto.timeout=120

Workarounds

Cause #1:

Disable/uninstall offending add-on.

Cause #2:

Git has a limit of 4096 characters for a filename, but on Windows, when the git client is compiled with Git for Windows (previously compiled with msysGit), it uses an older version of the Windows API and there's a limit of 260 characters for a file path. Possible workarounds:

• Open PowerShell or cmd.exe and run the following:

git config --system core.longpaths true

• Restructuring the Git repository so that the files in the pull request have shorter directory paths.
• Migrate Bitbucket Server to a different server running Linux.

There is a feature request open in Bitbucket Server that you may watch/vote for.

Cause #3:

Free up disk space.

Cause #4:

Map a network drive or use the mklink  command to create a symbolic link to the networked location and update  BITBUCKET_HOME to reference the new drive letter/path.

Cause #5:
This tends to occur when older versions of Bitbucket Server or Git are being used on the server side. Ideally you should upgrade to the latest version of Git and Bitbucket Server. To resolve this in a one off manner, manually delete these lock files from the file system of the server. The path in the error above should tell you where these lock files are located. In the example above, you would navigate to /bitbucket/home/shared/data/repositories/<REPO_NUMBER>/refs/heads and delete test.lock.

Cause #6:

• Edit your <BITBUCKET_HOME>/shared/bitbucket.properties
  • If your Bitbucket version is 6.10 or earlier, insert the parameter:
    plugin.bitbucket-git.pullrequest.merge.auto.timeout=360
  • If your Bitbucket version is 7.0 or later, insert the parameter:
    plugin.bitbucket-git.pullrequest.operation.timeout=360

• Restart your instance
• Go to the Pull Request again

Solution:

The reason why the merge is so slow needs to be addressed. A good start is reviewing the Git versions and possible performance issues outlined on our Supported platforms document.