Server Unreachable - Timeout when merging or viewing Diff of pull request
Symptoms
Upon merging a pull request the merge dialog will be displayed for around 30 seconds before timing out with "Server Unreachable – The server didn't respond. You may retry your request when the server comes back up." Note that refreshing the page in the browser actually shows that the pull request was merged. The "Delete Branch" option doesn't work and the timeout message is also seen when viewing the diff of a pull request. Also, pushing code via command-line gets to the point where all objects are written 100% and then pauses for ~45 seconds before finally completing.
Message seen on the screen:
Cause
Browser-side code will timeout after one minute. The merge completes because the HTTP POST leaves the customer browser after clicking "Merge". It arrives at the Stash server which processes it but the response to the POST is lost on the way back to the client. Possible causes:
- Proxy overload
- Network issues
- Stash running slow on a Windows server because it is being executed as a user that is part of the Administrator group.
- Plugins in your browser (i.e. reported similar issues proved to have been caused by AdBlock plugin)
- Add-ons on your Stash instance (i.e. reported similar issues proved to have been caused by Awesome Graphs for Stash plugin)
- Git 1.8.3.X and Git 2.2.x - 2.4.0 working with a STASH_HOME mounted on a NFS are not supported by Stash due to performance issues. Please consult our Supported platforms document.
- - STASH-7360Getting issue details... STATUS : if you have a large repository containing binary files merges often genuinely take longer than a minute. The error "Server Unreachable" is given even though the merge is still occurring behind the scenes.
- - STASH-7457Getting issue details... STATUS : after upgrading to Stash 3.9.X some pull requests hang for long periods trying to display the diff. When the diff is eventually displayed, the pull request indicates the diff has conflicts but none are shown. This can also manifest as a lengthy (~5 minute) delay merging pull requests, after which the pull request ultimately fails to merge. A "Server unreachable" error will be shown after ~1 minute due to STASH-7360, but refreshing shows that the merge didn't ever complete.
Diagnosis
- Bypass the reverse proxy to try and fix the problem.
- Check you network latency.
- Check your browser:
- Test access with different browsers/versions
- Run your browser in "safe mode" to isolate it from installed plugins that could be causing the issue (i.e.how to start firefox in "safe mode")
- Check your browser configuration for a proxy that could be causing the issue. Bypass this proxy.
- Check your logs for Stash Plugin Issues
Awesome Graphs: check for messages like the ones below:
2014-05-09 00:08:10,479 DEBUG [pool-8-thread-1] com.atlassian.stash.scm.BaseCommand Executed /usr/local/bin/git log 3472f5f866c30e31758853016401189e3b02b6d2 --pretty=format:%H --shortstat --no-merges 2014-05-09 00:08:10,479 WARN [pool-8-thread-1] c.s.s.g.s.LogThrowableExitHandler Unable to execute command completely: /usr/local/bin/git log 3472f5f866c30e31758853016401189e3b02b6d2 --pretty=format:%H --shortstat --no-merges. Throwable: com.atlassian.utils.process.ProcessTimeoutException: process timed out 2014-05-09 00:08:10,479 WARN [pool-8-thread-1] c.s.s.g.idx.GraphsIndexingService ((Diffs)) -> 359 2014-05-09 00:08:10,551 WARN [pool-8-thread-1] c.s.s.graphs.idx.GraphsIndexingJob GraphsIndexingJob --> stop() 2014-05-09 00:08:10,551 WARN [pool-8-thread-1] c.s.s.graphs.idx.GraphsIndexingJob GraphsIndexingJob --> start() 2014-05-09 00:08:26,898 DEBUG [pool-8-thread-1] com.atlassian.stash.scm.BaseCommand Executed /usr/local/bin/git rev-list --format=%H%x02%h%x02%P%x02%p%x02%aN%x02%aE%x02%at%n%B%n%x03 --stdin -- 2014-05-09 00:08:26,900 WARN [pool-8-thread-1] c.s.s.g.idx.GraphsIndexingService (()) -> 8592
- To see if the problem is being caused by "Git 1.8.3.X and Git 2.2.x - 2.4.0" or
-
STASH-7360Getting issue details...
STATUS
or
-
STASH-7457Getting issue details...
STATUS
- Enable your Stash Profiling logging
Look for the following profiling event:
2015-06-01 11:37:47,273 | drift:thread-1 | - | - | - [121430ms] - Drift: Calculate for com.atlassian.stash.internal.pull.comment.drift.DriftCommentUpdateProcessor$CommentDriftCalculator@76cec2d0 [121368ms] - Drift: Map anchors 3193 [16ms] - Set com.atlassian.stash.internal.pull.comment.CommentPostProcessor.processAll(InternalCommentable,Set) [16ms] - com.atlassian.stash.internal.comment.PluginCommentPostProcessor.processAll 1 comments [16ms] - com.atlassian.stash.internal.comment.like.property.CommentLikePropertyProvider.provideProperties for 1 comments [121336ms] - git: with pull request lock 2:2533 [15ms] - InternalPullRequest com.atlassian.stash.internal.pull.PullRequestDao.findByRepositoryScopedId(int,long) [121321ms] - git: resolve effective diff 2533@1 [120744ms] - git: create new merge 2533@1 [120681ms] - git: merge <USERNAME>/<REPOSITORY>:refs/heads/client_seeding_cleanup -> <PROJECT>/<REPOSITORY>:refs/heads/development [172ms] - git: clone 2 [172ms] - C:\Program Files (x86)\Git\cmd\git.exe clone --no-checkout --shared -b development D:\Atlassian\stash-home\shared\data\repositories\2 D:\Atlassian\stash-home\tmp\git\<REPOSITORY>-work7276869962633828049.git [140ms] - git: reset 818beae5e0bb5faa3998e7fe5bce3b64f2318185 [140ms] - C:\Program Files (x86)\Git\cmd\git.exe reset --quiet 818beae5e0bb5faa3998e7fe5bce3b64f2318185 -- [328ms] - git: merge 57fab50d40b6286ae12f20d8cfb4d96c3f47945b [328ms] - C:\Program Files (x86)\Git\cmd\git.exe merge -m Automatic merge --no-ff --log 57fab50d40b6286ae12f20d8cfb4d96c3f47945b [120041ms] - git: fetch 2 [120041ms] - C:\Program Files (x86)\Git\cmd\git.exe pack-objects --delta-base-offset -q --stdout [514ms] - git: resolve merge-base 2533@1 [483ms] - git: fetch client_seeding_cleanup for 2533@1 [483ms] - C:\Program Files (x86)\Git\cmd\git.exe fetch D:\Atlassian\stash-home\shared\data\repositories\183 refs/heads/client_seeding_cleanup: [31ms] - C:\Program Files (x86)\Git\cmd\git.exe merge-base 57fab50d40b6286ae12f20d8cfb4d96c3f47945b 818beae5e0bb5faa3998e7fe5bce3b64f2318185 [47ms] - git: update refs 2533@1 [32ms] - C:\Program Files (x86)\Git\cmd\git.exe update-ref stash-refs/pull-requests/2533/from 57fab50d40b6286ae12f20d8cfb4d96c3f47945b 3ea38ec83a47bb90eaca5774e062c561e296ffd3 [15ms] - C:\Program Files (x86)\Git\cmd\git.exe update-ref stash-refs/pull-requests/2533/merge 818beae5e0bb5faa3998e7fe5bce3b64f2318185 [62ms] - Drift: Strategy - Composite (FileComment, FastForward, Diff, OrphanAll) 3193 [31ms] - Drift: Strategy - FastForward 3193 [31ms] - Drift: FastForward - Merge changes [3ea38ec83a47bb90eaca5774e062c561e296ffd3]->[57fab50d40b6286ae12f20d8cfb4d96c3f47945b] 3193 [31ms] - C:\Program Files (x86)\Git\cmd\git.exe diff-tree -C -r --format=%H 3ea38ec83a47bb90eaca5774e062c561e296ffd3 57fab50d40b6286ae12f20d8cfb4d96c3f47945b -- [31ms] - Drift: Strategy - Diff 3193 [31ms] - Drift: Diff - Merge diff [3ea38ec83a47bb90eaca5774e062c561e296ffd3]->[57fab50d40b6286ae12f20d8cfb4d96c3f47945b] 3193 [31ms] - C:\Program Files (x86)\Git\cmd\git.exe diff -C --color=never -U0 --dst-prefix=dst:// --src-prefix=src:// 3ea38ec83a47bb90eaca5774e062c561e296ffd3 57fab50d40b6286ae12f20d8cfb4d96c3f47945b --
- Git 1.8.3.X
If the operation is taking a long time on the
[172ms] - git: clone
line, that's likely Git 1.8.3
-
-
STASH-7360Getting issue details...
STATUS
- If the operation is taking a long time on the
[328ms] - git: merge
line, that's STASH-7360
- If the operation is taking a long time on the
- Enable your Stash Profiling logging
Resolution
For each of the causes above, the solution would be:
- Correctly configure the timeout on the proxy server or fix its overload.
- Fix any latency issue that could be happening on your network.
- Apply the recommendations described in Git push operations extremely slow on Windows.
- Uninstall offending browser plugin.
- Uninstall offending Stash add-on. Instructions: Uninstalling apps
- Upgrade your Git. Instructions: Installing and upgrading Git
- Upgrade to a release that fixes - STASH-7360Getting issue details... STATUS .
- Upgrade to a release that fixes - STASH-7457Getting issue details... STATUS .