Git LFS fails with HTTP error over SSH: failed to push some refs to and Client error 413
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Problem
Pushing attempt with Git LFS results in when cloning, pushing or pulling/fetching, the client may fail like so:
$ git push origin master # using SSH
Git LFS: (186 of 187 files, 4 skipped) 268.25 MB / 448.34 MB, 17.66 MB skipped
Client error: https://your.bitbucket.com/rest/git-lfs/storage/proj/repo/84c72136e4d22c2c42819d902b4400a9cf73f2b04fd6f9f8666739c3bc0a0f79 from HTTP 413
Client error: https://your.bitbucket.com/rest/git-lfs/storage/proj/repo/1c9335681f70363edeb00820ef394fac66125856cca0c3c8cf2c66636739f277 from HTTP 413
Client error: https://your.bitbucket.com/rest/git-lfs/storage/proj/repo/e79f7abda0ba90f2be4e613f78d1ecb7344970711106fe80d0028106f97f613c from HTTP 413
error: failed to push some refs to 'ssh://git@your.bitbucket.com:7999/proj/repo.git'Cause
This indicates the repository does not have LFS enabled. A user with REPO_ADMIN permissions is able to enable LFS support.
Analysis
The following come up on different logs:
atlassian-bitbucket.log
2016-02-02 14:03:12,626 INFO [NioProcessor-8] c.a.b.i.s.s.SelfClosingChannelSession Executing command: git-receive-pack '/proj/repo.git'
atlassian-bitbucket-access.log
192.168.1.10 | ssh | i@1PQD53Ux843x157553x0 | user | 2016-02-02 14:03:12,626 | SSH - git-receive-pack '/proj/repo.git' | - | - | - | - | - | - | os4we8 |
atlassian-bitbucket.log
2016-02-02 14:03:13,050 INFO [NioProcessor-9] c.a.b.i.s.s.SelfClosingChannelSession Executing command: git-lfs-authenticate proj/repo.git upload
atlassian-bitbucket-access.log
192.168.1.10 | ssh | i@1PQD53Ux843x157554x1 | user | 2016-02-02 14:03:13,050 | SSH - git-lfs-authenticate proj/repo.git upload | - | - | - | - | - | - | 6pp10j |
192.168.1.10 | ssh | o@1PQD53Ux843x157554x1 | user | 2016-02-02 14:03:13,067 | SSH - git-lfs-authenticate proj/repo.git upload | - | 0 | 0 | 552 | ssh:user:id:1 | 17 | 6pp10j |
192.168.1.10,127.0.0.1 | https | i@1PQD53Ux843x157555x1 | - | 2016-02-02 14:03:13,575 | "POST /scm/proj/repo.git/info/lfs/objects/batch HTTP/1.0" | "" "git-lfs/1.1.0 (GitHub; darwin amd64; go 1.5.1)" | - | - | - | - | - | - |
192.168.1.10,127.0.0.1 | https | o@1PQD53Ux843x157555x1 | user | 2016-02-02 14:03:13,669 | "POST /scm/proj/repo.git/info/lfs/objects/batch HTTP/1.0" | "" "git-lfs/1.1.0 (GitHub; darwin amd64; go 1.5.1)" | 200 | 8907 | 12064 | - | 94 | 1d9nj3u |
nginx
2016/02/02 14:00:46 [error] 16062#0: *1224429 client intended to send too large body: 21693741 bytes, client: 4.3.2.1, server: stash.example.com, request: "PUT /rest/git-lfs/storage/MAIN/repo/e79f7abda0ba90f2be4e613f78d1ecb7344970711106fe80d0028106f97f613c HTTP/1.1", host: "stash.example.com"
In addition...
You can run a Git DEBUG. On the example we saw, the response by nginx was saying "too-large header".
$ GIT_TRACE=1 git push
14:44:49.161648 git.c:348 trace: built-in: git 'push'
...
trace git-lfs: tq: starting 3 transfer workers
trace git-lfs: tq: running as batched queue, batch size of 100
trace git-lfs: tq: sending batch of size 4
...
trace git-lfs: api: batch 4 files
trace git-lfs: HTTP: POST https://stash.example.com/scm/main/repo.git/info/lfs/objects/batch
trace git-lfs: HTTP: 200
trace git-lfs: HTTP: {"objects":[{"oid":"1bd91c03...62dbaa79","size":2746892,"actions":{}},{"oid":"58ef47926...70bb6ea89c","size":10639089,"actions":{}},{"oid":"2631c0...c6f6be538","size":2890471,"actions":{}},{"oid":"84c721...bc0a0f79","size":72016893,"actions":{"upload":{"href":"https://stash.example.com/rest/git-lfs/storage/MAIN/repo/84c72136
trace git-lfs: HTTP: e4d22c2...c0a0f79","header":{"Authorization":"JWT eyJ0e...NzA0NW
trace git-lfs: HTTP: IwOD...ifYKU"}}}}]}
trace git-lfs: HTTP:
Git LFS: (0 of 4 files, 3 skipped) 0 B / 84.20 MB, 15.52 MB skipped trace git-lfs: HTTP: PUT https://stash.example.com/rest/git-lfs/storage/MAIN/repo/84c72136e4...9c3bc0a0f79
trace git-lfs: HTTP: 413
trace git-lfs: HTTP: <html>
<head><title>413 Request Entity Too Large</title></head>
<body bgcolor="white">
<center><h1>413 Request Entity Too Large</h1></center>
<hr><center>nginx/1.9.3 (Ubuntu)</center>
</body>
</html>
...
Git LFS: (2 of 4 files, 3 skipped) 262.52 KB / 84.20 MB, 15.52 MB skipped
Client error: https://stash.example.com/rest/git-lfs/storage/MAIN/repo/84c7213...a0f79 from HTTP 413
error: failed to push some refs to 'ssh://git@stash.example.com:7999/main/repo.git'
$
Resolution
- This usually happens if you have a reverse proxy that does not allow a header that is too big. Please configure your HTTP proxy correctly.
The fix for nginx is to add/modify this server setting for the relevant vhost pointing to bitbucket server, modified for the maximum size of file you intend to store via git-lfs:
client_max_body_size 256M;
Even if you perform Git LFS operations over SSH, they are actually done of HTTPS.
Last modified on Sep 15, 2016
Powered by Confluence and Scroll Viewport.