Summary

When performing a git operation - such as cloning, pulling or pushing - the operation fails with a Smudge error. This only happens if there are LFS objects involved (i.e. they are stored in the repository and are being cloned or pulled, or if LFS objects are to be pushed) and if you are using a personal access token (or HTTP access token) with git (by specifying the -c http.extraHeader='Authorization: Bearer <token>' command line option in git). It does not occur if you provide your username, and your token as the password to git.

Environment

Bitbucket Server or Data Center version 5.5 or newer running behind Apache HTTP Server acting as a reverse proxy.

Note: So far this behaviour has only been observed with Apache HTTP Server but it may also occur with other reverse proxies.

Diagnosis

The following error is displayed in the git client:

Cloning into 'repo'...
remote: Enumerating objects: 4, done.
remote: Counting objects: 100% (4/4), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 4 (delta 0), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (4/4), done.
Downloading Skyline_Almere.jpg (49 KB)
Error downloading object: Skyline_Almere.jpg (57cc928): Smudge error: Error downloading Skyline_Almere.jpg (57cc928f8f924606af3674d00a64e6dd9a2fa152fc7f5bd0062632c87261439b): Authentication required: Authorization error: http://192.168.1.171/rest/git-lfs/storage/PROJ/repo/57cc928f8f924606af3674d00a64e6dd9a2fa152fc7f5bd0062632c87261439b
Check that you have proper access to the repository

Errors logged to '/Users/user/Downloads/repo/.git/lfs/logs/20221230T112235.911083.log'.
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: Skyline_Almere.jpg: smudge filter lfs failed
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry with 'git restore --source=HEAD :/'

The <BitbucketHome>/log/atlassian-bitbucket.log file will have errors similar to this:

2022-12-30 10:22:35,913 WARN  [http-nio-7990-exec-7] @K84A0Dx622x477x0 192.168.1.254,127.0.0.1 "GET /rest/git-lfs/storage/PROJ/repo/57cc928f8f924606af3674d00a64e6dd9a2fa152fc7f5bd0062632c87261439b HTTP/1.1" c.a.j.i.s.DefaultAuthenticationResultHandler Signature mismatch during JWT authentication, issuer: com.atlassian.bitbucket.server.bitbucket-git-lfs-storage
com.atlassian.jwt.exception.JwtSignatureMismatchException: eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIyIiwicmVwbyI6InByb2pcL3JlcG8iLCJxc2giOiJlMGJiYTw4YTQxMmUyZDU5N2Q1NGY5YmRlYmVkMmJkNTY3YTg1MGQ4MTk5ZTcyMzJjYzY2NWU2M2JkMGY3OWU1IiwiaXNzIjoiY29tLmF0bGFzc2lhbi5iaXRidWNrZXQuc2VydmVyLmJpdGJ1Y2tldC1naXQtbGZzLXN0b3JhZ2UiLCJjb250ZXh0Ijp7InVzZXIiOnsiaGlnaGVzdFBlcm1pc3Npb24iOiJTWVNfQURNSU4iLCJkaXNwbGF5TmFtZSI6IkNHIiwic2x1ZyI6ImNnIiwidXNlcktleSI6IjIiLCJ1c2VybmFtZSI6ImNnIn19LCJleHAiOjE2NzI0ODIxNTUsImlhdCI6MTY3MjM5NTc1NX0.xPVOoEKHDJlL_EwEz9JsmYoS0bDQMLNwtVNr7HhMwKs, Bearer BBDC-NDM5MzAxNTc4MzA1OqEEPqHU4oTScTVkDs/JUzB7ptIf

Cause

Git LFS relies on JSON Web Tokens (JWT) for authentication, and it sends the token using the Authorization request header. If you use also token authentication via the Authorization header, effectively two Authorization request headers are sent to Bitbucket Server. If there is no reverse proxy in front of Bitbucket Server this works fine, however if Apache HTTP Server is installed as a reverse proxy server in front of Bitbucket Server, then it will detect the two Authorization headers on the incoming request and it will merge these two into a single Authorization header, which is sent along with the request to Bitbucket Server. The effective Authorization header value that Bitbucket Server receives will look like this:

JWT eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIyIiwicmVwbyI6InByb2pcL3JlcG8iLCJxc2giOiJlMGJiYTw4YTQxMmUyZDU5N2Q1NGY5YmRlYmVkMmJkNTY3YTg1MGQ4MTk5ZTcyMzJjYzY2NWU2M2JkMGY3OWU1IiwiaXNzIjoiY29tLmF0bGFzc2lhbi5iaXRidWNrZXQuc2VydmVyLmJpdGJ1Y2tldC1naXQtbGZzLXN0b3JhZ2UiLCJjb250ZXh0Ijp7InVzZXIiOnsiaGlnaGVzdFBlcm1pc3Npb24iOiJTWVNfQURNSU4iLCJkaXNwbGF5TmFtZSI6IkNHIiwic2x1ZyI6ImNnIiwidXNlcktleSI6IjIiLCJ1c2VybmFtZSI6ImNnIn19LCJleHAiOjE2NzI0ODIxNTUsImlhdCI6MTY3MjM5NTc1NX0.xPVOoEKHDJlL_EwEz9JsmYoS0bDQMLNwtVNr7HhMwKs, Bearer BBDC-NDM5MzAxNTc4MzA1OqEEPqHU4oTScTVkDs/JUzB7ptIf

As you can see, the HTTP access token that was provided to git via the additional Authorization header is appended after the JWT, and Bitbucket Server interprets this entire value as a JWT, which it fails to parse.

We are tracking this as a bug as BSERV-13613 - Specifying HTTP Access Token via Authorization header causes git LFS operations to fail if a reverse proxy is deployed in front of Bitbucket Server.

Workaround

To work around this problem you have two options:

1. Instead of providing the HTTP access token via the Authorization header, specify the git credentials as the username, and the token as the password. Note: This workaround is only available for personal access tokens because there is no username for repository access tokens.

2. Modify your Apache HTTP Server proxy configuration to modify the Authorization header so it removes the HTTP access token from its value, but only for LFS object requests (because non-LFS requests still need the HTTP access token in the Authorization header). This can be done by adding the following directives to your Apache HTTP Server configuration:

   <Location "/rest/git-lfs/storage/">
   		# Remove HTTP Access Tokens in current format from version 8.x
           RequestHeader edit Authorization ",? ?Bearer BBDC-.{44}" ""
   		# Remove HTTP Access Tokens in the pre-8.x format
   		RequestHeader edit Authorization ",? ?Bearer [a-zA-Z\d+\/]{44}" ""
   </Location>