Summary:

While cloning a repository from Bitbucket Data Center, the following error message appears on the client terminal.

git clone http://10.211.XX.XX:7990/scm/pro1/repo5.git
Cloning into 'repo5'...
Username for 'http://10.211.XX.XX:7990': user1
Password for 'http://user1@10.211.XX.XX:7990':
remote: aborting due to possible repository corruption on the remote side.
fatal: early EOF
fatal: fetch-pack: invalid index-pack output

Environment:

The solution has been validated in Bitbucket Data Center 8.9.10 but may be applicable to other versions.

Diagnosis:

• The atlassian-bitbucket.log will have the following error stack trace:
  

  2024-08-07 12:26:32,692 DEBUG [http-nio-7990-exec-1] @9UT0S0x746x15x0 10.211.XX.XX "POST /scm/pro1/repo5.git/git-upload-pack HTTP/1.1" c.a.s.i.a.DefaultAuthenticationService user1: Authenticator 'com.atlassian.bitbucket.server.bitbucket-crowd-sso:crowdSsoAuthHandler' opted out
  2024-08-07 12:26:32,692 DEBUG [http-nio-7990-exec-1] @9UT0S0x746x15x0 10.211.XX.XX "POST /scm/pro1/repo5.git/git-upload-pack HTTP/1.1" c.a.s.i.a.DefaultAuthenticationService user1: Authenticator 'com.atlassian.bitbucket.server.bitbucket-git-lfs:git-lfs-jwt-auth-handler' opted out
  2024-08-07 12:26:32,695 DEBUG [http-nio-7990-exec-1] @9UT0S0x746x15x0 10.211.XX.XX "POST /scm/pro1/repo5.git/git-upload-pack HTTP/1.1" c.a.s.i.a.DefaultAuthenticationService user1: Authenticator 'com.atlassian.bitbucket.server.bitbucket-access-tokens:accessTokenHttpAuthHandler' opted out
  2024-08-07 12:26:32,697 DEBUG [http-nio-7990-exec-1] @9UT0S0x746x15x0 10.211.XX.XX "POST /scm/pro1/repo5.git/git-upload-pack HTTP/1.1" c.a.s.i.a.DefaultAuthenticationService user1: Authenticator 'com.atlassian.bitbucket.server.bitbucket-authentication:crowdHttpAuthHandler' authenticated user1 (2)
  2024-08-07 12:26:32,710 DEBUG [http-scmrequest-handler:thread-9] user1 @9UT0S0x746x15x0 10.211.XX.XX "POST /scm/pro1/repo5.git/git-upload-pack HTTP/1.1" c.a.s.i.s.g.p.http.CgiStdinHandler Closing request stream (12)
  2024-08-07 12:26:32,710 DEBUG [http-scmrequest-handler:thread-9] user1 @9UT0S0x746x15x0 10.211.XX.XX "POST /scm/pro1/repo5.git/git-upload-pack HTTP/1.1" c.a.s.i.s.g.p.http.CgiStdinHandler Closing stdin (12)
  2024-08-07 12:26:32,800 DEBUG [mesh-grpc-request:thread-3] user1 @9UT0S0x746x15x0 10.211.XX.XX "POST /scm/pro1/repo5.git/git-upload-pack HTTP/1.1" c.a.s.i.s.g.p.http.CgiStdinHandler [git http-backend /var/atlassian/application-data/bitbucket/shared/data/repositories/1] has completed (1; closed)
  2024-08-07 12:26:32,801 ERROR [http-scmrequest-handler:thread-9] user1 @9UT0S0x746x15x0 10.211.XX.XX "POST /scm/pro1/repo5.git/git-upload-pack HTTP/1.1" c.a.s.i.s.g.p.h.GitSmartExitHandler PRO1/repo5[1]: Read request from 10.211.XX.XX failed
  The following was written to stderr:
  error: git upload-pack: git-pack-objects died with error.
  fatal: git upload-pack: aborting due to possible repository corruption on the remote side.
  com.atlassian.bitbucket.scm.ProcessFailedException: 'git http-backend /var/atlassian/application-data/bitbucket/shared/data/repositories/1' exited with code 1 saying: error: git upload-pack: git-pack-objects died with error.
  fatal: git upload-pack: aborting due to possible repository corruption on the remote side.
  	at com.atlassian.stash.internal.scm.git.mesh.AbstractHostingFragmentResponseObserver.callExitHandler(AbstractHostingFragmentResponseObserver.java:170)
  	at com.atlassian.stash.internal.scm.git.mesh.AbstractHostingFragmentResponseObserver.asResult(AbstractHostingFragmentResponseObserver.java:150)
  	at com.atlassian.stash.internal.scm.git.mesh.AbstractBidirectionalHostingFragmentResponseObserver.asResult(AbstractBidirectionalHostingFragmentResponseObserver.java:72)
  	at com.atlassian.stash.internal.scm.git.mesh.GrpcHostingClient.httpBackend(GrpcHostingClient.java:85)
  	at com.atlassian.stash.internal.scm.git.protocol.http.AbstractRpcHttpScmRequest.handleRequest(AbstractRpcHttpScmRequest.java:141)
  	at com.atlassian.stash.internal.scm.git.protocol.http.MeshHttpUploadPack.handleRequest(MeshHttpUploadPack.java:36)
  	at com.atlassian.stash.internal.scm.DefaultHttpScmRequestExecutor.runSync(DefaultHttpScmRequestExecutor.java:174)
  	at com.atlassian.stash.internal.scm.DefaultHttpScmRequestExecutor.access$200(DefaultHttpScmRequestExecutor.java:49)
  	at com.atlassian.stash.internal.scm.DefaultHttpScmRequestExecutor$ScmAsyncRequest.run(DefaultHttpScmRequestExecutor.java:366)
  	at com.atlassian.stash.internal.request.DefaultRequestManager.lambda$wrapForAsync$0(DefaultRequestManager.java:169)
  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  	at java.lang.Thread.run(Thread.java:750)
  	... 21 frames trimmed
  2024-08-07 12:26:32,802 DEBUG [http-nio-7990-exec-2] user1 @9UT0S0x746x15x0 10.211.XX.XX "POST /scm/pro1/repo5.git/git-upload-pack HTTP/1.1" c.a.s.i.s.DefaultHttpScmRequestExecutor PRO1/repo5[1]: Async request completed

• The atlassian-mesh.log will have the following error stack trace:

  2024-08-07 12:26:32,616 DEBUG [grpc-server:thread-4] user1 7NGPOG08x746x39x2 @9UT0S0x746x13x0 127.0.0.1 "HostingService/HttpBackend" (>1 <0) c.a.b.m.r.DefaultRpcRepositoryAccessTracker all has 1 active requests
  2024-08-07 12:26:32,623 DEBUG [process-pump:thread-2] user1 7NGPOG08x746x39x2 @9UT0S0x746x13x0 127.0.0.1 "HostingService/HttpBackend" (>1 <5) c.a.bitbucket.mesh.process.OsProcess 102613: [git http-backend /var/atlassian/application-data/bitbucket/shared/data/repositories/1] exited 0 in 4ms (stdin: 0, stdout: 296, stderr: 0)
  2024-08-07 12:26:32,650 DEBUG [grpc-server:thread-4] user1 7NGPOG08x746x40x2 @9UT0S0x746x14x0 127.0.0.1 "HostingService/HttpBackend" (>1 <0) c.a.b.m.r.DefaultRpcRepositoryAccessTracker all has 1 active requests
  2024-08-07 12:26:32,683 DEBUG [process-pump:thread-2] user1 7NGPOG08x746x40x2 @9UT0S0x746x14x0 127.0.0.1 "HostingService/HttpBackend" (>2 <5) c.a.bitbucket.mesh.process.OsProcess 102618: [git http-backend /var/atlassian/application-data/bitbucket/shared/data/repositories/1] exited 0 in 28ms (stdin: 175, stdout: 314, stderr: 0)
  2024-08-07 12:26:32,708 DEBUG [grpc-server:thread-4] user1 7NGPOG08x746x41x2 @9UT0S0x746x15x0 127.0.0.1 "HostingService/HttpBackend" (>1 <0) c.a.b.m.r.DefaultRpcRepositoryAccessTracker all has 1 active requests
  2024-08-07 12:26:32,796 WARN  [git-hook:thread-2] user1 7NGPOG08x746x41x2 @9UT0S0x746x15x0 127.0.0.1 "HostingService/HttpBackend" (>2 <3) c.a.b.m.g.hook.DefaultGitHookService git-pack-objects: Hook handling failed
  java.lang.IllegalStateException: Could not create /var/atlassian/application-data/bitbucket/caches/pack/66bda179ab517af0709b-1
  	at com.atlassian.util.contentcache.internal.util.MoreFiles.mkdir(MoreFiles.java:30)
  	at com.atlassian.util.contentcache.internal.FileContentCache.<init>(FileContentCache.java:56)
  	at com.atlassian.util.contentcache.FileContentCacheManager.createContentCache(FileContentCacheManager.java:41)
  	at com.atlassian.util.contentcache.internal.AbstractContentCacheManager.getCache(AbstractContentCacheManager.java:66)
  	at com.atlassian.bitbucket.mesh.git.cache.DefaultPackCacheService.getCache(DefaultPackCacheService.java:73)
  	at com.atlassian.bitbucket.mesh.git.hook.handler.CachingPackObjectsHookHandler.streamFromCache(CachingPackObjectsHookHandler.java:241)
  	at com.atlassian.bitbucket.mesh.git.hook.handler.CachingPackObjectsHookHandler.maybeHandle(CachingPackObjectsHookHandler.java:213)
  	at com.atlassian.bitbucket.mesh.git.hook.handler.CachingPackObjectsHookHandler.handle(CachingPackObjectsHookHandler.java:85)
  	at com.atlassian.bitbucket.mesh.git.hook.DefaultGitHookService.handleRequest(DefaultGitHookService.java:317)
  	at com.atlassian.bitbucket.mesh.git.hook.DefaultGitHookService.lambda$handleRawRequest$1(DefaultGitHookService.java:256)
  	at io.grpc.Context.run(Context.java:536)
  	at com.atlassian.bitbucket.mesh.execution.GrpcExecutionManager$GrpcExecutionContext.run(GrpcExecutionManager.java:232)
  	at com.atlassian.bitbucket.mesh.git.hook.DefaultGitHookService$HookContext.applyAndRun(DefaultGitHookService.java:551)
  	at com.atlassian.bitbucket.mesh.git.hook.DefaultGitHookService.handleRawRequest(DefaultGitHookService.java:247)
  	at com.atlassian.bitbucket.mesh.git.hook.DefaultGitHookService$1.lambda$run$0(DefaultGitHookService.java:200)
  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  	at java.lang.Thread.run(Thread.java:750)
  Caused by: java.nio.file.AccessDeniedException: /var/atlassian/application-data/bitbucket/caches/pack/66bda179ab517af0709b-1
  	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
  	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
  	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
  	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
  	at java.nio.file.Files.createDirectory(Files.java:674)
  	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
  	at java.nio.file.Files.createDirectories(Files.java:767)
  	at com.atlassian.util.contentcache.internal.util.MoreFiles.mkdir(MoreFiles.java:22)
  	... 17 common frames omitted
  2024-08-07 12:26:32,798 DEBUG [process-pump:thread-2] user1 7NGPOG08x746x41x2 @9UT0S0x746x15x0 127.0.0.1 "HostingService/HttpBackend" (>2 <7) c.a.bitbucket.mesh.process.OsProcess 102622: [git http-backend /var/atlassian/application-data/bitbucket/shared/data/repositories/1] exited 1 in 89ms (stdin: 202, stdout: 250, stderr: 149)

Cause:

This error occurs when Bitbucket doesn't have read-write permission inside <Bitbucket-Home-Directory>/caches/pack/ directory.

Solution:

Make sure the Operating system user who is running the Bitbucket process has read, write, and execute in <Bitbucket-Home-Directory>/caches/pack/ and all of its subdirectories.