Summary

Unable to clone the repository from Bitbucket Mirror. The repository is shown as synchronised on the UI but when attempted to clone the repository from the mirror, it errors out

$ git clone <repository_clone_url_from_mirror> 
Cloning into '<repository>'...

fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output

Environment

Bitbucket Datacenter 8.9.5

Bitbucket Mirror 8.9.5

Diagnosis

• Enable debug logging on the Bitbucket mirror instance by adding the below properties in the bitbucket.properties file.

  logging.logger.com.atlassian.bitbucket.internal.mirroring=DEBUG
  logging.logger.ROOT=INFO
  logging.logger.com.atlassian.bitbucket=DEBUG
  logging.logger.com.atlassian.event.remote=DEBUG
  logging.logger.com.atlassian.stash=DEBUG
  logging.logger.org.hibernate=INFO
  logging.logger.org.springframework=INFO
  logging.logger.com.hazelcast.internal.diagnostics.HealthMonitor=DEBUG

  Restart the Bitbucket Mirror instance to get the changes to effect. 

• If the repositories are mirrored project-wise, remove the project containing the repository, wait for a few minutes and re-add the project to sync the repositories in the project from scratch.
• Verify the repository sync status from the logs or from the UI. 

• Attempt to clone the repository from the mirror instance and you will encounter the below error

  $ git clone <repository_clone_url_from_mirror> 
  Cloning into '<repository>'...
  
  fetch-pack: unexpected disconnect while reading sideband packet
  fatal: early EOF
  fatal: fetch-pack: invalid index-pack output

• The below error is seen in the Mirror logs

  ZZZZ-YY-XX XX:XX:XX,XXX DEBUG [http-nio-7990-exec-X]  c.a.s.i.w.filters.StreamGuardFilter Failed to flush buffer; the remote client aborted the connection
  org.apache.catalina.connector.ClientAbortException: java.io.IOException: java.net.SocketTimeoutException
  	at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:351)
  	at javax.servlet.ServletResponseWrapper.flushBuffer(ServletResponseWrapper.java:168)
  	at javax.servlet.ServletResponseWrapper.flushBuffer(ServletResponseWrapper.java:168)
  	at com.atlassian.stash.internal.web.util.web.CountingHttpServletResponseWrapper.flushBuffer(CountingHttpServletResponseWrapper.java:70)
  	at java.base/java.lang.Thread.run(Thread.java:829)
  	... 23 frames trimmed
  Caused by: java.io.IOException: java.net.SocketTimeoutException
  	at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1415)
  	... 5 common frames omitted
  Caused by: java.net.SocketTimeoutException: null

  Access log

  <IP_ADDRESS> | https | i@<REQUEST_ID> | - | <DATE_TIMESTAMP> | "POST /scm/<PROJ>/<REPOSITORY>/git-upload-pack HTTP/1.0" | "" "git/2.37.1.windows.1" | - | - | - | - | - | - | 
  <IP_ADDRESS> | https | o@<REQUEST_ID> | eatqs | <DATE_TIMESTAMP> | "POST /scm/<PROJ>/<REPOSITORY>/git-upload-pack HTTP/1.0" | "" "git/2.37.1.windows.1" | 200 | 2553112 | 1078887070 | async:error | 175758 | - | 

Cause

When both the client and server report the connection is aborted by a remote client, it is possible that the connection was aborted by the host that is in the middle, i.e. Load-balancer/ proxy server. In this case, the clone operation fails after fetching ~1GB (1078887070) of data. A size limitation in the load-balancer/ proxy server configuration may be causing the connection to abort.

Solution

Remove or increase the size limitation on the proxy server / load-balancer configuration before the Bitbucket Mirror server.