Clone fails with java.io.OutputStream.write for large repositories when scm-cache is enabled
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Problem
Clone fails for large repositories.
The following appears in the atlassian-bitbucket.log
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
2017-10-06 00:09:25,390 DEBUG [scm-request-poller:thread-1] c.a.b.scm.ssh.SshCommandExitHandler {}: Threw the following exception:
com.atlassian.utils.process.ProcessException: Non-zero exit code: 128
at com.atlassian.bitbucket.scm.SummarizingProcessHandler.complete(SummarizingProcessHandler.java:47)
at com.atlassian.utils.process.ExternalProcessImpl.wrapUpProcess(ExternalProcessImpl.java:667)
at com.atlassian.utils.process.ExternalProcessImpl.finish(ExternalProcessImpl.java:194)
..
..
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.lang.Thread.run(Thread.java:748)
... 5 frames trimmed
2017-10-06 00:09:25,390 INFO [ssh-scm-request-handler] <user> @15FI10Xx9x437x0 1izd1l1 10.0.0.254 SSH - git-upload-pack '/<proj>/<repo>.git' c.a.u.c.internal.AbstractCacheEntry Invalidating cache entry clone|1c9e1be2b6dd70b129814c141385a39d. An error occurred while writing to the cache Write to closed pipe
2017-10-06 00:09:25,392 DEBUG [ssh-scm-request-handler] <user> @15FI10Xx9x437x0 1izd1l1 10.0.0.254 SSH - git-upload-pack '/<proj>/<repo>.git c.a.b.s.c.i.s.CachingSshUploadPackRequest [GA/ipad[63]] Exception encountered while processing a git-upload-pack request
java.io.IOException: Write to closed pipe
at com.atlassian.util.contentcache.util.PipedStreams.writeBytesAvailable(PipedStreams.java:93)
at com.atlassian.util.contentcache.util.PipedStreams.access$1100(PipedStreams.java:23)
at com.atlassian.util.contentcache.util.PipedStreams$PipedOutputStream.awaitSpace(PipedStreams.java:284)
at com.atlassian.util.contentcache.util.PipedStreams$PipedOutputStream.write(PipedStreams.java:232)
at java.io.OutputStream.write(OutputStream.java:75)
at com.atlassian.bitbucket.scm.cache.internal.ssh.UploadPackProxy.streamPack(UploadPackProxy.java:202)
at com.atlassian.bitbucket.scm.cache.internal.ssh.CachingSshUploadPackRequest$PackContentProvider.lambda$apply$0(CachingSshUploadPackRequest.java:429)
at com.atlassian.bitbucket.util.TimerUtils.time(TimerUtils.java:54)Diagnosis
Diagnostic Steps
scm-cache is enabled and clone fails for large repositories
Cause
Buffer size limit have been enabled in sysctl.conf or in any other configuration files and set with low values
e.g.
1
2
3
net.core.rmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 16384 16777216Solution
Resolution
Remove if any buffer settings configured for hosting operating system.
Was this helpful?