Problem

• SSH git clone fail intermittently while HTTP(S) works as per normal.
• This is noticed after an upgrade to Bitbucket Server 3.6.0 - 3.10.x

The following appears in the catalina.log

30-Jun-2015 04:38:43.453 SEVERE [ajp-nio-127.0.0.1-8009-exec-11] org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun 
 java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)

The following appears in the atlassian-bitbucket.log

2015-06-29 05:17:36,139 ERROR [ajp-nio-127.0.0.1-8009-exec-31] xxxx @xxxx 127.0.0.1 "GET /scm/something/something.git/info/refs HTTP/1.1" c.a.s.i.s.g.p.h.GitSmartExitHandler something/something_shared[292]: Read request from 127.0.0.1 failed: com.atlassian.utils.process.ProcessException: Non-zero exit code: 255
The following was written to stderr:
error: cannot fork() for git-http-backend: Resource temporarily unavailable 

Diagnosis

Environment

• Large Enterprise server with similar environment as follows:

   <cpus>64</cpus>

Diagnostic Steps

• Enable git debug with below will show the debug logging stuck while trying to identity file .ssh cert

  export GIT_TRACE_PACKET=1
  export GIT_TRACE=1
  git clone ssh://admin@<mybitbucket.com>/something/something.git

Cause

The default max thread on Bitbucket Server 3.10.x is too high for an Enterprise server with lots of hyper-threaded cores. This changes was made for BSERV-6868 - Getting issue details... STATUS to fix insufficient threads. However, this is then to be an issue when internal thread pool can grow up to 100 + 20 * 64 = 1380 threads which at some point the server could not create any more threads because it ran out of (virtual) memory.

The following are the default set per Bitbucket Server version:

3.11.0: executor.max.threads=${scaling.concurrency}
3.6.0 - 3.10.x: executor.max.threads=100+20*${scaling.concurrency} (where the later is # of cpu)
2.11.x: executor.max.threads=100

Workaround

This is a known bug and a workaround would be to re-configure the default value of the executor.max.threads.

• Shutdown Bitbucket Server

• Add the following in bitbucket.properties

  executor.max.threads=2*cpu
  

• Startup Bitbucket Server

Resolution