Summary

Bamboo is encountering (Too many open files) errors after recent Java (JDK) update. At this point, it's known to affect JDK 1.8.0_271 and 1.8.0_281. This can cause operations to error out frequently, render the application unusable or cause it to crash. Analysis of the lsof output for the Bamboo process shows that a disproportionate number of open files are related to TCP sockets that are never released / leaking.

Please note, there are other causes for this such as having ulimits set too low for what is required or a bug that was fixed in Bamboo 6.9.2. This is discussed more indepth in the below knowledge-base article and bug reports

• Subversion Indexer Paused with Too Many Open files error
• BAM-16283 - Getting issue details... STATUS

Environment

• JDK 1.8.0_271 < b37
• JDK 1.8.0_281 < b31

Diagnosis

$BAMBOO_HOME/logs/atlassian-bamboo.log contains many errors about Too many open files. Example:

Caused by: java.lang.IllegalStateException: java.io.FileNotFoundException: /opt/atlassian/bamboo/atlassian-bamboo/WEB-INF/lib/... (Too many open files)

Check the lsof output for the Bamboo Process ID (PID):

BAMBOO_PID=$(ps aux | grep bamboo | grep Bootstrap | awk '{print $2}')
lsof -p $BAMBOO_PID

It shows a high number of open file descriptors are unbound sockets. Example:

...
java    50129 bamboo  253u     sock                0,9      0t0 3282247 protocol: TCPv6
java    50129 bamboo  254u     sock                0,9      0t0 3282248 protocol: TCPv6
java    50129 bamboo  255u     sock                0,9      0t0 3282249 protocol: TCPv6
java    50129 bamboo  256u     sock                0,9      0t0 3282250 protocol: TCPv6
java    50129 bamboo  257u     sock                0,9      0t0 3282251 protocol: TCPv6
java    50129 bamboo  258u     sock                0,9      0t0 3282252 protocol: TCPv6
java    50129 bamboo  259u     sock                0,9      0t0 3282253 protocol: TCPv6
java    50129 bamboo  260u     sock                0,9      0t0 3282254 protocol: TCPv6
java    50129 bamboo  261u     sock                0,9      0t0 3282255 protocol: TCPv6
java    50129 bamboo  262u     sock                0,9      0t0 3282256 protocol: TCPv6
java    50129 bamboo  263u     sock                0,9      0t0 3282257 protocol: TCPv6
...

 This count will increase overtime when leaking sockets:

lsof -p $BAMBOO_PID | grep sock | grep protocol | wc -l
1144

(If Bamboo is currently running) Check the Java version your Bamboo is using:

• Bamboo Administration >> System Information >> Java / JVM Information: JVM version

 If Bamboo is not running, check the Java version your Bamboo typically uses:

$JAVA_HOME/jre/bin/java -version

Affected versions are listed in the Environment section above.

Cause

• [JDK-8256818] SSLSocket that is never bound or connected leaks socket resources

Solution

Upgrade the JDK that your Bamboo installation uses to JDK 1.8.0_282 or more recent. AdoptOpenJDK provides distributions of this which you can get from the below location:

• https://adoptopenjdk.net/releases.html?variant=openjdk8&jvmVariant=hotspot

Ensure that the $JAVA_HOME environment variable is updated to this new JDK for the user and/or service that launches Bamboo.

Alternatively, the first affected version is claimed to be 271, so downgrading to Java 8 pre 271 may also be a viable option – but we recommend being on the later version with a full fix.