Summary

Error “Too many open files in system” is written to the logs when running Bamboo or a Bamboo Remote Agent.

2021-04-26 10:00:00,000 ERROR \[0-BAM::ln-dockerbuild01.dev.atlassian.com: 1::Agent:pool-3-thread-1] \[BuildLogFileWriter]
java.io.FileNotFoundException: <path> \(Too many open files in system\)
        at java.io.FileOutputStream.open0(Native Method\)
        at java.io.FileOutputStream.open(FileOutputStream.java:270\)
        at java.io.FileOutputStream.<init>\(FileOutputStream.java:213\)
        at java.io.FileWriter.<init>\(FileWriter.java:107\)

Environment

Linux 

Diagnosis

Ulimit setting or a single user issue

If the command below will show the value for "open files" and "max user processes." If this value is set to 16384 or lower the system could log the error and the build will fail. The default for this setting in most linux distributions is between 1024 and 4096.

sudo -u <atlbitbucket> bash -c "ulimit -n -u"

System value or if ulimit is set high

For the system, Please see your Linux distro for the default value.  This error can be logged when multiple users have many open files.  For instance, if using docker all of the files are opened as different users.  So the number of files can increase significantly.    This is how to find the current settings.

cat /proc/sys/fs/file-max

Cause

For the user environment nofile is set too low, use solution Ulimit Settings.  Or for the system fs.file-max is set too low, use solution File-max.

Note that the max value for file-max, at time of writing, is 2147483584 on x86_64.  If not using x86_64, please see your documentation for the max value.

Solutions

Ulimit Settings

Remote Agent had a "Too many open files in system errors" in the system logs. You are looking for the recommend ulimits settings for remote agents.

Our current recommendation to disable this check by setting the value very high for the user running the Remote Agent for noproc and nofile. We suggest to modify the permitted number of file handles per process, edit /etc/security/limits.conf and add or edit the following lines:

Bamboo_USER           soft    noproc          16384
Bamboo_USER           hard    noproc          16384
Bamboo_USER           soft    nofile          16384
Bamboo_USER           hard    nofile          16384

Replacing Bamboo_USER with the name of the user running Bamboo.  Please note that noproc is the number of processes that a user may open.  This is a similar setting to nofile but can be hard to diagnose if hitting this value.  A user will need to be logged out and logged back in for all login environments for this to take effect, stopping and starting may not be enough.  If unsure, please reboot the system.

File-max

For the system value needs to be high enough for all users of the system.  On a current Ubuntu system the default is 6549203. This is usually high enough.  But if you run into this issue, increasing the number with a 0 at the end of the current limit, for example 65492030, should be enough.  To set this value for a live system, use this command as root, sudo will probably not work:

~$ sudo -s
[sudo] password for user:
~# cat /proc/sys/fs/file-max
6549203
~# echo 65492030 >> /proc/sys/fs/file-max
~# cat /proc/sys/fs/file-max
65492030
~#

To make this change permanent, edit or add the following line in the /etc/sysctl.conf file:

fs.file-max = 65492030

If you have increased this value to the max or have increased the default number more than once, please submit a support ticket.