Bamboo Indexer Fails with "Too Many Open Files" Error

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.

Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible

Symptoms

While indexing or building bamboo displays Too many open files error. The error can also be presented in UI even when Bamboo is idle:

Cause

Bamboo has hit the the maximum permitted number of open files in Linux/UNIX operating systems.

If you are encountering this after a recent Java update to JDK 1.8.0_271 or 281, it may be caused by a bug in Java. Please see the below knowledge-base article that covers it:

Resolution based on the limits.conf file

Increase the permitted number of open file handles for the Bamboo user and/or whole system. These setting may vary between operating systems, so you may need to talk to your system administrator or do your own research. We'll describe the most common approach to troubleshooting and fixing this problem.

In general, there are two limits to the the number of file handles that will affect Bamboo: the total number of file handles allowed on the system, and the number of file handles allowed per process for the user that runs Bamboo.

Currently Bamboo can hold a high number of open file handles, exceeding common per-process default limits of 1000 to 4000 open files. At this point we recommend that if you're experiencing issues, increase this limit to 40000. The total number of file handles allowed on the whole system is usually set high enough by default, but on a heavily loaded system may also need to be increased.

Setting the permitted number of file handles per process

To check the number of currently open file handles, run sudo lsof -u Bamboo_USER (restricted to the Bamboo running user) or sudo lsof -p Bamboo_PROCESS (restricted to the Bamboo process).

To check the permitted number of file handles per process, run ulimit -n

To modify the permitted number of file handles per process, edit /etc/security/limits.conf and add (replacing Bamboo_USER with the name of the user running Bamboo or * for all users):

Bamboo_USER           soft    nofile          40000
Bamboo_USER           hard    nofile          40000
Setting the permitted number of file handles over the whole system

To check the number of open file handles, run sudo lsof.

To check the total number of open file permitted in the whole system, run cat /proc/sys/fs/file-max.

To modify the total number of open file permitted in the whole system, edit /etc/sysctl.conf and add the line:

fs.file-max = 65535

PS: After making these changes you need to reboot your OS.

Solution when installed as a systemd service

When Bamboo is installed as a systemd service you may need to update the service unit file as described below.
If you are sure Bamboo is running as a systemd service, go straight to Step 2.

  1. Check if Bamboo is configured as a systemd service. This helps to identify if this is the case and the service name, in case you are not sure.

    grep -i bamboo/etc/systemd/system/*.service /lib/systemd/system/*.service

    In this example, the name of our service is bamboo.service, which is the name of the unit file itself and is located in the standard folder /etc/systemd/system.


  2. Edit the service unit file (/etc/systemd/system/bamboo.service in our example) and add the following line.

    LimitNOFILE=65536
  3. Reboot the server.

  4. To ensure the new value is being used by the application, take a Support Zip (link here) and search for the max-file-descriptor attribute in application-properties/application.xml file.

    <max-file-descriptor>65,536</max-file-descriptor>


Last modified on Jun 28, 2021

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.