Summary

Under heavy load, Bitbucket may log errors of "Too many open files".
At the same time, Git operations may fail and web interface may not be fully functional.

Environment

This issue may occur in various versions of Bitbucket Server and Data Center on Linux.
For example, it has been seen in the version 5.16.11.

Diagnosis

The application logs (eg. atlassian-bitbucket.log) may show multiple errors similar to the following:

2021-02-02 23:59:58,584 ERROR https-jsse-nio-8443-Acceptor-0 o.apache.tomcat.util.net.NioEndpoint Socket accept failed
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source)
at sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source)
at java.lang.Thread.run(Unknown Source)
... 1 frame trimmed

Cause

Bitbucket's process, combined with its child (Git) processes, has too many open files, and has reached the maximum limit set in the Operating System.
UNIX systems have a limit on the number of files that can be concurrently open by any one process. This error indicates that the limit has been reached and Bitbucket is unable to open additional files to complete the on-going operations.

Solution based on the limits.conf file

In order to resolve this issue, you will need to allow Bitbucket to open more files than it is currently allowed to.
This involves a change in the configuration of the Operating System and a change in Bitbucket's startup procedure.

1. Log into Bitbucket's Linux server as Bitbucket's user (eg. atlbitbucket), open terminal window and run the following command, to find the currently applied limit:

   ulimit -aS | grep open

   The returned value is too low for your Bitbucket's process.

2. Use sudo command, or switch user to root, and edit /etc/security/limits.conf
   Specify the new soft and hard limit for Bitbucket's user. For example, to set both limits to 64K for user atlbitbucket, you need to specify:

   atlbitbucket      soft    nofile  65536
   atlbitbucket      hard    nofile  65536

   In some instances, the limit of 64K may still be too restrictive, and you may need to raise it to 128K (134217728).
   We do not recommend specifying an unlimited value, even if your Linux Operating System allows it.

3. Edit Bitbucket's $BITBUCKET_INSTALL_DIR/bin/_start-webapp.sh script.
   Find the line that begins with: MAX_OPEN_FILES= and change it to the same limit you specified in the previous step.
   In this example, this line should be:
   

   MAX_OPEN_FILES=65536

4. Restart the server (including the Operating System) for the new settings to be applied.

5. If this is Bitbucket Data Center with multiple nodes, repeat the steps 1 through 4 for each node, one-by-one.
   In step 4, to prevent the downtime, you may use the rolling restart of all nodes.

6. To verify if the new limit has been correctly applied, you may use one of the following two methods:

   • Generate a Support Zip (in Data Center, one from each Bitbucket node), extract it, open application-properties/application.xml and look for the line similar to:

     <max-file-descriptor>65,536</max-file-descriptor>

     This method allows you to confirm the change on each node of Bitbucket Data Center.

   • In Bitbucket's web interface, navigate to Administration > Troubleshooting and support tools > System Information.
     Next, scroll down to the last section: Operating System, where you should see:

     Max File Descriptors   65,536

     This method only allows to confirm the change on the node to which your browser is currently connected.

Solution when installed as a systemd service

When Bitbucket is installed as a systemd service you may need to update the service unit file as described below.
If you are sure Bitbucket is running as a systemd service, go straight to Step 2.

1. Check if Bitbucket is configured as a systemd service. This helps to identify if this is the case and the service name, in case you are not sure.

   grep -i bitbucket /etc/systemd/system/*.service /lib/systemd/system/*.service

   In this example, the name of our service is bitbucket.service, which is the name of the unit file itself and is located in the standard folder /etc/systemd/system.
   
   
   

2. Edit the service unit file (/etc/systemd/system/bitbucket.service in our example) and add the following line.

   LimitNOFILE=65536

3. Reboot the server.
   
   

4. To ensure the new value is being used by the application, take a Support Zip and search for the max-file-descriptor attribute in application-properties/application.xml file.
   You should find the line reflecting the change, as shown below:

   <max-file-descriptor>65,536</max-file-descriptor>