Jira server: IO exception "Too many open files" then crashes

Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.

Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible

Summary

Problem

Jira is running out of open files descriptors in Linux environment eventually failing completely to function leading to a crash.

You can see similar exceptions in the logs:

1 2 3 4 5 6 7 8 9 2017-05-14 14:43:25,674 http-nio-8080-exec-10 ERROR anonymous 883x135283x1 1lds68b 192.168.0.121,192.168.0.242 /rest/usermanagement/1/search [c.a.p.r.c.error.jersey.ThrowableExceptionMapper] Uncaught exception thrown by REST service: java.io.FileNotFoundException: /opt/atlassian/jira/atlassian-jira/WEB-INF/lib/spring-security-crypto-3.1.0.RELEASE.jar (Too many open files) java.lang.IllegalStateException: java.io.FileNotFoundException: /opt/atlassian/jira/atlassian-jira/WEB-INF/lib/spring-security-crypto-3.1.0.RELEASE.jar (Too many open files) at org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.getArchiveEntry(AbstractSingleArchiveResourceSet.java:97) at org.apache.catalina.webresources.AbstractArchiveResourceSet.getResource(AbstractArchiveResourceSet.java:260) at org.apache.catalina.webresources.StandardRoot.getResourceInternal(StandardRoot.java:280) at org.apache.catalina.webresources.CachedResource.validateResource(CachedResource.java:95) at org.apache.catalina.webresources.Cache.getResource(Cache.java:69) ... 4 filtered
1 2 3 4 5 2022-08-03 12:59:03,285+0300 http-nio-8080-exec-907 url: /rest/idalko-grid/1.0/servicedesk/names/ITHD-22412; user: charlie ERROR charlie 779x16193639x2 14tlt8c 10.155.240.180,10.155.16.159 /rest/idalko-grid/1.0/servicedesk/names/ITHD-22412 [c.a.p.r.c.error.jersey.ThrowableExceptionMapper] Uncaught exception thrown by REST service: failed to create a child event loop java.lang.IllegalStateException: failed to create a child event loop at play.shaded.ahc.io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:88) at play.shaded.ahc.io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58) at play.shaded.ahc.io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47)

Diagnosis

Environment

Diagnostic Steps

  • Confirm the Open files limits on the OS and for Jira using below commands:

    1 2 cat /proc/<Jira_PID>/limits cat /proc/sys/fs/file-max

    Open files should show as more than 1024 for default and something like 8192 or more for Jira.

    OS one coming from file-max should be much bigger, it is set on kernel level typically can be bigger than 500k and can reach more than 1 million files.

  • Look for what kind of files are opened by Jira using the command:

    1 lsof -p <Jira_PID> >Jira_open_files.txt

    Usually you might see some files like the below in the list:

    1 2 3 4 java 14317 xpkjir1p 78r DIR 253,30 4096 7212034 /usr/local/pr/Jira_Software/atlassian/application-data/jira/com.deiser.jira.exporter java 14317 xpkjir1p 79r DIR 253,30 4096 7212034 /usr/local/pr/Jira_Software/atlassian/application-data/jira/com.deiser.jira.exporter java 14317 xpkjir1p 80r DIR 253,30 4096 7212034 /usr/local/pr/Jira_Software/atlassian/application-data/jira/com.deiser.jira.exporter java 14317 xpkjir1p 81r DIR 253,30 4096 7212034 /usr/local/pr/Jira_Software/atlassian/application-data/jira/com.deiser.jira.exporter

Cause

Jira installed with one these plugins on the respective versions:

They cause Jira to create a lot of open temp files and thus will consume the whole open files allowed for the Jira process eventually.

Solution

Workaround

Exporter

Table Grid Next Generation and Table Grid Editor

Not available.

Increasing the file limit will not stop the open files from increasing. Open file count will increase until the limit is reached.

Resolution

Exporter

For Exporter plugin upgrade to version 2.1.5:

  • Below are the steps provided by the plugin Vendor:

Version 2.1.5 • Released 2017-07-01 •Supported By DEISER•Paid via Atlassian• Commercial

Resolved the problem that was affecting only Jira instances installed on a CentOS/RedHat and was causing that with the time a lot of processes keep opened and was causing an impact on the instance.

IMPORTANT! If you have a CentOS/RedHat distribution, we strongly recommend following these instructions to prevent malfunctions:

1) Uninstall Exporter

2) Reboot Jira (this will close the previous open connections)

3) Install Exporter again, the latest version (2.1.5)

Thank you so much for your patience and reporting.If you have further problems don't hesitate to contact us at our Service Management.

Table Grid Next Generation and Table Grid Editor

Please downgrade to a lower version than the affected versions. Known versions that do no have the problem are:

Table Grid Editor: 1.29.17

Table Grid Next Generation: 1.13.2-jira8

After downgrading the plugin(s), you'll still need to reboot Jira for the open files to be closed.

Updated on April 2, 2025
Platform
Data Center only
Product
Jira Software Data Center
On this pageSummaryDiagnosisCauseSolution

Still need help?

The Atlassian Community is here for you.
Ask the Community