Indexer Paused with "Too many open files" Error

Problem

While indexing a repository the indexer stops and the following error can be seen in the logs (example below for an SVN repository):

ERROR [FishEye SVN Diff Fetcher] fisheye.app DiffProcessor-getRequiredInfo - SVN Error diffing revision 4127
com.cenqua.fisheye.rep.RepositoryClientException: org.tigris.subversion.javahl.ClientException: svn: Can't read length line from file /artimi/db/revs/138: /artimi/db/revs/138 (Too many open files)
	at com.cenqua.fisheye.svn.SvnThrottledClient.diff(SvnThrottledClient.java:119)
	at com.cenqua.fisheye.svn.diff.DiffProcessor.doDiff(DiffProcessor.java:55)
	at com.cenqua.fisheye.svn.diff.DiffProcessor.processChangePaths(DiffProcessor.java:194)
	at com.cenqua.fisheye.svn.diff.DiffProcessor.getRequiredInfo(DiffProcessor.java:317)
	at com.cenqua.fisheye.svn.diff.DiffProcessor.processMessage(DiffProcessor.java:268)
	at com.cenqua.fisheye.svn.diff.DiffFetcher.process(DiffFetcher.java:64)
	at com.cenqua.fisheye.svn.diff.DiffFetcher.run(DiffFetcher.java:45)
	at java.lang.Thread.run(Unknown Source)
Caused by: org.tigris.subversion.javahl.ClientException: svn: Can't read length line from file /artimi/db/revs/138: /artimi/db/revs/138 (Too many open files)
	at org.tigris.subversion.javahl.JavaHLObjectFactory.throwException(JavaHLObjectFactory.java:435)
	at org.tmatesoft.svn.core.javahl.SVNClientImpl.throwException(SVNClientImpl.java:1309)
	at org.tmatesoft.svn.core.javahl.SVNClientImpl.diff(SVNClientImpl.java:783)
	at org.tigris.subversion.javahl.SVNClient.diff(SVNClient.java:206)
	at com.cenqua.fisheye.svn.SvnThrottledClient.diff(SvnThrottledClient.java:116)
	... 7 more

Cause

The indexer has hit the the maximum permitted number of open files in Linux/UNIX operating systems.

Diagnosis

The current values for open file descriptors and maximum allowed file descriptors can be determined by turning on debug logging under Administration >> Global Settings >> Server >> Debug Logging

After debug logging as been turned on, the Resource Monitor will provide values for ofd (open file descriptors) and mfd (maximum file descriptors):

DEBUG [Resource Monitor ] fisheye SystemResourceMonitor$MonitoringTask-run - tc=243, ofd=4080, mfd=4096, fm=114635672, tm=847773696

The above output indicates that FishEye can have a maximum of 4096 file descriptors and currently is using 4080 file descriptors.

Resolution

There are two possible resolutions:

  1. Optimize the Search Indexes on restart.

    More information can be found on the following page: Optimizing the Search Indexes on Restart

  2. Increase the number of allowed file handles per process (and possibly for the entire system).
     

    There are two file handle limits that can affect FishEye/Crucible: the total number of file handles allowed per process and the number of file handles allowed on the entire system.

    FishEye/Crucible can hold a high number of open file handles which often exceeds common per-process default limits – usually between 1000 and 4000 open files. If these limits are reached we recommend increasing the limit to 40000. The total number of file handles allowed on the entire system is usually set high enough by default but on an server with many other running applications this too may need to be increased.

    These settings may vary between operating systems so please check with a system administrator or perform additional research before making any changes. Described below is the most common approach to troubleshooting and fixing this problem.


    Setting the permitted number of file handles per process

    To check the number of currently open file handles, run sudo lsof -u FISHEYE_USER (restricted to the FisheEye running user) or sudo lsof -p FISHEYE_PROCESS (restricted to the fisheye process).

    To check the permitted number of file handles per process, run ulimit -n

    To modify the permitted number of file handles per process, edit /etc/security/limits.conf and add (replace FISHEYE_USER with the name of the user that will run FishEye/Crucible or use * to make the change for all users):

    Single user: FISHEYE_USER

     It is recommended to set the limits for the FISHEYE_USER explicitly first before increasing the limit for all users.
    FISHEYE_USER           soft    nofile          40000
    FISHEYE_USER           hard    nofile          40000

    All users:

    *           soft    nofile          40000
    *           hard    nofile          40000

    Setting the permitted number of file handles over the whole system

    To check the number of open file handles, run sudo lsof.

    To check the total number of open file permitted in the whole system, run cat /proc/sys/fs/file-max.

    To modify the total number of open file permitted in the whole system, edit /etc/sysctl.conf and add the line:

    fs.file-max = 40000
    

    Note: After making these changes the OS will need to be restarted. For further information please see this page.

    Ubuntu note

    The Ubuntu server requires additional additional steps in order have the larger limits register (based on this article and this article):  add “session required pam_limits.so” in /etc/pam.d/common-session.

    1. sudo vi /etc/pam.d/common-session
    2. Add session required pam_limits.so to the end of the file.
    3. Reboot the OS.

    j2ee note

    If FishEye/Crucible is running under j2ee, add the ulimit into FishEye/Crucible's daemontool runscript.

    ulimit -n 40000

 

Was this helpful?

Thanks for your feedback!

Why was this unhelpful?

Have a question about this article?

See questions about this article

Powered by Confluence and Scroll Viewport