Indexer Paused with "Too many open files" Error
Problem
While indexing a repository the indexer stops and the following error can be seen in the logs (example below for an SVN repository):
ERROR [FishEye SVN Diff Fetcher] fisheye.app DiffProcessor-getRequiredInfo - SVN Error diffing revision 4127
com.cenqua.fisheye.rep.RepositoryClientException: org.tigris.subversion.javahl.ClientException: svn: Can't read length line from file /artimi/db/revs/138: /artimi/db/revs/138 (Too many open files)
at com.cenqua.fisheye.svn.SvnThrottledClient.diff(SvnThrottledClient.java:119)
at com.cenqua.fisheye.svn.diff.DiffProcessor.doDiff(DiffProcessor.java:55)
at com.cenqua.fisheye.svn.diff.DiffProcessor.processChangePaths(DiffProcessor.java:194)
at com.cenqua.fisheye.svn.diff.DiffProcessor.getRequiredInfo(DiffProcessor.java:317)
at com.cenqua.fisheye.svn.diff.DiffProcessor.processMessage(DiffProcessor.java:268)
at com.cenqua.fisheye.svn.diff.DiffFetcher.process(DiffFetcher.java:64)
at com.cenqua.fisheye.svn.diff.DiffFetcher.run(DiffFetcher.java:45)
at java.lang.Thread.run(Unknown Source)
Caused by: org.tigris.subversion.javahl.ClientException: svn: Can't read length line from file /artimi/db/revs/138: /artimi/db/revs/138 (Too many open files)
at org.tigris.subversion.javahl.JavaHLObjectFactory.throwException(JavaHLObjectFactory.java:435)
at org.tmatesoft.svn.core.javahl.SVNClientImpl.throwException(SVNClientImpl.java:1309)
at org.tmatesoft.svn.core.javahl.SVNClientImpl.diff(SVNClientImpl.java:783)
at org.tigris.subversion.javahl.SVNClient.diff(SVNClient.java:206)
at com.cenqua.fisheye.svn.SvnThrottledClient.diff(SvnThrottledClient.java:116)
... 7 more
Cause
The indexer has hit the the maximum permitted number of open files in Linux/UNIX operating systems.
Diagnosis
The current values for open file descriptors and maximum allowed file descriptors can be determined by turning on debug logging under Administration >> Global Settings >> Server >> Debug Logging
After debug logging as been turned on, the Resource Monitor will provide values for ofd (open file descriptors) and mfd (maximum file descriptors):
DEBUG [Resource Monitor ] fisheye SystemResourceMonitor$MonitoringTask-run - tc=243, ofd=4080, mfd=4096, fm=114635672, tm=847773696The above output indicates that Fisheye can have a maximum of 4096 file descriptors and currently is using 4080 file descriptors.
Resolution
There are two possible resolutions:
- Optimize the Search Indexes on restart.
More information can be found on the following page: Optimizing the Search Indexes on Restart Increase the number of allowed file handles per process (and possibly for the entire system).
There are two file handle limits that can affect Fisheye/Crucible: the total number of file handles allowed per process and the number of file handles allowed on the entire system.
Fisheye/Crucible can hold a high number of open file handles which often exceeds common per-process default limits – usually between 1000 and 4000 open files. If these limits are reached we recommend increasing the limit to 40000. The total number of file handles allowed on the entire system is usually set high enough by default but on an server with many other running applications this too may need to be increased.
These settings may vary between operating systems so please check with a system administrator or perform additional research before making any changes. Described below is the most common approach to troubleshooting and fixing this problem.
Setting the permitted number of file handles per process
To check the number of currently open file handles, runsudo lsof -u FISHEYE_USER(restricted to the FisheEye running user) orsudo lsof -p FISHEYE_PROCESS(restricted to the fisheye process).
To check the permitted number of file handles per process, runulimit -n
To modify the permitted number of file handles per process, edit/etc/security/limits.confand add (replace FISHEYE_USER with the name of the user that will run Fisheye/Crucible or use * to make the change for all users):Single user: FISHEYE_USER
It is recommended to set the limits for theFISHEYE_USERexplicitly first before increasing the limit for all users.FISHEYE_USER soft nofile 40000 FISHEYE_USER hard nofile 40000All users:
* soft nofile 40000 * hard nofile 40000Setting the permitted number of file handles over the whole system
To check the number of open file handles, run
sudo lsof.To check the total number of open file permitted in the whole system, run
cat /proc/sys/fs/file-max.To modify the total number of open file permitted in the whole system, edit
/etc/sysctl.confand add the line:fs.file-max = 40000Note: After making these changes the OS will need to be restarted. For further information please see this page.
Ubuntu note
The Ubuntu server requires additional additional steps in order have the larger limits register (based on this article): add “session required pam_limits.so” in
/etc/pam.d/common-session.sudo vi /etc/pam.d/common-session- Add
session required pam_limits.soto the end of the file. - Reboot the OS.
j2ee note
If Fisheye/Crucible is running under j2ee, add the ulimit into Fisheye/Crucible's daemontool runscript.
ulimit -n 40000