FishEye Documentation

FishEye 3.4 Documentation
FishEye 3.3 Documentation

FishEye 3.2 Documentation

More...  

 

FishEye Knowledge Base Resources

You're visiting the Fisheye and Crucible Knowledge Base. Visit the FishEye and Crucible Knowledge Base Home for an overview.

Skip to end of metadata
Go to start of metadata

Symptoms

While indexing a repository the indexer stops and the following error can be seen in the logs (example below for an SVN repository):

Cause

The indexer has hit the the maximum permitted number of open files in Linux/UNIX operating systems.

Resolution

There are two possible resolutions:

  1. Optimising the Search Indexes on Restart

    More information can be found on the following page: Optimising the Search Indexes on Restart

  2. Increase the Number of Allowed System/User File Handles

    Increase the permitted number of open file handles for the FishEye user and/or whole system. These settings may vary between operating systems, please check with a system administrator or research beforehand. Below is described the most common approach to troubleshooting and fixing this problem.

    In general, there are two limits to the the number of file handles that will affect FishEye: the total number of file handles allowed on the system, and the number of file handles allowed per process for the user that runs FishEye.

    Currently FishEye can hold a high number of open file handles, exceeding common per-process default limits of 1000 to 4000 open files. At this point we recommend that if experiencing issues, to increase this limit to 40000. The total number of file handles allowed on the whole system is usually set high enough by default, but on a heavily loaded system may also need to be increased.

    Setting the permitted number of file handles per process

    To check the number of currently open file handles, run sudo lsof -u FISHEYE_USER (restricted to the FisheEye running user) or sudo lsof -p FISHEYE_PROCESS (restricted to the fisheye process).

    To check the permitted number of file handles per process, run ulimit -n

    To modify the permitted number of file handles per process, edit /etc/security/limits.conf and add (replacing FISHEYE_USER with the name of the user running fisheye or * for all users):

    Setting the permitted number of file handles over the whole system

    To check the number of open file handles, run sudo lsof.

    To check the total number of open file permitted in the whole system, run cat /proc/sys/fs/file-max.

    To modify the total number of open file permitted in the whole system, edit /etc/sysctl.conf and add the line:

    Note: After making these changes the OS will need to be restarted. For further information please see this page.

    Ubuntu note

    Icon

    The Ubuntu server requires additional additional steps in order have the larger limits register (based on this article and this article):  add “session required pam_limits.so” in /etc/pam.d/common-session.

    1. sudo vi /etc/pam.d/common-session
    2. Add session required pam_limits.so to the end of the file.
    3. Reboot the OS.

    j2ee note

    Icon

    If FishEye is running under j2ee, add the ulimit into Fisheye's daemontool runscript.



Related Content

 Expand to see related content
Help us improve!
Is this article helpful?
Is it well written?
Is the content complete?

8 Comments

  1. How many open files are to be expected?

    On which factors does the number of open files depend?

    1. Hello, Dennis.

      The default number of open files, according to this Linux guide, is 4096, but it may vary depending on your server configuration.

      If you're experiencing issues related to the number of open files, don't hesitate to open a support ticket at https://support.atlassian.com

      Kind regards,
      Felipe Kraemer
      Atlassian Support

      1. Hm, seems like my question wasn't clear enough.

        I wanted to know how many open files FishEye usually has.
        And what influences this number. Size of the repository? Number of repositories?

        1. Hi Dennis,

          We're currently investigating this — you might like to follow this jira issue.

          Anna

  2. This certainly worked for us - we had the default of 1024 set.  I had restarted Fisheye a couple of times which would only ever temporarily fix the issue.  Upping to 40000 for the user seems to be doing the trick.  I'll guess we'll see in time!  Thanks for the article though!!

  3. I've set the limit to 65535 files and I still get the following for Git repos:

    com.cenqua.fisheye.rep.DbException: java.io.FileNotFoundException: /usr/local/fisheye/home/cache/globalfe/idx0/_3z4.tii (Too many open files)

    To be sure, I've rebooted the server and checked with ulimit.  I also monitor the number of open files used by the Atlassian user which never goes over about 2600.

    This completely prevents indexing.

  4. On Ubuntu you need to edit another file to enable the user specific limits

    Source

  5. I don't see in the documentation above or any of the replies a way to see the number of files fisheye has open - lsof -p <pid> and lsof -u FISHEYE_USER return all the files open, but not a count. So, I did this:

    sudo lsof -p <pid> | wc -l

    and it (eventually) returned 4024. Much more useful. And if 4096 is the limit, that would explain why we're getting the "Too many open files" error message. However, ulimit -n returns 1024? How does that reconcile with lsof? I followed the instructions and updated /etc/security/limits.conf to 40k, rebooted, and now ulimit -n returns 40k. We'll see if this is resolved. 

    -Kelly