On this page:

Updating JVM options when running Fisheye / Crucible on Windows

Updating JVM options when running Fisheye / Crucible on Linux or MacOS

Troubleshooting

There are a number of different Out Of Memory Errors that the JVM can encounter. Solving these errors might require that you update the default memory settings by following the steps from the previous sections of this article, according to Fisheye's host operating system.

The most common types of Out Of Memory Errors are listed below:

OutOfMemoryError: Java Heap Space

• Meaning: The JVM had to allocate more memory than it is currently allowed to.
• Default value: 1024 MB / 1 GB
• Known scenarios: 
  • During the initial repository scan period
  • While syncing a large number of users from an external user directory
  • While a large backup is being performed
  • When working on large reviews
  • Installed add-ons
• How to fix it: This usually indicates you need to increase the amount of memory the JVM can allocate (also known as memory heap) to Fisheye, however doing so may mask and / or postpone the real issue. In addition to that, there are many things to consider when increasing the available memory to your application. For example, the amount of memory being allowed cannot be greater than the RAM configured on the server and also needs to take into consideration the memory required by all the other applications installed on the system and the amount of memory required by the operating system itself. It may be needed to increase the RAM on the server before actually increasing the amount of memory the JVM used by Fisheye can allocate.
  When needed, though, to increase the amount of memory the JVM can allocate you need to:
  • If Fisheye runs on Windows and the Windows service for Fisheye exists, set / increase the Maximum memory pool field in the Apache procrun monitor application (see instructions in the section above).
  • If Fisheye runs on Windows and the Windows service for Fisheye does not exist, set / increase the -Xmx option in the FISHEYE_OPTS environment variable (see instructions in the section above).
  • If Fisheye runs on Linux or Mac, set / increase the -Xmx option in the FISHEYE_OPTS environment variable (see instructions in the section above).
  • Restart Fisheye afterwards.

OutOfMemoryError: unable to create new native thread

• Meaning: This error occurs when the operating system is unable to create new threads.
• Default value: not set by default.
• Known scenarios:
  • The JVM heap may be taking up the available RAM on the system. Big heaps take away from the space that can be allocated for the stack of a new thread.
  • The OS limits for the maximum number of threads are too low
• How to fix it:
  • Option 1: Reduce the heap: 

    The size of the stack per thread (-Xss) can also contribute to this problem. The stack size can reduce the number of threads that can be created. 
    The default value of -Xss depends on the platform: 
    - On Windows the default value depends on virtual memory.

    - On Linux / MacOS the default value is 1024 KB.
    
    To fix this problem, you should reduce the amount of memory the JVM can allocate (-Xmx) and also reduce the size of the JVM stack per thread (-Xss).
    • If Fisheye runs on Windows and the Windows service for Fisheye exists, open the Apache procrun monitor application and:
      • Set / decrease the Maximum memory pool field
      • Set the -Xss option in the Java Options list to, for example, -Xss512k.
      • See instructions in the section above.
    • If Fisheye runs on Windows and the Windows service for Fisheye does not exist, or if Fisheye runs on Linux or MacOS:
      • Set / decrease the -Xmx option in the FISHEYE_OPTS environment variable
      • Set the -Xss option to -Xss512k in the FISHEYE_OPTS environment variable
      • See instructions in the section above for each operating system.
    • Restart Fisheye afterwards.

• • Option 2: Increase the number of allowed threads:
    In Linux / MacOS systems the maximum number of allowed threads can be checked by executing ulimit -u as the user running Fisheye/Crucible.
    This limit can be increased by executing ulimit -u 2048 for example, which will set the maximum number of threads to 2048.
    Restart Fisheye afterwards.

OutOfMemoryError: GC overhead limit exceeded

• Meaning: This error indicates the JVM took too long to free up memory during its garbage collection process.
• Default value: not set by default.
• Known scenarios: The Oracle Java documentation states that this error is reported "if more than 98% of the total time is spent in garbage collection and less than 2% of the heap is recovered", and that "this feature is designed to prevent applications from running for an extended period of time while making little or no progress because the heap is too small". This may happen if the JVM starts to use the swapped memory for its heap. This will cause the JVM to take longer than normal to perform normal GC operations which will eventually cause a timeout to occur.

• How to fix it: Even though this feature can be disabled by adding the -XX:-UseGCOverheadLimit JVM option to stop this kind of error to be written in the Fisheye log files, it is not a good idea to do so. To overcome this issue, you actually need to make sure that all processes can't allocate more memory than the RAM available on the system. In practice this is impossible to calculate that for all processes so at a minimum you should make sure that the JVM does not have a total maximum memory allocation than your normally available system memory.
  To analyze the garbage collection activity and to explore the contents the application's memory, the JVM options below should be used. Please check in the sections above how to add these options according to your use case:

  -XX:+HeapDumpOnOutOfMemoryError
  -XX:HeapDumpPath=<absolute_path_to_file>
  -XX:+PrintGCTimeStamps
  -XX:+PrintGCDateStamps 
  -verbose:gc
  -Xloggc:<absolute_path_to>/atlassian-fecru-gc-$(date +%Y-%m-%d_%H%M).log 
  -XX:+PrintGCDetails

  Restart your Fisheye instance after adding these options above and enable debug logging via Fisheye web interface by going to Administration > Global Settings > Server > Debug Logging and clicking the Turn Debugging ON button.
  
  The atlassian-fecru-gc-<timestamp>.log file will be created automatically on restart, however the heap dump file named like java_pid<process PID here>.hprof (e.g. java_pid12345.hprof) that is the size of the heap will only be created on the next OutOfMemoryError.

  If you open a support request with the following items the Atlassian Support team will be able to investigate this problem further:
  - The atlassian-fecru-gc-<timestamp>.log
  - The java_pid<process PID here>.hprof
  - The support zip with debug logging enabled before the error was reported.

  The garbage collection settings (excluding the -Xloggc one) can be enabled at runtime as explained on the How to change JVM arguments at runtime to avoid application restart guide.

java.lang.OutOfMemoryError: requested 32756 bytes for ChunkPool::allocate. Out of swap space?

• Meaning: The native objects do not have enough memory to use.
• Known scenarios: This usually happens when you have allocated too much memory to your heap, reducing the amount available for native objects. See this article.
• How to fix it: Reduce the amount of heap memory you have allocated. For example, if you currently allow the JVM to allocate up to 4096m of memory, you should consider reducing this to 3072m or even 2048m. 

Additional reading

The following documentation provides additional clarifications on adjusting limits and performance settings in Fisheye:

• Tuning Fisheye performance
• How Fisheye uses memory