Fix Out of Memory errors

Still need help?

The Atlassian Community is here for you.

Ask the community

On this page:

 

Updating JVM options when running the application as a Windows service

For Fisheye/Crucible 3.4 and later, 32-bit and 64-bit Windows installers are available. Each installer sets up the service wrapper, adds Fisheye as a Windows service, and starts the service, automatically. Prior to the introduction of the installers, running the application as a service required the use of the Java Service Wrapper. See Running Fisheye as a Windows service for more information.

Installer - versions: 3.4.3+

The tool to edit the service is installed automatically and you can find it here: Windows Start Menu > All Programs > FishEye > Configure FishEye. Modify the JVM options by selecting the tab labeled Java and then restart the service so the settings take effect.

Installer - versions: 3.4 to 3.4.3

You must download the tool mentioned above to edit the service. Download it here:

http://www.apache.org/dist/commons/daemon/binaries/windows/commons-daemon-1.0.15-bin-windows.zip

To use it, you need to rename "prunmgr.exe" from the zip file to "Atlassian Fisheye.exe" or "Atlassian Crucible.exe" depending on which product you initially installed. 

Java Service Wrapper - versions: all (not recommended for new installations)

Updating the JVM options with this setup requires updating the wrapper configuration file, wrapper.conf. Refer to Running Fisheye as a Windows Service under Setting Fisheye environment variables for Windows Services for instructions.

 

Out Of Memory Errors (OOMEs)

There are a number of different memory errors that the JVM can encounter – the most common ones are listed below. Solving these errors might require that you update the default memory settings which is accomplished by modifying the FISHEYE_OPTS environment variable (see the page on Environment variables) and then restarting the application. You can confirm that the settings were updated correctly by confirming that the updated parameters and values appear in the UI by navigating to Administration > Sys Info/Support > System Info and looking at "JVM Input Arguments".

OutOfMemoryError: Java Heap Space

In some cases you need to increase the amount of memory allocated to Fisheye during the initial repository scan period but once this has completed you can reduce back down to the default value. There are many variables to consider when increasing the available memory to your application (total system memory, other applications' memory requirements, etc.) but this can be accomplished using the Xmx parameter:

FISHEYE_OPTS="-Xms128m -Xmx1024m -XX:MaxPermSize=256m"

After having set the FISHEYE_OPTS and restarting your server, go to Administration > Sys Info/Support > System Info, and check your JVM Input Arguments to ensure that your server is picking up your FISHEYE_OPTS as expected.

OutOfMemoryError: PermGen space, or Permanent Generation Size

If you get the error message: java.lang.OutOfMemoryError: PermGen space this means that you have exceeded Java's fixed 64MB block for loading class files. You will need to add the argument -XX:MaxPermSize=256m to FISHEYE_OPTS, in addition to any argument you use to set the heap size.

FISHEYE_OPTS="-Xms128m -Xmx512m -XX:MaxPermSize=256m"

After having set the FISHEYE_OPTS and restarting Fisheye/Crucible, go to Administration > Sys Info/Support > System Info, and check your JVM Input Arguments to ensure that your server is picking up your FISHEYE_OPTS as expected.

OutOfMemoryError: unable to create new native thread

This error occurs when the operating system is unable to create new threads. This can be due to the JVM Heap taking up the available RAM or that the OS limits for the maximum number of threads are too low.

Big heaps take away from the space that can be allocated for the stack of a new thread

Option 1: Reducing Heap

For Linux the maximum heap size of the 32bit JVM cannot be greater than 2GB. If you only have 2GB RAM in your server, it is not recommended to set the Max size of the JVM that high.
The size of the stack per thread can also contribute to this problem. The stack size can reduce the number of threads that can be created.

To fix this problem, you should reduce the size of your JVM Heap (Xmx) and also the size of the JVM stack per thread (Xss) by adding the following parameters to FISHEYE_OPTS, e.g.:

FISHEYE_OPTS="-Xms128m -Xmx1024m -XX:MaxPermSize=256m -Xss512k"

After having set the FISHEYE_OPTS and restarting your server, go to Administration > Sys Info/Support > System Info, and check your JVM Input Arguments to ensure that your server is picking up your FISHEYE_OPTS as expected.

Option 2: Increase the number of allowed threads

The maximum number of allowed threads can be checked by executing ulimit -u as the user running Fisheye/Crucible. This limit can be increased by executing ulimit -u 2048 for example, which will set the maximum number of threads to 2048.

OutOfMemoryError: GC overhead limit exceeded

This kind of OutOfMemoryError can be caused if your java process is starting to use swapped memory for its heap. This will cause the JVM to take longer than normal to perform normal GC operations which will eventually cause a timeout to occur and cause this error.

To overcome this issue, you need to make sure that all processes can't allocate more memory than there is system memory. In practice this is impossible to do for all processes so at a minimum you should make sure that all your application JVMs do not have a total maximum memory allocation than your normally available system memory.

To analyze the garbage collection activity and to explore the contents the application's memory the following JVM options should be added to FISHEYE_OPTS, e.g.:

FISHEYE_OPTS="-Xms128m -Xmx1024m -XX:MaxPermSize=256m -Xss512k -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=<fullpath_to_file> -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -verbose:gc -Xloggc:<fullpath_to_file>/atlassian-fecru-gc-$(date +%Y-%m-%d_%H%M).log

 -XX:+PrintGCDetails"

The file gc.log will be created automatically on restart and but only on the next OutOfMemoryError will a file (java_pid<process PID here>.hprof, e.g. java_pid12345.hprof) be created that is the size of the heap. If you open a support request with the following items the Atlassian Support team will be able to investigate:

  • gc.log
  • java_pidXXXX.hprof
  • Support Zip

Please refer to this guide for more information.

The garbage collection settings (excluding the -Xloggc one) can be enabled at runtime as explained on the How to change JVM arguments at runtime to avoid application restart guide.

java.lang.OutOfMemoryError: requested 32756 bytes for ChunkPool::allocate. Out of swap space?

Essentially the native objects does not have enough memory to use. This is usually because you have allocated too much memory to your heap reducing the amount available for native objects. See this article.

The solution is to reduce the amount of heap memory you have allocated. For example if you have set -Xmx4096, you should consider reducing this to -Xmx2048m.

Remember if you are using a 32bit JVM you cannot allocate more than -Xmx2048m for linux (and even less for windows). Using a 64 bit JVM resolves this problem.



Read the Tuning Fisheye page for more detail on adjusting resource limits and performance settings in Fisheye.

 

Last modified on Nov 2, 2018

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.