Fatal JVM Crash - 'java.lang.OutOfMemoryError .. Out of Swap Space?'
Symptoms
Confluence (and other applications running on the JVM) become unavailable. Files with names like hs_err_pid123456.log
may be created in the application server's bin
directory (wherever you start it from), containing text like:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 17179869200 bytes for Chunk::new. Out of swap space?
#
# Internal Error (allocation.cpp:215), pid=30169, tid=1080936768
# Error: Chunk::new
#
# JRE version: 6.0_19-b04
# Java VM: Java HotSpot(TM) 64-Bit Server VM (16.2-b04 mixed mode linux-amd64 )
# An error report file with more information is saved as:
# /usr/confluence-3.0.0_01-std/bin/hs_err_pid30169.log
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#
This indicates Java has failed to acquire more memory from the operating system.
The hs_err Logs may contain:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 17179869200 bytes for Chunk::new. Out of swap space?
#
# Internal Error (allocation.cpp:215), pid=31336, tid=1090754912
# Error: Chunk::new
#
# JRE version: 6.0_18-b07
# Java VM: Java HotSpot(TM) 64-Bit Server VM (16.0-b13 mixed mode linux-amd64 )
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#
--------------- T H R E A D ---------------
Current thread (0x0000002bda85e000): JavaThread "CompilerThread1" daemon [_thread_in_native, id=31351, stack(0x0000000040f39000,0x000000004103a000)]
Stack: [0x0000000040f39000,0x000000004103a000], sp=0x0000000041033520, free space=3e90000000000000018k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x70e5f0]
V [libjvm.so+0x2e3f98]
..
..
..
Current CompileTask:
C2:9245 ! org.apache.velocity.runtime.directive.Foreach.render(Lorg/apache/velocity/context/InternalContextAdapter;Ljava/io/Writer;Lorg/apache/velocity/runtime/parser/node/Node;)Z (514 bytes)
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x0000002be3a25000 JavaThread "Thread-16770" daemon [_thread_in_native, id=8727, stack(0x0000000047ea8000,0x0000000047fa9000)]
Diagnosis
- Confluence is running on Java Version 6 Update 18 or higher.
- The Java process isn't even running on the system, and it's been confirmed it was not manually shut down.
- Files named hs_err_pid* exist on the system or there are one or more segmentation fault messages in /var/log/messages.
Cause
This is usually caused by a bug in the JVM. From this forum discussion, it appears to be the Hot Spot compiler, which is failing to request more memory.
Workaround
There are two workaround options:
- Downgrade to Java 6 update 17 or lower.
- Disable the JVM HotSpot Compilation for the class and method that is causing the crash.
In the above example crash log, the compilation crash was caused by the compile task trying to process the class-method: org.apache.velocity.runtime.directive.Foreach.render
Add the following to the startup system properties.
-XX:CompileCommand=exclude,org/apache/velocity/runtime/directive/Foreach,render
A restart of Confluence is required to apply the changes.
Resolution
Upgrade to Java 6 Update 26 or higher.