Application crashes with SIGSEGV jvm.dll ciObjectFactory::create_new_metadata

Problem

Java application (including Atlassian products) may sporadically crash with the following message appearing in the logs.

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f9026d3cbd7, pid=6999, tid=140252587140864
#
# JRE version: Java(TM) SE Runtime Environment (8.0_60-b27) (build 1.8.0_60-b27)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.60-b23 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x436bd7]  ciObjectFactory::create_new_metadata(Metadata*)+0x327
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x00007f9022889800):  JavaThread "C2 CompilerThread4" daemon [_thread_in_vm, id=7173, stack(0x00007f8f1992a000,0x00007f8f19a2b000)]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000013

Diagnosis

Environment

• Running on Java 8 update 40 or later (1.8.0_40+)
• Have a Java agent enabled for monitoring the JVM

Diagnostic Steps

• You can verify your Java version in every Atlassian product on the Administration section, navigating to the Troubleshooting and Support Tools >> System Information tab, under the heading Java Runtime Environment
• Your JVM startup arguments should contain a Java agent, see the following example highlighting it:
  •  -Xmx8g -Xms8g -XX:MaxMetaspaceSize=512m -Djava.awt.headless=true -XX:+UseCodeCacheFlushing -XX:+UseG1GC -XX:+UseNUMA -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -Xloggc:/usr/local/atlassian/confluence/current/logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=50M -Dcom.atlassian.confluence.extra.calendar3.jira.timeout.socket=30000 -javaagent:/usr/local/newrelic/current/newrelic.jar 

• The above arguments can be checked in the <Bitbucket-installation-directory>/bin/_start-webapp.sh file or in the application.xml file in the Support Zip shared by the customer.

Cause

The cause is not yet identified, there are a number of Java bugs that may be related to the issue (listed below). What we have identified so far that it does not only affect Atlassian applications, but Java application running Java 8 update 40 or later in general, in combination with Java agent(s) using Spring AOP instrumentation. Although it may not be the only possible way to reproduce the issue, what we have seen in all issues reported to us so far, producing these symptoms is that all instances affected were having a Java agent enabled.

So far we have encountered the crash with the following Java agents (however the affected ones may not be limited to these):

• AppDynamics
  • Customer reports state that modifying the agent configurations can work around the problem that is related to instrumentation (for JIRA):

    • Agent configuration change: edit the file AgentDir/ver**/conf/app-agent-config.xml and add the following snippet into <fork-config> element.

      <excludes filter-type="STARTSWITH" filter-value="com.atlassian,net.sf.ehcache.Cache"/>

    • Controller changes: From the Controller UI go to Configure -> Instrumentation -> Callgraph Settings and Uncheck the Enable Aggressive Snapshot Collection.

      • Node Property: exclude-interceptors Type: String Value: com.singularity.tm.TurboCustomExitPointInterceptor 

      • Node Property: disable-custom-exit-points-for Type: String Value: EHCache

      • Node Property: disable-ootb-turbo-interceptors Type: Boolean Value: true

• NewRelic 
  • Customer reports state, that version 3.17.0 contains a fix that eliminated the problem: The Spring AOP instrumentation is now disabled by default. It was exercising a bug in certain versions of Java 8 that caused the JVM to crash)

Workaround

Please note that the workaround you apply is done at your own risk, you need to evaluate the possible impacts of either.

• Disable/remove Java agent from JVM startup arguments
• See the agent-specific possible workarounds above
• Contact your Java agent vendor

Resolution

Contact the vendor of the Java agent you use and inquire about this bug and see if they already have a version with a fix available.

Watch the following Java bugs and monitor the progress to see when a version with the fix for it is released. 

• Crash in HotSpot with jvm.dll+0x42b48 ciObjectFactory::create_new_metadata
• Fatal error in ciObjectFactory::create_new_metadata with Java8u40

References

JRA-59362 - Getting issue details... STATUS