Application crashes with SIGSEGV jvm.dll ciObjectFactory::create_new_metadata

This Knowledge Base article was written specifically for the Atlassian Server platform. Due to the Functional differences in Atlassian Cloud, the contents of this article cannot be applied to Atlassian Cloud applications.

Problem

Java application (including Atlassian products) may sporadically crash with the following message appearing in the logs.

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f9026d3cbd7, pid=6999, tid=140252587140864
#
# JRE version: Java(TM) SE Runtime Environment (8.0_60-b27) (build 1.8.0_60-b27)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.60-b23 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x436bd7]  ciObjectFactory::create_new_metadata(Metadata*)+0x327
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x00007f9022889800):  JavaThread "C2 CompilerThread4" daemon [_thread_in_vm, id=7173, stack(0x00007f8f1992a000,0x00007f8f19a2b000)]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000013

Diagnosis

Environment

  • Running on Java 8 update 40 or later (1.8.0_40+)
  • Have a Java agent enabled for monitoring the JVM

Diagnostic Steps

  • You can verify your Java version in every Atlassian product on the Administration section, navigating to the Support Tools Plugin's System Information tab
  • Your JVM startup arguments should contain a Java agent, see the following example highlighting it:
    •  -Xmx8g -Xms8g -XX:MaxMetaspaceSize=512m -Djava.awt.headless=true -XX:+UseCodeCacheFlushing -XX:+UseG1GC -XX:+UseNUMA -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -Xloggc:/usr/local/atlassian/confluence/current/logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=50M -Dcom.atlassian.confluence.extra.calendar3.jira.timeout.socket=30000 -javaagent:/usr/local/newrelic/current/newrelic.jar 

Cause

The cause is not yet identified, there are a number of Java bugs that may be related to the issue (listed below). What we have identified so far that it does not only affect Atlassian applications, but Java application running Java 8 update 40 or later in general, in combination with Java agent(s) using Spring AOP instrumentation. Although it may not be the only possible way to reproduce the issue, what we have seen in all issues reported to us so far, producing these symptoms is that all instances affected were having a Java agent enabled.

So far we have encountered the crash with the following Java agents (however the affected ones may not be limited to these):

  • AppDynamics
    • Customer reports state that modifying the agent configurations can work around the problem that is related to instrumentation (for JIRA):
      • Agent configuration change: edit the file AgentDir/ver**/conf/app-agent-config.xml and add the following snippet into <fork-config> element.

        <excludes filter-type="STARTSWITH" filter-value="com.atlassian,net.sf.ehcache.Cache"/>
      • Controller changes: From the Controller UI go to Configure -> Instrumentation -> Callgraph Settings and Uncheck the Enable Aggressive Snapshot Collection.

        • Node Property: exclude-interceptors Type: String Value: com.singularity.tm.TurboCustomExitPointInterceptor 

        • Node Property: disable-custom-exit-points-for Type: String Value: EHCache

        • Node Property: disable-ootb-turbo-interceptors Type: Boolean Value: true

  • NewRelic 
    • Customer reports state, that version 3.17.0 contains a fix that eliminated the problem: The Spring AOP instrumentation is now disabled by default. It was exercising a bug in certain versions of Java 8 that caused the JVM to crash)

 

Workaround

Please note that the workaround you apply is done at your own risk, you need to evaluate the possible impacts of either.

  • Disable/remove Java agent from JVM startup arguments
  • See the agent-specific possible workarounds above
  • Contact your Java agent vendor

Resolution

Contact the vendor of the Java agent you use and inquire about this bug and see if they already have a version with a fix available.

Watch the following Java bugs and monitor the progress to see when a version with the fix for it is released. 

References

JRA-59362 - Getting issue details... STATUS

 

Last modified on Sep 20, 2017

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.