Summary

When running Jira Data Center with 2 nodes or more, 1 of the node throws "500 error" on random pages (Dashboard page, Versions & Licenses page, Issue filter page...), and that same node eventually crashes and throws a JVM error in the tomcat logs.

Environment

• Jira Data Center (any version), with 2 nodes or more

Diagnosis

• The issue not replicable when only 1 node is added to the cluster
• The issue starts to occur as soon as a 2nd node is added to the cluster
• Random pages are throwing the 500 error code, with a different error message. For example:

  • The Dashboard page throws the following error:

    com.google.template.soy.tofu.SoyTofuException: In 'print' tag, expression "$dashboardTitle" evaluates to undefined.
    at JIRA.Dashboard.page.dashboard(dashboard.soy:12) [?:?]
    at com.google.template.soy.tofu.internal.BaseTofu.renderMainHelper(BaseTofu.java:369) [?:?]
    at com.google.template.soy.tofu.internal.BaseTofu.renderMain(BaseTofu.java:322) [?:?]
    at com.google.template.soy.tofu.internal.BaseTofu.access$100(BaseTofu.java:66) [?:?]
    at com.google.template.soy.tofu.internal.BaseTofu$RendererImpl.render(BaseTofu.java:476) [?:?]
    at com.atlassian.soy.impl.DefaultSoyManager.render(DefaultSoyManager.java:154) [?:?]

  • The ⚙ > Applications > Versions & Licenses page throws the following error:

    com.atlassian.templaterenderer.RenderingException: org.apache.velocity.exception.MethodInvocationException: Invocation of method 'render' in class com.sun.proxy.$Proxy1810 threw ...
    at com.atlassian.templaterenderer.velocity.one.six.internal.VelocityTemplateRendererImpl.render(VelocityTemplateRendererImpl.java:104)
    at sun.reflect.GeneratedMethodAccessor897.invoke(Unkown Source)

  • The Issue Search page (/issues/?filter=-1) throws the following error:

    java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:62)

• The catalina.out log file shows the following JVM crash report:

  #
  # A fatal error has been detected by the Java Runtime Environment:
  #
  #  SIGBUS (0x7) at pc=0x00007f3c6b38a4e2, pid=2545, tid=0x00007f2efecf6700
  #
  # JRE version: Java(TM) SE Runtime Environment (8.0_181-b13) (build 1.8.0_181-b13)
  # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.181-b13 mixed mode linux-amd64 )
  # Problematic frame:
  # C  [libzip.so+0x124e2]  newEntry+0x62
  #
  # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
  #
  # An error report file with more information is saved as:
  # /opt/jira/atlassian/jira/bin/hs_err_pidXXXX.log
  Compiled method (nm) 13135685  207     n 0       java.util.zip.ZipFile::getEntry (native)
   total in heap  [0x00007f3c4b69a3d0,0x00007f3c4b69a738] = 872
   relocation     [0x00007f3c4b69a4f8,0x00007f3c4b69a540] = 72
   main code      [0x00007f3c4b69a540,0x00007f3c4b69a738] = 504
  #
  # If you would like to submit a bug report, please visit:
  #   http://bugreport.java.com/bugreport/crash.jsp
  # The crash happened outside the Java Virtual Machine in native code.
  # See problematic frame for where to report the bug.
  #

Cause

Each Jira node in the cluster is configured with a symlink which is pointing from the Jira home folder to the Jira shared folder. Due to the nature of symlinks, whenever an update is made to the local home folder of a node, this change will also be reflected in the shared home folder. If at least 2 nodes are up and running, both nodes might end up modifying their own local home folder at the same time, and as a result "compete" to update the same shared home filter, ultimately causing the JVM to crash.

 Note that symlinks should never be created between the Jira nodes and the shared folder, in order to prevent such issues from happening. As explained in the documentation Installing Jira Data Center, when creating the shared home directory, the only requirement for the Jira Data Center cluster to work properly is to ensure that the shared directory is readable and writable by all nodes in the cluster.

Solution

Remove the symlinks that were linking the Jira local home folders to the shared home folder.