Jira fails to shutdown due to insufficient memory for the JRE

For Atlassian eyes only

This article is Not Validated and cannot be shared with customers.

Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.

Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible

    

Summary

When shutting down Jira using built-in scripts (stop-jira.sh, shutdown.sh or catalina.sh stop), the same memory allocation settings used by Jira will be read by the auxiliary Java process (org.apache.catalina.startup.Bootstrap) spun up to execute shutdown tasks.

If the lower boundary is set too high - sometimes equals to the upper boundary as recommended by Atlassian Support - it may not be able to execute as it will require effectively double the allocated memory available on the OS.

Observe that the shutdown process will fail and throw an error similar to the following:

Java HotSpot(TM\) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000XXXXXXX, XXXXXXXXX, 0\) failed; error='Cannot allocate memory' \(errno=12\)
There is insufficient memory for the Java Runtime Environment to continue.
Native memory allocation \(mmap\) failed to map XXXXXXXXXX bytes for committing reserved memory.
An error report file with more information is saved as:
/opt/jira/bin/hs_err_pidXXXX.log

Diagnosis

There is a discussion around this Bug JRASERVER-35398 regarding the root cause. We have some clues that the causer might be the setenv.sh using JAVA_OPTS to set the memory limits.

By the Tomcat documentation, the CATALINA_OPTS should the correct one for this memory configuration.

CATALINA_OPTS   (Optional) Java runtime options used when the "start",
#                   "run" or "debug" command is executed.
#                   Include here and not in JAVA_OPTS all options, that should
#                   only be used by Tomcat itself, not by the stop process,
#                   the version command etc.
#                   Examples are heap size, GC logging, JMX ports etc.
...
#   JAVA_OPTS       (Optional) Java runtime options used when any command
#                   is executed.
#                   Include here and not in CATALINA_OPTS all options, that
#                   should be used by Tomcat and also by the stop process,
#                   the version command etc.
#                   Most options should go into CATALINA_OPTS.

Solution

We will need to perform some changes in the setenv.sh file to follow the Tomcat recommendations.

PLEASE READ: As with all recommendations made by Atlassian Support, please follow best practices for Change Management and test and validate these settings in a Test/Development and Staging environment prior to rolling any changes into a Production environment. This is to validate these changes and ensure that they will function well within your infrastructure prior to placing these changes in production.


In a test environment:

  1. Create a copy of your current setenv.sh;
  2. Rename the original "setenv.sh" to setenvORI.sh;
  3. Apply the following modifications in the new setenv.sh file:

These should be the differences between your new setenv.sh and the original one: ( Lines starting with " - " were removed | Lines starting with " + " were added)

➜  diff -u setenvORI.sh NEWsetenv.sh
--- setenvORI.sh	2020-12-29 18:02:02.000000000 -0300
+++ NEWsetenv.sh	2020-12-29 17:55:57.000000000 -0300
@@ -64,10 +64,29 @@
     fi
 fi

-JAVA_OPTS="-Xms${JVM_MINIMUM_MEMORY} -Xmx${JVM_MAXIMUM_MEMORY} ${JVM_CODE_CACHE_ARGS} ${JAVA_OPTS} ${JVM_REQUIRED_ARGS} ${DISABLE_NOTIFICATIONS} ${JVM_SUPPORT_RECOMMENDED_ARGS} ${JVM_EXTRA_ARGS} ${JIRA_HOME_MINUSD} ${START_JIRA_JAVA_OPTS}"

+##
+# Taking into account 2 things:
+# 1. Order JAVA_OPTS CATALINA_OPTS:
+#       exec "$_RUNJDB" "$LOGGING_CONFIG" $LOGGING_MANAGER $JAVA_OPTS $CATALINA_OPTS
+# 2. Last -Xmx2G wins
+#     > java -version; java -Xmx1G -XX:+PrintFlagsFinal -Xmx2G 2>/dev/null | grep MaxHeapSize
+#
+# We set JVM XMX 2 times:
+#  - generic, for Tomcat only through JAVA_OPTS
+#  - specific for Jira, one more time through CATALINA_OPTS
+#
+# 512MB should be enough for Tomcat stop process
+
+TOMCAT_JVM_OPTS="-Xms512m -Xmx512m"
+
+JAVA_OPTS="${JAVA_OPTS} ${TOMCAT_JVM_OPTS} ${JVM_REQUIRED_ARGS} ${DISABLE_NOTIFICATIONS} ${JVM_EXTRA_ARGS} ${JIRA_HOME_MINUSD} ${START_JIRA_JAVA_OPTS}"
 export JAVA_OPTS

+JVM_OPTS_CATALINA="-Xms${JVM_MINIMUM_MEMORY} -Xmx${JVM_MAXIMUM_MEMORY} ${JVM_SUPPORT_RECOMMENDED_ARGS} "
+CATALINA_OPTS="${JVM_OPTS_CATALINA} ${CATALINA_OPTS}"


After applying those changes, try to stop Jira. If it does not work, you can kill the Jira process, start Jira and try to stop Jira again.




Last modified on Jan 26, 2021

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.