How to use Java Native Memory Tracking (NMT) to obtain JVM memory usage details
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Purpose
Atlassian applications generally consume more physical server memory than what is configured using the -Xmx or -XX:MaxHeapSize JVM parameters. This is expected and intended behavior due to how the JVM allocates memory, as there are more memory categories that Java uses beyond just Heap. However, it is not easy nor enabled by default to identify these other memory categories in order to account for all memory used by the Java process, and as such, administrators are often left wondering where the rest of the consumed memory is.
The purpose of this article is not to explain how to tune or otherwise optimize JVM memory usage; rather, this article serves to help administrators account for unexpected memory overhead beyond the Heap memory that they have configured for their application. Troubleshooting and tuning Java memory usage based on the results of NMT is out of scope of Atlassian Support, and the support team may not be able to assist with questions and concerns that arise based on these results.
Solution
To address this, it is possible to use Java's Native Memory Tracking functionality in order to track a JVM's internal memory usage. This will break down a JVM's memory consumption by category, which helps explain how other memory categories beyond Heap contribute to a JVM's memory consumption. This feature is not enabled by default, and enabling it involves a restart of the JVM. Further details and examples of this functionality in action can also be found in Oracle's documentation here: Native Memory Tracking - Usage and Examples
Per Native Memory Tracking - Usage and Examples, enabling NMT can cause a performance drop, and memory usage will be increased due to monitoring/tracking overhead.
Enabling NMT will result in a 5-10 percent JVM performance drop and memory usage for NMT adds 2 machine words to all malloc memory as malloc header. NMT memory usage is also tracked by NMT.
We do not recommend enabling NMT for extended periods of time in Production environments, as a result. We'd suggest only enabling it as needed in order to obtain the necessary information before removing it.
To enable the NMT feature:
- Shut down the application
- Add the following argument to the application's JVM parameters:
- -XX:NativeMemoryTracking=summary
- Start the application
To obtain the NMT details once the feature is enabled:
- Perform the following command as the OS user who is running the application, being sure to replace '<pid>' with the PID of the application being analyzed
- jcmd <pid> VM.native_memory summary
- The categories and their descriptions are defined here: NMT Memory Categories
Example Output