Generating a thread dump

Cross Product Knowledge

On this page

Still need help?

The Atlassian Community is here for you.

Ask the community

If your Atlassian application stops responding or is showing poor performance, Atlassian Support may ask for a set of thread dumps to help diagnose the problem. A thread dump is a log containing information about currently running threads and processes within the Java Virtual Machine. Taking thread-dumps is a non-destructive process that can be run on live systems. This document describes the steps necessary to retrieve a thread dump.

The steps necessary to retrieve the thread dump are dependent on the server operating system, Windows or Linux. You may also choose to manually retrieve the thread dump, or use the Support Scripts to automatically retrieve the thread dump. We recommend using the Support Scripts as they're designed to retrieve the information required by Support to better assist you.

It's best to generate a set of thread dumps so that the threads can be compared over time, Atlassian Support typically recommends taking 6 thread dumps in 10 second intervals for a total of one minute.

Automatically generating a thread dump

Windows and Linux environment using Java Flight Recorder (JFR)

JFR is bundled in the following Atlassian applications (via Troubleshooting and Support Tools app), and continuous diagnostic data recording begins as soon as you start up your instance:

  • Jira Data Center and Server 9.0 and newer

  • Bitbucket Data Center and Server 8.3 and newer

  • Confluence Data Center and Server 7.20 and newer

Thread dumps are a part of the diagnostic data that is automatically captured by JFR. Within our applications, we've changed the default frequency for gathering thread dumps to once every 10 seconds.
This set of thread dumps would then be automatically saved to a binary .jfr file (in <Application-Home-Directory>/log/jfr folder) every five minutes, together with the other diagnostics data (runtime events). Collected thread dumps within the binary .jfr file would need to be parsed using JDK Mission Control.

In addition to that, when a Support zip is being created, a new binary .jfr file would be created in the <Application-Home-Directory>/log/jfr folder as well.
This latest .jfr binary file and thread dumps that are extracted from it will be included in the Support zip file (under jfr-bundle folder). This set of extracted thread dumps are already in human-readable format and can be parsed by common thread analysis tools.

(info) For more information on this, please refer to our Diagnosing runtime issues using the Java Flight Recorder documentation.

Manually generating a thread dump

Windows environment

Follow the steps below to manually retrieve your thread dump.

The application running in the console window

  1. In the Command Console window where the application is running, open the properties dialog box by right-clicking on the title bar and selecting "Properties".
  2. Select the Layout tab.
  3. Under Screen Buffer Size, set the Height to 3000.
  4. Click OK.
  5. With the same command console in focus, press CTRL-BREAK. This will output the thread dump to the command console.
  6. Scroll back in the command console until you reach the line containing "Full thread dump".
  7. Right-click the title bar and select Edit -> Mark. Highlight the entire text of the thread dump.
  8. Right-click the title bar and select Edit -> Copy. The thread dump can then be pasted into a text file.

The application runs as a Windows Service

Using jstack

The JDK ships with a tool named jstack for generating thread dumps.

If you're using the JVM bundled with the product, please download and install a supported JDK, such as AdoptOpenJDK.
(info) Please make sure that the JDK is of the same major version as the JVM running the application, typically Java 8 or 11. If in doubt, run java -version to check

  1. Launch Windows Powershell as administrator
  2. Run this command to set the PID to a variable

    $APP_PID = (Get-Process -name "tomcat8.exe.x64").id

    Alternatively, launch the task manager by, pressing Ctrl + Shift + Esc and find the Process ID of the Java process. You may need to add the PID column using View -> Select Columns ...
    (info) 
    If there are multiple Tomcat/application instances running, please get the PID manually for the right process.

  3. Run this 

    1..6|foreach{jstack -l $APP_PID|Out-File -FilePath "app_threads.$(Get-Date -uformat %s).txt";sleep 10}

    This will create the app_threads files in the current directory.


Common issues with jstack:

  • You must run jstack as an admin or as the same user that is running the application.
  • If you get the error "Not enough storage is available to process this command", download the 'psexec' utility from here, then run the following command using:
    psexec -s jstack <pid> >> threaddumps.txt
  • If you are connecting to the server through RDP and getting "Not enough storage is available to process this command", you will need to open an RDP session in console mode: 
    mstsc /admin
  • If the jstack executable is not in your $PATH, then please look for it in your <JDK_HOME>/bin directory
  • If you receive java.lang.NoClassDefFoundError: sun/tools/jstack/JStack check that tools.jar is present in your JDK's lib directory. If it is not, download a full version of the JDK (ensure the JDK version you download is the same version as the JRE the process is running on).

Linux/Unix/OS X Environment

Using jstack

This is the preferred method as it provides both the threads and their CPU usage. jstack is a java utility that will output stack traces of Java threads for a given process. The JDK is required to be on the server as it ships with jstack.

  1. Identify the Java process that the application is running in and store it in the APP_PID environment variable

    Bamboo Server
    APP_PID=`ps aux | grep -i bamboo | grep -i java | awk  -F '[ ]*' '{print $2}'`;
    APP_USER=`ps aux | grep -i bamboo | grep -i java | awk  -F '[ ]*' '{print $1}'`;
    Bitbucket Server
    APP_PID=`ps aux | grep -i bitbucket | grep -i java | awk  -F '[ ]*' '{print $2}'`;
    APP_USER=`ps aux | grep -i bitbucket | grep -i java | awk  -F '[ ]*' '{print $1}'`;
    Confluence
    APP_PID=`ps aux | grep -i confluence | grep -i java | awk  -F '[ ]*' '{print $2}'`;
    APP_USER=`ps aux | grep -i confluence | grep -i java | awk  -F '[ ]*' '{print $1}'`;
    Fisheye/Crucible
    APP_PID=`ps aux | grep -i fisheye | grep -i java | awk  -F '[ ]*' '{print $2}'`;
    APP_USER=`ps aux | grep -i fisheye | grep -i java | awk  -F '[ ]*' '{print $1}'`;
    JIRA
    APP_PID=`ps aux | grep -i jira | grep -i java | awk  -F '[ ]*' '{print $2}'`;
    APP_USER=`ps aux | grep -i jira | grep -i java | awk  -F '[ ]*' '{print $1}'`;
  2. Run the following command:

    # seq 6, will run the command 6 times
    # sleep 10, will wait for 10 seconds
    for i in $(seq 6); do top -b -H -p $APP_PID -n 1 > app_cpu_usage.`date +%s`.txt; jstack $APP_PID > app_threads.`date +%s`.txt; sleep 10; done

    If running the application as a service, run the following command providing the user running the application service

    # seq 6, will run the command 6 times
    # sleep 10, will wait for 10 seconds
    for i in $(seq 6); do top -b -H -p $APP_PID -n 1 > app_cpu_usage.`date +%s`.txt; sudo -u $APP_USER jstack $APP_PID > app_threads.`date +%s`.txt; sleep 10; done
  3. The resulting app_cpu_usage.<DATE>.txt and app_threads.<DATE>.txt can be found in the current directory.

Analyzing the resulting thread dumps
  • Look in the CPU usage files to identify which threads are consistently using a lot of CPU time

  • Take the PID of the top 10 threads which are using CPU time and convert them to Hexadecimal, eg: 11159 becomes 0x2b97

  • Search up the Hex values in the thread dumps to figure out which threads are using CPU

Using the kill signal

If the JDK and jstack are not available on the server, a kill signal can be sent to the Java process to produce a thread dump.

  1. Identify the Java process that the application is running in. This can be achieved by running a command similar to:

    ps -ef | grep java

    The process will appear similarly as follows:

    keithb     910   873  1 17:01 pts/3    00:00:18 /usr/java/jdk/bin/java -Xms128m -Xmx256m
    -Xms128m -Xmx256m -Djava.awt.headless=true -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
    -Djava.awt.headless=true -Djava.endorsed.dirs=/tmp/atlassian-jira-enterprise-3.6-standalone/common/endorsed
    -classpath :
  2. In order to retrieve the thread dump, execute the command

    # seq 6, will run the command 6 times
    # sleep 10, will wait for 10 seconds
    for i in $(seq 6); do top -b -H -p $APP_PID -n 1 > app_cpu_usage.`date +%s`.txt; kill -3 $APP_PID ; sleep 10; done

    where $APP_PID is the process id — in this case, 910.

  3. The thread dump will be written to the Tomcat console output. The console output is redirected to the logs/catalina.out file, which can be found in the application installation directory.

Common Analysis Tools to analyze your thread dump

Try TDA, IBM TMDA (JCA) or Samurai to inspect your thread dump.

TDA

  1. Download TDA
  2. CD to the directory where the JAR exists
  3. Run:

    java -jar -Xmx512M ~/tda-bin-1.6/tda.jar
  4. Open your catalina.out file, containing the thread dump

Issue processing thread dump with TDA (NumberFormatException)

Should you get an error on the TDA console like:

Exception in thread "AWT-EventQueue-0" java.lang.NumberFormatException: For input string: "5 os_prio=0"
        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
        at java.lang.Long.parseLong(Long.java:441)
        at java.lang.Long.<init>(Long.java:702)
        at com.pironet.tda.utils.ThreadsTableModel.getValueAt(ThreadsTableModel.java:80)
        at com.pironet.tda.utils.TableSorter.getValueAt(TableSorter.java:285)
        at javax.swing.JTable.getValueAt(JTable.java:2717)
        at javax.swing.JTable.prepareRenderer(JTable.java:5719)
        at javax.swing.plaf.basic.BasicTableUI.paintCell(BasicTableUI.java:2114)
        at javax.swing.plaf.basic.BasicTableUI.paintCells(BasicTableUI.java:2016)
        at javax.swing.plaf.basic.BasicTableUI.paint(BasicTableUI.java:1812)
        at javax.swing.plaf.ComponentUI.update(ComponentUI.java:161)
        at javax.swing.JComponent.paintComponent(JComponent.java:778)
        at javax.swing.JComponent.paint(JComponent.java:1054)
        at javax.swing.JComponent.paintChildren(JComponent.java:887)
        at javax.swing.JComponent.paint(JComponent.java:1063)
        at javax.swing.JViewport.paint(JViewport.java:731)
        at javax.swing.JComponent.paintChildren(JComponent.java:887)
        at javax.swing.JComponent.paint(JComponent.java:1063)
        at javax.swing.JComponent.paintChildren(JComponent.java:887)
        at javax.swing.JSplitPane.paintChildren(JSplitPane.java:1047)
        at javax.swing.JComponent.paint(JComponent.java:1063)
        at javax.swing.JComponent.paintToOffscreen(JComponent.java:5230)
        at javax.swing.BufferStrategyPaintManager.paint(BufferStrategyPaintManager.java:295)
        at javax.swing.RepaintManager.paint(RepaintManager.java:1249)
        at javax.swing.JComponent._paintImmediately(JComponent.java:5178)
        at javax.swing.JComponent.paintImmediately(JComponent.java:4989)
        at javax.swing.RepaintManager$3.run(RepaintManager.java:808)
        at javax.swing.RepaintManager$3.run(RepaintManager.java:796)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
        at javax.swing.RepaintManager.paintDirtyRegions(RepaintManager.java:796)
        at javax.swing.RepaintManager.paintDirtyRegions(RepaintManager.java:769)
        at javax.swing.RepaintManager.prePaintDirtyRegions(RepaintManager.java:718)
        at javax.swing.RepaintManager.access$1100(RepaintManager.java:62)
        at javax.swing.RepaintManager$ProcessingRunnable.run(RepaintManager.java:1677)
        at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:312)
        at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:733)
        at java.awt.EventQueue.access$200(EventQueue.java:103)
        at java.awt.EventQueue$3.run(EventQueue.java:694)
        at java.awt.EventQueue$3.run(EventQueue.java:692)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
        at java.awt.EventQueue.dispatchEvent(EventQueue.java:703)
        at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:242)
        at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:161)
        at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:150)
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:146)
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:138)
        at java.awt.EventDispatchThread.run(EventDispatchThread.java:91)

Apply the following command on the thread dump(s) to fix the thread header format to make it processable:

Linux
sed -i 's/prio=[0-9]\{1,2\} os_prio=-\?[0-9]\{1,2\}/prio=5/g' <filename> 
Mac
sed -i '' -E 's/prio=[0-9]{1,2} os_prio=-?[0-9]{1,2}/prio=5/g' <filename>

Last modified on Apr 17, 2023

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.