Generating a thread dump
If your Atlassian application stops responding or is showing poor performance, Atlassian Support may ask for a set of thread dumps to help diagnose the problem. A thread dump is a log containing information about currently running threads and processes within the Java Virtual Machine. Taking thread-dumps is a non-destructive process that can be run on live systems. This document describes the steps necessary to retrieve a thread dump.
The steps necessary to retrieve the thread dump are dependent on the server operating system, Windows or Linux. You may also choose to manually retrieve the thread dump, or use the Support Scripts to automatically retrieve the thread dump. We recommend using the Support Scripts as they're designed to retrieve the information required by Support to better assist you.
It's best to generate a set of thread dumps so that the threads can be compared over time, Atlassian Support typically recommends taking 6 thread dumps in 10 second intervals for a total of one minute.
On this page:
Automatically generating a thread dump
Windows environment
Please visit the Windows thread dump Bitbucket repository and follow the full instructions there for running the thread dump generation scripts.
Linux/Unix/OS X environment
Please visit the Linux thread dump Bitbucket repository and follow the full instructions there for running the thread dump generation scripts.
Manually generating a thread dump
Windows environment
Follow the steps below to manually retrieve your thread dump.
Application running in console window
- In the Command Console window where the application is running, open the properties dialog box by right clicking on the title bar and select "Properties".
- Select the Layout tab.
- Under Screen Buffer Size, set the Height to 3000.
- Click OK.
- With the same command console in focus, press CTRL-BREAK. This will output the thread dump to the command console.
- Scroll back in the command console until you reach the line containing "Full thread dump".
- Right click the title bar and select Edit -> Mark. Highlight the entire text of the thread dump.
- Right click the title bar and select Edit -> Copy. The thread dump can then be pasted into a text file.
Application running as a Windows Service
Using jstack
The JDK ships with a tool named jstack for generating thread dumps.
- Identify the process. Launch the task manager by, pressing
Ctrl + Shift + Esc
and find the Process ID of the Java process. You may need to add the PID column usingView
->Select Columns ...
- Run
jstack <pid>
to Capture a Single Thread Dump. This command will take one thread dump of the process id <pid>, in this case the pid is 22668:
C:\Users\Administrator>jstack.exe -l 22668 > threaddump.txt
This will output a file called threaddump.txt
to your current directory.
Common issues with jstack:
- You must run jstack as the same user that is running JIRA.
- If you get the error "Not enough storage is available to process this command", download the 'psexec' utility from here, then run the following command using:
psexec -s jstack <pid> >> threaddumps.txt
- If you are connecting to the server through RDP and getting "Not enough storage is available to process this command", you will need to open a RDP session in console mode:
mstsc /admin
- If the jstack executable is not in your $PATH, then please look for it in your <JDK_HOME>/bin directory
- If you receive
java.lang.NoClassDefFoundError: sun/tools/jstack/JStack
check that tools.jar is present in your JDK's lib directory. If it is not, download a full version of the JDK (ensure the JDK version you download is the same version as the JRE the process is running on).
Linux/Unix/OS X Environment
Using jstack
This is the preferred method as it provides both the threads and their CPU usage. jstack is a java utility that will output stack traces of Java threads for a given process. The JDK is required to be on the server as it ships with jstack.
Identify the Java process that the application is running in and store it in the
APP_PID
environment variableRun the following command:
# seq 6, will run the command 6 times # sleep 10, will wait for 10 seconds for i in $(seq 6); do top -b -H -p $APP_PID -n 1 > app_cpu_usage.`date +%s`.txt; jstack $APP_PID > app_threads.`date +%s`.txt; sleep 10; done
If running the application as a service, run the following command providing the user running the application service
# seq 6, will run the command 6 times # sleep 10, will wait for 10 seconds for i in $(seq 6); do top -b -H -p $APP_PID -n 1 > app_cpu_usage.`date +%s`.txt; sudo -u $APP_USER jstack $APP_PID > app_threads.`date +%s`.txt; sleep 10; done
The resulting
app_cpu_usage.<DATE>.txt
andapp_threads.<DATE>.txt
can be found in the current directory.
Analyzing the resulting thread dumps
Look in the cpu usage files to identify which threads are consistently using a lot of CPU time
Take the PID of the top 10 threads which are using CPU time and convert them to Hexadecimal, eg: 11159 becomes 0x2b97
Search up the Hex values in the thread dumps to figure out which threads are using CPU
Using the kill signal
If the JDK and jstack is not available on the server, a kill signal can be sent to the Java process to produce a thread dump.
Identify the Java process that the application is running in. This can be achieved by running a command similar to:
ps -ef | grep java
The process will appear similarly as follows:
keithb 910 873 1 17:01 pts/3 00:00:18 /usr/java/jdk/bin/java -Xms128m -Xmx256m -Xms128m -Xmx256m -Djava.awt.headless=true -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless=true -Djava.endorsed.dirs=/tmp/atlassian-jira-enterprise-3.6-standalone/common/endorsed -classpath :
In order to retrieve the thread dump, execute the command
kill -3 <pid>
where pid is the process id — in this case, 910.
The thread dump will be written to the Tomcat console output. The console output is redirected to the
logs/catalina.out
file, which can be found in the application installation directory.
Common Analysis Tools to analyze your thread dump
Try TDA or Samurai to inspect your thread dump.
TDA
- Download TDA
CD
to the directory where the JAR existsRun:
java -jar -Xmx512M ~/tda-bin-1.6/tda.jar
- Open your
catalina.out
file, containing the thread dump
Issue processing thread dump with TDA (NumberFormatException)
Should you get an error on TDA console like:
Exception in thread "AWT-EventQueue-0" java.lang.NumberFormatException: For input string: "5 os_prio=0"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:441)
at java.lang.Long.<init>(Long.java:702)
at com.pironet.tda.utils.ThreadsTableModel.getValueAt(ThreadsTableModel.java:80)
at com.pironet.tda.utils.TableSorter.getValueAt(TableSorter.java:285)
at javax.swing.JTable.getValueAt(JTable.java:2717)
at javax.swing.JTable.prepareRenderer(JTable.java:5719)
at javax.swing.plaf.basic.BasicTableUI.paintCell(BasicTableUI.java:2114)
at javax.swing.plaf.basic.BasicTableUI.paintCells(BasicTableUI.java:2016)
at javax.swing.plaf.basic.BasicTableUI.paint(BasicTableUI.java:1812)
at javax.swing.plaf.ComponentUI.update(ComponentUI.java:161)
at javax.swing.JComponent.paintComponent(JComponent.java:778)
at javax.swing.JComponent.paint(JComponent.java:1054)
at javax.swing.JComponent.paintChildren(JComponent.java:887)
at javax.swing.JComponent.paint(JComponent.java:1063)
at javax.swing.JViewport.paint(JViewport.java:731)
at javax.swing.JComponent.paintChildren(JComponent.java:887)
at javax.swing.JComponent.paint(JComponent.java:1063)
at javax.swing.JComponent.paintChildren(JComponent.java:887)
at javax.swing.JSplitPane.paintChildren(JSplitPane.java:1047)
at javax.swing.JComponent.paint(JComponent.java:1063)
at javax.swing.JComponent.paintToOffscreen(JComponent.java:5230)
at javax.swing.BufferStrategyPaintManager.paint(BufferStrategyPaintManager.java:295)
at javax.swing.RepaintManager.paint(RepaintManager.java:1249)
at javax.swing.JComponent._paintImmediately(JComponent.java:5178)
at javax.swing.JComponent.paintImmediately(JComponent.java:4989)
at javax.swing.RepaintManager$3.run(RepaintManager.java:808)
at javax.swing.RepaintManager$3.run(RepaintManager.java:796)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
at javax.swing.RepaintManager.paintDirtyRegions(RepaintManager.java:796)
at javax.swing.RepaintManager.paintDirtyRegions(RepaintManager.java:769)
at javax.swing.RepaintManager.prePaintDirtyRegions(RepaintManager.java:718)
at javax.swing.RepaintManager.access$1100(RepaintManager.java:62)
at javax.swing.RepaintManager$ProcessingRunnable.run(RepaintManager.java:1677)
at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:312)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:733)
at java.awt.EventQueue.access$200(EventQueue.java:103)
at java.awt.EventQueue$3.run(EventQueue.java:694)
at java.awt.EventQueue$3.run(EventQueue.java:692)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:703)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:242)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:161)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:150)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:146)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:138)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:91)
Apply the following command on the thread dump(s) to fix the thread header format to make it processable:
Linux
sed -i 's/prio=[0-9]\{1,2\} os_prio=-\?[0-9]\{1,2\}/prio=5/g' <filename>
Mac
sed -i '' -E 's/prio=[0-9]{1,2} os_prio=-?[0-9]{1,2}/prio=5/g' <filename>