Generating a thread dump

Automatically generating a thread dump

Windows environment

Please visit the Windows thread dump Bitbucket repository and follow the full instructions there for running the thread dump generation scripts. 

Linux/Unix/OS X environment

Please visit the Linux thread dump Bitbucket repository and follow the full instructions there for running the thread dump generation scripts.

Manually generating a thread dump

Windows environment

Follow the steps below to manually retrieve your thread dump.

Application running in console window

1. In the Command Console window where the application is running, open the properties dialog box by right clicking on the title bar and select "Properties".
2. Select the Layout tab.
3. Under Screen Buffer Size, set the Height to 3000.
   
4. Click OK.
5. With the same command console in focus, press CTRL-BREAK. This will output the thread dump to the command console.
6. Scroll back in the command console until you reach the line containing "Full thread dump".
7. Right click the title bar and select Edit -> Mark. Highlight the entire text of the thread dump.
8. Right click the title bar and select Edit -> Copy. The thread dump can then be pasted into a text file.

Application running as a Windows Service

Using jstack

The JDK ships with a tool named jstack for generating thread dumps.

1. Identify the process. Launch the task manager by, pressing Ctrl + Shift + Esc and find the Process ID of the Java process. You may need to add the PID column using View -> Select Columns ...
2. Run jstack <pid> to Capture a Single Thread Dump. This command will take one thread dump of the process id <pid>, in this case the pid is 22668:

C:\Users\Administrator>jstack.exe -l 22668 > threaddump.txt

This will output a file called threaddump.txt to your current directory.

Linux/Unix/OS X Environment

Using jstack

This is the preferred method as it provides both the threads and their CPU usage. jstack is a java utility that will output stack traces of Java threads for a given process. The JDK is required to be on the server as it ships with jstack.

1. Identify the Java process that the application is running in and store it in the APP_PID environment variable

   Bamboo Server

   APP_PID=`ps aux | grep -i bamboo | grep -i java | awk  -F '[ ]*' '{print $2}'`;

   APP_USER=`ps aux | grep -i bamboo | grep -i java | awk  -F '[ ]*' '{print $1}'`;

   Bitbucket Server

   APP_PID=`ps aux | grep -i bitbucket | grep -i java | awk  -F '[ ]*' '{print $2}'`;

   APP_USER=`ps aux | grep -i bitbucket | grep -i java | awk  -F '[ ]*' '{print $1}'`;

   Confluence

   APP_PID=`ps aux | grep -i confluence | grep -i java | awk  -F '[ ]*' '{print $2}'`;

   APP_USER=`ps aux | grep -i confluence | grep -i java | awk  -F '[ ]*' '{print $1}'`;

   Fisheye/Crucible

   APP_PID=`ps aux | grep -i fisheye | grep -i java | awk  -F '[ ]*' '{print $2}'`;

   APP_USER=`ps aux | grep -i fisheye | grep -i java | awk  -F '[ ]*' '{print $1}'`;

   JIRA

   APP_PID=`ps aux | grep -i jira | grep -i java | awk  -F '[ ]*' '{print $2}'`;

   APP_USER=`ps aux | grep -i jira | grep -i java | awk  -F '[ ]*' '{print $1}'`;

2. Run the following command:

   # seq 6, will run the command 6 times
   # sleep 10, will wait for 10 seconds
   for i in $(seq 6); do top -b -H -p $APP_PID -n 1 > app_cpu_usage.`date +%s`.txt; jstack $APP_PID > app_threads.`date +%s`.txt; sleep 10; done

   If running the application as a service, run the following command providing the user running the application service

   # seq 6, will run the command 6 times
   # sleep 10, will wait for 10 seconds
   for i in $(seq 6); do top -b -H -p $APP_PID -n 1 > app_cpu_usage.`date +%s`.txt; sudo -u $APP_USER jstack $APP_PID > app_threads.`date +%s`.txt; sleep 10; done

3. The resulting app_cpu_usage.<DATE>.txt and app_threads.<DATE>.txt can be found in the current directory.

Analyzing the resulting thread dumps

• Look in the cpu usage files to identify which threads are consistently using a lot of CPU time

• Take the PID of the top 10 threads which are using CPU time and convert them to Hexadecimal, eg: 11159 becomes 0x2b97

• Search up the Hex values in the thread dumps to figure out which threads are using CPU

Using the kill signal

If the JDK and jstack is not available on the server, a kill signal can be sent to the Java process to produce a thread dump.

1. Identify the Java process that the application is running in. This can be achieved by running a command similar to:

   ps -ef | grep java

   The process will appear similarly as follows:

   keithb     910   873  1 17:01 pts/3    00:00:18 /usr/java/jdk/bin/java -Xms128m -Xmx256m
   -Xms128m -Xmx256m -Djava.awt.headless=true -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
   -Djava.awt.headless=true -Djava.endorsed.dirs=/tmp/atlassian-jira-enterprise-3.6-standalone/common/endorsed
   -classpath :

2. In order to retrieve the thread dump, execute the command

   kill -3 <pid>

   where pid is the process id — in this case, 910.

3. The thread dump will be written to the Tomcat console output. The console output is redirected to the logs/catalina.out file, which can be found in the application installation directory.

Common Analysis Tools to analyze your thread dump

Try TDA or Samurai to inspect your thread dump.

TDA

1. Download TDA
2. CD to the directory where the JAR exists

3. Run:

   java -jar -Xmx512M ~/tda-bin-1.6/tda.jar

4. Open your catalina.out file, containing the thread dump

Issue processing thread dump with TDA (NumberFormatException)

Should you get an error on TDA console like:

Exception in thread "AWT-EventQueue-0" java.lang.NumberFormatException: For input string: "5 os_prio=0"
        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
        at java.lang.Long.parseLong(Long.java:441)
        at java.lang.Long.<init>(Long.java:702)
        at com.pironet.tda.utils.ThreadsTableModel.getValueAt(ThreadsTableModel.java:80)
        at com.pironet.tda.utils.TableSorter.getValueAt(TableSorter.java:285)
        at javax.swing.JTable.getValueAt(JTable.java:2717)
        at javax.swing.JTable.prepareRenderer(JTable.java:5719)
        at javax.swing.plaf.basic.BasicTableUI.paintCell(BasicTableUI.java:2114)
        at javax.swing.plaf.basic.BasicTableUI.paintCells(BasicTableUI.java:2016)
        at javax.swing.plaf.basic.BasicTableUI.paint(BasicTableUI.java:1812)
        at javax.swing.plaf.ComponentUI.update(ComponentUI.java:161)
        at javax.swing.JComponent.paintComponent(JComponent.java:778)
        at javax.swing.JComponent.paint(JComponent.java:1054)
        at javax.swing.JComponent.paintChildren(JComponent.java:887)
        at javax.swing.JComponent.paint(JComponent.java:1063)
        at javax.swing.JViewport.paint(JViewport.java:731)
        at javax.swing.JComponent.paintChildren(JComponent.java:887)
        at javax.swing.JComponent.paint(JComponent.java:1063)
        at javax.swing.JComponent.paintChildren(JComponent.java:887)
        at javax.swing.JSplitPane.paintChildren(JSplitPane.java:1047)
        at javax.swing.JComponent.paint(JComponent.java:1063)
        at javax.swing.JComponent.paintToOffscreen(JComponent.java:5230)
        at javax.swing.BufferStrategyPaintManager.paint(BufferStrategyPaintManager.java:295)
        at javax.swing.RepaintManager.paint(RepaintManager.java:1249)
        at javax.swing.JComponent._paintImmediately(JComponent.java:5178)
        at javax.swing.JComponent.paintImmediately(JComponent.java:4989)
        at javax.swing.RepaintManager$3.run(RepaintManager.java:808)
        at javax.swing.RepaintManager$3.run(RepaintManager.java:796)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
        at javax.swing.RepaintManager.paintDirtyRegions(RepaintManager.java:796)
        at javax.swing.RepaintManager.paintDirtyRegions(RepaintManager.java:769)
        at javax.swing.RepaintManager.prePaintDirtyRegions(RepaintManager.java:718)
        at javax.swing.RepaintManager.access$1100(RepaintManager.java:62)
        at javax.swing.RepaintManager$ProcessingRunnable.run(RepaintManager.java:1677)
        at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:312)
        at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:733)
        at java.awt.EventQueue.access$200(EventQueue.java:103)
        at java.awt.EventQueue$3.run(EventQueue.java:694)
        at java.awt.EventQueue$3.run(EventQueue.java:692)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
        at java.awt.EventQueue.dispatchEvent(EventQueue.java:703)
        at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:242)
        at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:161)
        at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:150)
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:146)
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:138)
        at java.awt.EventDispatchThread.run(EventDispatchThread.java:91)

Apply the following command on the thread dump(s) to fix the thread header format to make it processable:

Linux

sed -i 's/prio=[0-9]\{1,2\} os_prio=-\?[0-9]\{1,2\}/prio=5/g' <filename> 

Mac

sed -i '' -E 's/prio=[0-9]{1,2} os_prio=-?[0-9]{1,2}/prio=5/g' <filename>