Generating a thread dump

Automatically generating a thread dump

Windows and Linux environment using Java Flight Recorder (JFR)

JFR is bundled in the following Atlassian applications (via Troubleshooting and Support Tools app), and continuous diagnostic data recording begins as soon as you start up your instance:

• Jira Data Center and Server 9.0 and newer

• Bitbucket Data Center and Server 8.3 and newer

• Confluence Data Center and Server 7.20 and newer

Thread dumps are a part of the diagnostic data that is automatically captured by JFR. Within our applications, we've changed the default frequency for gathering thread dumps to once every 10 seconds.
This set of thread dumps would then be automatically saved to a binary .jfr file (in <Application-Home-Directory>/log/jfr folder) every five minutes, together with the other diagnostics data (runtime events). Collected thread dumps within the binary .jfr file would need to be parsed using JDK Mission Control.

In addition to that, when a Support zip is being created, a new binary .jfr file would be created in the <Application-Home-Directory>/log/jfr folder as well.
This latest .jfr binary file and thread dumps that are extracted from it will be included in the Support zip file (under jfr-bundle folder). This set of extracted thread dumps are already in human-readable format and can be parsed by common thread analysis tools.

For more information on this, please refer to our Diagnosing runtime issues using the Java Flight Recorder documentation.

Manually generating a thread dump

Windows environment

Follow the steps below to manually retrieve your thread dump.

The application running in the console window

1. In the Command Console window where the application is running, open the properties dialog box by right-clicking on the title bar and selecting "Properties".
2. Select the Layout tab.
3. Under Screen Buffer Size, set the Height to 3000.
   
4. Click OK.
5. With the same command console in focus, press CTRL-BREAK. This will output the thread dump to the command console.
6. Scroll back in the command console until you reach the line containing "Full thread dump".
7. Right-click the title bar and select Edit -> Mark. Highlight the entire text of the thread dump.
8. Right-click the title bar and select Edit -> Copy. The thread dump can then be pasted into a text file.

The application runs as a Windows Service

Using jstack

The JDK ships with a tool named jstack for generating thread dumps.

If you're using the JVM bundled with the product, please download and install a supported JDK, such as AdoptOpenJDK.
 Please make sure that the JDK is of the same major version as the JVM running the application, typically Java 8 or 11. If in doubt, run java -version to check

1. Launch Windows Powershell as administrator

2. Run this command to set the PID to a variable

   $APP_PID = (Get-Process -name "tomcat8.exe.x64").id

   Alternatively, launch the task manager by, pressing Ctrl + Shift + Esc and find the Process ID of the Java process. You may need to add the PID column using View -> Select Columns ...
 If there are multiple Tomcat/application instances running, please get the PID manually for the right process.


3. Run this 

   1..6|foreach{jstack -l $APP_PID|Out-File -FilePath "app_threads.$(Get-Date -uformat %s).txt";sleep 10}

   This will create the app_threads files in the current directory.

Linux/Unix/OS X Environment

Using jstack

This is the preferred method as it provides both the threads and their CPU usage. jstack is a java utility that will output stack traces of Java threads for a given process. The JDK is required to be on the server as it ships with jstack.

1. Identify the Java process that the application is running in and store it in the APP_PID environment variable

   Bamboo Server

   APP_PID=`ps aux | grep -i bamboo | grep -i java | awk  -F '[ ]*' '{print $2}'`;

   APP_USER=`ps aux | grep -i bamboo | grep -i java | awk  -F '[ ]*' '{print $1}'`;

   Bitbucket Server

   APP_PID=`ps aux | grep -i bitbucket | grep -i java | awk  -F '[ ]*' '{print $2}'`;

   APP_USER=`ps aux | grep -i bitbucket | grep -i java | awk  -F '[ ]*' '{print $1}'`;

   Confluence

   APP_PID=`ps aux | grep -i confluence | grep -i java | awk  -F '[ ]*' '{print $2}'`;

   APP_USER=`ps aux | grep -i confluence | grep -i java | awk  -F '[ ]*' '{print $1}'`;

   Fisheye/Crucible

   APP_PID=`ps aux | grep -i fisheye | grep -i java | awk  -F '[ ]*' '{print $2}'`;

   APP_USER=`ps aux | grep -i fisheye | grep -i java | awk  -F '[ ]*' '{print $1}'`;

   JIRA

   APP_PID=`ps aux | grep -i jira | grep -i java | awk  -F '[ ]*' '{print $2}'`;

   APP_USER=`ps aux | grep -i jira | grep -i java | awk  -F '[ ]*' '{print $1}'`;

2. Run the following command:

   # seq 6, will run the command 6 times
   # sleep 10, will wait for 10 seconds
   for i in $(seq 6); do top -b -H -p $APP_PID -n 1 > app_cpu_usage.`date +%s`.txt; jstack $APP_PID > app_threads.`date +%s`.txt; sleep 10; done

   If running the application as a service, run the following command providing the user running the application service

   # seq 6, will run the command 6 times
   # sleep 10, will wait for 10 seconds
   for i in $(seq 6); do top -b -H -p $APP_PID -n 1 > app_cpu_usage.`date +%s`.txt; sudo -u $APP_USER jstack $APP_PID > app_threads.`date +%s`.txt; sleep 10; done

3. The resulting app_cpu_usage.<DATE>.txt and app_threads.<DATE>.txt can be found in the current directory.

Analyzing the resulting thread dumps

• Look in the CPU usage files to identify which threads are consistently using a lot of CPU time

• Take the PID of the top 10 threads which are using CPU time and convert them to Hexadecimal, eg: 11159 becomes 0x2b97

• Search up the Hex values in the thread dumps to figure out which threads are using CPU

Using the kill signal

If the JDK and jstack are not available on the server, a kill signal can be sent to the Java process to produce a thread dump.

1. Identify the Java process that the application is running in. This can be achieved by running a command similar to:

   ps -ef | grep java

   The process will appear similarly as follows:

   keithb     910   873  1 17:01 pts/3    00:00:18 /usr/java/jdk/bin/java -Xms128m -Xmx256m
   -Xms128m -Xmx256m -Djava.awt.headless=true -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
   -Djava.awt.headless=true -Djava.endorsed.dirs=/tmp/atlassian-jira-enterprise-3.6-standalone/common/endorsed
   -classpath :

2. In order to retrieve the thread dump, execute the command

   # seq 6, will run the command 6 times
   # sleep 10, will wait for 10 seconds
   for i in $(seq 6); do top -b -H -p $APP_PID -n 1 > app_cpu_usage.`date +%s`.txt; kill -3 $APP_PID ; sleep 10; done

   where $APP_PID is the process id — in this case, 910.

3. The thread dump will be written to the Tomcat console output. The console output is redirected to the logs/catalina.out file, which can be found in the application installation directory.

Common Analysis Tools to analyze your thread dump

Try TDA, IBM TMDA (JCA) or Samurai to inspect your thread dump.

TDA

1. Download TDA
2. CD to the directory where the JAR exists

3. Run:

   java -jar -Xmx512M ~/tda-bin-1.6/tda.jar

4. Open your catalina.out file, containing the thread dump

Issue processing thread dump with TDA (NumberFormatException)

Should you get an error on the TDA console like:

Exception in thread "AWT-EventQueue-0" java.lang.NumberFormatException: For input string: "5 os_prio=0"
        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
        at java.lang.Long.parseLong(Long.java:441)
        at java.lang.Long.<init>(Long.java:702)
        at com.pironet.tda.utils.ThreadsTableModel.getValueAt(ThreadsTableModel.java:80)
        at com.pironet.tda.utils.TableSorter.getValueAt(TableSorter.java:285)
        at javax.swing.JTable.getValueAt(JTable.java:2717)
        at javax.swing.JTable.prepareRenderer(JTable.java:5719)
        at javax.swing.plaf.basic.BasicTableUI.paintCell(BasicTableUI.java:2114)
        at javax.swing.plaf.basic.BasicTableUI.paintCells(BasicTableUI.java:2016)
        at javax.swing.plaf.basic.BasicTableUI.paint(BasicTableUI.java:1812)
        at javax.swing.plaf.ComponentUI.update(ComponentUI.java:161)
        at javax.swing.JComponent.paintComponent(JComponent.java:778)
        at javax.swing.JComponent.paint(JComponent.java:1054)
        at javax.swing.JComponent.paintChildren(JComponent.java:887)
        at javax.swing.JComponent.paint(JComponent.java:1063)
        at javax.swing.JViewport.paint(JViewport.java:731)
        at javax.swing.JComponent.paintChildren(JComponent.java:887)
        at javax.swing.JComponent.paint(JComponent.java:1063)
        at javax.swing.JComponent.paintChildren(JComponent.java:887)
        at javax.swing.JSplitPane.paintChildren(JSplitPane.java:1047)
        at javax.swing.JComponent.paint(JComponent.java:1063)
        at javax.swing.JComponent.paintToOffscreen(JComponent.java:5230)
        at javax.swing.BufferStrategyPaintManager.paint(BufferStrategyPaintManager.java:295)
        at javax.swing.RepaintManager.paint(RepaintManager.java:1249)
        at javax.swing.JComponent._paintImmediately(JComponent.java:5178)
        at javax.swing.JComponent.paintImmediately(JComponent.java:4989)
        at javax.swing.RepaintManager$3.run(RepaintManager.java:808)
        at javax.swing.RepaintManager$3.run(RepaintManager.java:796)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
        at javax.swing.RepaintManager.paintDirtyRegions(RepaintManager.java:796)
        at javax.swing.RepaintManager.paintDirtyRegions(RepaintManager.java:769)
        at javax.swing.RepaintManager.prePaintDirtyRegions(RepaintManager.java:718)
        at javax.swing.RepaintManager.access$1100(RepaintManager.java:62)
        at javax.swing.RepaintManager$ProcessingRunnable.run(RepaintManager.java:1677)
        at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:312)
        at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:733)
        at java.awt.EventQueue.access$200(EventQueue.java:103)
        at java.awt.EventQueue$3.run(EventQueue.java:694)
        at java.awt.EventQueue$3.run(EventQueue.java:692)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
        at java.awt.EventQueue.dispatchEvent(EventQueue.java:703)
        at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:242)
        at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:161)
        at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:150)
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:146)
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:138)
        at java.awt.EventDispatchThread.run(EventDispatchThread.java:91)

Apply the following command on the thread dump(s) to fix the thread header format to make it processable:

Linux

sed -i 's/prio=[0-9]\{1,2\} os_prio=-\?[0-9]\{1,2\}/prio=5/g' <filename> 

Mac

sed -i '' -E 's/prio=[0-9]{1,2} os_prio=-?[0-9]{1,2}/prio=5/g' <filename>