Generating thread dumps using the Performance Data Collector

Generating thread dumps on Linux

To generate a thread dump on Linux (or Solaris or other Unixes):

1. Identify the Process ID for Confluence and set it as the local variable $CONF_PID.
   This can be achieved by running the following command in the terminal:
   

   CONF_PID=`ps aux | grep -i confluence | grep -i java | grep -i catalina | awk  -F '[ ]*' '{print $2}'`; 

2. Using the $CONF_PID while logged in as the user that runs Confluence, run the following script in the terminal, to generate a set of CPU and thread dumps (taken at 10 seconds interval for the period of 1 minute):

   for i in $(seq 6); do top -b -H -p $CONF_PID -n 1 > conf_cpu_usage.`date +%s`.txt; kill -3 $CONF_PID; sleep 10; done

   • This command won't kill your Confluence application (as long as you included the "-3" option with no space in between).
   • The thread dumps will be exported to the existing <Confluence-Installation>/logs/catalina.out log file.

   • The CPU usage statistics will be exported to conf_cpu_usage* files, in the directory from where this command was run.

3. Alternatively, if you have trouble using kill -3 <Confluence-pid> to obtain a thread dump and JDK is installed in your server - try using jstack a java utility that will output stack traces of Java threads for a given process.

   for i in $(seq 6); do top -b -H -p $CONF_PID -n 1 > conf_cpu_usage.`date +%s`.txt; jstack $CONF_PID > conf_threads.`date +%s`.txt; sleep 10; done

   • The thread dumps will be exported to conf_conf_threads* files, in the directory from where this command was run

   • The CPU usage statistics will be exported to conf_cpu_usage* files, in the directory from where this command was run.

Generating thread dumps on windows 

Generating a thread dump from the console

If you are not running Confluence as a service, you can generate a thread dump directly in the console. 

Click the console window and press <CTRL>+BREAK (or SHIFT+CTRL+PAUSE on some keyboards).  The thread dump will print directly to the console.

Generating a thread dump using jstack

The JDK (Java Development Kit) includes jstack, which is used for generating thread dumps.  

To generate a thread dump using jstack:

1. Identify the process. Launch the task manager by, pressing Ctrl + Shift + Esc and find the Process ID of the Java (Confluence) process. (If you can't see the PID column right click a column heading in Task Manager and choose PID).

2. Run jstack <pid> to Capture a Single Thread Dump. This command will take one thread dump of the process id <pid>, in this case the pid is 22668:

   Capture a single thread dump

   jstack.exe <CONFLUENCE_PID> > threaddump.txt

   For example, if the PID is 22668, enter:

   jstack.exe 22668 > threaddump.txt
   

   This will create a file called threaddump.txt to your current directory.

   Capture multiple thread dumps at set intervals

   Use the following command to capture 6 thread dumps of the process id <CONFLUENCE_PID> in 10-second intervals between each thread dump:

   for /L %n in (1,1,6) do timeout 10 | jstack.exe <CONFLUENCE_PID> > threaddump-%n.txt

   For example, if the PID is 22668, enter:

   for /L %n in (1,1,6) do timeout 10 | jstack.exe 22668 > threaddump-%n.txt

   This will output to threaddump-%n.txt, where %n is the number of the loop iteration (starting at 1).

   
   

   You can modify the timeout command parameter to adjust the time between each thread dump and the range in the in (start,step,end) clause to adjust the number of thread dumps to capture. The (1,1,6) range in the example above means the following:

   • Start at 1
   • Increment by 1
   • End at 6

   
   

   
   

Generating thread dumps for containerized Atlassian applications

The Atlassian docker images come with a bundled thread dump collector that can capture thread dump files. This script can be run via docker exec to easily trigger the collection of thread
dumps from the containerized application. For example:

$ docker exec my_confluence /opt/atlassian/support/thread-dumps.sh

By default this script will collect 10 thread dumps at a 5 second interval. This can
be overridden by passing a custom value for the count and interval, respectively. For
example, to collect 20 thread dumps at a 3 second interval:

$ docker exec my_confluence /opt/atlassian/support/thread-dumps.sh -c 20 -i 3

Once the script successfully terminates it will output the folder where the thread and cpu dumps are stored with a message like:

Thread dumps have been written to /var/atlassian/application-data/confluence/thread_dumps/YYYY-MM-DD_HH-MM-SS

To extract the thread dumps from the docker container the below command can be used:

$ docker cp my_confluence:/var/atlassian/application-data/confluence/thread_dumps/YYYY-MM-DD_HH-MM-SS/ /destination_folder

Output

Thread dumps appear in the catalina.out file in the application directory's logs folder. Search for "thread dump" in the log file for the beginning of the dump. 

Often Support may ask you to generate a sequence of thread dumps over a short period, so that they can compare what each dump contains and to look for any long running threads that could be the cause of the performance issue.

You can manually generate multiple thread dumps by executing the command repeatedly, but it is often easier to use a small script to automate the process. Here's an example that you can adapt to run on your Linux server:

  for i in `seq 1 10` ; do
  echo ${i}
  your/path/to/jstack `ps aux | grep java | grep confluence | grep -v grep | awk '{print $2}'` >> threaddump.log
  sleep 10
done

Thread dump analysis tools

• Thread Dump Analyzer TDA 
• Samurai