How to troubleshoot performance issues with thread dumps
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Purpose
When Confluence is behaving slowly, and you need more information as to what part of it is being slow, this page provides a way of collecting thread dumps (a breakdown of what every thread is doing for a Java process) and the output of top (shows what each native OS thread is consuming as far as resources are concerned). This breakdown could normally be collected with something like jProfiler as per Use jProfiler to analyse Jira application performance - in this example we're using native (free) tools to collect information.
This will only work for *nix systems and needs
jstack
to be installed (should be by default). For Windows please see Generating a Thread Dump Externally.
The commands below also work for JIRA, Bamboo, CROWD, Confluence, etc. You just need to change the CONFLUENCE variable for the name of the new application and change the "grep -i confluence" for "grep -i <application_name>”
Resolution
Create a script called dump_reports on /tmp directory and paste the block command below:
#!/bin/bash CONFLUENCE_PID=`ps aux |grep -i confluence |awk '{print $2}' |head -n1` DUMP_HOME="/tmp/dump_home" COUNT=0 if [ ! -d $DUMP_HOME ]; then mkdir $DUMP_HOME fi while true; do top -b -H -p $CONFLUENCE_PID -n 1 |grep CPU -A 5 | awk '{print $(2-1), "\t", $9}' |head -n 6 > $DUMP_HOME/confluence_cpu_usage.`date +%s`.txt; jstack $CONFLUENCE_PID > $DUMP_HOME/confluence_threads.`date +%s`.txt sleep 10 COUNT=$(($COUNT + 1)); if [ $COUNT -gt 5 ]; then echo "" break exit 1 fi done LIST=`ls -l $DUMP_HOME/confluence_cpu* |awk '{print $9}'` for b in $LIST; do echo "" > $DUMP_HOME/threads.txt CONVERT=`cat $b |grep -v PID | awk '{print $1}'` for i in $CONVERT; do printf '%x\n' $i >> $DUMP_HOME/threads.txt done paste $b $DUMP_HOME/threads.txt >> $DUMP_HOME/threads_result.txt done
Then give permissions to the script with the following command:
chmod +x /tmp/dump_reports
Now you can run the script:
/tmp/dump_reports
The result will be something like that:
PID %CPU 6149 2.0 1805 5872 0.0 16f0 5873 0.0 16f1 5874 0.0 16f2 5875 0.0 16f3 PID %CPU 5880 57.4 16f8 6393 45.5 18f9
- Above we can see the PID of the thread, the percentual of CPU usage and the hexadecimal PID which can be used to read the confluence_threads files within the logs more detailed about what is happening.