How to troubleshoot performance issues with thread dumps

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.

Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible

Purpose

When Confluence is behaving slowly, and you need more information as to what part of it is being slow, this page provides a way of collecting thread dumps (a breakdown of what every thread is doing for a Java process) and the output of top (shows what each native OS thread is consuming as far as resources are concerned). This breakdown could normally be collected with something like jProfiler as per Use jProfiler to analyse Jira application performance - in this example we're using native (free) tools to collect information.

(warning) This will only work for *nix systems and needs jstack to be installed (should be by default). For Windows please see Generating a Thread Dump Externally.

The commands below also work for JIRA, Bamboo, CROWD, Confluence, etc. You just need to change the CONFLUENCE variable for the name of the new application and change the "grep -i confluence" for "grep -i <application_name>”

Resolution

  • Create a script called dump_reports on /tmp directory and paste the block command below:

    #!/bin/bash
    CONFLUENCE_PID=`ps aux |grep -i confluence |awk '{print $2}' |head -n1`
    DUMP_HOME="/tmp/dump_home"
    COUNT=0
    if [ ! -d $DUMP_HOME ]; then
    	mkdir $DUMP_HOME
    fi
    while true; do
    	top -b -H -p $CONFLUENCE_PID -n 1 |grep CPU -A 5 | awk '{print  $(2-1), "\t", $9}' |head -n 6 > $DUMP_HOME/confluence_cpu_usage.`date +%s`.txt; jstack $CONFLUENCE_PID > $DUMP_HOME/confluence_threads.`date +%s`.txt
    	sleep 10 
    	COUNT=$(($COUNT + 1));
    	if [ $COUNT -gt 5 ]; then
    		echo ""
    		break
    		exit 1
    	fi
    done
    LIST=`ls -l $DUMP_HOME/confluence_cpu* |awk '{print $9}'`
    for b in $LIST; do
    	echo "" > $DUMP_HOME/threads.txt
    	CONVERT=`cat $b |grep -v PID | awk '{print $1}'`
    		for i in $CONVERT; do
    			printf '%x\n' $i >> $DUMP_HOME/threads.txt
    		done
    	paste $b $DUMP_HOME/threads.txt >> $DUMP_HOME/threads_result.txt
    done
  • Then give permissions to the script with the following command:

    chmod +x /tmp/dump_reports
  • Now you can run the script:

    /tmp/dump_reports
  • The result will be something like that:

    PID 	 %CPU	
    6149 	 2.0	1805
    5872 	 0.0	16f0
    5873 	 0.0	16f1
    5874 	 0.0	16f2
    5875 	 0.0	16f3
    
    PID 	 %CPU	
    5880 	 57.4	16f8
    6393 	 45.5	18f9
    
  • Above we can see the PID of the thread, the percentual of CPU usage and the hexadecimal PID which can be used to read the confluence_threads files within the logs more detailed about what is happening.
Last modified on Feb 26, 2016

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.