Generate a Thread Dump Externally

Still need help?

The Atlassian Community is here for you.

Ask the community


If Bitbucket Server stops responding or is showing poor performance, providing thread dumps to Support can help diagnose the problem.

If you were asked by Atlassian technical support to create thread dumps please take six thread dumps – one every 10 seconds for one minute – so we can identify patterns. Attach all generated files to the support ticket in addition to a Support Zip (the Support Zip will contain catalina.out).

(info) If you are using Bitbucket Server 5.x, catalina.out file will no longer exist. The thread dumps will be written in the $BITBUCKET_HOME/log/launcher.log file instead.

Generating a Thread Dump for Windows

Method 1 - Windows scripts

We now have scripts for generating thread dumps externally on Windows. Download them from this Bitbucket Repository.

Method 2 - CTRL+BREAK (only if Bitbucket Server is running in a console window)

    1. In the Command Console window where Bitbucket Server is running, open the properties dialog box by right clicking on the title bar and select "Properties".
    2. Select the Layout tab.
    3. Under Screen Buffer Size, set the Height to 3000.
    4. Click OK.
    5. With the same command console in focus, press CTRL-BREAK. This will output the thread dump to the command console.
    6. Scroll back in the command console until you reach the line containing "Full thread dump".
    7. Right click the title bar and select Edit -> Mark. Highlight the entire text of the thread dump.
    8. Right click the title bar and select Edit -> Copy. The thread dump can then be pasted into a text file.

Method 3 - jstack:

The JDK ships with a tool named jstack for generating thread dumps.

    1. Identify the process. Launch the task manager by, pressing Ctrl + Shift + Esc and find the Process ID of the Java (JIRA) process. You may need to add the PID column using View -> Select Columns ...
    2. Run jstack <pid> to Capture a Single Thread Dump. This command will take one thread dump of the process id <pid>, in this case the pid is 22668:


C:\Users\Administrator>jstack.exe -l 22668 > threaddump.txt


This will output a file called threaddump.txt to your current directory.

Common issues with jstack:

    • You must run jstack as the same user that is running Bitbucket Server.
    • If you get the error "Not enough storage is available to process this command", download the 'psexec' utility from here, then run the following command using:
      psexec -s jstack <pid> >> threaddumps.txt
    • If the jstack executable is not in your $PATH, then please look for it in your <JDK_HOME>/bin directory
    • If you receive java.lang.NoClassDefFoundError: sun/tools/jstack/JStack check that tools.jar is present in your JDK's lib directory. If it is not, download a full version of the JDK.


Method 4 - VisualVM:

VisualVM is only provided with the JDK distribution so you must have this installed regardless of the scenario describing your environment below. You don't have to change the Java version used by the application to use VisualVM but you will have to add JMX parameters if this is the case or if you are running Bitbucket Server as a service.

Scenario 1: If Bitbucket Server is using the JDK and Bitbucket Server is running as a console application:
    1. Start VisualVM: <JDK installation directory>\bin\jvisualvm.exe
    2. Select the tomcat process in the left-hand pane under "Local".
    3. Select the "Threads" tab in the right pane.
    4. Click the button "Thread Dump" (and then select the "Threads" tab to get back to the button).

Scenario 2: If Bitbucket Server is not using the JDK and Bitbucket Server is running as a console application:
    1. Stop Bitbucket Server.
    2. Edit the <Bitbucket Server installation directory>\bin\setenv.bat and add the following JMX parameters to those already defined for the JVM_SUPPORT_RECOMMENDED_ARGS property. You must choose a free port for JMX to bind to – the following example uses port 7995:

      rem
      rem Occasionally Atlassian Support may recommend that you set some specific JVM arguments.  You can use this variable
      rem below to do that.
      rem
      set JVM_SUPPORT_RECOMMENDED_ARGS=-Dcom.sun.management.jmxremote.port=7995 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false

      (info) As of Bitbucket Server 5.x, the JVM_SUPPORT_RECOMMENDED_ARGS property is defined in the _start-webapp.sh ( for Linux/Mac) and _start-webapp.bat ( for Windows) in the bin directory
       

    3. Restart Bitbucket Server.
    4. Start VisualVM: <JDK installation directory>\bin\jvisualvm.exe
    5. Select "File" > "Add JMX Connection".
    6. Type: localhost:7995
    7. Select the tomcat process in the left-hand pane under "Local".
    8. Select the "Threads" tab in the right pane.
    9. Click the button "Thread Dump" (and then select the "Threads" tab to get back to the button).

    10. As alternative, the "Thread Dump" option is available when you right click on the tomcat process.
Scenario 3: If Bitbucket Server is running as a service:
    1. Stop Bitbucket Server.
    2. Open the command window from Start >> Run >> type in 'cmd' >> Enter
    3. cd to the bin directory the Bitbucket Server Installation (<Bitbucket Server Installation dir>/bin).

      tomcat8w //ES//AtlassianBitbucket 
    4. A window will pop up. Click on the Java tab to see the list of current start-up options. Under "Java Options:" form, input the JMX parameters to the service configuration in addition to what is already there and apply the changes. You must choose a free port for JMX to bind to – the following example uses port 7995:

      -Dcom.sun.management.jmxremote.port=7995
      -Dcom.sun.management.jmxremote.authenticate=false
      -Dcom.sun.management.jmxremote.ssl=false
    5. Restart Bitbucket Server.

    6. Start VisualVM: <JDK installation directory>\bin\jvisualvm.exe
    7. Select "File" > "Add JMX Connection".
    8. Type: localhost:7995
    9. Select the tomcat process in the left-hand pane under "Local".
    10. Select the "Threads" tab in the right pane.
    11. Click the button "Thread Dump" (and then select the "Threads" tab to get back to the button).

Generating a Thread Dump on Linux, including Solaris and other Unixes

Method 1 - jstack:

Sun JDK 1.5 and above ship with native tool called jstack to perform thread dump.

    • Find the process ID of Bitbucket Server the JVM using the ps command:


BITBUCKET_PID=`ps aux | grep -v elasticsearch | grep -i bitbucket | grep -i java | awk  -F '[ ]*' '{print $2}'`;
    • Generate 6 thread dumps and cpu usage snapshots with a 10 second interval:
for i in $(seq 6); do
  top -b -H -p $BITBUCKET_PID -n 1 > bitbucket_cpu_usage.`date +%s`.txt; jstack $BITBUCKET_PID > bitbucket_threads.`date +%s`.txt
  sleep 10
done


    • Look in the resulting cpu usage files to identify which threads are consistently using a lot of CPU time.

    • Take the PID of the top 10 threads which are using CPU time and convert them to Hexadecimal. Eg: 11159 becomes 0x2b97
    • Search up the Hex values in the thread dumps to figure out which threads are using up all the CPU
Method 2 - kill signal:

An alternative to the method above is:

    • Find the process ID of Bitbucket Server the JVM using the ps command:

BITBUCKET_PID=`ps aux | grep -v elasticsearch | grep -i bitbucket | grep -i java | awk  -F '[ ]*' '{print $2}'`
    • Generate 6 thread dumps and cpu usage snapshots with a 10 second interval:

for i in $(seq 6); do
  top -b -H -p $BITBUCKET_PID -n 1 > app_cpu_usage.`date +%s`.txt; kill -3 $BITBUCKET_PID
  sleep 10
done

Note: This will not kill your server (so long as you included the "-3" option, no space in between).


The thread dumps will be written to Bitbucket Server's standard output ($BITBUCKET_HOME/log/launcher.log for Bitbucket Server 5.+ or catalina.out for Bitbucket Server up to version 4.14.x).
If you have trouble generating the thread dumps with this method, then use the method "Generating a Thread dump for Windows" as they can also apply for linux, etc.

The preferred method is the first one as we can cross-check running processes and threads.

Generating a Thread Dump in the Docker deployments

  1. Switch to the user running Bitbucket inside the container (daemon for pre-6.0 and atlbitbucket for 6.0 and newer):

    $ docker exec -it bitbucket bash
    # su -s /bin/bash daemon
  2. Find Java process PID for Bitbucket inside the container and set it as environment variable 

    export BITBUCKET_PID='76'
  3. Take the thread dumps with the following command: 

    for i in $(seq 10); do
      top -b -H -p $BITBUCKET_PID -n 1 > bitbucket_cpu_usage.`date +%s`.txt; jcmd $BITBUCKET_PID Thread.print > bitbucket_threads.`date +%s`.txt
      sleep 10
    done
  4. The thread dumps will be stored in the directory where the previous command has been executed. You can fetch them via persistent storage or with docker cp.

Thread Dump Tools for analyzing the data


Last modified on Oct 22, 2019

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.