If Bitbucket Server stops responding or is showing poor performance, providing thread dumps to Support can help diagnose the problem.

Generating a Thread Dump for Windows

Method 1 - Windows scripts

We now have scripts for generating thread dumps externally on Windows and Linux. Download them from this Bitbucket Repository.

Method 2 - CTRL+BREAK (only if Bitbucket Server is running in a console window)

1. 1. In the Command Console window where Bitbucket Server is running, open the properties dialog box by right clicking on the title bar and select "Properties".
   2. Select the Layout tab.
   3. Under Screen Buffer Size, set the Height to 3000.
   4. Click OK.
   5. With the same command console in focus, press CTRL-BREAK. This will output the thread dump to the command console.
   6. Scroll back in the command console until you reach the line containing "Full thread dump".
   7. Right click the title bar and select Edit -> Mark. Highlight the entire text of the thread dump.
   8. Right click the title bar and select Edit -> Copy. The thread dump can then be pasted into a text file.

Method 3 - jstack:

The JDK ships with a tool named jstack for generating thread dumps.

1. 1. Identify the process. Launch the task manager by, pressing Ctrl + Shift + Esc and find the Process ID of the Java (JIRA) process. You may need to add the PID column using View -> Select Columns ...
   2. Run jstack <pid> to Capture a Single Thread Dump. This command will take one thread dump of the process id <pid>, in this case the pid is 22668:

C:\Users\Administrator>jstack.exe -l 22668 > threaddump.txt

This will output a file called threaddump.txt to your current directory.

Method 4 - VisualVM:

VisualVM is only provided with the JDK distribution so you must have this installed regardless of the scenario describing your environment below. You don't have to change the Java version used by the application to use VisualVM but you will have to add JMX parameters if this is the case or if you are running Bitbucket Server as a service.

Scenario 1: If Bitbucket Server is using the JDK and Bitbucket Server is running as a console application:

1. 1. Start VisualVM: <JDK installation directory>\bin\jvisualvm.exe
   2. Select the tomcat process in the left-hand pane under "Local".
   3. Select the "Threads" tab in the right pane.

   4. Click the button "Thread Dump" (and then select the "Threads" tab to get back to the button).

Scenario 2: If Bitbucket Server is not using the JDK and Bitbucket Server is running as a console application:

1. 1. Stop Bitbucket Server.

   2. Edit the <Bitbucket Server installation directory>\bin\setenv.bat and add the following JMX parameters to those already defined for the JVM_SUPPORT_RECOMMENDED_ARGS property. You must choose a free port for JMX to bind to – the following example uses port 7995:

      rem
      rem Occasionally Atlassian Support may recommend that you set some specific JVM arguments.  You can use this variable
      rem below to do that.
      rem
      set JVM_SUPPORT_RECOMMENDED_ARGS=-Dcom.sun.management.jmxremote.port=7995 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false

       As of Bitbucket Server 5.x, the JVM_SUPPORT_RECOMMENDED_ARGS property is defined in the _start-webapp.sh ( for Linux/Mac) and _start-webapp.bat ( for Windows) in the bin directory
       

   3. Restart Bitbucket Server.
   4. Start VisualVM: <JDK installation directory>\bin\jvisualvm.exe
   5. Select "File" > "Add JMX Connection".
   6. Type: localhost:7995
   7. Select the tomcat process in the left-hand pane under "Local".
   8. Select the "Threads" tab in the right pane.

   9. Click the button "Thread Dump" (and then select the "Threads" tab to get back to the button).

   10. As alternative, the "Thread Dump" option is available when you right click on the tomcat process.

Scenario 3: If Bitbucket Server is running as a service:

1. 1. Stop Bitbucket Server.
   2. Open the command window from Start >> Run >> type in 'cmd' >> Enter

   3. cd to the bin directory the Bitbucket Server Installation (<Bitbucket Server Installation dir>/bin).

      tomcat8w //ES//AtlassianBitbucket 

   4. A window will pop up. Click on the Java tab to see the list of current start-up options. Under "Java Options:" form, input the JMX parameters to the service configuration in addition to what is already there and apply the changes. You must choose a free port for JMX to bind to – the following example uses port 7995:

      -Dcom.sun.management.jmxremote.port=7995
      -Dcom.sun.management.jmxremote.authenticate=false
      -Dcom.sun.management.jmxremote.ssl=false

   5. Restart Bitbucket Server.

   6. Start VisualVM: <JDK installation directory>\bin\jvisualvm.exe
   7. Select "File" > "Add JMX Connection".
   8. Type: localhost:7995
   9. Select the tomcat process in the left-hand pane under "Local".
   10. Select the "Threads" tab in the right pane.

   11. Click the button "Thread Dump" (and then select the "Threads" tab to get back to the button).
       
       

Generating a Thread Dump on Linux, including Solaris and other Unixes

Method 1 - jstack:

Sun JDK 1.5 and above ship with native tool called jstack to perform thread dump.

• • Find the process ID of Bitbucket Server the JVM using the ps command:

BITBUCKET_PID=`ps aux | grep -v search | grep -v mesh | grep -i bitbucket | grep -i java | awk  -F '[ ]*' '{print $2}'`;

• • Generate 6 thread dumps and cpu usage snapshots with a 10 second interval:

for i in $(seq 6); do
  top -b -H -p $BITBUCKET_PID -n 1 > bitbucket_cpu_usage.`date +%s`.txt; jstack $BITBUCKET_PID > bitbucket_threads.`date +%s`.txt
  sleep 10
done

• • Look in the resulting cpu usage files to identify which threads are consistently using a lot of CPU time.

  • Take the PID of the top 10 threads which are using CPU time and convert them to Hexadecimal. Eg: 11159 becomes 0x2b97
    
  • Search up the Hex values in the thread dumps to figure out which threads are using up all the CPU

Optional:  Generate Thread Dump and CPU dumps of Mesh Sidecar or Remote Mesh process

If you are on Bitbucket 8.x version, Git SCM requests are handled by a separate Java process called Bitbucket Mesh sidecar. When troubleshooting issues related to Git SCM processes, we need to generate the thread dumps and CPU dumps for the Mesh Sidecar, as well as for the main Bitbucket process.

• To find the process ID of the Bitbucket Server Mesh Sidecar JVM, use the `ps` command:

BITBUCKET_MESH_PID=`ps aux | grep -v search | grep -i java | grep 'mesh-app.jar' | awk  -F '[ ]*' '{print $2}'`;

• Generate 6 thread dumps and cpu usage snapshots with a 10 second interval:

for i in $(seq 6); do
  top -b -H -p $BITBUCKET_MESH_PID -n 1 > bitbucket_mesh_cpu_usage.`date +%s`.txt; jstack $BITBUCKET_MESH_PID > bitbucket_mesh_threads.`date +%s`.txt
  sleep 10
done

Method 2 - kill signal:

An alternative to the method above is:

• • Find the process ID of Bitbucket Server the JVM using the ps command:
    
    

BITBUCKET_PID=`ps aux | grep -v search | grep -v mesh | grep -i bitbucket | grep -i java | awk  -F '[ ]*' '{print $2}'`

• • Generate 6 thread dumps and cpu usage snapshots with a 10 second interval:
    
    

for i in $(seq 6); do
  top -b -H -p $BITBUCKET_PID -n 1 > app_cpu_usage.`date +%s`.txt; kill -3 $BITBUCKET_PID
  sleep 10
done

Note: This will not kill your server (so long as you included the "-3" option, no space in between).

The thread dumps will be written to Bitbucket Server's standard output ($BITBUCKET_HOME/log/launcher.log for Bitbucket Server 5.+ or catalina.out for Bitbucket Server up to version 4.14.x).
If you have trouble generating the thread dumps with this method, then use the method "Generating a Thread dump for Windows" as they can also apply for linux, etc.

Generate Thread Dump and CPU dumps of Mesh Sidecar or Remote Mesh process

• To find the process ID of the Bitbucket Server Mesh Sidecar JVM, use the `ps` command:

BITBUCKET_MESH_PID=`ps aux | grep -v search | grep -i java | grep 'mesh-app.jar' | awk  -F '[ ]*' '{print $2}'`;

• Generate 6 thread dumps and cpu usage snapshots with a 10 second interval:

for i in $(seq 6); do
  top -b -H -p $BITBUCKET_MESH_PID -n 1 > bitbucket_mesh_app_cpu_usage.`date +%s`.txt; kill -3 $BITBUCKET_MESH_PID
  sleep 10
done

The thread dumps will be written to Bitbucket Server's Mesh standard output file $BITBUCKET_HOME/log/mesh-launcher.log

Generating a Thread Dump in the Docker deployments

1. Switch to the user running Bitbucket inside the container (daemon for pre-6.0 and atlbitbucket for 6.0 and newer):

   $ docker exec -it bitbucket bash
   # su -s /bin/bash <bitbucket_user>

2. Find Java process PID for Bitbucket and Bitbucket Mesh Sidecar inside the container and set it as environment variable 

   export BITBUCKET_PID='<bitbucket_main_pid>'
   export BITBUCKET_MESH_PID='<bitbucket_mesh_pid>'

3. Take the thread dumps for the Bitbucket Main process with the following command: 

   for i in $(seq 10); do
     top -b -H -p $BITBUCKET_PID -n 1 > bitbucket_cpu_usage.`date +%s`.txt; jcmd $BITBUCKET_PID Thread.print > bitbucket_threads.`date +%s`.txt
     sleep 10
   done

4. Take the thread dumps for the Bitbucket Mesh Sidecar process with the following command: 

5. for i in $(seq 10); do
     top -b -H -p $BITBUCKET_MESH_PID -n 1 > bitbucket_mesh_cpu_usage.`date +%s`.txt; jcmd $BITBUCKET_MESH_PID Thread.print > bitbucket_mesh_threads.`date +%s`.txt
     sleep 10
   done

6. The thread dumps will be stored in the directory where the commands has been executed. You can fetch them via persistent storage or with docker cp.

Thread Dump Tools for analyzing the data

• Samurai
• Thread Dump Analyzer