Generate a Thread Dump Externally

'How Do I...' and 'How to...' Guide to Stash

On this page

Still need help?

The Atlassian Community is here for you.

Ask the community

 

If Stash stops responding or is showing poor performance, providing thread dumps to Support can help diagnose the problem.

If you were asked by Atlassian technical support to create thread dumps please take six thread dumps – one every 10 seconds for one minute – so we can identify patterns. Attach all generated files to the support ticket in addition to a Support Zip (the Support Zip will contain catalina.out).

Generating a Thread Dump for Windows

Method 1 - Windows scripts

We now have scripts for generating thread dumps externally on Windows. Download them from this Bitbucket Repository.

Method 2 - CTRL+BREAK (only if Stash is running in a console window)
    1. In the Command Console window where Stash is running, open the properties dialog box by right clicking on the title bar and select "Properties".
    2. Select the Layout tab.
    3. Under Screen Buffer Size, set the Height to 3000.
    4. Click OK.
    5. With the same command console in focus, press CTRL-BREAK. This will output the thread dump to the command console.
    6. Scroll back in the command console until you reach the line containing "Full thread dump".
    7. Right click the title bar and select Edit -> Mark. Highlight the entire text of the thread dump.
    8. Right click the title bar and select Edit -> Copy. The thread dump can then be pasted into a text file.
Method 3 - jstack:

The JDK ships with a tool named jstack for generating thread dumps.

    1. Identify the process. Launch the task manager by, pressing Ctrl + Shift + Esc and find the Process ID of the Java (JIRA) process. You may need to add the PID column using View -> Select Columns ...
    2. Run jstack <pid> to Capture a Single Thread Dump. This command will take one thread dump of the process id <pid>, in this case the pid is 22668:

 

C:\Users\Administrator>jstack.exe -l 22668 > threaddump.txt

 

This will output a file called threaddump.txt to your current directory.

Common issues with jstack:

    • You must run jstack as the same user that is running Stash.
    • If you get the error "Not enough storage is available to process this command", download the 'psexec' utility from here, then run the following command using:
      psexec -s jstack <pid> >> threaddumps.txt
    • If the jstack executable is not in your $PATH, then please look for it in your <JDK_HOME>/bin directory
    • If you receive java.lang.NoClassDefFoundError: sun/tools/jstack/JStack check that tools.jar is present in your JDK's lib directory. If it is not, download a full version of the JDK.

 

Method 4 - VisualVM:

VisualVM is only provided with the JDK distribution so you must have this installed regardless of the scenario describing your environment below. You don't have to change the Java version used by the application to use VisualVM but you will have to add JMX parameters if this is the case or if you are running Stash as a service.

Scenario 1: If Stash is using the JDK and Stash is running as a console application:

 

    1. Start VisualVM: <JDK installation directory>\bin\jvisualvm.exe
    2. Select the tomcat process in the left-hand pane under "Local".
    3. Select the "Threads" tab in the right pane.
    4. Click the button "Thread Dump" (and then select the "Threads" tab to get back to the button).

 

 

Scenario 2: If Stash is not using the JDK and Stash is running as a console application:
    1. Stop Stash.
    2. Edit the <Stash installation directory>\bin\setenv.bat and add the following JMX parameters to those already defined for the JVM_SUPPORT_RECOMMENDED_ARGS property. You must choose a free port for JMX to bind to – the following example uses port 7995:

      rem
      rem Occasionally Atlassian Support may recommend that you set some specific JVM arguments.  You can use this variable
      rem below to do that.
      rem
      set JVM_SUPPORT_RECOMMENDED_ARGS=-Dcom.sun.management.jmxremote.port=7995 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
    3. Restart Stash.
    4. Start VisualVM: <JDK installation directory>\bin\jvisualvm.exe
    5. Select "File" > "Add JMX Connection".
    6. Type: localhost:7995
    7. Select the tomcat process in the left-hand pane under "Local".
    8. Select the "Threads" tab in the right pane.
    9. Click the button "Thread Dump" (and then select the "Threads" tab to get back to the button).

 

Scenario 3: If Stash is running as a service:
    1. Stop Stash.
    2. Edit the existing service – the default service name is AtlassianStash: <Stash installation directory>\bin\tomcat8w.exe //ES//AtlassianStash  and add the JMX parameters to the service configuration on the "Java" tab under "Java Options" in addition to what is already there and apply the changes. You must choose a free port for JMX to bind to – the following example uses port 7995:

      -Dcom.sun.management.jmxremote.port=7995
      -Dcom.sun.management.jmxremote.authenticate=false
      -Dcom.sun.management.jmxremote.ssl=false
    3. Restart Stash.

    4. Start VisualVM: <JDK installation directory>\bin\jvisualvm.exe
    5. Select "File" > "Add JMX Connection".
    6. Type: localhost:7995
    7. Select the tomcat process in the left-hand pane under "Local".
    8. Select the "Threads" tab in the right pane.
    9. Click the button "Thread Dump" (and then select the "Threads" tab to get back to the button).

Generating a Thread Dump on Linux, including Solaris and other Unixes

Method 1 - jstack:

Sun JDK 1.5 and above ship with native tool called jstack to perform thread dump.

    • Find the process ID of Stash the JVM using the ps command:

 

STASH_PID=`ps aux | grep -i stash | grep -i java | awk  -F '[ ]*' '{print $2}'`;
    • Then run the following command 6 times with a 10 second interval.

 

top -b -H -p $STASH_PID -n 1 > stash_cpu_usage.`date +%s`.txt; jstack $STASH_PID > stash_threads.`date +%s`.txt

 

    • Look in the resulting cpu usage files to identify which threads are consistently using a lot of CPU time.

    • Take the PID of the top 10 threads which are using CPU time and convert them to Hexadecimal. Eg: 11159 becomes 0x2b97
    • Search up the Hex values in the thread dumps to figure out which threads are using up all the CPU
Method 2 - kill signal:

An alternative to the method above is:

    • Find the process ID of Stash the JVM using the ps command:

ps aux | grep -i stash | grep -i java | awk  -F '[ ]*' '{print $2}'
    • Then run the following command 6 times with a 10 second interval:

kill -3 <pid>

Note: This will not kill your server (so long as you included the "-3" option, no space in between).
The thread dump will be printed to Stash's standard output (catalina.out).
If you have trouble generating the thread dumps with this method, then use the method "Generating a Thread dump for Windows" as they can also apply for linux, etc.

The preferred method is the first one as we can cross-check running processes and threads.

Thread Dump Tools for analyzing the data


Last modified on Mar 30, 2016

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.