Windows environment

Jira running from startup.bat

1. In the Command console window where Jira is running, open the properties dialog box by right-clicking on the title bar and select Properties.
2. Select the Layout tab.
3. Under Screen buffer size, set the Height to 3000.
4. Select Ok.
5. With the same command console in focus, press CTRL-BREAK. This will output the thread dump to the command console.
6. Scroll back in the command console until you reach the line containing "Full thread dump".
7. Right-click the title bar and select Edit > Mark. Highlight the entire text of the thread dump.
8. Right-click the title bar and select Edit > Copy. The thread dump can then be pasted into a text file.
    

Jira running as a Windows service

Using jstack

The JDK ships with a tool named jstack for generating thread dumps.

1. Identify the process. Launch the task manager by, pressing Ctrl + Shift + Esc and find the Process ID of the Java (Jira) process. You may need to add the PID column using View -> Select Columns ...

2. Run jstack to capture a single thread dump or multiple thread dumps at set intervals:

   Capture a single thread dump...

   Use the following command to capture a single thread dump of the process ID <JIRA_PID>:

   jstack.exe -l <JIRA_PID> > threaddump.txt

   For example, if the PID is 22668, enter:

   jstack.exe -l 22668 > threaddump.txt

   This will output to threaddump.txt in your current directory.

   Capture multiple thread dumps at set intervals...

   Use the following command to capture 6 thread dumps of the process id <JIRA_PID> in 10-second intervals between each thread dump:

   for /L %n in (1,1,6) do timeout 10 | jstack.exe -l <JIRA_PID> > threaddump-%n.txt

   For example, if the PID is 22668, enter:

   for /L %n in (1,1,6) do timeout 10 | jstack.exe -l 22668 > threaddump-%n.txt

   This will output to threaddump-%n.txt, where %n is the number of the loop iteration (starting at 1).

   You can modify the timeout command parameter to adjust the time between each thread dump and the range in the in (start,step,end) clause to adjust the number of thread dumps to capture. The (1,1,6) range in the example above means the following:

   • Start at 1
   • Increment by 1
   • End at 6

   Common issues with jstack:

   • You must run jstack as the same user that is running Jira.

   • If you get the error "Not enough storage is available to process this command", download the 'psexec' utility from here, and then run one of the following commands, where <JIRA_PID> is the Jira process ID (for example, 22668):

     • To capture a single thread dump:

       jstack.exe -l <JIRA_PID> > threaddump.txt

     • To capture multiple thread dumps at set intervals:

       1..6|foreach{jstack -l <JIRA_PID> |Out-File -FilePath "app_threads.$(Get-Date -uformat %s).txt";sleep 10}

   • If the jstack executable is not in your $PATH, then look for it in your <JDK_HOME>/bin directory
   • If you receive java.lang.NoClassDefFoundError: sun/tools/jstack/JStack check that tools.jar is present in your JDK's lib directory. If it is not, download a full version of the JDK.

Linux/Unix/OS X environment

Linux/Unix command line

1. Identify the java process that Jira is running in. This can be achieved by running a command similar to:

   ps -ef | grep java

   The process will appear similarly as follows:

   keithb     910   873  1 17:01 pts/3    00:00:18 /usr/java/jdk/bin/java -Xms128m -Xmx256m
   -Xms128m -Xmx256m -Djava.awt.headless=true -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
   -Djava.awt.headless=true -Djava.endorsed.dirs=/tmp/atlassian-jira-enterprise-3.6-standalone/common/endorsed
   -classpath :

2. In order to retrieve the thread dump, execute the command:
   
   For a single capture:

   kill -3 <pid>

   For multiple captures:

   for i in $(seq 6); do top -b -H -p <pid> -n 1 > jira_cpu_usage.`date +%s`.txt; kill -3 <pid>; sleep 10; done

   where pid is the process id — in this case, 910.

3. The thread dump will be written to the Tomcat console output. The console output is redirected to the logs/catalina.out file, which can be found in the Jira application installation directory for JIRA Standalone / Installer.

Linux/Unix Alternative: Generating thread dumps using jstack

If you have trouble using kill -3 <pid> to obtain a thread dump, try using jstack a java utility that will output stack traces of Java threads for a given process.

1. Identify the java process that Jira is running in. This can be achieved by running a command similar to:

   ps -ef | grep java

   The process will appear similarly as follows:

   adam 22668 0.3 14.9 1691788 903928 ? Sl Jan27 9:36 /usr/lib/jvm/java-6-sun-1.6.0.14/bin/java -Djava.util.logging.config.file=/home/adam/Products/installs/atlassian-jira-enterprise-4.0.1-standalone/conf/logging.properties -XX:MaxPermSize=256m -Xms128m -Xmx1048m -Djava.awt.headless=true -Datlassian.standalone=JIRA -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true -Dmail.mime.decodeparameters=true -Datlassian.mail.senddisabled=false -Datlassian.mail.fetchdisabled=false -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/home/adam/Products/installs/atlassian-jira-enterprise-4.0.1-standalone/common/endorsed -classpath /home/adam/Products/installs/atlassian-jira-enterprise-4.0.1-standalone/bin/bootstrap.jar -Dcatalina.base=/home/adam/Products/installs/atlassian-jira-enterprise-4.0.1-standalone -Dcatalina.home=/home/adam/Products/installs/atlassian-jira-enterprise-4.0.1-standalone -Djava.io.tmpdir=/home/adam/Products/installs/atlassian-jira-enterprise-4.0.1-standalone/temp org.apache.catalina.startup.Bootstrap start

2. Run jstack to capture a single thread dump or multiple thread dumps at set intervals:

   Capture a single thread dump...

   Use the following command to capture a single thread dump of the process ID <JIRA_PID>:

   jstack <JIRA_PID> > threaddump.txt

   For example, if the PID is 22668, enter:

   jstack 22668 > threaddump.txt

   The output will be saved to threaddump.txt in the current directory.

   Capture multiple thread dumps at set intervals...

   Use the following command to capture 6 thread dumps of the process ID <JIRA_PID> in 10-second intervals between each thread dump:

   for i in $(seq 6); do top -b -H -p <JIRA_PID> -n 1 > threaddump.$(date +%s).txt; kill -3 <JIRA_PID>; sleep 10; done

   For example, if the PID is 22668, enter:

   for i in $(seq 6); do top -b -H -p 22668 -n 1 > threaddump.$(date +%s).txt; kill -3 22668; sleep 10; done

   The output will be saved to threaddump.$(date +%s).txt in the current directory, where date +%s is the current Unix timestamp.

   You can modify the seq command parameter to adjust the the number of thread dumps to capture and the sleep command parameter to adjust the time between each thread dump.

   If you are connecting to the server through RDP, jstack might fail with following error:

   Not enough storage is available to process this command 

   You will need to open a RDP session in console mode: mstsc /admin

Linux/Unix Alternative: Generating thread dumps using scripts

You can also generate a thread dump by using scripts prepared by our Support team. That's an easy process, and the scripts will do everything for you. As an addition to generating a thread dump, the scripts also allow you to generate heap dumps, check the disk access speed, or the Java SSL connection.

1. Make sure that the Thread diagnostics feature is enabled in your app. Learn more about generating thread dumps with thread diagnostics 
2. Download and install the scripts from https://bitbucket.org/atlassianlabs/atlassian-support/.
3. Execute the scripts when your Jira instance behaves slowly or is unresponsive. 
   
   1. (Optional) The scripts also allow you to test the disk access speed. This is described in more detail in Testing disk access speed. 
   2. When asked whether you want to capture thread dumps, enter Y.
   3. When asked whether you want to capture heap dumps, enter N.
   4. (Optional) The scripts also allow you to check the Java SSL connection.
4. After running the scripts, the thread dump will be captured and compressed. You can now open a Support ticket and attach the generated package.

Steps for Atlassian Docker containers

If you're running Jira on a container, follow the steps below:

/opt/atlassian/support/thread-dumps.sh can be run via docker exec to easily trigger the collection of thread dumps from the containerized application. For example:

docker exec my_container /opt/atlassian/support/thread-dumps.sh

By default this script will collect 10 thread dumps at 5 second intervals. This can be overridden by passing a custom value for the count and interval, by using -c / --count and -i / --interval respectively. For example, to collect 20 thread dumps at 3 second intervals:

docker exec my_container /opt/atlassian/support/thread-dumps.sh --count 20 --interval 3

If you're running the Docker container in a Kubernetes environment, you can execute the command as below:

kubectl exec -it jira-1 -n jira -- bash -c "/opt/atlassian/support/thread-dumps.sh --count 20 --interval 3"

Replace -it jira-1  with the pod name, and -n jira  with the namespace where the Jira pods are running.

Thread dumps will be written to $APP_HOME/thread_dumps/<date>.

Note: By default this script will also capture output from top run in 'Thread-mode'. This can be disabled by passing -n / --no-top

The Troubleshooting section on https://hub.docker.com/r/atlassian/jira-software has additional information. 

Analysis tools

Try Watson, TDA, or Samurai to inspect your thread dump.

TDA

1. Download TDA.
2. CD to the directory where the JAR exists.

3. Run:

   java -jar -Xmx512M ~/tda-bin-1.6/tda.jar

4. Open your catalina.out file, containing the thread dump.

Issue processing thread dump with TDA (NumberFormatException)

Should you get an error on TDA console like:

Exception in thread "AWT-EventQueue-0" java.lang.NumberFormatException: For input string: "5 os_prio=0"
        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
        at java.lang.Long.parseLong(Long.java:441)
        at java.lang.Long.<init>(Long.java:702)
        at com.pironet.tda.utils.ThreadsTableModel.getValueAt(ThreadsTableModel.java:80)
        at com.pironet.tda.utils.TableSorter.getValueAt(TableSorter.java:285)
        at javax.swing.JTable.getValueAt(JTable.java:2717)
        at javax.swing.JTable.prepareRenderer(JTable.java:5719)
        at javax.swing.plaf.basic.BasicTableUI.paintCell(BasicTableUI.java:2114)
        at javax.swing.plaf.basic.BasicTableUI.paintCells(BasicTableUI.java:2016)
        at javax.swing.plaf.basic.BasicTableUI.paint(BasicTableUI.java:1812)
        at javax.swing.plaf.ComponentUI.update(ComponentUI.java:161)
        at javax.swing.JComponent.paintComponent(JComponent.java:778)
        at javax.swing.JComponent.paint(JComponent.java:1054)
        at javax.swing.JComponent.paintChildren(JComponent.java:887)
        at javax.swing.JComponent.paint(JComponent.java:1063)
        at javax.swing.JViewport.paint(JViewport.java:731)
        at javax.swing.JComponent.paintChildren(JComponent.java:887)
        at javax.swing.JComponent.paint(JComponent.java:1063)
        at javax.swing.JComponent.paintChildren(JComponent.java:887)
        at javax.swing.JSplitPane.paintChildren(JSplitPane.java:1047)
        at javax.swing.JComponent.paint(JComponent.java:1063)
        at javax.swing.JComponent.paintToOffscreen(JComponent.java:5230)
        at javax.swing.BufferStrategyPaintManager.paint(BufferStrategyPaintManager.java:295)
        at javax.swing.RepaintManager.paint(RepaintManager.java:1249)
        at javax.swing.JComponent._paintImmediately(JComponent.java:5178)
        at javax.swing.JComponent.paintImmediately(JComponent.java:4989)
        at javax.swing.RepaintManager$3.run(RepaintManager.java:808)
        at javax.swing.RepaintManager$3.run(RepaintManager.java:796)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
        at javax.swing.RepaintManager.paintDirtyRegions(RepaintManager.java:796)
        at javax.swing.RepaintManager.paintDirtyRegions(RepaintManager.java:769)
        at javax.swing.RepaintManager.prePaintDirtyRegions(RepaintManager.java:718)
        at javax.swing.RepaintManager.access$1100(RepaintManager.java:62)
        at javax.swing.RepaintManager$ProcessingRunnable.run(RepaintManager.java:1677)
        at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:312)
        at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:733)
        at java.awt.EventQueue.access$200(EventQueue.java:103)
        at java.awt.EventQueue$3.run(EventQueue.java:694)
        at java.awt.EventQueue$3.run(EventQueue.java:692)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
        at java.awt.EventQueue.dispatchEvent(EventQueue.java:703)
        at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:242)
        at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:161)
        at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:150)
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:146)
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:138)
        at java.awt.EventDispatchThread.run(EventDispatchThread.java:91)

Apply the following command on the thread dump(s) to fix the thread header format to make it processable:
sed -i 's/prio=[0-9]\{1,2\} os_prio=[0-9]\{1,2\}/prio=5/g' <filename>

Check the known thread dump knowledge base articles:

• Page:
  Troubleshoot index problems in Jira server
• Page:
  OutOfMemory or Poor Performance due to XML View of a Filter
• Page:
  JIRA Deadlocks when Running Tomcat 6.0.24
• Page:
  Jira slow with dangerous use of multiple connections error in log
• Page:
  How to ignore updates in the issues that were made during the weekend while closing them via Automation for Jira
• Page:
  Jira server crashes with OutofMemory Java heap space error