Troubleshoot performance issues in Jira server
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Verify that your JIRA applications are running on appropriate Supported Platforms & the JIRA applications installation requirements are met prior to proceeding with further troubleshooting.
Overview
Troubleshooting Performance Problems is not a trivial process and can be quite complicated when it comes to identifying problems with JIRA applications. Given the number of different configurations possible for JIRA applications, such as different versions, plugins, network, hardware configurations and other various factors, it increases the difficulty of solving these problems. This Knowledge Base article is designed to assist in making that job easier to do and also provide guidance in identifying problems and also giving information to Support.
Troubleshooting performance problems often means isolating out each possible cause one-by-one, based off one or more symptoms and verifying if that is the root cause (or one of them). We do this with discovery (collecting diagnostic information on the instance), then analyze that information to see if we can identify problems. It often means changing something small, then testing if that change affected the stability of the instance - this can take quite some time depending upon the complexity of the instance (plugins, hardware configuration and so on).
Commonly Used Terms
Term | Description |
---|---|
Java Virtual Machine (JVM) | A Java virtual machine (JVM) is a virtual machine that can execute the compiled version of a Java application (known as bytecode). This allows for cross-platform functionality and the JVM is the code execution component of the Java platform that powers JIRA applications. |
Garbage Collection (GC) | In computer science, garbage collection (GC) is a form of automatic memory management. The garbage collector, or just collector, attempts to reclaim garbage, or memory occupied by objects that are no longer in use by the program. This is done automatically by the JVM. |
OutOfMemoryError (OOME) | This is thrown when the Java Virtual Machine cannot allocate an object because it is out of memory, and no more memory could be made available by the garbage collector. This can cause a number of problems and it's recommended to restart JIRA applications ASAP if one is thrown, as the JVM will likely be in an unstable state. |
Exception | An exception is an event (typically an error), which occurs during the execution of a program that disrupts the normal flow of that program's instructions. For example, a NullPointerException indicates JIRA applications have encountered a |
Stack Trace | A stack trace (also called stack backtrace or stack traceback) is a report of the active stack frames at a certain point in time during the execution of a program. It is commonly used during interactive and post-mortem debugging. It can also be displayed to the user of a program as part of an error message, which a user can report to a programmer. |
Thread Dump | This is a dump of what all the different Java threads are doing at the time the dump is taken and can be used to identify why a Java application is experiencing problems, as it gives a breakdown of the current state of every single thread. These are only useful when they are created when the system is experiencing problems (JIRA applications are locked up/hanging or running slow). |
Heap Dump | A dump of the entire contents of all objects within the memory of your JIRA applications. These can be used to identify what is causing a JVM to throw an OutOfMemoryError as it contains a breakdown of every single object within memory at the time the heap dump is produced. Much like a thread dump, they are only useful when an OOME is thrown. |
JIRA Application Hardware Configuration
JIRA Software Configuration
Symptom
JIRA applications are generally slow to respond and load in the browser.
At-capacity / Overused Instance or Server
Application bug (JIRA, JVM, Kernel / OS)
Plugin (third-party or bundled)
JIRA applications are unresponsive.
At-capacity / Overused Instance or Server
Application bug (JIRA, JVM, Kernel / OS)
Plugin (third-party or bundled)
CPU usage spikes significantly (e.g.: 80-100% usage)
JVM Configuration / Insufficient Memory
At-capacity / Overused Instance or Server
Application bug (JIRA, JVM, Kernel / OS)
Plugin (third-party or bundled)
JIRA application crash due to OutOfMemoryError
- Jira server crashes with OutofMemory Java heap space error
- GC overhead limit exceeded error crashes Jira server
JVM Configuration / Insufficient Memory
Application bug (JIRA, JVM, Kernel / OS)
Plugin (third-party or bundled)
Unable to obtain a connection from the underlying connection pool
Operations fail in Jira server with connection pool errors in logs
In order to proceed through this step, please step-through each common problem one-by-one, determining whether it is the cause of the relevant symptom.
Network Connectivity
Verify with the Sys Admin / hosting provider if there are any problems with the network and/or server. You can also perform checks by pinging or using netcat or telnet to connect to the server on the port where JIRA applications are hosted, such as:
ping 192.168.1.123
telnet 192.168.1.123 8080
Or,
root@kali:~# nc -vnz -w 1 192.168.1.123 8080
Connection to 192.168.1.123 8080 port [tcp/*] succeeded!
root@kali:~#
Netcat and telnet can be used to verify if the Firewall is blocking that port, and tools such as http://just-ping.com/ can be used to diagnose geographic-specific latency problems. For a comprehensive isolation test, connect to the server over the local network to verify if there are any network problems.
Application
Check the application is up (the Tomcat process will be running).
- Review the logs to see if there are any errors or exceptions and stack traces. Check https://jira.atlassian.com, the Jira Server, Jira Core and Jira Service Desk Document Archives and Google for those Exceptions/errors.
- Take 3 thread dumps, 10 seconds apart as per our Generating a Thread Dump docs to analyse. This will give a breakdown of exactly what the instance is doing when it's locked up.
The thread dumps can be analysed to see what may be causing the application to lock up. Thread dumps are only useful when they are taken during application failure or slow performance. If they're taken when the JIRA application instance is stable they are not useful as they will show that JIRA applications are stable, rather than experiencing problems. For examples on how to analyse thread dumps, please review the following sites:
- http://architects.dzone.com/articles/how-analyze-java-thread-dumps
- http://www.javacodegeeks.com/2012/03/jvm-how-to-analyze-thread-dump.html
Additionally:
- Enabling profiling in Administration > System > Troubleshooting and Support > Logging and Profiling and then testing can identify any problems.
- Also, profiling the instance with jProfiler, as per our Use jProfiler to analyse Jira application performance KB will give an exact breakdown of the classes and methods that are the slowest to respond.
Reverse-proxy / Load-balancer
If using a reverse-proxy / load-balancer, try bypassing it and access the instance directly, e.g.: http://192.168.1.123:8080/jira to see if there is any difference.
This will not work if the Tomcat connector has been set up as a proxy connector as per our Integrating JIRA applications with Apache using SSL docs.
Analyzing the access logs with our Jira HTTP Requests Log Analyzer is an excellent way of diagnosing if the instance is being overloaded. If so, it can increase response times and those logs will also assist in identifying the culprits.
Hardware Resources
- Verify the JIRA application Requirements are met and check the CPU load of the JIRA application instance to see if it is high, as if the CPU is consumed by another process (or by a JIRA application), it may be slow to respond.
- If there are multiple applications on the one server, verify that there are enough resources for all applications. To calculate the required resources for a JVM, add the maximum heap size to the permgen and add 128mb for the JVM. For example:
Application | Max Mem (Xmx) | PermGen | JVM Overhead | Total |
---|---|---|---|---|
JIRA | 1024 | 256 | 128 | 1408 |
Confluence | 1536 | 256 | 128 | 1920 |
For a server with Confluence and JIRA applications installed, it would require 3328mb just to provide to those two applications, and you would need to add the overhead of the operating system. At minimum, there should be 4gb available on that machine and it would need to be 64-bit. Also, take into account the requirements of other applications on the same server (for example, a database).
There are known problems if the server hosting JIRA applications is running 32-bit and other JVM applications, as in our JIRA crashes when running as a Windows Service KB.
Browser Performance Problems
Try using a different browser to see if there is any difference in the response - IE (especially IE8) will typically be significantly slower than Chrome to render content. For further assistance diagnosing problems, take a look through our Troubleshooting Client-Side Browser Problems KB.
A definitive isolation test for browser problems is to test cross-browser - if the problem persists across browsers it likely points to the server-side performance.
At-capacity / Overused Instance or Server
- There is no hard and fast rule on this as it often depends on the database stats (issues, custom fields, workflows, users, security schemes and so on) and the JIRA application version. Providing a Support ZIP to Atlassian Support is enough for us to provide some rough recommendations on the instance. It may be that an upgrade is required if the instance is too large for the version.
- Collect and review access logs to see if the instance is experiencing high load.
- Verify if an external application integration using SOAP or REST is overloading the instance.
- Upgrading a test server to the latest version and restoring production onto it is a sure-fire way of identifying if performance problems will be fixed when upgrading.
Analyzing the access logs with our Jira HTTP Requests Log Analyzer is an excellent way of diagnosing if the instance is being overloaded. If so, it can increase response times and those logs will also assist in identifying the culprits.
Disk Speed
Poor disk I/O will cause poor performance with searches, gadgets, and issues with 10+ attachments. Test the disk speed with the utility and instructions from our Testing Disk Access Speed KB and compare your results with the good/bad/average ranges.
Anti-virus "on access" does not get along with JIRA applications and the indexing directory must be excluded from on-access scanning. In some cases (such as certain versions of Symantec) these exclusions are ignored and the problems will continue. In this case it may be necessary to remove anti-virus from the server and seek another solution.
For Linux installations consider mounting your filesystems with noatime. This instructs the OS not to set a timestamp each time a file is read. When applications access Lucene indexes and cache files thousands of times a second the atime writes can add up to a noticeable slowdown... as much as 30% according to one case study. Before you change your mount options you must be sure no other application is dependant on atime. A more conservative setting that would still yield benefits is "relatime". For more info see Gain 30% Linux Disk Performance with noatime, nodiratime, and relatime.
Database
- Verify the load on the database and the DB server - our Test database performance for Jira Server KB can be used for this. Additionally, this should be raised with the DBA for the server, as Atlassian Support are not experts on diagnosing and troubleshooting database problems.
- Verify the
validationQuery
has been set as per Surviving Connection Closures and the database connection is configured as per the Connecting JIRA applications to a Database documentation.
It is critical that the correct version of the documentation is followed, as using incorrect parameters from the wrong versions can cause significant problems with the DB connection.
Known Issues
- JRA-9979 - Getting issue details... STATUS - pre JIRA 5.2, the notificationinstance could grow to massive sizes. In these cases, if JIRA accesses these tables, it would drastically slow down the response time between the database and JIRA. Clearing out this table has helped bring performance back.
JVM Configuration / Insufficient Memory
- Were JIRA applications recently upgraded? If so, verify that the previous JVM configurations were migrated from the older version.
- If experiencing OOMEs, increase the maximum memory by small amounts (256mb) as per Increasing JIRA application memory to verify if the problem is still occurring.
- Garbage Collection logs are required to analyze what's happening with the JVM memory that JIRA applications are using. These can be enabled as in our Troubleshoot Jira Server performance with GC logs KB article.
- Set up the JVM so that it generates a heap dump automagically when an OOME is thrown, as per our Analyze OutofMemory errors in Jira server with heap dumps KB.
- Check the logs for possible CodeCache warnings, addressed in more details in the Full CodeCache causes Jira to crash or perform slowly article.
Java VisualVM can also be installed and used to bind to the process that is running JIRA applications. This can be used to analyse CPU utilisation by GC, along with the heap usage. The visualGC plugin is recommended to be installed as it will provide a visual breakdown of each memory space.
- Analysing the GC logs and comparing when an instance is performing slowly is an excellent way of isolating out any memory problems. For example, if the instance is slow to perform during a specific time check if there are any Full GCs occurring during that time, as this could cause the application to respond slowly.
- If the CPU is spiking, check if there are consistent full GCs occurring, as garbage collection is CPU intensive. If there is insufficient memory provided to the JVM, it will need to go through more GC cycles, so it will be using more CPU. Ensure that the JVM has enough memory - increasing it in small increments can help isolate if this is a cause.
Using a tool such as GCViewer is very helpful for analysing GC logs.
Application bug (JIRA, JVM, Kernel/OS)
Check jira.atlassian.com, the Jira Server, Jira Core and Jira Service Desk Document Archives and also the DB/OS/Java vendor to see if there are any bugs that present with performance symptoms. Some examples of OS bugs that have crippled JIRA application performance:
- khugepaged eating 100%CPU
- Linux glibc >= 2.10 (RHEL 6) malloc may show excessive virtual memory usage
Plugin (third-party or bundled)
Atlassian does not provide support for third-party plugins. Please check the Atlassian Marketplace for the plugin developer's details and contact them directly, or raise a question on Atlassian Answers to receive assistance from our amazing community.
Plugins, especially third-party or in-house (developed internally), can cause a significant number of problems with JIRA applications. They can consume the DBCP, memory (heap usage), slow down issue operations and cause any number of other problems. Best practice is to verify if the performance problems are still experienced when no plugins are enabled. This can be done by enabling Safe Mode in the UPM will verify whether or not a plugin is a problem, however it comes at the detrimental impact of reducing the instance's functionality. This can be a hard case to make to the business, so we recommend this be done on a replicated test instance (that is, one identical to production). This can also be done by isolation testing as below:
- Enable Safe Mode and test to see if the problem is still reproducible. If it is not, it implies a plugin is causing a slow-down. If the problem is no longer reproducible, continue with the below steps, otherwise this rules out plugins altogether.
- Enable one single plugin and test.
- Repeat step 2 until the problem replicates itself - the most recently enabled plugin can be considered the culprit.
- Disable all other plugins except for the one identified in step 3 and then verify if it is indeed the problem (rather than that plugin being used in combination with any others). If so, raise it with the plugin developers as from the Atlassian Marketplace or with Atlassian if it's one of our plugins.
Additionally, verify that all your plugins are valid and up-to-date for the JIRA application version. It is highly common that an incompatible plugin will cause problems with a JIRA application instance.
Reviewing a heap dump from an OOME or memory utilisation in jProfiler (Use jProfiler to analyse Jira application performance) / VisualVM are suitable methods to identify if a plugin is chewing up a significant amount of heap, and thread dumps can indicate if the plugin is causing problems during lock ups, however these are no substitute for definitive isolation testing.
Providing Information to Support
Often when dealing with performance problems, Atlassian Support will request further info on the instance so we can troubleshoot the performance problems. In order to do so, please provide the following information:
- What are the specs of the server running JIRA applications (CPU/RAM/OS & Architecture/Physical or Virtual)?
- What other applications are running on that server?
- Which other applications use the database server?
- Which browser and version does the error happen in?
- Does it happen in any other browsers?
- If the instance is performing slowly, which pages are performing slowly (provide some sample URLs).
- When was the instance performing slowly (so we can check the logs)?
- Provide GC logs, as in JVM Configuration / Insufficient Memory.
- Provide Access logs if using a Reverse-proxy, as in Reverse-proxy / Load-balancer.
- The results of Disk Speed tests, as from that section.
- Generate a Support ZIP and provide it along with the above information.