Test disk access speed for Jira server performance troubleshooting
Platform Notice: Server and Data Center Only - This article only applies to Atlassian products on the server and data center platforms.
The Atlassian Server application appears to be experiencing performance issues and is running slowly. These applications could include Java based software such as JIRA, Confluence, Bitbucket Server, Bamboo, and Fisheye/Crucible.
For example in JIRA, this will be particularly noticeable in the following areas:
- The Issue Navigator
- Opening Issues
Disk access speed is critical for the Atlassian applications performance.
If the application is running slowly, disk speed can be a potential root cause. It's possible to isolate that cause using this Support Tools Utility to benchmark the disk access speed.
Download the Support Tools Utility to any writable folder on the actual JIRA server (for example ~/$username$/temp on Linux or c:\temp on Windows).
This is a file with a .jar extension and sometimes Internet Explorer renames these to .zip. If this is the case, rename the file to .jar; do not unzip the file.
- For Windows: Open a terminal window (go to
Start >> Run >> type in 'cmd').
- For Linux/Unix: Open a command line.
- When running the tool, write access to the temporary directory is required - it will create and delete a large number of files in order to calculate the disk speed.
CD to the directory where the tools was downloaded and run it with the following:
java -Djava.io.tmpdir=<index directory> -jar support-tools.jar
Depending on your application please place the .jar in the following directories <index directory> will be:
For JIRA make sure to test against the default index directory
<jira_home>/caches/indexes on JIRA 7.x and below, <jira_home>/caches/indexesV1 folder on JIRA 8.x and above. Notice if you have Insight - run the check against
For Confluence make sure to test against
- For Bitbucket the location is $BITBUCKET_HOME/shared/search/data
For Fisheye/Crucible, the test should be run against
If you're running Data Center version of JIRA or Confluence - please run the speed check on each node as the indexes are stored locally on each node
- Pick the average of the results to assess and review our Grading the Results section for further info on how to determine the results of the speed tests.
For example in JIRA, disk access performance is particularly important due to the heavy use of disk I/O to the indexes, because they are read each time a search is made and written each time an update to an issue is committed (see Troubleshoot index problems in Jira server for further information).
This basic test will perform the open, read/write, close, and delete operations 1000 times to a temporary file in the
java.io.tmpdir directory for each measurement, reporting the results as in the assessment section below. This test is the closest we can get to simulating the performance of a Java process when it comes to accessing the file system, allowing it to be an accurate measurement for the application performance when utilizing the disk. Other disk measurement tools may provide different results if they do not measure the specific performance of the Java JDK/JRE.
The Java source can be found within the JAR - if the file is uncompressed it can be viewed.
This tool has been designed primarily to perform basic disk assessment of Java running on the operating system, not the actual speed of the disk itself. It is highly possible a very fast disk can run Java slowly due to OS-level issues. This is not a substitute for a full disk analysis software suite and if the disk appears to be experiencing hardware issues or is slow, please raise this with the System Administrator or hosting provider of the server.
When reviewing the benchmark results:
- Focus primarily on the average result.
- When the open & close speeds are bad and the r/w is good, the likely cause is anti-virus.
- A high minimum and average speed may be indicative of a low speed disk or hardware fault.
- If a disk has a high average and low minimum speed, there may be an environmental factor at play (see below for further information).
- For JIRA open and close are important values because the indexes are accessed very often - the average search will close a file many times.
Grading the Results
Below is a breakdown of approximate average speeds that can be used to grade the results.
|Open||< 40,000||40,000 - 150,000||> 150,000|
|Read/Write||< 40,000||40,000 - 100,000||> 100,000|
|Close||< 20,000||20,000 - 100,000||> 100,000|
|Delete||< 50,000||50,000 - 300,000||> 300,000|
Examples of environmental factors that can cause slow disk access are as follows:
- Anti-Virus software. As in our Common causes for Jira Server crashes and performance issues KB article, Anti-Virus software can be detrimental to the operation of the application as it will limit open and close times.
- A remote disk or shared drive.
- Synchronization to another machine over a slow network.
- A virtualized OS, such as VMWare (see Run Jira server in a virtualized environment for further information).
- A disk defragments job may be running.
- Hardware issues such as disk failure.
- File system encryption.
- Automated compression of files controlled by the OS.
- Specific issues with the Java version and OS. This is a rare occurrence, however, a bug or known issue within the JVM may cause it to perform poorly on a specific OS.
- Other applications or operations that are currently using the disk.
- The disk capacity may be nearing full, which on some OS can slow the performance of the disk (in this particular example, it was on Solaris).
- Lucene has some issues with file deletion in Windows.
Since Lucene uses mmapdirectory and specifically to the Windows OS, it keeps the handles open to these mapped files even after close, not allowing these files to be deleted. Lucene Class MMapDirectory
We have included some sample benchmarks below.
TOTALS ---- ---- ---- ---- ---- stat avg median min max ---- ---- ---- ---- ---- open 35,289 36,000 29,000 472,000 r/w 30,934 31,000 23,000 77,000 close 14,988 5,000 5,000 9,796,000 delete 39,135 40,000 31,000 123,000 ---- ---- ---- ---- ---- All times are in nanoseconds.
This is a premier hosting provider using disk hardware technology equivalent to an SSD.
TOTALS ---- ---- ---- ---- ---- stat avg median min max ---- ---- ---- ---- ---- open 75,830 62,000 44,000 2,869,000 r/w 46,407 40,000 35,000 907,000 close 5,751 5,000 4,000 336,000 delete 118,942 81,000 65,000 22,864,000 ---- ---- ---- ---- ---- All times are in nanoseconds.
TOTALS ---- ---- ---- ---- ---- stat avg median min max ---- ---- ---- ---- ---- open 275,886 267,184 246,412 2,544,101 r/w 92,956 81,538 70,727 7,358,896 close 3,194,958 3,177,771 3,076,064 4,331,872 delete 542,650 530,271 514,226 1,771,416 ---- ---- ---- ---- ---- All times are in nanoseconds.
This benchmark is of very fast disk with a virus scanner enabled.
TOTALS ---- ---- ---- ---- ---- stat avg median min max ---- ---- ---- ---- ---- open 133,704 121,000 83,000 12,787,000 r/w 30,892 26,000 22,000 3,176,000 close 40,652 40,000 10,000 108,000 delete 274,710 254,000 211,000 15,151,000 ---- ---- ---- ---- ---- All times are in nanoseconds.
This is a WD Black 3.0GB/s 7.2k RPM drive running 5 JIRA application instances during low load (whilst we would not ever recommend to do this, it does highlight the impact of multiple applications running on the same server).
*Notice. The above results are given as a guideline for local <index directory> folder only. There is no need to perform these tests on $SHARED_HOME in Data Center configurations as the indexes are stored locally on each node and not on the SHARED drive. Sometimes support may request to run the same tests on the $SHARED_HOME, but that's only if performance issues relevant to attachments or index recovery mechanism (such as when a node starts up after a prolonged period of downtime and if the index recovery is enabled in the respective application) .