Test disk access speed for a Java application

Problem

The Atlassian Server application appears to be experiencing performance issues and is running slowly. These applications could include Java based software such as JIRA, Confluence, Bamboo, and Fisheye/Crucible.

For example in JIRA, this will be particularly noticeable in the following areas:

• Reindexing
• Searching
• The Issue Navigator
• Gadgets
• Opening Issues
• Adding/Accessing Attachments

Diagnosis

Disk access speed is critical for the Atlassian applications performance.

If the application is running slowly, disk speed can be a potential root cause. It's possible to isolate that cause using this Support Tools Utility to benchmark the disk access speed.

Download the Support Tools Utility to any writable folder on the actual JIRA server (for example ~/$username$/temp on Linux or c:\temp on Windows).

1. For Windows: Open a terminal window (go to Start >> Run >> type in 'cmd').
2. For Linux/Unix: Open a command line.
3. When running the tool, write access to the temporary directory is required - it will create and delete a large number of files in order to calculate the disk speed.

4. CD to the directory where the tools was downloaded and run it with the following:

   java -Djava.io.tmpdir=<index directory> -jar support-tools.jar

    Depending on your application please test the following directories using the <index directory> path below:

   • For JIRA make sure to test against the default index directory <jira_home>/caches/indexes on JIRA 7.x and below, <jira_home>/caches/indexesV1 folder on JIRA 8.x and above. Notice if you have Insight - run the check against <jira_home>/caches/insight_indexes too.

   • For Confluence make sure to test against <confluence-home>/index.

   • For Fisheye/Crucible, the test should be run against <FISHEYE_INST>/cache and <FISHEYE_INST>/var/cache.

   If you're running Data Center version of JIRA or Confluence - please run the speed check on each node as the indexes are stored locally on each node

5. Pick the average of the results to assess and review our Grading the Results section for further info on how to determine the results of the speed tests. 
   For example in JIRA, disk access performance is particularly important due to the heavy use of disk I/O to the indexes, because they are read each time a search is made and written each time an update to an issue is committed (see Troubleshoot index problems in Jira server for further information).

Operation

This basic test will perform the open, read/write, close, and delete operations 1000 times to a temporary file in the java.io.tmpdir directory for each measurement, reporting the results as in the assessment section below. This test is the closest we can get to simulating the performance of a Java process when it comes to accessing the file system, allowing it to be an accurate measurement for the application performance when utilizing the disk. Other disk measurement tools may provide different results if they do not measure the specific performance of the Java JDK/JRE.

The Java source can be found within the JAR - if the file is uncompressed it can be viewed.

Resolution

When reviewing the benchmark results:

• Focus primarily on the average result.
• When the open & close speeds are bad and the r/w is good, the likely cause is anti-virus.
• A high minimum and average speed may be indicative of a low speed disk or hardware fault.
• If a disk has a high average and low minimum speed, there may be an environmental factor at play (see below for further information).
• For JIRA open and close are important values because the indexes are accessed very often - the average search will close a file many times.
• Make sure that the disk is not in use by any other processes running on your system.

Grading the Results 

Below is a breakdown of approximate average speeds that can be used to grade the results.

• Anti-Virus software. As in our Common causes for Jira Server crashes and performance issues KB article, Anti-Virus software can be detrimental to the operation of the application as it will limit open and close times.
• A remote disk or shared drive.
• Synchronization to another machine over a slow network.
• A virtualized OS, such as VMWare (see Run Jira server in a virtualized environment for further information).
• A disk defragments job may be running.
• Hardware issues such as disk failure.
• File system encryption.
• Automated compression of files controlled by the OS.
• Specific issues with the Java version and OS. This is a rare occurrence, however, a bug or known issue within the JVM may cause it to perform poorly on a specific OS.
• Other applications or operations that are currently using the disk.
• The disk capacity may be nearing full, which on some OS can slow the performance of the disk (in this particular example, it was on Solaris).
• Lucene has some issues with file deletion in Windows.
  Since Lucene uses mmapdirectory and specifically to the Windows OS, it keeps the handles open to these mapped files even after close, not allowing these files to be deleted. Lucene Class MMapDirectory

Sample Benchmarks*

We have included some sample benchmarks below.

TOTALS
----    ----    ----    ----    ----
stat    avg     median  min     max
----    ----    ----    ----    ----
open    35,289  36,000  29,000  472,000
r/w     30,934  31,000  23,000  77,000
close   14,988  5,000   5,000   9,796,000
delete  39,135  40,000  31,000  123,000
----    ----    ----    ----    ----
All times are in nanoseconds.

This is a premier hosting provider using disk hardware technology equivalent to an SSD.

OK

TOTALS
----    ----    ----    ----    ----
stat    avg     median  min     max
----    ----    ----    ----    ----
open    75,830  62,000  44,000  2,869,000
r/w     46,407  40,000  35,000  907,000
close   5,751   5,000   4,000   336,000
delete  118,942 81,000  65,000  22,864,000
----    ----    ----    ----    ----
All times are in nanoseconds.

Bad

TOTALS
----    ----    	----    	----    	----
stat    avg     	median  	min     	max
----    ----    	----    	----    	----
open    275,886 	267,184 	246,412 	2,544,101
r/w     92,956  	81,538  	70,727  	7,358,896
close   3,194,958   3,177,771   3,076,064   4,331,872
delete  542,650 	530,271 	514,226 	1,771,416
----    ----    	----    	----    	----
All times are in nanoseconds.

This benchmark is of very fast disk with a virus scanner enabled.

Average

TOTALS
----	----	----	----	----
stat	avg		median	min		max
----	----	----	----	----
open	133,704	121,000	83,000	12,787,000
r/w		30,892	26,000	22,000	3,176,000
close	40,652	40,000	10,000	108,000
delete	274,710	254,000	211,000	15,151,000
----	----	----	----	----
All times are in nanoseconds.

This is a WD Black 3.0GB/s 7.2k RPM drive running 5 JIRA application instances during low load (whilst we would not ever recommend to do this, it does highlight the impact of multiple applications running on the same server).

TOTALS
---- ---- ---- ---- ----
stat avg median min max
---- ---- ---- ---- ----
open 4,790,174 4,638,824 3,943,820 13,139,894
r/w 14,947 13,312 3,519 83,294
close 4,398,125 4,297,768 3,708,446 16,138,393
delete 4,323,586 4,178,885 3,581,537 33,963,856
---- ---- ---- ---- ----
All times are in nanoseconds.