Symptoms

We have noticed a variety of symptoms that have been ameliorated by a switch to Java 8 (and, correspondingly, JIRA 6.3 or above). An affected instance may exhibit general slowness during use, but the issue may also manifest as a rapid degradation of performance resulting in pages failing to load. 

Discussion

Java 8

Java 8 exposes (via Unsafe) a fetch-and-add (XADD) instruction that classes such as AtomicInteger can take advantage of to implement their atomicity guarantees. This has a significant advantage over the Java 7 implementation, which instead relies on compare-and-swap (CAS). As the CAS instruction can fail if the value changes in between loading and the CAS instruction, it must be retried in a loop. This leads to problems in highly contended scenarios where branch prediction can start predicting the failure path for the loop, making it a significantly more expensive operation. Fetch-and-add always succeeds, so it does not suffer from this drawback.

http://ashkrit.blogspot.com.au/2014/02/atomicinteger-java-7-vs-java-8.html shows a comparison of AtomicInteger.getAndIncrement() performance under Java 7 and Java 8.

JIRA 6.3 or above

Between JIRA 6.2 and 6.3, most of the caches were migrated from using simple Maps to the Atlassian Cache API. The API is implemented either with Guava or Ehcache depending on if it is clustered. In both cases this increases the usage of the CAS operations mentioned above. These characteristics are also inherited by sequential iterations, such as JIRA 6.4.

There are other places that use the CAS operations too which may be improved by Java 8 but the caches are the key area that changed within JIRA applications recently.

Identifying if there is a performance issue where Java 8 could help

This issue shows up in thread dumps taken during periods of degraded performance as a significant number of threads in the process of doing cache reads. These threads will appear to be RUNNABLE and will likely show up as long running threads in the various analysis tools.

When Guava is providing the cache implementation (i.e. when JIRA applications are not clustered) the java.util.concurrent.atomic.* and java.util.concurrent.ConcurrentLinkedQueue classes will show up in the traces:

"http-bio-8080-exec-717" daemon prio=10 tid=0x000000001521b000 nid=0x299a runnable [0x00002b17b8069000]
   java.lang.Thread.State: RUNNABLE
	at sun.misc.Unsafe.compareAndSwapInt(Native Method)
	at java.util.concurrent.atomic.AtomicInteger.compareAndSet(AtomicInteger.java:135)
	at java.util.concurrent.atomic.AtomicInteger.incrementAndGet(AtomicInteger.java:206)
	at com.google.common.cache.LocalCache$Segment.postReadCleanup(LocalCache.java:3451)
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2238)
	at com.google.common.cache.LocalCache.get(LocalCache.java:3970)
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
	at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4834)
	at com.atlassian.cache.memory.DelegatingCachedReference.get(DelegatingCachedReference.java:56)
"http-bio-8443-exec-1718" daemon prio=10 tid=0x00007fc9f68b4800 nid=0x4741 runnable [0x00007fc9ca923000]
   java.lang.Thread.State: RUNNABLE
	at java.util.concurrent.ConcurrentLinkedQueue$Node.casNext(Unknown Source)
	at java.util.concurrent.ConcurrentLinkedQueue.offer(Unknown Source)
	at java.util.concurrent.ConcurrentLinkedQueue.add(Unknown Source)
	at com.google.common.collect.CustomConcurrentHashMap$Segment.recordRead(CustomConcurrentHashMap.java:2290)
	at com.google.common.collect.ComputingConcurrentHashMap$ComputingSegment.getOrCompute(ComputingConcurrentHashMap.java:87)
	at com.google.common.collect.ComputingConcurrentHashMap.getOrCompute(ComputingConcurrentHashMap.java:69)
	at com.google.common.collect.ComputingConcurrentHashMap$ComputingMapAdapter.get(ComputingConcurrentHashMap.java:393)
	at com.atlassian.cache.memory.DelegatingCachedReference.get(DelegatingCachedReference.java:39)

A similar effect can be observed when Ehcache is used (i.e. in a JIRA application cluster):

"http-bio-18009-exec-1854" daemon prio=10 tid=0x00007fed9aece000 nid=0x7751 runnable [0x00007febcdc2a000]
   java.lang.Thread.State: RUNNABLE
	at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.fullTryAcquireShared(ReentrantReadWriteLock.java:505)
	at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryAcquireShared(ReentrantReadWriteLock.java:491)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1281)
	at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:731)
	at net.sf.ehcache.concurrent.ReadWriteLockSync.lock(ReadWriteLockSync.java:50)
	at net.sf.ehcache.constructs.blocking.BlockingCache.acquiredLockForKey(BlockingCache.java:196)
	at net.sf.ehcache.constructs.blocking.BlockingCache.get(BlockingCache.java:158)
	at net.sf.ehcache.constructs.blocking.SelfPopulatingCache.get(SelfPopulatingCache.java:68)
	at net.sf.ehcache.constructs.blocking.BlockingCache.get(BlockingCache.java:318)
	at com.atlassian.cache.ehcache.DelegatingCachedReference.get(DelegatingCachedReference.java:59)

• The threads won't necessarily all be querying the same cache.

JIRA 6.2.x

As per our Supported Platforms document, Java 8 is not compatible with JIRA 6.2.x.

Resolution

Atlassian recommends upgrading to Java 8 for a variety of performance related symptoms, as described above. We have seen significant performance improvements to large scale instances by upgrading to Java 8, including cases where the performance was a lot worse with JIRA 6.3 and above than earlier versions of JIRA when using Java 6/7.

Due to the JVM bug described in the JIRA Crashes due to Segmentation Fault in Java 8 JVM document, we recommend upgrading Java to 1.8.0_40 or later. For more information about how to switch the Java Virtual Machine used by JIRA applications, please refer to the links below:

• How to Use System JRE Instead of Embedded JRE
• Configuring JIRA Service to Run on a Different JVM