Summary

Jira page-load-times are taking more time than usual.

This mainly happens after a Jira upgrade or after plugin upgrades.

The pages show loading transactions that take seconds to finish, this is specially noticed with loading of batch.js and batch.css files.

The issue gets worse if we use browser incognito mode or if we disable browser caching.

The load times are showing as below if we generate a har file from the browser:

Environment

All versions of Jira Core 7.x, 8.x and 9.x

Any of the below environment conditions is relevant to this problem:

• Jira was upgraded recently or Jira plugins were updated recently.
• Jira runs under load, many concurrent users 100+.
• Jira has many plugins injecting code in batch.js/batch.css
• Jira runs on default Java CodeCache value - 240MB
• Jira runs behind a reverse proxy on a high latency network.

Diagnosis

Diagnostic Steps

• Need to generate HAR files for the slow pages in Jira, we need to check the loading times of batch.js and batch.css
• Verify the loading times by checking Jira Tomcat access logs.
• Need to collect thread dumps and thread CPU information on the impacted instance.
• Need to check the tomcat console for any related CodeCache messages.
• Need to verify if the customer browser has caching enabled or not.

Causes

There could be multiple causes that contribute to this problem:

Cause #1: Browser caching disabled/Slow network.

• Jira generates Javascript and Styles on the fly based on the user language and on project permissions and the plugins being used.
• The files can get to be large in size, sometimes reaching more than 7MB compressed, Jira has gzip compression enabled by default to help make the network transfer fast.
• Jira also caches the files on the server side to help speed up processing, but the cache could expire, if the user properties change quickly, eg: user changes his language, changes his project permissions.
• Also plugin update / installation flushes the Jira server side cache, which could make things slower.
• Jira will rely on the browse to cache those resources also on client side as it will take time to transfer those files on the network, especially if the network has high latency.
• Caching the files on the browser will have those files loaded from the client disk, and will save the time of the network transfer.
• The browser should get a 304 not modified (loaded from disk) on checking if the file is still the same, this means that Jira still has the same version of the file as the file cached by browser, this will save the network time.
• If the file is changed, the browser should stream the file from Jira and you should see a full transfer happening with a normal http 200 signaling success.
• If the file needs to be transferred every time a request is made on a slow network (no browser caching), it will cause significant delay, Jira will use multiple batch.js and batch.css files on the same page depending on the number of plugins and the page structure missing browser cache on a slow network will lead to very long network delays.

Cause#2: Jira is running out of CodeCache

• Jira server will show very high CPU usages, and the whole Jira instance will be very slow.

• Looking at Jira thread dumps, you will see that the Java Compiler threads are runnable all the time:

  "C1 CompilerThread10" #39 daemon prio=9 os_prio=0 tid=0x00007f66a9c59000 nid=0x58d8 runnable [0x0000000000000000]
     java.lang.Thread.State: RUNNABLE
  
  "C2 CompilerThread9" #38 daemon prio=9 os_prio=0 tid=0x00007f66a9c56800 nid=0x58d7 runnable [0x0000000000000000]
     java.lang.Thread.State: RUNNABLE

• The Compiler threads will be using most of the available CPU, starving other threads for longer times, specifically, we will also see many thread trying to render statics as long running threads, having below stack traces:

  "http-nio-10.28.113.26-8080-exec-1012" #592901 daemon prio=5 os_prio=0 tid=0x00007f64bc06d800 nid=0x5f7b runnable [0x00007f5f9f21d000]
     java.lang.Thread.State: RUNNABLE
          at org.mozilla.javascript.gen._js_less_less_rhino_js_1._c_anonymous_72(/js/less/less-rhino.js:1527)
  
  ...
          at org.mozilla.javascript.gen._js_less_less_patches_js_2.call(/js/less/less-patches.js)
          at com.atlassian.lesscss.RhinoLessCompiler.compile(RhinoLessCompiler.java:69)
          at com.atlassian.plugins.less.LessResource.transform(LessResource.java:42)
          at com.atlassian.plugin.webresource.transformer.CharSequenceDownloadableResource$1.apply(CharSequenceDownloadableResource.java:44)
  
  

• One other contributing factor will be the loss of optimized code for the Mozilla Rhino compiler library from CodeCache, this code renders the statics for each request - if they are not cached on the server. If the optimized Rhino code is not in the CodeCache, it will run extremely slowly and would cause further slowness. This becomes a big bottleneck especially for very busy Jira instances as the Mozilla Rhino compiler is used very frequently, which meets one the JIT compilation criteria and is supposed to be optimized by the JVM.

Cause#3: Jira's JIT Compiler got disabled due to full CodeCache, this is related to Cause#2.

• In this case CodeCache would be overwhelmed and got filled up, the Compiler threads are not able to store any more code in the CodeCache and the compiler will get disabled.

• We will see similar logs in the tomcat console logs:

  Java HotSpot(TM) 64-Bit Server VM warning: CodeCache is full. Compiler has been disabled.
  Java HotSpot(TM) 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize=
  CodeCache: size=245760Kb used=244153Kb max_used=244189Kb free=1606Kb
   bounds [0x00007f2709000000, 0x00007f2718000000, 0x00007f2718000000]
   total_blobs=45381 nmethods=43794 adapters=1490
   compilation: disabled (not enough contiguous free space left)

• Looking into Jira thread dumps will not show any Compiler threads since the compiler was disabled.
• Jira will not run any optimized code, it will use standard JVM interpreted byte code for all operations, this will cause huge performance hit.
• We might not notice a lot of CPU usage in this case, since we have no compiler threads running.
• Once more we will also see long running Mozilla Rhino compiler threads in the thread dumps.

Cause#4: Incompatible custom HTML/CSS/Javascript is being injected into the page.

• The issue persists after disabling installed apps by entering Safe Mode.
• Disabling the HTML markup support "Enable HTML in ..." options within ⚙️ → System → General configuration resolves the issue.
• In this case, there is custom HTML/CSS/Javascript being injected into the browser from either the customfield, fieldconfiguration, fieldconfigscheme, fieldlayoutitem, or the Announcement Banner that is no longer compatible with the application.

• You can verify this by reviewing the entries in these tables: The following query checks the descriptions of custom fields for scripts:

  select * from customfield
  where lower(cast(description as varchar)) like '%<javascript%' 
    or lower(cast(description as varchar)) like '%<script%' 
    or lower(cast(description as varchar)) like '%html%' 
    or lower(cast(description as varchar)) like '%css%';

  Custom fields can also have alternate descriptions specified by field configurations:

  select * from fieldlayoutitem 
  where lower(cast(description as varchar)) like '%<javascript%' 
    or lower(cast(description as varchar)) like '%<script%' 
    or lower(cast(description as varchar)) like '%html%' 
    or lower(cast(description as varchar)) like '%css%';

  The following queries check for any scripts in custom field contexts:

  select * from fieldconfigscheme 
  where lower(cast(description as varchar)) like '%<javascript%' 
    or lower(cast(description as varchar)) like '%<script%' 
    or lower(cast(description as varchar)) like '%html%' 
    or lower(cast(description as varchar)) like '%css%';
  
  select * from fieldconfiguration 
  where lower(cast(description as varchar)) like '%<javascript%' 
    or lower(cast(description as varchar)) like '%<script%' 
    or lower(cast(description as varchar)) like '%html%' 
    or lower(cast(description as varchar)) like '%css%';

  It can also be worth checking the announcement banner for any scripts, as it's a known potential cause of interference when it contains custom scripts or HTML code:

  select * from propertytext 
  where id in (select id from propertyentry 
    where property_key='jira.alertheader');

  
  

  Here are the same queries above all in one block for easier copying and pasting:

  select * from customfield
  where lower(cast(description as varchar)) like '%<javascript%' 
    or lower(cast(description as varchar)) like '%<script%' 
    or lower(cast(description as varchar)) like '%html%' 
    or lower(cast(description as varchar)) like '%css%';
  
  select * from fieldlayoutitem 
  where lower(cast(description as varchar)) like '%<javascript%' 
    or lower(cast(description as varchar)) like '%<script%' 
    or lower(cast(description as varchar)) like '%html%' 
    or lower(cast(description as varchar)) like '%css%';
  
  select * from fieldconfigscheme 
  where lower(cast(description as varchar)) like '%<javascript%' 
    or lower(cast(description as varchar)) like '%<script%' 
    or lower(cast(description as varchar)) like '%html%' 
    or lower(cast(description as varchar)) like '%css%';
  
  select * from fieldconfiguration 
  where lower(cast(description as varchar)) like '%<javascript%' 
    or lower(cast(description as varchar)) like '%<script%' 
    or lower(cast(description as varchar)) like '%html%' 
    or lower(cast(description as varchar)) like '%css%';
  
  select * from propertytext 
  where id in (select id from propertyentry 
    where property_key='jira.alertheader');
  
  

Resolution

• Cause#1:
  • We need to make sure that browser caching is enabled all the time.
  • We need to make sure that Gzip compression is enabled ether on JIRA or better on the reserve proxy, this should save JIRA from losing CPU cycles on compression.
  • We might want to introduce an extra caching layer by using a caching office proxy: eg: Squid, this should help keep the large statics cached close to the client if the connection to Jira has high latency.
  • In case Jira runs behind a reverse proxy, we might want to enable mod_expires, this could help with setting content expiration times, please check this KB for more details: Jira is Slow Due to High Latency Connections While Using Reverse Proxy.
  • Make sure that the client machine has enough resources from CPU and memory perspective to be able to work with compressed content.
  • Need to avoid doing any plugin updates or installations during Jira usage peak hours.

• Cause#2 & Cause#3:

  • We need to increase CodeCache by adding the below start up option to JIRA.

    -XX:ReservedCodeCacheSize=512m

    Once added, Jira needs to be restarted.

  • Please make sure you check your instance needs, for the current versions of Jira (Jira 7.x), using 512MB should be enough, though you might want to increase if you still see problems with compiler threads CPU usage or if you continue to see CodeCache being disabled in the Tomcat console output.
  • Also please take note of this bug:  JRASERVER-68149 - Getting issue details... STATUS  the bug would have direct impact causing this issue, so far, increasing code cache helps, but eventually when the bug is fixed, the problem should be far less frequent.

• Cause#4
  • We need to remove the offending code from either the UI at the relevant admin page (or directly from the database).

The following query checks the descriptions of custom fields for scripts:

select * from customfield
where lower(cast(description as varchar)) like '%<javascript%' 
  or lower(cast(description as varchar)) like '%<script%' 
  or lower(cast(description as varchar)) like '%html%' 
  or lower(cast(description as varchar)) like '%css%';

Custom fields can also have alternate descriptions specified by field configurations:

select * from fieldlayoutitem 
where lower(cast(description as varchar)) like '%<javascript%' 
  or lower(cast(description as varchar)) like '%<script%' 
  or lower(cast(description as varchar)) like '%html%' 
  or lower(cast(description as varchar)) like '%css%';

The following queries check for any scripts in custom field contexts:

select * from fieldconfigscheme 
where lower(cast(description as varchar)) like '%<javascript%' 
  or lower(cast(description as varchar)) like '%<script%' 
  or lower(cast(description as varchar)) like '%html%' 
  or lower(cast(description as varchar)) like '%css%';

select * from fieldconfiguration 
where lower(cast(description as varchar)) like '%<javascript%' 
  or lower(cast(description as varchar)) like '%<script%' 
  or lower(cast(description as varchar)) like '%html%' 
  or lower(cast(description as varchar)) like '%css%';

It can also be worth checking the announcement banner for any scripts, as it's a known potential cause of interference when it contains custom scripts or HTML code:

select * from propertytext 
where id in (select id from propertyentry 
  where property_key='jira.alertheader');

Here are the same queries above all in one block for easier copying and pasting:

select * from customfield
where lower(cast(description as varchar)) like '%<javascript%' 
  or lower(cast(description as varchar)) like '%<script%' 
  or lower(cast(description as varchar)) like '%html%' 
  or lower(cast(description as varchar)) like '%css%';

select * from fieldlayoutitem 
where lower(cast(description as varchar)) like '%<javascript%' 
  or lower(cast(description as varchar)) like '%<script%' 
  or lower(cast(description as varchar)) like '%html%' 
  or lower(cast(description as varchar)) like '%css%';

select * from fieldconfigscheme 
where lower(cast(description as varchar)) like '%<javascript%' 
  or lower(cast(description as varchar)) like '%<script%' 
  or lower(cast(description as varchar)) like '%html%' 
  or lower(cast(description as varchar)) like '%css%';

select * from fieldconfiguration 
where lower(cast(description as varchar)) like '%<javascript%' 
  or lower(cast(description as varchar)) like '%<script%' 
  or lower(cast(description as varchar)) like '%html%' 
  or lower(cast(description as varchar)) like '%css%';

select * from propertytext 
where id in (select id from propertyentry 
  where property_key='jira.alertheader');