Context

Jira 8.0 has a new version of the Lucene library. In this version, we have periodic Lucene data commits done every 30 sec by default. On top of that, Lucene can also trigger flushes according to memory usage: maxrambuffermb – RAM usage of documents added to the IndexWriter. We have also noticed that Lucene flushes can consume a lot of memory (exceeding maxrambuffermb), especially when flushing a low number of documents with a large number of Lucene Doc Values of type SortedDocValues and SortedSetDocValues (for example in multiple custom fields).

Performance tests

Example test results for different  maxrambuffermb sizes on Jira 8.0 (1M issues, 5000 custom fields, 100 projects, 6CF per issue, 16GB heap):

Changes

Jira 8.0 comes with a larger maxrambuffermb size (Jira Config property: jira.index.batch.maxrambuffermb, default 1024) to handle issues with a large number of CF by default. As per Jira Software 8.0.x upgrade notes:

There are also new logs and warnings in the application logs (atlassian-jira.log) to help diagnose the problem:

• New Lucene stats about flushes (logged every 5 minuntes for every indexer):

  INFO [lucene-stats] flush stats: snapshotCount=59, totalCount=1649, periodSec=308, flushIntervalMillis=5229, 
      indexDirectory=/opt/atlassian/jira/jira_home/caches/indexesV1/issues

  
  

  • snapshotCount – the number of flushes in this period (~5 minutes by default).
  • totalCount – the number of flushes since this index writer was created.
  • periodSec – this period in seconds.
  • flushIntervalMillis – the average flush interval in milliseconds.
  • indexDirectory – the index directory path

The most interesting part of this log is the flushIntervalMillis value. With very small flushIntervalMillis (less than 1 second), we can see young GC start having issues with cleaning the garbage created by the flush. As a result, GC pauses start to affect the indexing time and performance.

• There is a new warning logged when flushIntervalMillis is below 1 second (by default). If this warning appears, you should review and, probably, increase the maxrambuffermb value to avoid any potential performance problems:

  WARN - Detected frequent flushes (every 600 millis) of lucene index which is below
      warning limit: jira.index.warn.flush.min.interval.millis=1000 millis. This may 
      affect indexing performance. Please visit https://confluence.atlassian.com/x/w0VwOQ 
      for more information.

False alarm

The log statistics and warnings only apply to "real indexes" stored on the disk. Jira also uses in-memory indexes (RAMDirectory) for temporary indexing. Statistics and warnings for those indexes can be ignored: JRASERVER-69389 - Getting issue details... STATUS