Optimizing custom fields

A high number of custom fields can negatively impact performance, index size, and indexing time in Jira. A big chunk of this impact is caused by the time it takes to index certain custom fields. To minimize the impact we introduced improvements on how custom fields are indexed.

Reducing the number of fields indexers executions when indexing an issue

Whenever an issue was being indexed, Jira would retrieve all registered custom field indexers and calls them regardless of whether the custom field was applicable to the issue or not (visible and in the context). This generated additional overhead and affected indexing time.

To reduce this overhead we’ve introduced two improvements:

1. Only the indexers for the custom fields that have values assigned to the issue will be called
   Note: Currently, this optimization works only for the Jira built-in fields.

2. Only the indexers for the custom fields that are applicable to the issue will be called. What is meant by “applicable” is that the custom field is in the context of a particular issue and is visible. The custom field is considered visible when it is not hidden in the Field configuration menu. Screens are not used to calculate field visibility - the field may be not assigned to any screen, but still, be considered visible.

As a result, you should expect a reduction in the re-indexing time.

Visibility in the logs

You can view the results of the optimizations, such as which indexers are filtered out by monitoring entries, in the atlassian-jira.log file. You can use the log entries to troubleshoot potential custom field search issues or the issue data not being timely updated. The log entries show which indexers are involved in certain processes, which allows for in-depth monitoring.

To see the entries, change the logging level to DEBUG (or TRACE) for com.atlassian.jira.issue.index.DefaultIssueDocumentFactory directly in log4j.properties or by going to Troubleshooting and Support > Logging & Profiling. For more details on changing the logging level and the log file location, see Logging and profiling.

Log entry examples

Refer to the examples below to see which entries can display in your log file:

• An entry listing the custom fields that are applicable for the issue (not hidden and in the context):

DEBUG: Visible custom fields: [customfield_10104, customfield_10105, issuetype, customfield_10000, priority, customfield_10100]

• An entry with custom fields using the new NonnullCustomFieldProvider API (currently only applicable to Jira system fields) and with existing value for the issue:

DEBUG: Persisted custom fields: []

• An entry showing the number of matched indexers compared to the total number of indexers for the issue:

DEBUG: Number of visible indexers 3 out of 8 custom field indexes

• A trace level entry with details on called indexers versus all available indexers for an issue:

TRACE: Visible custom field indexers: [CustomFieldIndexerAdapter{delegate=FieldIndexerWithStats{delegate=com.atlassian.greenhopper.customfield.epiclink.EpicLinkCustomFieldIndexer@49d88503, isKnown=false, stats=com.atlassian.jira.issue.index.indexers.FieldIndexerWithStats$MutableFieldIndexerStats@4c3e7b67}, field=Epic Link}] TRACE: All custom field indexers: [CustomFieldIndexerAdapter{delegate=FieldIndexerWithStats{delegate=com.atlassian.jira.issue.index.indexers.impl.NumberCustomFieldIndexer@197d6f82, isKnown=false, stats=com.atlassian.jira.issue.index.indexers.FieldIndexerWithStats$MutableFieldIndexerStats@7e26c275}, field=Story Points}]

How do I disable this feature?

In case any problems arise, you may turn these features off by using system properties (requires restart):

• jira.cfv.driven.indexing.disabled=true to disable filtering indexers by value

• jira.local.context.indexing.disabled=true to disable filtering indexers by context

Note: a full re-index is not required unless you would also like to disable sorting optimization (see below).

Removing redundant data from Lucene index

JQL supports sorting by the custom field values with the ‘ORDER BY’ clause:

project = MyProject ORDER BY MyCustomField

Before Jira 8, Lucene required adding data to the index regardless if a custom field had a value assigned or not to handle sorting properly. However, with new Lucene, we could get rid of this and store data only when a field has a value assigned.

This reduces the index size on your instance (depending on the number of custom fields without any values assigned).

How do I disable this feature?

You can use the system property jira.skip.indexing.null.disabled=true to disable this feature (requires restart). You will also need to trigger full re-index to move the index back to the original state. Note that it will also turn the filtering field indexers by values off.

Technical details

If you are interested to find out more about the improvements or know how your app can be affected by these changes, see our blog post on custom fields' optimizations.