Lucene upgrade

Preparing for Jira 8.0

On this page

Related content

  • No related content found

Still need help?

The Atlassian Community is here for you.

Ask the community

This page contains breaking changes related to the upgrade of the Lucene library from version 3.3 to 7.3. 

Changes

Jira code and jira-lucene-dmz.jar

Several methods have been removed from the Jira code, because they're no longer needed. For some, we've provided alternatives you can use.

com.atlassian.jira.util.BuildUtilsInfo#getLuceneVersion

Changes:

Class removed. 

com.atlassian.jira.issue.index.indexers.impl.VersionCustomFieldIndexer

Method being changed:

public void addDocumentFields(final Document doc, final Issue issue, final Field.Index indexType)

Changes:

  • To index and store a field, use this method:

    addDocumentFieldsSearchable(final Document doc, final Issue issue)

  • To store a field without indexing it, use this method:

    addDocumentFieldsNotSearchable(final Document doc, final Issue issue)

com/atlassian/jira/config/properties/APKeys$JiraIndexConfiguration

Changes:

The getMaxFieldLength has been removed. 

com/atlassian/jira/config/properties/APKeys$JiraIndexConfiguration$MergePolicy

Changes:

The EXPUNGE_DELETES_PCT_ALLOWED has been removed. 

com.atlassian.jira.web.component.IssuePage 
com.atlassian.jira.web.component.IssuePager

Changes:

Renamed → com.atlassian.jira.web.component.ResultPage

Renamed → com.atlassian.jira.web.component.Pager

com.atlassian.jira.issue.search.SearchResults

Changes: 

 - used to hold List<Issue>

  • Now → holds List<T>

  • SearchService uses List<Issue>, SearchProvider uses List<DocumentWithId>

- method getIssues → Renamed to getResults

- new method transform for transforming results of one type to another

- class used to have 3 constructors → Old constructor logic moved to com.atlassian.jira.issue.search.IssueSearchResultsFactory and com.atlassian.jira.issue.search.DocumentSearchResultsFactory

com.atlassian.jira.issue.search.SearchProvider

Changes:

  • searchCount → Renamed → getHitCount

  • search:

    • parameters packed into the com.atlassian.jira.issue.search.SearchQuery object

    • new parameter fieldsToLoad to restrict which document fields are loaded, ok to pass empty set to load only document

  • New return type com.atlassian.jira.issue.search.DocumentWithId that holds loaded Lucene document and its internal Lucene id. This id is only valid in the context of current  IndexReader  - don't cache it. 

  • searchAndSort with Collector removed (usual search with Collector remains)

    Collector collector = new MyCollector(); searchProvider.searchAndSort(query, user, collector, pager); return collector.getResults();

com.atlassian.jira.index.ManagedIndexSearcher

In the org.apache.lucene.search.IndexSearcher it was not clear who manages the searcher's lifecycle.

Changes: 

Now we have 2 new classes:

  • com.atlassian.jira.index.ManagedIndexSearcher that does not need to be closed manually

  • com.atlassian.jira.index.UnmanagedIndexSearcher which has to be closed by its user

AbstractOneDimensionalHitCollector
DocumentHitCollector

Changes: 

changed to public void doSetNextReader(LeafReaderContext context)

FieldableDocumentHitCollector

Changes:

Class renamed to FieldDocumentHitCollector

MappedSortComparator

public FieldComparator newComparator(String fieldname, int numHits, int sortPos, boolean reversed) throws IOException 

Changes:

public FieldComparator newComparator(String fieldname, int numHits, int sortPos, boolean reversed)

OneDimensionalTermHitCollector


public OneDimensionalTermHitCollector (final String fieldId, final FieldVisibilityManager fieldVisibilityManager, final ReaderCache readerCache, final FieldManager fieldManager, final ProjectManager projectManager)

Changes:

public OneDimensionalTermHitCollector(final String fieldId, final FieldVisibilityManager fieldVisibilityManager, final ReaderCache readerCache, final FieldManager fieldManager

ReaderCache

Collection<String>[] get(LeafReader reader, String key, Supplier<Collection<String>[]> supplier);

JiraDocValues getDocValues(LeafReader indexReader, String field); 

Changes:

{@link ReaderCache} is using Lucene {@link org.apache.lucene.index.DocValues}

JiraDocValues getDocValues(IndexReader indexReader, String field);

CustomFieldStattable

StatisticsMapper getStatisticsMapper(CustomField customField);

Changes:

Contract change; starting from Jira 8.0 onwards a "stattable" field must have a doc value of type:

{@link org.apache.lucene.index.SortedDocValues},

{@link org.apache.lucene.index.SortedSetDocValues}, or {@link org.apache.lucene.index.BinaryDocValues}

com.atlassian.jira.jql.util.JqlDateSupport

Changes:

Removed getIndexedValue → Now dates are stored directly as longs in the index.

com.atlassian.jira.jql.util.JqlLocalDateSupport

Changes:

Removed getIndexedValue → Now local dates are stored directly as longs in the index.

com.atlassian.jira.util.LuceneUtils

Various methods for converting dates and local dates to and from the old String-based index formats.

Changes:

This class was removed. Now dates & local dates are added to the index as longs. 

com.atlassian.jira.issue.customfields.NaturallyOrderedCustomFieldsSearcher

Changes:

Expanded SPI by one method: public SortField.Type getSortFieldType();

com.atlassian.jira.index.NumberTools
com.atlassian.jira.util.LuceneNumericUtils

moved → com.atlassian.jira.lucenelegacy.NumberTools

moved → com.atlassian.jira.lucenelegacy.NumericUtils

com.atlassian.jira.issue.search.QueryPermissions

Removed → see  com.atlassian.jira.issue.search.SearchQuery

com.atlassian.jira.issue.search.SearchQuery

Changes:

  • QueryPermissions field removed 
    • Set ApplicationUser user and boolean overrideSecurity directly
    • user is used for permissions and to create com.atlassian.jira.jql.query.QueryCreationContext. It is needed even when overrideSecurity = true.
  • create methods removed:
  1. create(Query,QueryPermissions, org.apache.lucene.search.Query)
  2. create(Query,QueryPermissions)
  3. createNoPermissions(Query)  

    Changes:
    To further configure SearchQuery, use the following:
    • SearchQuery.create(query, user)
    • overrideSecurity(true)
    • luceneQuery(andQuery);
See the changes in a table

Affected class

Code/items being changed

Changes and new code/items

com.atlassian.jira.util.BuildUtilsInfo#getLuceneVersion

com.atlassian.jira.util.BuildUtilsInfo#getLuceneVersion

None, no longer needed.

com.atlassian.jira.issue.index.indexers.impl.VersionCustomFieldIndexer

Method:

public void addDocumentFields(final Document doc, final Issue issue, final Field.Index indexType)

  • To index and store a field, use this method:

    addDocumentFieldsSearchable(final Document doc, final Issue issue)

  • To store a field without indexing it, use this method:

    addDocumentFieldsNotSearchable(final Document doc, final Issue issue)

com/atlassian/jira/config/properties/APKeys$JiraIndexConfiguration

Method:

getMaxFieldLength

None, no longer needed.

com/atlassian/jira/config/properties/APKeys$JiraIndexConfiguration$MergePolicy

Field: EXPUNGE_DELETES_PCT_ALLOWED

None, no longer needed.

com.atlassian.jira.web.component.IssuePage

com.atlassian.jira.web.component.IssuePager

com.atlassian.jira.web.component.IssuePage

com.atlassian.jira.web.component.IssuePager

Renamed → com.atlassian.jira.web.component.ResultPage

Renamed → com.atlassian.jira.web.component.Pager

com.atlassian.jira.issue.search.SearchResults

  • used to hold List<Issue>

  • Now → holds List<T>

  • SearchService uses List<Issue>, SearchProvider uses List<DocumentWithId>

  • method getIssues

Renamed → getResults

New method transform for transforming results of one type to another.

  • class used to have 3 constructors

Old constructor logic moved to com.atlassian.jira.issue.search.IssueSearchResultsFactory and com.atlassian.jira.issue.search.DocumentSearchResultsFactory

com.atlassian.jira.issue.search.SearchProvider

searchCount  

Renamed → getHitCount

search

  • parameters packed into the com.atlassian.jira.issue.search.SearchQuery object

  • new parameter fieldsToLoad to restrict which document fields are loaded, ok to pass empty set to load only document id


New return type com.atlassian.jira.issue.search.DocumentWithId  that holds loaded Lucene document and its internal Lucene id. This id is only valid in the context of current  IndexReader  - don't cache it. 

searchAndSort with Collector removed (usual search with Collector remains)

Collector collector = new MyCollector(); searchProvider.searchAndSort(query, user, collector, pager); return collector.getResults();
MyLoader loader = new MyLoader(indexReader) { public void myCollect( int docId) { // your business logic with indexReader and docId } }; searchProvider.search(SearchQuery.create(query, user), pager, Collections.emptySet()) .getResults() .stream() .map(DocumentWithId::getDocId) .forEach(loader::myCollect); return loader.getResults();

com.atlassian.jira.index.ManagedIndexSearcher

In the org.apache.lucene.search.IndexSearcher, it was not clear who manages the searcher's lifecycle

New classes:

  • com.atlassian.jira.index.ManagedIndexSearcher that does not need to be closed manually

  • com.atlassian.jira.index.UnmanagedIndexSearcher which has to be closed by its user

AbstractOneDimensionalHitCollector DocumentHitCollector

AbstractOneDimensionalHitCollector DocumentHitCollector

public void doSetNextReader(LeafReaderContext context)
FieldableDocumentHitCollector class renamedFieldDocumentHitCollector
MappedSortComparator

public FieldComparator newComparator(String fieldname, int numHits, int sortPos, boolean reversed) 

throws IOException

public FieldComparator newComparator(String fieldname, int numHits, int sortPos, boolean reversed)
OneDimensionalTermHitCollector public OneDimensionalTermHitCollector (final String fieldId, final FieldVisibilityManager fieldVisibilityManager, final ReaderCache readerCache, final FieldManager fieldManager, final ProjectManager projectManager) public OneDimensionalTermHitCollector(final String fieldId, final FieldVisibilityManager fieldVisibilityManager, final ReaderCache readerCache, final FieldManager fieldManager)

ReaderCache

Collection<String>[] get(LeafReader reader, String key, Supplier<Collection<String>[]> supplier);

JiraDocValues getDocValues(LeafReader indexReader, String field);

Starting Jira 8.0 {@link ReaderCache} is using Lucene

{@link org.apache.lucene.index.DocValuJiraDocValues getDocValues(IndexReader indexReader, String field);


CustomFieldStattable

StatisticsMapper getStatisticsMapper(CustomField customField);

Contract change; starting Jira 8.0 onwards a "stattable" field must have a doc value of type

{@link org.apache.lucene.index.SortedDocValues},

{@link org.apache.lucene.index.SortedSetDocValues}, or {@link org.apache.lucene.index.BinaryDocValues}
com.atlassian.jira.jql.util.JqlDateSupport

getIndexedValue

Removed. Now dates are stored directly as longs in the index.
com.atlassian.jira.jql.util.JqlLocalDateSupport

getIndexedValue

Removed. Now local dates are stored directly as longs in the index.
com.atlassian.jira.util.LuceneUtils Various methods for converting dates and local dates to and from the old String-based index formats.Entire class removed. Now dates & local dates are added to the index as longs.
com.atlassian.jira.issue.customfields.NaturallyOrderedCustomFieldsSearcher
N/A

Expanded SPI by one method: 

public SortField.Type getSortFieldType();

com.atlassian.jira.index.NumberTools

com.atlassian.jira.util.LuceneNumericUtils

classes moved

moved → com.atlassian.jira.lucenelegacy.NumberTools

moved → com.atlassian.jira.lucenelegacy.NumericUtils

com.atlassian.jira.issue.search.QueryPermissions


Removed → see  com.atlassian.jira.issue.search.SearchQuery
com.atlassian.jira.issue.search.SearchQuery
  • QueryPermissions field removed
  • create methods removed:
  1. create(Query,QueryPermissions, org.apache.lucene.search.Query)
  2. create(Query,QueryPermissions)
  3. createNoPermissions(Query)
  • Set  ApplicationUser user  and  boolean overrideSecurity  directly
  • user  is used for permissions and to create  com.atlassian.jira.jql.query.QueryCreationContext. It is needed even when  overrideSecurity = true .

To further configure  SearchQuery, use the following :
SearchQuery.create(query, user)
    .overrideSecurity(true)
    .luceneQuery(andQuery);

Lucene API exposed by Jira

Creating BooleanQuery

Using constructors to create boolean queries is no longer supported. Instead, you'll need to use the query builder.

See code examples...



Jira 7.x 
BooleanQuery boolQuery = new BooleanQuery();
boolQuery.add(TermQueryFactory.nonEmptyQuery(fieldName), BooleanClause.Occur.MUST);
boolQuery.add(TermQueryFactory.visibilityQuery(fieldName), BooleanClause.Occur.MUST);
return boolQuery;
Jira 8.0 
BooleanQuery.Builder boolQueryBuilder = new BooleanQuery.Builder();
boolQueryBuilder.add(TermQueryFactory.nonEmptyQuery(fieldName), BooleanClause.Occur.MUST);
boolQueryBuilder.add(TermQueryFactory.visibilityQuery(fieldName), BooleanClause.Occur.MUST);
return boolQueryBuilder.build();

Creating Documents

To specify fields in the document, you'll now need to use the subclasses of the Field class (i.e. StringField, TextField, StoredField).

See code examples...
Jira 7.x 
Document document = new Document();       
document.add(new Field(ACTIVE, Boolean.toString(user.isActive()), YES, NOT_ANALYZED_NO_NORMS));
document.add(new Field(EMAIL, user.getEmailAddress(), YES, ANALYZED_NO_NORMS));
document.add(new Field(EXACT_DISPLAY_NAME, IdentifierUtils.toLowerCase(user.getDisplayName()), NO, NOT_ANALYZED_NO_NORMS));
return document;
Jira 8.0 
Document document = new Document();      
document.add(new StringField(ACTIVE, Boolean.toString(user.isActive()), YES));document.add(new TextField(DISPLAY_NAME, user.getDisplayName(), YES));
document.add(new TextField(EMAIL, user.getEmailAddress(), YES));
document.add(new SortedDocValuesField(EXACT_USER_NAME, new BytesRef(IdentifierUtils.toLowerCase(user.getName()))));
return document;


Creating Filters

The org.apache.lucene.search.Filter class has changed to the regular org.apache.lucene.search.Query with BooleanClause.Occur.FILTER occurrence.

See code examples...
Jira 7.x 
Query query = new TermQuery(new Term(fieldName, fieldValue));
Query filterQuery = new TermQuery(new Term(filterFieldName, filterFieldValue));
Filter filter = new QueryWrapperFilter(filterQuery);
issueSearcher.search(query, filter, maxHits);
Jira 8.0 
Query query = new TermQuery(new Term(fieldName, fieldValue));
Query filterQuery = new TermQuery(new Term(filterFieldName, filterFieldValue));
Query queryWithFilter = new BooleanQuery.Builder()
                    .add(query, BooleanClause.Occur.MUST)
                    .add(filterQuery, BooleanClause.Occur.FILTER)
                    .build();
issueSearcher.search(queryWithFilter, maxHits);

Configuration

Several configuration items have been removed, and substituted with the new ones.

Obsolete configurationNew configuration

mergePolicy.setExpungeDeletesPctAllowed

mergePolicy.setForceMergeDeletesPctAllowed
mergePolicy.setUseCompoundFile indexWriterConfig.setUseCompoundFile
-mergePolicy.setReclaimDeletesWeight
-mergePolicy.setMaxCFSSegmentSizeMB

JQL changes


Fuzzy Search

Old Lucene
New Lucene

text ~ "foo~0.5"

text ~ "foo~2"

Where 0 is an exact match and 2 allows 2 changes.

Index is seen as a swap file

Doc values are stored in virtual address space, so it might look like Java process takes up lots of memory (3x index file), but this is not real RAM, just memory-mapped files, which are swapped in and out as needed.

Re-indexing

When moving from any version of Jira older than EAP 04 to a version higher than EAP 04 or the actual EAP 04, a full foreground reindex must be performed. In EAP 04, we changed how we're storing Dates in the Lucene index. A background re-index won't do the trick, as it leaves Lucene in a bad state (with a mix of types under the same field name).

Last modified on Aug 13, 2019

Was this helpful?

Yes
No
Provide feedback about this article

Related content

  • No related content found
Powered by Confluence and Scroll Viewport.