Indexing fails due to immense field
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Problem
Both background and foreground reindex requests fail due to a field that has more than 32766 bytes in the field.
The following appears in the atlassian-jira.log
1
2
2019-07-02 16:32:24,247 IssueIndexer:thread-9 WARN 10.x.x.x /secure/admin/jira/IndexReIndex!reindex.jspa [c.a.jira.index.AccumulatingResultBuilder] Document contains at least one immense term in field="<fieldname>" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[list of entries in the field]...', original message: bytes can be at most 32766 in length; got 32767
2019-07-02 16:32:24,248 JiraTaskExectionThread-2 WARN 10.x.x.x /secure/admin/jira/IndexReIndex!reindex.jspa [c.a.jira.index.AccumulatingResultBuilder] Indexing failed for Issue - '<#####>'Diagnosis
Environment
This affects Jira 8.0 and above.
In Jira 8.0, Lucene was upgraded from 3.3 to 7.3. Instead of silently failing, Lucene now throws an exception.
See Lucene upgrade for more information.
Cause
Lucene limits the length of indexed fields to 32766 bytes. Fields that are continually incremented (such as SLA, individual permissions, or custom fields) are more likely to exceed the length limit. The application and the database allow these to increment and do not limit the field length, but Lucene cannot parse immense fields that exceed the byte length, so it fails to index them.
In older versions of Lucene, it would silently fail, but after LUCENE-5472, this will throw an exception.
Solution
Workaround
There is no workaround at this time.
There is bug report: JSWSERVER-20133 - Fix the Lucene immense field indexing failure
Resolution
In order to allow indexing to run, you must either truncate, disable, or delete the problematic field.
If the field is managed by a plugin, you can disable that plugin.
Limit the input points. For example, if the error indicates a permission field, limit the individuals or groups that are included in the permission field.
If it is a custom field that is no longer needed, remove it from active screens, then trim it in the database.
Query the database
Always back up your data before modifying the database. If possible, try your modifications on a test server.
If you would like to query your database on what values are hitting this byte size threshold, you can query the database. Typically, this often happens in the customfield table.
Below is a query tested on PostgreSQL to check the byte size for a given custom field ID:
1
2
3
4
5
6
7
8
9
SELECT pg_column_size(cv.stringvalue) as stringbytesize,pg_column_size(cv.numbervalue),pg_column_size(cv.textvalue) as textbytesize,p.pkey AS "Project", ji.id AS "Issue Key"
FROM customfield c
JOIN customfieldvalue cv ON c.id = cv.customfield
JOIN jiraissue ji ON cv.issue = ji.id
JOIN project p ON ji.project = p.id
WHERE cv.customfield = examplefield
ORDER BY length(stringvalue) desc;
-- examplefield should be a numeric value of your customfield_id eg: 10001. stringvalue can be replaced for numbervalue or textvalue.Was this helpful?