Summary

When performing a full reindex in Jira, it fails with several WARN messages about "Duplicate keys mapped" like this below:

2021-03-12 15:59:56,965-0300 IssueIndexer:thread-1 WARN admin 959x1839x1 1wtf9jg 0:0:0:0:0:0:0:1 /secure/admin/jira/IndexReIndex!reindex.jspa [c.a.jira.index.AccumulatingResultBuilder] java.lang.IllegalStateException: Duplicate keys mapped to com.atlassian.jira.issue.customfields.vdi.CustomFieldPrefetchedData@6934d589 / com.atlassian.jira.issue.customfields.vdi.CustomFieldPrefetchedData@5c73dbed for custom field id customfield_10221

And later the reindex failure message:

2021-03-12 15:59:57,136-0300 JiraTaskExecutionThread-3 ERROR admin 959x1839x1 1wtf9jg 0:0:0:0:0:0:0:1 /secure/admin/jira/IndexReIndex!reindex.jspa [c.a.j.util.index.CompositeIndexLifecycleManager] Reindex All FAILED. ...

Environment

All versions of Jira 8.

Diagnosis

1) Look for the reindex all failure in the logs:

cat atlassian-jira.log | grep -i "reindex all failed"

2) Look for "Duplicate keys mapped" messages and their respective customfields:

cat atlassian-jira.log | grep "Duplicate keys mapped to" | egrep -o "customfield_.*" | sort | uniq -c

This should output each customfield and the number of messages for each:

3025 customfield_10250
 150 customfield_10251

3) Check if there are any fields that have values in both the label and customfieldvalue tables:

select distinct f.id, f.cfname
from label l
join customfieldvalue v on v.customfield = l.fieldid
join customfield f on f.id = v.customfield
where l.fieldid is not null and v.stringvalue is not null
and f.customfieldtypekey = 'com.atlassian.jira.plugin.system.customfieldtypes:labels';

Confirm their ids are the same that showed up on the logs (in step #2).

Cause

There's a label-type customfield with values stored in the customfieldvalue table instead of the label table only.

This scenario shouldn't exist and Jira's reindex doesn't completely succeeds, leaving the index incomplete or in a non-reliable state (eg. filter results show different values than the issue screen).

Root cause

We don't know yet how label-type fields could have values stored in the customfieldvalue table. Maybe it's the work of 3rd party apps or custom scripts running in Jira (through Script-enabling apps) — or a direct database insert.

Solution

We should remove the invalid entries from the customfieldvalue for those label-type fields table and perform a full reindex again:

delete from customfieldvalue where customfield in (10250, 10251); -- example Ids