Duplicate key error in Jira server on creating issue via REST API or accessing Custom Fields page in JIRA
Platform Notice: Cloud, Server, and Data Center - This article applies equally to all platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
This issue might surface in various scenarios as follows (but not limited to):
Scenario 1: REST API
When checking the Required fields to create an issue in a project using a REST API below, it will return an error.
Example GET REST Call:
<JIRA Base URL>/rest/api/2/issue/createmeta?projectIds=<project id>&expand=projects.issuetypes.fields
The following error will be returned
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><status><status-code>500</status-code><message>Duplicate key com.atlassian.jira.rest.v2.issue.FieldMetaBean@174a0eaf</message><stack-trace>java.lang.IllegalStateException: Duplicate key com.atlassian.jira.rest.v2.issue.FieldMetaBean@174a0eaf
at java.util.stream.Collectors.lambda$throwingMerger$0(Collectors.java:133)
at java.util.HashMap.merge(HashMap.java:1253)
at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1320)
at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.LinkedList$LLSpliterator.forEachRemaining(LinkedList.java:1235)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:270)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at com.atlassian.jira.rest.v2.issue.AbstractMetaFieldBeanBuilder.build(AbstractMetaFieldBeanBuilder.java:97)
at com.atlassian.jira.rest.v2.issue.CreateMetaIssueTypeBean$1.expand(CreateMetaIssueTypeBean.java:31)
...
</stack-trace></status>Scenario 2: Custom Fields or editing Screen
This issue might manifest itself when accessing Custom Fields configuration page from JIRA → Settings → Custom Fields.
The exception will be slightly different in this scenario, however, the Resolution below is still applicable.
Here's an example of an exception that comes up in the logs in Scenario 2:
20-05-29 13:18:24,736 http-nio-8080-exec-7 ERROR admin 798x1632x2 1b7ddq4 127.0.0.1 /rest/api/2/customFields [c.a.j.rest.exception.ExceptionInterceptor] Returning internal server error in response
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
...
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Duplicate key com.atlassian.jira.issue.fields.screen.FieldScreenLayoutItemImpl@d8014b05The editing page of a Screen may also present errors, in which case it's the exact Screen we'll fix below.
Additional Details
After the above errors occur, you may also experience errors related to System Fields, like so:
2023-03-10 16:55:01,396-0500 http-nio-8080-exec-13 url: /secure/admin/ViewSystemFields.jspa; user: admin ERROR admin 1015x10612704x7 12dxhmp 127.0.0.1 /secure/admin/ViewSystemFields.jspa [o.a.c.c.C.[Catalina].[localhost].[/]] Unhandled exception occurred whilst decorating page com.google.template.soy.tofu.SoyTofuException: In 'foreach' command {foreach $systemField in $systemFields}{call .sytemFieldsTableRow}{param systemField: $systemField /}{/call}{/foreach}, the data reference does not resolve to a SoyList (encountered type com.google.template.soy.data.restricted.UndefinedData).
at JIRA.Templates.Admin.Systemfields.systemfieldsPage(Unknown Source)
at com.google.template.soy.tofu.internal.BaseTofu.renderMainHelper(BaseTofu.java:369)
at com.google.template.soy.tofu.internal.BaseTofu.renderMain(BaseTofu.java:322)
at com.google.template.soy.tofu.internal.BaseTofu.access$100(BaseTofu.java:66)
at com.google.template.soy.tofu.internal.BaseTofu$RendererImpl.render(BaseTofu.java:476)
at com.atlassian.soy.impl.DefaultSoyManager.render(DefaultSoyManager.java:147)
at com.atlassian.soy.impl.DefaultSoyTemplateRenderer.render(DefaultSoyTemplateRenderer.java:45)
at com.atlassian.soy.impl.DefaultSoyTemplateRenderer.render(DefaultSoyTemplateRenderer.java:39)
...
Caused by: com.google.template.soy.sharedpasses.render.RenderException: In 'foreach' command {foreach $systemField in $systemFields}{call .sytemFieldsTableRow}{param systemField: $systemField /}{/call}{/foreach}, the data reference does not resolve to a SoyList (encountered type com.google.template.soy.data.restricted.UndefinedData).Environment
All versions of Jira Core 7.x and 8.x.
Diagnosis
If the issue is integration related, please open the page "Logging & Profiling" on the remote application, and add the package com.atlassian.internal.integration.jira with the "DEBUG" level to let the error message parse in the server log.
Run the following SQL query towards the database to see if there is a duplicate field in a screen:
select f.name, i.fieldidentifier, count(*)
from fieldscreen f, fieldscreenlayoutitem i, fieldscreentab t
where f.id = t.fieldscreen
and i.fieldscreentab = t.id
group by f.name, i.fieldidentifier having count(*) > 1;If there is a duplicate, an example of a result is below:
name | fieldidentifier | count
---------------------------------+-----------------+-------
SSD: Scrum Default Issue Screen | assignee | 2Cause
There is a field that is appearing on a screen more than once. The cause for the duplicate is not yet known.
- One of the possible identified causes for the field to be duplicated is caused by an upgrade of the Xray plugin to 6.5.
The field can be duplicate in the same Screen tab or a different tab: the errors will vary slightly (error message Vs endless loading spinner), but the Custom Fields page or editing Screen page unusable all the same.
Resolution
Through Jira's UI
If there is a result shown by running the query above, it means that there is a duplicate in a screen. To remove it, follow the steps below:
- Navigate to JIRA Administration > Issues > Screens
- Look for the Screen mentioned in the SQL query result e.g. "SSD: Scrum Default Issue Screen"
- Click Configure and look for the field e.g. "assignee"
- Remove one of the fields from the screen. In case there are multiple tabs, check for each tab whether the field is called twice.
Through the Database
If the duplicate field is not shown on the screen or if the screen cannot be opened _(error message or endless loading spinner)_, the solution is to delete duplicate field reference directly from the database.
Please back up your database before proceeding with this workaround. Also, if possible, test the workaround on a staging environment before applying it on production.
To resolve specific duplicate records, the steps are as follows:
Identify and write down ID value for each duplicate field reference; screen name and field identifier need to be modified accordingly. For the current example, the ID can be identified as follows:
select f.name, i.id, i.fieldidentifier from fieldscreen f, fieldscreenlayoutitem i, fieldscreentab t where f.id = t.fieldscreen and i.fieldscreentab = t.id and f.name like '%SSD: Scrum Default Issue Screen%' -- the affected screen name from the diagnosis query and i.fieldidentifier='assignee'; -- the affected customfield from the diagnosis queryDelete all duplicate rows from fieldscreenlayoutitem table and retain only one row; replace <duplicate_id> with the ID that you wrote down in previous step (in this example, there is only one duplicate row that needs to be deleted):
delete from fieldscreenlayoutitem where id = <duplicate_id>;Restart Jira. For Jira DC, a rolling restart of the cluster nodes will suffice.
Through the Database (bulk resolution)
If the diagnostic query returned too many duplicates to correct individually, a bulk operation is possible. To resolve large numbers of duplicates, the steps are as follows:
Run the following query to confirm that the IDs are the right ones to delete. A simple check is to confirm that the count is the same as the count from the diagnostic query. Comparing the IDs against those in a "Select * from fieldscreenlayoutitem;" is a good idea too.
select f.name, i.fieldidentifier, max(i.id) from fieldscreen f, fieldscreenlayoutitem i, fieldscreentab t where f.id = t.fieldscreen and i.fieldscreentab = t.id group by f.name, i.fieldidentifier having count(*) > 1;Run the following command to delete any duplicate rows (those with higher values in the ID field), while leaving behind a single row (with a lower ID) for that field/screen/tab:
delete from fieldscreenlayoutitem where id in (select max(i.id) as deleteme from fieldscreen f, fieldscreenlayoutitem i, fieldscreentab t where f.id = t.fieldscreen and i.fieldscreentab = t.id group by f.name, i.fieldidentifier having count(*) > 1);- Restart Jira. For Jira DC, a rolling restart of the cluster nodes will suffice.