Bulk updating tickets in Jira service management Data center is slow
Platform Notice: Data Center - This article applies to Atlassian products on the Data Center platform.
Note that this knowledge base article was created for the Data Center version of the product. Data Center knowledge base articles for non-Data Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Jira service management with Assets bulk editing performance is unexplainably slow even though all performance indicators are in acceptable values, Jira might take up to 2 minutes to bulk edit (Add comment for example) on just 10 issues.
Environment
Jira Service Management 5.X
Diagnosis
- There are no bottlenecks in the memory, CPU, disk or database performance
- The issue type have Assets customfields even if not in use by the affected issues.
- Taking a thread dump during the bulk edit process shows multiple threads in a running state started from the bulk editing classes such as "BulkOperationProgress" most of them going to the database
12:52:34 - JiraTaskExecutionThread-3
State:RUNNABLE
CPU usage:0.00%
Running for: 0:00.00
Waiting for
This thread is not waiting for notification on any lock
Locks held
This thread does not hold any locks
Stack trace
java.lang.reflect.AccessibleObject.slowVerifyAccess(java.base@11.0.13/Unknown Source)
java.lang.reflect.AccessibleObject.verifyAccess(java.base@11.0.13/Unknown Source)
java.lang.reflect.AccessibleObject.checkAccess(java.base@11.0.13/Unknown Source)
java.lang.reflect.Method.invoke(java.base@11.0.13/Unknown Source)
com.atlassian.plugin.osgi.bridge.external.HostComponentFactoryBean$DynamicServiceInvocationHandler.invoke(HostComponentFactoryBean.java:130)
com.sun.proxy.$Proxy696.formatter(Unknown Source)
com.riadalabs.jira.plugins.insight.common.tools.InsightDateTimeFormatterInJira.getDateFormatter(InsightDateTimeFormatterInJira.java:99)
com.riadalabs.jira.plugins.insight.common.tools.InsightDateTimeFormatterInJira.formatDateToString(InsightDateTimeFormatterInJira.java:78)
com.riadalabs.jira.plugins.insight.services.predicate.attribute.AttributeValuePredicateBase.formatDate(AttributeValuePredicateBase.java:399)
com.riadalabs.jira.plugins.insight.services.predicate.attribute.AttributeValuePredicateBase.getValue(AttributeValuePredicateBase.java:450)
com.riadalabs.jira.plugins.insight.services.predicate.attribute.AnyAttributeValuePredicate.searchValueInObjectAttributeValues(AnyAttributeValuePredicate.java:54)
com.riadalabs.jira.plugins.insight.services.predicate.attribute.AnyAttributeValuePredicate.compareObjectAttributeValues(AnyAttributeValuePredicate.java:47)
com.riadalabs.jira.plugins.insight.services.predicate.attribute.AttributeValuePredicateBase.testObject(AttributeValuePredicateBase.java:171)
com.riadalabs.jira.plugins.insight.services.predicate.attribute.AttributeValuePredicateBase.testObjectBean(AttributeValuePredicateBase.java:82)
com.riadalabs.jira.plugins.insight.services.predicate.ObjectBeanPredicateBase.test(ObjectBeanPredicateBase.java:20)
com.riadalabs.jira.plugins.insight.services.predicate.ObjectBeanPredicateBase.test(ObjectBeanPredicateBase.java:10)
com.riadalabs.jira.plugins.insight.services.predicate.MultiplePredicates.testAnyOf(MultiplePredicates.java:62)
com.riadalabs.jira.plugins.insight.services.predicate.MultiplePredicates.test(MultiplePredicates.java:53)
com.riadalabs.jira.plugins.insight.services.predicate.MultiplePredicates.test(MultiplePredicates.java:11)
java.util.function.Predicate.lambda$and$0(java.base@11.0.13/Unknown Source)
java.util.function.Predicate$$Lambda$3411/0x0000000803df9840.test(java.base@11.0.13/Unknown Source)
com.riadalabs.jira.plugins.insight.services.predicate.ObjectTypeBucketPredicate.test(ObjectTypeBucketPredicate.java:33)
com.riadalabs.jira.plugins.insight.services.predicate.ObjectTypeBucketPredicate.test(ObjectTypeBucketPredicate.java:9)
com.riadalabs.jira.plugins.insight.services.core.ObjectFilterManagerImpl.filterObjectBeansInThread(ObjectFilterManagerImpl.kt:125)
com.riadalabs.jira.plugins.insight.services.core.ObjectFilterManagerImpl.findObjectsBasedOnFilter(ObjectFilterManagerImpl.kt:84)
com.riadalabs.jira.plugins.insight.services.core.ObjectFilterManagerImpl.findObjectsByFilter(ObjectFilterManagerImpl.kt:62)
com.riadalabs.jira.plugins.insight.services.core.ObjectFilterManagerImpl.findObjectsByFilter(ObjectFilterManagerImpl.kt:53)
com.riadalabs.jira.plugins.insight.services.core.iql.impl.IQLManagerImpl.retrieveObjectsThatShouldReference(IQLManagerImpl.java:674)
com.riadalabs.jira.plugins.insight.services.core.iql.impl.IQLManagerImpl.getBeansReferencedByIQL(IQLManagerImpl.java:664)
com.riadalabs.jira.plugins.insight.services.core.iql.impl.IQLManagerImpl.getAttributesPredicate(IQLManagerImpl.java:500)
com.riadalabs.jira.plugins.insight.services.core.iql.impl.IQLManagerImpl.acceptTerminalClause(IQLManagerImpl.java:171)
com.riadalabs.jira.plugins.insight.services.core.iql.impl.IQLManagerImpl.acceptTerminalClause(IQLManagerImpl.java:89)
com.riadalabs.jira.plugins.insight.services.core.iql.model.IQLClauseVisitor.acceptIQLClause(IQLClauseVisitor.java:28)
com.riadalabs.jira.plugins.insight.services.core.iql.model.IQLClausePredicateVisitor.acceptAndClause(IQLClausePredicateVisitor.java:12)
com.riadalabs.jira.plugins.insight.services.core.iql.model.IQLClausePredicateVisitor.acceptAndClause(IQLClausePredicateVisitor.java:6)
com.riadalabs.jira.plugins.insight.services.core.iql.model.IQLClauseVisitor.acceptIQLClause(IQLClauseVisitor.java:22)
com.riadalabs.jira.plugins.insight.services.core.iql.impl.IQLManagerImpl.appendIQLBeansAndSchemaToFilter(IQLManagerImpl.java:474)
com.riadalabs.jira.plugins.insight.services.core.iql.impl.IQLManagerImpl.toObjectFilter(IQLManagerImpl.java:360)
com.riadalabs.jira.plugins.insight.services.core.iql.impl.IQLManagerImpl.findObjects(IQLManagerImpl.java:272)
com.riadalabs.jira.plugins.insight.services.core.impl.CustomFieldServiceImpl.findObjectsByCustomFieldConnectBean(CustomFieldServiceImpl.java:363)
com.riadalabs.jira.plugins.insight.services.core.impl.CustomFieldServiceImpl.findObjectsByCustomFieldConfig(CustomFieldServiceImpl.java:157)
com.riadalabs.jira.plugins.insight.services.jira.listeners.job.CustomFieldEventListenerJob.setAssignObjectsOnCustomField(CustomFieldEventListenerJob.java:150)
com.riadalabs.jira.plugins.insight.services.jira.listeners.job.CustomFieldEventListenerJob.doUpdate(CustomFieldEventListenerJob.java:130)
com.riadalabs.jira.plugins.insight.services.jira.listeners.CustomFieldEventListener.onIssueEvent(CustomFieldEventListener.java:91)
jdk.internal.reflect.GeneratedMethodAccessor1449.invoke(Unknown Source)
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.13/Unknown Source)
java.lang.reflect.Method.invoke(java.base@11.0.13/Unknown Source)
com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:42)
com.atlassian.diagnostics.internal.platform.monitor.event.EventSystemMonitor.invokeMonitored(EventSystemMonitor.java:105)
com.atlassian.diagnostics.internal.platform.monitor.event.MonitoredListenerInvoker.invoke(MonitoredListenerInvoker.java:38)
com.atlassian.event.internal.ComparableListenerInvoker.invoke(ComparableListenerInvoker.java:48)
com.atlassian.event.internal.AsynchronousAbleEventDispatcher.lambda$null$0(AsynchronousAbleEventDispatcher.java:37)
com.atlassian.event.internal.AsynchronousAbleEventDispatcher$$Lambda$691/0x00000008009c3840.run(Unknown Source)
com.atlassian.event.internal.AsynchronousAbleEventDispatcher$$Lambda$211/0x00000008003c5c40.execute(Unknown Source)
com.atlassian.event.internal.AsynchronousAbleEventDispatcher.dispatch(AsynchronousAbleEventDispatcher.java:85)
com.atlassian.diagnostics.internal.platform.monitor.event.MonitoredEventDispatcher.dispatch(MonitoredEventDispatcher.java:36)
com.atlassian.event.internal.EventPublisherImpl.publish(EventPublisherImpl.java:114)
com.atlassian.event.internal.LockFreeEventPublisher.publish(LockFreeEventPublisher.java:40)
com.atlassian.jira.event.issue.DefaultIssueEventManager.publishEventIfNotificationsAreEnabled(DefaultIssueEventManager.java:180)
com.atlassian.jira.event.issue.DefaultIssueEventManager.publishEvent(DefaultIssueEventManager.java:162)
com.atlassian.jira.event.issue.DefaultIssueEventManager.publishAsRedundant(DefaultIssueEventManager.java:204)
com.atlassian.jira.event.issue.DefaultIssueEventManager.dispatchRedundantEvent(DefaultIssueEventManager.java:121)
com.atlassian.jira.issue.util.DefaultIssueUpdater.storeModifiedFields(DefaultIssueUpdater.java:158)
com.atlassian.jira.issue.util.DefaultIssueUpdater.doUpdate(DefaultIssueUpdater.java:94)
com.atlassian.jira.issue.util.DefaultIssueUpdater.doUpdate(DefaultIssueUpdater.java:68)
com.atlassian.jira.issue.managers.DefaultIssueManager.doUpdate(DefaultIssueManager.java:702)
com.atlassian.jira.issue.managers.DefaultIssueManager.updateIssue(DefaultIssueManager.java:673)
com.atlassian.jira.issue.managers.DefaultIssueManager.updateIssue(DefaultIssueManager.java:652)
com.atlassian.jira.issue.managers.RequestCachingIssueManager.updateIssue(RequestCachingIssueManager.java:217)
com.atlassian.jira.bulkedit.operation.BulkEditOperation.perform(BulkEditOperation.java:216)
com.atlassian.jira.web.action.issue.bulkedit.BulkOperationProgress$BulkEditCallable.call(BulkOperationProgress.java:174)
com.atlassian.jira.web.action.issue.bulkedit.BulkOperationProgress$BulkEditCallable.call(BulkOperationProgress.java:144)Cause
Assets have long-running automation rules that go through a high number of objects and take time to finish causing interference (Concurrent access to Assets data by the automation and bulk editing threads), specifically :
An issue update triggers "on issue update" event, customfieldeventListner is called for the Assets customfield. This is expected, if there are any assets customfields on the issue that have filterIssueScope set, (this means that the value of the assets customfield can be dependent on the other fields of Jira issue ). If one of these customfields have a complex AQL that takes more time to run, or fetches a lot of objects, then the update operation could also be delayed especially that there are automation rules that parse the same data in parallel.
Solution
- Lower the automation rule execution frequency or move it outside of working hours if possible to avoid concurrent data access
- Remove Assets cusomfield(s) from the issue if it's not in actual use.
Note
If the suggested resolution doesn't bring the wanted impact and lower the bulk editing time to a few seconds, please reach out to Atlassian support with a support zip with a thread dump recorded during the slow bulk editing, thread dumps can be automatically recorded using Java flight recorder
Last modified on Jul 15, 2024
Powered by Confluence and Scroll Viewport.