Database Deadlocks due to Third Party Plugins

Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.

Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible

Summary

APPARENT DEADLOCK messages frequently seen under Confluence logs (<Confluence-Home>/logs/atlassian-confluence.log).

Environment

Confluence version: 6.15.10

Diagnosis


Following messages appear frequently under atlassian-confluence.log file. 

2020-10-05 10:04:56,123 WARN [Timer-0] [mchange.v2.async.ThreadPoolAsynchronousRunner] run com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@35e3a02c -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 3
Active Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@415b4e32 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@691164f5 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@10cec7bb (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
Pending Tasks:

Cause

This error can indicate issues with the database, it's connection pool and the database's ability to cleanly acquire a new connection. There is a possibility that deadlocks are happening due to some third party plugins. 

Check for the stuck threads under catalina logs and also capture the thread dumps. If you see stuck threads from third party plugin, dig further to understand the cause. As an example, below is the snippet from Comala Workflow Document plugin that was causing deadlock -

Below stack trace can be seen for Comala Workflow Document plugin

10-Nov-2020 14:17:10.088 WARNING [Catalina-utility-1] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected Thread [https-jsse-nio-443-exec-648] (id=[303560]) has been active for [61,513] milliseconds (since [11/10/20 2:16 PM]) to serve the same request for [https:/baseurl/rest/cw/1/content/387046976/approvals/approve?expand=state%2Cstates%2Cactions%2Capprovals&_=1605014166153] and may be stuck (configured threshold for this StuckThreadDetectionValve is [60] seconds). There is/are [1] thread(s) in total that are monitored by this Valve and may be stuck.
java.lang.Throwable
at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown Source)
...
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)
at com.comalatech.confluence.workflow.dao.SaxApprovalCheckDao.deserializeApprovals(SaxApprovalCheckDao.java:66)
at com.comalatech.confluence.workflow.dao.DefaultApprovalCheckDao.getApprovalChecks(DefaultApprovalCheckDao.java:52)
at com.comalatech.confluence.workflow.DefaultApprovalAccessor.setApprovalsAndRejections(DefaultApprovalAccessor.java:409)
at com.comalatech.confluence.ipc.api.DefaultStateService.getStates(DefaultStateService.java:199)
at com.comalatech.confluence.dataextractor.extractor.WorkflowsPropertyExtractor.getFullWorkflowsContentPropertyModel(WorkflowsPropertyExtractor.java:180)
at com.comalatech.confluence.dataextractor.DefaultDataExtractorManager.createOrUpdateContentProperties(DefaultDataExtractorManager.java:231)
at com.comalatech.confluence.dataextractor.DefaultDataExtractorManager.updateContentProperty(DefaultDataExtractorManager.java:166)
at com.comalatech.confluence.dataextractor.DefaultDataExtractorManager.updateWorkflowsContentProperty(DefaultDataExtractorManager.java:135)
at com.comalatech.confluence.dataextractor.DataExtractorListener.updatePage(DataExtractorListener.java:80)
at com.comalatech.confluence.dataextractor.DataExtractorListener.approvalEvent(DataExtractorListener.java:34)
...
at com.atlassian.confluence.vcache.VCacheRequestContextOperations.lambda$doInRequestContext$0(VCacheRequestContextOperations.java:50)
at com.atlassian.confluence.vcache.VCacheRequestContextOperations$$Lambda$409/1766567016.perform(Unknown Source)
at com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContextInternal(VCacheRequestContextManager.java:87)
at com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContext(VCacheRequestContextManager.java:71)
at com.atlassian.confluence.vcache.VCacheRequestContextOperations.doInRequestContext(VCacheRequestContextOperations.java:49)
...
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
...

at com.atlassian.event.internal.EventPublisherImpl.publish(EventPublisherImpl.java:114)
at com.atlassian.confluence.event.TimingEventPublisher.publish(TimingEventPublisher.java:76)
...
at com.sun.proxy.$Proxy499.publish(Unknown Source)
...

Above threads looks to be linked to approve workflow and this was causing lots of other threads to stuck. 

Looking at Comala Workflow Document release history, below bug fixes were done under Comala version 6.12.2 directly linked to approval. Snippet from the same is as below

Comala Document Management 6.12.2 is a bug fix release.

Updates and Bug Fixes

Improved responsiveness of approve / reject actions on pages with many versions

Solution

Check for the bug fixes under the new versions of the third party plugin. If fix has already been made, update to the latest version of the plugin and see if that helps. If no bug fix around the respective functionality of the stuck thread are seen, reach out to the third party plugin vendor. 


Last modified on Oct 3, 2022

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.