Confluence Unresponsive due to Blocked Threads when Comala Workflows Plugin 5.5.0, 5.5.1, 5.5.2, 5.7.2 or 5.7.3 Has Been Installed
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Problem
Confluence becomes gradually unresponsive under load when Comala Workflows plugin 5.5.0, 5.5.1, 5.5.2, 5.7.2 or 5.7.3 has been installed on this instance.
Diagnosis
Thread dumps taken during the time of incident reveal that most HTTP threads (look for those with "-exec-" in its thread name) are in BLOCKED status, all "waiting to lock" the same object and each containing a reference to org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton, for example:
"http-nio-8093-exec-302" #12461 daemon prio=5 os_prio=0 tid=0x00007f4058160000 nid=0x215d waiting for monitor entry [0x00007f3f7e8dc000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:187)
- waiting to lock <0x0000000462c907c8> (a java.util.concurrent.ConcurrentHashMap)
at org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:489)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:447)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:423)
at org.springframework.beans.factory.BeanFactoryUtils.beanNamesForTypeIncludingAncestors(BeanFactoryUtils.java:220)
...
Tomcat logs at <confluence_install>/logs/catalina.out may also report many stuck threads with similar stacktraces, for instances:
18-Jul-2018 20:45:03.998 WARNING [ContainerBackgroundProcessor[StandardEngine[Standalone]]] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected Thread "http-nio-8093-exec-149" (id=12303) has been active for 68,146 milliseconds (since 7/18/18 8:43 PM) to serve the same request for http://confluence.example.com/display/SPACEKEY/Example+Page and may be stuck (configured threshold for this StuckThreadDetectionValve is 60 seconds). There is/are 68 thread(s) in total that are monitored by this Valve and may be stuck.
java.lang.Throwable
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:187)
at org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:489)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:447)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:423)
at org.springframework.beans.factory.BeanFactoryUtils.beanNamesForTypeIncludingAncestors(BeanFactoryUtils.java:220)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1177)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1116)
...
Cause
Comala Workflows plugin versions 5.5.0, 5.5.1, 5.5.2, 5.7.2 and 5.7.3 contained a bug which can cause performance/stability issues given sufficient load. This issue has been resolved in plugin version 5.5.3 and 5.8.1.
Resolution
Upgrade to plugin version 5.5.3 or 5.8.1 and above. If you are unable to upgrade right away, disabling the existing Workflows plugin may be used as a temporary workaround.