Confluence Unresponsive due to Blocked Threads when Comala Workflows Plugin 5.5.0, 5.5.1, 5.5.2, 5.7.2 or 5.7.3 Has Been Installed

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform Notice: Server and Data Center Only - This article only applies to Atlassian products on the server and data center platforms.

Problem

Confluence becomes gradually unresponsive under load when Comala Workflows plugin 5.5.0, 5.5.1, 5.5.2, 5.7.2 or 5.7.3 has been installed on this instance.

Diagnosis

Thread dumps taken during the time of incident reveal that most HTTP threads (look for those with "-exec-" in its thread name) are in BLOCKED status, all "waiting to lock" the same object and each containing a reference to org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton, for example:

"http-nio-8093-exec-302" #12461 daemon prio=5 os_prio=0 tid=0x00007f4058160000 nid=0x215d waiting for monitor entry [0x00007f3f7e8dc000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:187)
	- waiting to lock <0x0000000462c907c8> (a java.util.concurrent.ConcurrentHashMap)
	at org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:489)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:447)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:423)
	at org.springframework.beans.factory.BeanFactoryUtils.beanNamesForTypeIncludingAncestors(BeanFactoryUtils.java:220)
...

Tomcat logs at <confluence_install>/logs/catalina.out may also report many stuck threads with similar stacktraces, for instances:

18-Jul-2018 20:45:03.998 WARNING [ContainerBackgroundProcessor[StandardEngine[Standalone]]] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected Thread "http-nio-8093-exec-149" (id=12303) has been active for 68,146 milliseconds (since 7/18/18 8:43 PM) to serve the same request for http://confluence.example.com/display/SPACEKEY/Example+Page and may be stuck (configured threshold for this StuckThreadDetectionValve is 60 seconds). There is/are 68 thread(s) in total that are monitored by this Valve and may be stuck.
 java.lang.Throwable
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:187)
	at org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:489)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:447)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:423)
	at org.springframework.beans.factory.BeanFactoryUtils.beanNamesForTypeIncludingAncestors(BeanFactoryUtils.java:220)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1177)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1116)
...


Cause

Comala Workflows plugin versions 5.5.0, 5.5.1, 5.5.2, 5.7.2 and 5.7.3 contained a bug which can cause performance/stability issues given sufficient load. This issue has been resolved in plugin version 5.5.3 and 5.8.1.

Resolution

Upgrade to plugin version 5.5.3 or 5.8.1 and above. If you are unable to upgrade right away, disabling the existing Workflows plugin may be used as a temporary workaround.


Last modified on Oct 22, 2018

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.