Confluence Unresponsive due to stuck Workbox Notifications threads

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform Notice: Server and Data Center Only - This article only applies to Atlassian products on the server and data center platforms.

Problem

Confluence went unresponsive and returning HTTP 503 errors while being accessed from the Load Balancer. The issue continues to occur even after Confluence restart.

In addition, the following stuck threads detections messages are also being thrown in thcatalina.out log:

WARNING [ContainerBackgroundProcessor[StandardEngine[Standalone]]] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected Thread "http-nio-8090-exec-147" (id=5169) has been active for 63,774 milliseconds (since 11/6/18 7:42 PM) to serve the same request for https://example.atlassian.com/rest/mywork/latest/status/notification/count?_=1541562126288 and may be stuck (configured threshold for this StuckThreadDetectionValve is 60 seconds). There is/are 185 thread(s) in total that are monitored by this Valve and may be stuck.
 java.lang.Throwable
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394)

Diagnosis

Confluence had been configured to have a maximum of 200 HTTP threads:

<Confluence-Installation>/conf/server.xml
	<Connector port="8090"
                   ...
                   maxThreads="200"
                   ...
	/>

Looking at the thread dumps taken during the incident window, a vast number of long-running HTTP threads are shown and most of them are stuck waiting on SSL connection. In specific, these RUNNABLE threads are for Workbox Notifications:

Thread counts per status
$ grep -A1 '-exec-' conf_threads.* | grep State | sort | uniq -c
 200 conf_threads.1541570464.txt-   java.lang.Thread.State: RUNNABLE
  10 conf_threads.1541570464.txt-   java.lang.Thread.State: WAITING (parking)
 200 conf_threads.1541570479.txt-   java.lang.Thread.State: RUNNABLE
  10 conf_threads.1541570479.txt-   java.lang.Thread.State: WAITING (parking)
 200 conf_threads.1541570490.txt-   java.lang.Thread.State: RUNNABLE
  10 conf_threads.1541570490.txt-   java.lang.Thread.State: WAITING (parking)
 200 conf_threads.1541570501.txt-   java.lang.Thread.State: RUNNABLE
  10 conf_threads.1541570501.txt-   java.lang.Thread.State: WAITING (parking)
 200 conf_threads.1541570512.txt-   java.lang.Thread.State: RUNNABLE
  10 conf_threads.1541570512.txt-   java.lang.Thread.State: WAITING (parking)
 200 conf_threads.1541570523.txt-   java.lang.Thread.State: RUNNABLE
  10 conf_threads.1541570523.txt-   java.lang.Thread.State: WAITING (parking)
Example RUNNABLE thread
http-nio-8090-exec-200 - priority:5 - threadId:0x00007f6f9e2e6800 - nativeId:0x355e - state:RUNNABLE
stackTrace:
java.lang.Thread.State: RUNNABLE
     at java.net.SocketInputStream.socketRead0(Native Method)
     at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
     at java.net.SocketInputStream.read(SocketInputStream.java:171)
     at java.net.SocketInputStream.read(SocketInputStream.java:141)
     at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
     at sun.security.ssl.InputRecord.read(InputRecord.java:503)
     at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
     - locked <0x00000004f564e008> (a java.lang.Object)
     at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
     - locked <0x00000004f564e018> (a java.lang.Object)
     at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
     at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
     ...
     at com.atlassian.sal.core.net.HttpClientRequest.executeAndReturn(HttpClientRequest.java:103)
     at com.atlassian.plugins.rest.module.jersey.JerseyRequest.executeAndReturn(JerseyRequest.java:131)
     at com.atlassian.applinks.core.auth.ApplicationLinkRequestAdaptor.execute(ApplicationLinkRequestAdaptor.java:58)
     at com.atlassian.applinks.oauth.auth.OAuthRequest.execute(OAuthRequest.java:58)
     at com.atlassian.mywork.host.service.AppLinkHelper.execute(AppLinkHelper.java:64)
     at com.atlassian.mywork.host.service.AppLinkHelper.execute(AppLinkHelper.java:42)
     at com.atlassian.mywork.host.service.ClientServiceImpl.verifyAuth(ClientServiceImpl.java:171)
     at com.atlassian.mywork.host.service.ClientServiceImpl.verifyAuth(ClientServiceImpl.java:136)
     ...
     at com.atlassian.mywork.host.service.LocalNotificationServiceImpl.loadCount(LocalNotificationServiceImpl.java:385)
   	 at com.atlassian.mywork.host.service.LocalNotificationServiceImpl.lambda$_getCount$2(LocalNotificationServiceImpl.java:375)
	 at com.atlassian.mywork.host.service.LocalNotificationServiceImpl$$Lambda$1096/27140524.get(Unknown Source)
	 ...
     - locked <0x0000000438ab9e90> (a org.apache.tomcat.util.net.NioChannel)

Cause

This issue can occur when a linked application (via Application Link) is down, for example Jira. The issue occurs as each request coming from the Confluence's Workbox notifications is waiting to connect to the linked application (e.g. Jira). This then results in Confluence not having enough threads free in order to service users, leading to performance degradation.

Resolution

  • Please make sure that the linked application has been configured correctly in the Application Link page.
  • Please ensure that Confluence can reach all linked instances (via Application Link) over https. 
  • If the other end (e.g. Jira) is down or no longer in service, remove the application link 

After resolving the issue above, you'll need to restart Confluence for the threads to be released and available to users again.


DescriptionConfluence Unresponsive due to stuck Workbox Notifications threads
ProductConfluence

Last modified on Jan 22, 2020

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.