Applications connected with Crowd SSO sending huge amount of requests to Crowd

Still need help?

The Atlassian Community is here for you.

Ask the community

This article only applies to Atlassian's server products. Learn more about the differences between cloud and server.

Problem

Atlassian application such as Confluence, JIRA which are connected using Crowd Single Sign-On (SSO) are observed to be generating a huge amount of requests to Crowd on every user action. Depending on the number of concurrent users, applications with a high user base may be observed to have send 10000+ calls per minute to Crowd to validate user sessions. This can result in Crowd becoming a bottleneck, resulting in negative performance impact for Crowd-connected applications downstream.

Diagnosis

Check for misconfiguration in crowd.properties

  1. Open the file "crowd.properties" file inside the Atlassian Application connected to Crowd. They should be located in the following path:

    1. For Confluence: CONFLUENCE/confluence/WEB-INF/classes/crowd.properties

    2. For JIRA: JIRA/atlassian-jira/WEB-INF/classes/crowd.properties

  2. Check the value of the parameter session.validationinterval inside this file. This should be set to a non-zero value.

Validation through thread dumps

When the problem performance problems occur in downstream applications due to this issue, thread dumps (JIRA, Confluence) from these applications will show that most or all HTTP threads in RUNNABLE status are waiting on Crowd for authentication. Most or all threads in Crowd itself appear to be validating tokens.

Example JIRA stacktrace
Example JIRA stacktrace
"https-jsse-nio-8443-exec-53" #2039 daemon prio=5 os_prio=0 tid=0x00007fc4b8127000 nid=0x63a9 runnable [0x00007fc46b82f000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
...
	at com.atlassian.crowd.integration.rest.service.RestExecutor$MethodExecutor.executeCrowdServiceMethod(RestExecutor.java:574)
	at com.atlassian.crowd.integration.rest.service.RestExecutor$MethodExecutor.andReceive(RestExecutor.java:413)
	at com.atlassian.crowd.integration.rest.service.RestCrowdClient.validateSSOAuthenticationAndGetSession(RestCrowdClient.java:1146)
	at com.atlassian.crowd.integration.http.CrowdHttpAuthenticatorImpl.checkAuthenticated(CrowdHttpAuthenticatorImpl.java:155)
	at com.atlassian.crowd.integration.http.CacheAwareCrowdHttpAuthenticator.checkAuthenticated(CacheAwareCrowdHttpAuthenticator.java:82)
	at com.atlassian.crowd.integration.seraph.CrowdAuthenticator.checkAuthenticated(CrowdAuthenticator.java:271)
	at com.atlassian.crowd.integration.seraph.CrowdAuthenticator.getUser(CrowdAuthenticator.java:429)
	at com.atlassian.jira.security.login.SSOSeraphAuthenticator.getUser(SSOSeraphAuthenticator.java:63)
	at com.atlassian.seraph.auth.AbstractAuthenticator.getUser(AbstractAuthenticator.java:45)
...
Example Confluence stacktrace
Example Confluence stacktrace
"http-nio-8443-exec-110" daemon prio=5 tid=0x0000000000000351 nid=0 runnable 
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
...
	at com.atlassian.crowd.integration.rest.service.RestExecutor$MethodExecutor.executeCrowdServiceMethod(RestExecutor.java:574)
	at com.atlassian.crowd.integration.rest.service.RestExecutor$MethodExecutor.andReceive(RestExecutor.java:413)
	at com.atlassian.crowd.integration.rest.service.RestCrowdClient.validateSSOAuthenticationAndGetSession(RestCrowdClient.java:1146)
	at com.atlassian.crowd.integration.http.CrowdHttpAuthenticatorImpl.checkAuthenticated(CrowdHttpAuthenticatorImpl.java:155)
	at com.atlassian.crowd.integration.http.CacheAwareCrowdHttpAuthenticator.checkAuthenticated(CacheAwareCrowdHttpAuthenticator.java:82)
	at com.atlassian.crowd.integration.seraph.CrowdAuthenticator.checkAuthenticated(CrowdAuthenticator.java:271)
	at com.atlassian.crowd.integration.seraph.CrowdAuthenticator.getUser(CrowdAuthenticator.java:429)
	at com.atlassian.confluence.user.ConfluenceCrowdSSOAuthenticator.lambda$getUser$2(ConfluenceCrowdSSOAuthenticator.java:91)
	at com.atlassian.confluence.user.ConfluenceCrowdSSOAuthenticator$$Lambda$719/240855331.get(Unknown Source)
	at com.atlassian.confluence.impl.seraph.TimingAccumulator.accumulateOperation(TimingAccumulator.java:51)
	at com.atlassian.confluence.impl.seraph.AuthenticatorMetrics.measureGetUser(AuthenticatorMetrics.java:31)
	at com.atlassian.confluence.user.ConfluenceCrowdSSOAuthenticator.getUser(ConfluenceCrowdSSOAuthenticator.java:91)
	at com.atlassian.seraph.auth.AbstractAuthenticator.getUser(AbstractAuthenticator.java:45)
...
Example Crowd stacktrace
Example Crowd stacktrace
"http-bio-8444-exec-245" daemon prio=5 tid=0x00000000000025bf nid=0 runnable 
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at java.io.DataInputStream.readFully(DataInputStream.java:195)
	at java.io.DataInputStream.readFully(DataInputStream.java:169)
	at net.sourceforge.jtds.jdbc.SharedSocket.readPacket(SharedSocket.java:850)
	at net.sourceforge.jtds.jdbc.SharedSocket.getNetPacket(SharedSocket.java:731)
...
	- locked <0x000000005c8ad030> (a net.sourceforge.jtds.jdbc.JtdsConnection)
...
	at com.atlassian.crowd.manager.application.AbstractDelegatingApplicationService.storeUserAttributes(AbstractDelegatingApplicationService.java:178)
...
	at com.atlassian.crowd.manager.authentication.AliasingAwareTokenAuthenticationManager.validateUserToken(AliasingAwareTokenAuthenticationManager.java:101)
...



Cause

If the "session.validationinterval" is being set to 0, each individual request made by users in the SSO-participating application (such as JIRA or Confluence) will result in requests to Crowd to check for the validity of the user's session. This can lead to Crowd becoming flooded and overwhelmed with repeated requests to validation user tokens. Please also note, that session validity check will significantly increase page loading time for the application, since check will be done in Tomcat filter for each request before executing application code. 

The following is the description for the session.validationinterval parameter, according to the document for the crowd.properties file:

The number of minutes to cache authentication validation in the session. If this value is set to 0, each HTTP request will be authenticated with the Crowd server.

Resolution

Change session.validationinterval in crowd.properties for each downstream application to a non-zero value. By default, JIRA and Confluence ships with the value of "2", which means the session will be validated against Crowd Server every 2 minutes. A higher value means lower load on Crowd, but there is a security factor. If a Crowd administrator expires a user's session on the Crowd side, the user will still have access to downstream applications until the session needs to be re-validated.

Last modified on Apr 16, 2018

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.