Crowd performance issues with Miniorange SSO Plugin
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
The Crowd becomes unresponsive when it receives high traffic from Jira, Confluence, or any other linked applications. Users would be unable to login into these applications or they may be able to log in intermittently. It may also not be possible to log in inside Crowd. There might be high CPU and database usage.
Environment
Crowd server or Data Center 4.4.1
Diagnosis
The following patterns might be noted from server logs:
1
2
3
2022-06-27 09:07:48,167 https-jsse-nio-8443-exec-184 url: /crowd/rest/usermanagement/1/session/dYPuZ-Wg7biwbwgZzjNfyAAAAAABAYABYWdhcmNpYTNANzdnbG9iYWwuYml6 WARN [v2.c3p0.impl.NewPooledConnection] [c3p0] Another error has occurred [ org.postgresql.util.PSQLException: This connection has been closed. ] which will not be reported to listeners!
org.postgresql.util.PSQLException: This connection has been closed.
at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:877)There may be multiple database connection closures. The database thread pool may also be fully occupied leaving no db threads for new connections:
1
acquire test -- pool is already maxed out. [managed: 60; max: 60]And there would be multiple waiting threads from the thread dumps:
1
2
3
4
5
6
7
at java.lang.Object.wait(java.base@11.0.15/Native Method)
- waiting on [no object reference available]
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1503)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:644)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:554)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutAndMarkConnectionInUse(C3P0PooledConnectionPool.java:758)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:685)But most importantly, you might notice the following trace from the thread dumps, which will mostly be stuck threads. And these stuck threads might be close to the maximum number of database threads assigned to that node, clearly indicating that all database threads are occupied. And you might also find a trace of the miniOrange SSO app in these snippets
1
2
3
4
5
6
...
com.sun.proxy.$Proxy83.searchUsers(Unknown Source)com.atlassian.crowd.manager.license.CrowdLicenseManagerImpl.getCurrentResourceUsageTotal(CrowdLicenseManagerImpl.java:199)
...
com.sun.proxy.$Proxy326.getCurrentResourceUsageTotal(Unknown Source)
com.miniorange.sso.saml.crowd.MoSAMLSettings.isNoOfUserExceed(MoSAMLSettings.java:835)
com.miniorange.sso.saml.crowd.servlet.MoSAMLLoginServlet.doPost(MoSAMLLoginServlet.java:150)Cause
Probably the miniOrange app, that during most (or maybe all) of the requests that need authentication checks if Crowd license usage is not exceeded. All seems to be good, except for one fact. The miniOrange app usesCrowdLicenseManager#getCurrentResourceUsageTotal (see this JavaDoc that includes a warning) that recalculates the user number every time from scratch (a very resource-expensive operation). This method should be avoided for this scenario. Instead of this, app developers should utilize a pre-calculated value that is recalculated every 6 hours by a scheduler. They can access it byPropertyManager#getCurrentLicenseResourceTotal
All the above are true only for versions of Crowd lesser than 5.0.0. As in 5.0, this method was refactored and this part of logic andCrowdLicenseManager#getCurrentResourceUsageTotal returns a pre-calculated value (see its JavaDocs).
Solution
A) Upgrade to miniOrange Crowd SAML SSO plugin v2.0.0, where this issue has been fixed.
B) Upgrade Crowd to 5.0.0 or higher.
In case of issues or concerns to follow this procedure, please reach out to the Atlassian Support team and share the information that's been gathered so far along with a fresh Support zip file.
Was this helpful?