Crowd not able to start up due to database pool max out
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
The crowd does not start after an upgrade or restart when multiple applications try to synchronize their directories. There is no error messages written in the application logs.
Environment
- Crowd 4.4.0
- Crowd 5.2.0
Diagnosis
Diagnosis 1
To locate the issue we need to enable debug logging by following Logging and Profiling KB and changing the log4j2.properties file by adding the below properties:
log4j.rootLogger=DEBUG, console, crowdlog
com.atlassian.crowd.startup=DEBUG
com.atlassian.crowd=DEBUGAfter that, we will see the below debug log is written in the logs:
http-nio-8095-exec-82 DEBUG [mchange.v2.resourcepool.BasicResourcePool] acquire test -- pool is already maxed out. [managed: 90; max: 90]Diagnosis 2
1. Running curl on the Crowd Tomcat localhost will block, e.g.:
curl -k -i http://localhost:8095/crowd/console/login.action
(will just hang)2. Capture CPU and Thread dumps of the Crowd java process when the above Crowd URL fails to respond. There will be some threads in this BLOCKED stack trace:
"http-nio-8095-exec-1" #211 daemon prio=5 os_prio=0 tid=0x00007f7999cd1234 nid=1234 waiting for monitor entry [0x00007f78123a0000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.atlassian.hibernate.extras.ResettableTableHiLoGenerator.generate(ResettableTableHiLoGenerator.java:67)
- waiting to lock <0x000000008284ec30> (a com.atlassian.hibernate.extras.ResettableTableHiLoGenerator)
at org.hibernate.event.internal.AbstractSaveEventListener.saveWithGeneratedId(AbstractSaveEventListener.java:115)
at org.hibernate.event.internal.DefaultSaveOrUpdateEventListener.saveWithGeneratedOrRequestedId(DefaultSaveOrUpdateEventListener.java:194)
at org.hibernate.event.internal.DefaultSaveEventListener.saveWithGeneratedOrRequestedId(DefaultSaveEventListener.java:38)
at org.hibernate.event.internal.DefaultSaveOrUpdateEventListener.entityIsTransient(DefaultSaveOrUpdateEventListener.java:179)
at org.hibernate.event.internal.DefaultSaveEventListener.performSaveOrUpdate(DefaultSaveEventListener.java:32)
at org.hibernate.event.internal.DefaultSaveOrUpdateEventListener.onSaveOrUpdate(DefaultSaveOrUpdateEventListener.java:75)
at org.hibernate.internal.SessionImpl$$Lambda$1294/90049633.accept(Unknown Source)
at org.hibernate.event.service.internal.EventListenerGroupImpl.fireEventOnEachListener(EventListenerGroupImpl.java:99)
at org.hibernate.internal.SessionImpl.fireSave(SessionImpl.java:636)
at org.hibernate.internal.SessionImpl.save(SessionImpl.java:629)
at org.hibernate.internal.SessionImpl.save(SessionImpl.java:624)
at com.atlassian.crowd.util.persistence.hibernate.HibernateDao.save(HibernateDao.java:298)
...and also many threads blocked waiting for a database connection:
Cause
Multiple applications try to synchronize their user directories which causes Crowd database connection pool to max up and Crowd cannot start its Web Interface.
Solution
To fix the problem we can follow the below steps:
- Shutdown Crowd
Open <crowd_home>/shared/crowd.cfg.xml file and check the below property:
<property name="hibernate.c3p0.max_size">30</property>Then we need to check <crowd-installation>/apache-tomcat/conf/server.xml file for the maxThread property:
<Connector acceptCount="100" connectionTimeout="20000" disableUploadTimeout="true" enableLookups="false" maxHttpHeaderSize="8192" maxThreads="150" minSpareThreads="25" port="8095" redirectPort="8443" useBodyEncodingForURI="true" URIEncoding="UTF-8" scheme="http" />- Finally, update maxThread or/and max_size value accordingly to have 25% more max_size than maxThread.
- e.g. if Tomcat maxThreads is 150, we would want DB to be 188 max_size
The recommendation is to have Tomcat maxThreads 48 and DB to be max_size of 60
- Start Crowd.