JIRA applications stalls due to lost lock in dbcp pool caused by StackOverflowError
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Problem
JIRA application stops responding (stalls) due to problem obtaining connection from DBCP pool. Problem is caused by lost lock, which lead to dead-lock for this particular part of code. Note that there is no problem with DB configuration or network.
You may see the following in the thread dump: large number of thread waiting in the same code with same stack-trace:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
"http-nio-8000-exec-1003" #388843 daemon prio=5 tid=0x00007fbd7c0e3000 nid=0x7e80 waiting on condition [0x00007fbcfe331000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006810eba00> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
at org.apache.commons.pool2.impl.LinkedBlockingDeque.size(LinkedBlockingDeque.java:989)
at org.apache.commons.pool2.impl.GenericObjectPool.getNumIdle(GenericObjectPool.java:693)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:415)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)
at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1533)
...In this case Lock have been lost: <0x00000006810eba00> - java.util.concurrent.locks.ReentrantLock$NonfairSync
Diagnosis
Diagnostic Steps
Collect thread dump and check waiting threads, see example of stack-trace above.
Check logs for StackOverflowError exception
Cause
Problem is caused by the following chain of actions:
Thread obtains the lock for the DBpool
Thread executes the code
Then it gets StackOverflowError exception (see Jira Data Center stalls due to StackOverflowError)
Thread still holds the lock, but it was not able to release it
For whole application this lock is lost, which lead to dead-lock for this particular part of code.
Solution
Workaround
Restart the application.
Resolution
Fix the problem in the code which has caused StackOverflowError.
See related KB - Jira Data Center stalls due to StackOverflowError
Was this helpful?