Confluence triggers cluster panic while updating the database - Unable to acquire JDBC Connection
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
An error message appears when trying to access Confluence:
Database is being updated by an instance which is not part of the current cluster.Environment
Confluence Data Center
Diagnosis
When accessing Confluence, some nodes will throw errors in the logs, making the application inaccessible. When looking at the atlassian-confluence.log, will find entries like below:
2021-05-25 01:50:42,150 WARN [http-nio-8090-exec-9 url: /status] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1621921699643, threadId=403, threadName='http-nio-8090-exec-9 url: /status', userId='<unknown>', type='web-request', summary='/status'}
-- url: /status | traceId: 3990689cfb713b1a | userName: anonymous
2021-05-25 01:50:42,151 WARN [http-nio-8090-exec-59 url: /status] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1621921701223, threadId=13882, threadName='http-nio-8090-exec-59 url: /status', userId='<unknown>', type='web-request', summary='/status'}
-- url: /status | traceId: 98643738cbe68254 | userName: anonymous
2021-05-25 01:50:42,154 WARN [http-nio-8090-exec-13 url: /status] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1621921736283, threadId=411, threadName='http-nio-8090-exec-13 url: /status', userId='<unknown>', type='web-request', summary='/status'}
-- url: /status | traceId: e9b11240652a7c46 | userName: anonymous
2021-05-25 01:50:42,154 WARN [http-nio-8090-exec-69 url: /status] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1621921781252, threadId=25435, threadName='http-nio-8090-exec-69 url: /status', userId='<unknown>', type='web-request', summary='/status'}
-- url: /status | traceId: e2396d2c8415c734 | userName: anonymous
2021-05-25 01:50:42,156 WARN [http-nio-8090-exec-46 url: /status] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1621921721253, threadId=623, threadName='http-nio-8090-exec-46 url: /status', userId='<unknown>', type='web-request', summary='/status'}
-- url: /status | traceId: 0eba7f5425983dab | userName: anonymous
2021-05-25 01:50:42,158 WARN [http-nio-8090-exec-68 url: /status] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1621921779637, threadId=25434, threadName='http-nio-8090-exec-68 url: /status', userId='<unknown>', type='web-request', summary='/status'}
-- url: /status | traceId: 20f56684c4c282c3 | userName: anonymous
2021-05-25 01:50:42,162 WARN [http-nio-8090-exec-11 url: /status] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1621921733614, threadId=408, threadName='http-nio-8090-exec-11 url: /status', userId='<unknown>', type='web-request', summary='/status'}
-- url: /status | traceId: 37fdb3f28d2429ed | userName: anonymous
2021-05-25 01:51:01,158 WARN [hz.confluence.cached.thread-30] [internal.cluster.impl.ClusterHeartbeatManager] log [142.136.96.235]:5801 [CONFLUENCE_APP_NCW] [3.11.6] Suspecting Member [142.136.96.236]:5801 - 1a46fbe9-da3a-4469-be93-e2250c7a73f5 because it has not sent any heartbeats since 2021-05-25 01:50:00.516. Now: 2021-05-25 01:51:01.158, heartbeat timeout: 60000 ms, suspicion level: 1.00
Following that, the error below will be spammed in the application logs, until the connection with the database is re-established.
2021-05-25 04:25:23,567 WARN [Caesium-1-4] [impl.schedule.caesium.JobRunnerWrapper] runJob Scheduled job EhCacheCompactionJob#EhCacheCompactionJob completed unsuccessfully with response JobRunnerResponse[runOutcome=FAILED,message='CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is net.sf.hibernate.exception.GenericJDBCException: Unable to acquire JDBC Connection
GenericJDBCException: Unable to acquire JDBC Connection
SQLException']Cause
This is caused when a network glitch or maintenance happens, breaking the communication between the confluence server and the database.
When in a clustered instance, after the nodes re-establish the communication with the database, only the first node will be recognized as part of the cluster, and all of the others will be in a database state conflict, making them not be recognized as valid and part of the current data center connected to it.
Solution
Restart all impacted nodes, so the cache on the nodes is cleared and the connection with the database is re-established.