Confluence Stuck Threads due to oracle.net.nt.TimeoutSocketChannel.read
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
It has been observed that the Confluence application experiences multiple stuck threads on method oracle.net.nt.TimeoutSocketChannel.read
As a result, the application's performance is adversely affected.
Environment
Oracle JDBC driver version 12.2.0.0.0 and later
Diagnosis
If Confluence is impacted by this issue, we would observe the following stack for multiple stuck threads:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
15-Jun-2023 00:04:29.587 WARNING [Catalina-utility-1] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected Thread [http-nio2-8090-exec-457 url: /display/CTH/xys, /pages/viewpage.action; user: gobster] (id=[1313082]) has been active for [60,841] milliseconds (since [6/15/23 12:03 AM]) to serve the same request for [https://confluence.com/display/CTH/xyz] and may be stuck (configured threshold for this StuckThreadDetectionValve is [60] seconds). There is/are [66] thread(s) in total that are monitored by this Valve and may be stuck.
java.lang.Throwable
at java.base@11.0.19/sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at java.base@11.0.19/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at java.base@11.0.19/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276)
at java.base@11.0.19/sun.nio.ch.IOUtil.read(IOUtil.java:245)
at java.base@11.0.19/sun.nio.ch.IOUtil.read(IOUtil.java:223)
at java.base@11.0.19/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:353)
at oracle.net.nt.TimeoutSocketChannel.read(TimeoutSocketChannel.java:144)
at oracle.net.ns.NIOHeader.readHeaderBuffer(NIOHeader.java:82)
at oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:139)
at oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:101)
at oracle.net.ns.NIONSDataChannel.readDataFromSocketChannel(NIONSDataChannel.java:80)
at oracle.jdbc.driver.T4CMAREngineNIO.prepareForReading(T4CMAREngineNIO.java:98)
at oracle.jdbc.driver.T4CMAREngineNIO.unmarshalUB1(T4CMAREngineNIO.java:534)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:485)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:252)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:612)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:226)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:59)
at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:910)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1119)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3780)
at oracle.jdbc.driver.T4CPreparedStatement.executeInternal(T4CPreparedStatement.java:1343)
at oracle.jdbc.driver.OraclePreparedStatement.executeLargeUpdate(OraclePreparedStatement.java:3865)
at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:3845)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeUpdate(OraclePreparedStatementWrapper.java:1061)
at org.apache.tomcat.dbcp.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:135)
at org.apache.tomcat.dbcp.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:135)
...
...This indicates that there is an issue with the database connection itself, causing all Confluence connections using JDBC to get stuck at the same point (oracle.net.nt.TimeoutSocketChannel.read)
Cause
According to the following Oracle Document, the JDBC Driver version 12.2.0.1 and above may encounter failures with the error: "IO Error: Socket Read Interrupted."
The document (attached PDF below) explains that the 12.2.0.1 and above, JDBC driver uses Java NIO calls in blocking mode, which can be impacted by any interrupt() calls made by the application. This differs from previous versions of the JDBC driver that used stream-based I/O API calls unaffected by calls to interrupt() . Please note that this is a deliberate and intentional change beginning in the Oracle 12.2.0.1 JDBC driver, rather than a bug.
Further details can be found in the Oracle Support website, under the document:
Solution
With Oracle JDBC driver version 12.2.0.x and above, set the following connection property: oracle.jdbc.javaNetNio=false (in the startup parameters).
Add the following System Property to add in the
setenv.shfile. (Check the Configure System Properties document for more information on how to add the system properties. )This change will revert the driver behavior so it does not block
interrupt()calls in Java.1-Doracle.jdbc.javaNetNio=falseRestarting the Confluence will fix the issue.
Was this helpful?