Unable to edit or save pages due to multiple reconnections between the browser and Synchrony
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Problem
While attempting to edit or save pages with Collaborative Editing enabled, the following timeout/connection errors are intermittently shown in the UI:
While trying to edit the page, you see this message when the auto-save interval is reached:
Can't reach the server. Check your internet connection, and we'll keep trying to reconnect you.
While trying to manually update the page :
Unable to communicate with server. Saving is not possible at the moment.
Or:
The editor didn’t load this time The connection timed out. If it happens again, speak to your Confluence admin. You may be using a proxy server that prevents WebSocket connections.
Also, the following messages can be seen in the atlassian-synchrony.
log
:
2018-11-11 15:03:28,451 DEBUG [271:StdOutHandler [/Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home/jre/bin/java]] {"synchrony":{"message":"tilt.bayeux.server [warn] error rerouting request","ns":"tilt.bayeux.server","throwable":"clojure.lang.ExceptionInfo: in channel closed prematurely
...
2018-11-11 22:01:55,549 DEBUG [143:StdOutHandler [/usr/lib/jvm/java-8-oracle/jre/bin/java]] {"synchrony":{"message":"tilt.bayeux.server [warn] error rerouting request","ns":"tilt.bayeux.server","throwable":"clojure.lang.ExceptionInfo: unprocessed request {:type :no-target, :source nil}\n\tat clojure.core$ex_info.invokeStatic(core.clj:4725)
...
2018-11-11 22:01:56,326 DEBUG [143:StdOutHandler [/usr/lib/jvm/java-8-oracle/jre/bin/java]] {"synchrony":{"client":{"clientId":"UCBLNUcexJSH20I71GOLRcw","sk":"pong timeout\nError: pong timeout\n at new Xj (eval at <anonymous> (https://domain/s/202a4eb254bd334279a734425f2367bc-CDN/
...
Environment
Tomcat version below 8.5. You can check this with the following command on your Confluence install folder:
java -cp lib/catalina.jar org.apache.catalina.util.ServerInfo
The following protocol is set in the active connector inside <Confluence-Install>/conf/server.xml file:
protocol="org.apache.coyote.http11.Http11Protocol"
Diagnosis
The Synchrony logs above mention some timeouts and errors rerouting requests, which explain the connectivity issues that are observed in the browser. To better understand what is happening, the following data needs to be verified:
Browser console logs will show messages like the ones below, which indicate multiple connection drops with Synchrony:
Synchrony connected. batch.js:633:165 Synchrony: Warning: reconnect (xhr error) batch.js:540:344 warn reinit pong timeout Synchrony disconnected! batch.js:633:165
- The HAR file will show multiple canceled requests (HTTP return code 0) for the bayeux server:
02:51:20.782 0 708 — [MIME][NOT AVAILABLE][RESP W][CACHE ??] 918 ms ‡ POST https://domain/synchrony-proxy/v1/bayeux-sync1
If you have a reverse proxy forwarding the requests to Confluence, you may see messages like this one (example from Apache):
[Tue Oct 30 08:32:23.707353 2018] [proxy_http:error] [pid 1111:tid 11111111] (103)Software caused connection abort: [client 1.1.1.1:11111] AH01102: error reading status line from remote server 2.2.2.2:8090, referer: https://domain/pages/resumedraft.action?draftId=<ID>&draftShareId=<ID> [Tue Oct 30 08:32:23.707372 2018] [proxy:error] [pid 1111:tid 111111111] [client 1.1.1.1:111111] AH00898: Error reading from remote server returned by /synchrony-proxy/v1/bayeux-sync1, referer: https://domain/pages/resumedraft.action?draftId=<ID>&draftShareId=<ID>
Cause
This behaviour will occur if Confluence is using the old BIO HTTP Protocol instead of the newer NIO version. The NIO version is the default one for Confluence 6 and above, but the BIO protocol may be in use if the connector was copied from an old version of Confluence after an upgrade, for example.
In the documents below, you can find more details about the BIO protocol issues:
- What is the difference between Tomcat's BIO Connector and NIO Connector?
- Why is NIO much more efficient than BIO, even though NIO doesn't save any CPU circles?
As we can see, the BIO protocol has blocking semantics, which is a big problem for WebSockets or HTTP long pooling connections. Since those are the two possible connection types to Synchrony, the usage of this protocol can severely degrade its communication with the browser. This explains the timeouts and errors rerouting requests that can be seen in the Synchrony logs, as well as the multiple disconnections between the browser and Synchrony.
This problem is aggravated as the load of the instance increases. This is due to the inefficiency of the BIO protocol, which is unable to handle WebSockets or HTTP long pooling requests properly, especially at a large scale.
To confirm that you are using this protocol, check the <Confluence-install>/logs/catalina.out file and find the protocol startup, it will look like this:
31-Oct-2018 14:23:39.027 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-bio-8090"]
Resolution
With that in mind, we should replace the protocol on the Tomcat connector to use the new version. Here are the steps to achieve that:
- Edit the <Confluence-Install>/conf/server.xml file
Find the protocol attribute in the active connector, it will look like this:
protocol="org.apache.coyote.http11.Http11Protocol"
Replace the BIO protocol above with the NIO2 version, so it looks like this:
protocol="org.apache.coyote.http11.Http11Nio2Protocol"
- Save the file and restart Confluence
- Check if the editor works properly after the change