Synchrony Heartbeat fails due to a TimedOutException
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Synchrony fails and it is reported in Confluence UI as stopped or having errors.
Environment
Confluence 7.19.X and Confluence 8.X
Diagnosis
Reviewing the atlassian-confluence.log, it can be observed that the Synchrony heartbeat check failed due to a TimedOutException:
2023-11-07 08:41:54,560 WARN [synchrony-interop-executor:thread-8] [plugins.synchrony.bootstrap.DefaultSynchronyMonitor] lambda$pollHeartbeat$1 Rescheduled heartbeat check failed.
-- url: /rest/synchrony-interop/restart | userName: admin | referer: https://CONFLUENCE_URL/admin/confluence-collaborative-editor-plugin/configure.action | traceId: 403fa3a71a94ee0f
io.atlassian.util.concurrent.TimedOutException: Timed out after: 30076280311 NANOSECONDS
at com.atlassian.confluence.plugins.synchrony.bootstrap.DefaultSynchronyMonitor.lambda$pollHeartbeat$0(DefaultSynchronyMonitor.java:137)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
...
...Cause
1) Synchrony is using the URL, http://127.0.0.1:8091/synchrony which should be escaped by the forward proxy.
2) There is a misconfiguration on the reverse proxy level such as the wrong Synchrony port being used.
Solution
1) Follow the configuration described in our Configuring Web Proxy Support for Confluence documentation:
If you're using Confluence 6.0 or later with Synchrony, you'll need to pass the following to ensure Confluence can connect directly to Synchrony.
Replace localhost|127.0.0.1 with your Synchrony IP if you have used the synchrony.host system property to change the IP Synchrony uses.
-Dhttp.nonProxyHosts=localhost|127.0.0.1 -Dhttps.nonProxyHosts=localhost|127.0.0.1Note: You may need to escape the pipe character | in some command-line environments.
If the http.nonProxyHosts property is not configured, all web requests will be sent to the proxy.
Remember to restart Confluence once the change is implemented.
2) If you are using the direct connection to the Synchrony port at the reverse proxy level, check your reverse proxy configuration and make sure that /synchrony requests are redirected to the correct Synchrony port which is 8091 by default.