Synchrony process get stuck and not cleared in Confluence Upgrade process
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Sometimes, while doing Confluence upgrade Synchrony process get stuck and not being killed or cleared by the upgrade process. This creates a problem of starting the Synchrony after an upgrade and we get the below error in the atlassian-synchrony.log
2020-12-08 15:56:47,437 DEBUG [225:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 2020-12-08 14:56:47,435 ERROR [main] [com.hazelcast.instance.AddressPicker] [LOCAL] [confluencesd_HA-Synchrony] [3.11.4] Cannot bind to a given address: /x.x.x.x Hazelcast cannot start. Port [5701] is already in use and auto-increment is disabled.
2020-12-08 15:56:47,437 DEBUG [225:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] com.hazelcast.core.HazelcastException: Cannot bind to a given address: /x.x.x.x. Hazelcast cannot start. Port [5701] is already in use and auto-increment is disabled.Environment
This is a common behavior in Server and Datacenter environment.
Diagnosis
Synchrony runs on 8091 (Default) port in Server and 5701 (default) in Data Center. If we see below error message in atlassian-synchrony.log similar to 8091 port as well:
2020-12-08 15:56:47,437 DEBUG [225:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 2020-12-08 14:56:47,435 ERROR [main] [com.hazelcast.instance.AddressPicker] [LOCAL] [confluencesd_HA-Synchrony] [3.11.4] Cannot bind to a given address: /x.x.x.x Hazelcast cannot start. Port [5701] is already in use and auto-increment is disabled.
2020-12-08 15:56:47,437 DEBUG [225:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] com.hazelcast.core.HazelcastException: Cannot bind to a given address: /x.x.x.x. Hazelcast cannot start. Port [5701] is already in use and auto-increment is disabled.To diagnose, Stop the Confluence and try to check if the port is currently initialized and used by the old Synchrony process. We can run a couple of commands:
Linux
1. netstat -tulpn
2. lsof -i @localhost:8091 (If Synchrony Port is 8091)
Windows
1. netstat -an
Solution
If we get any process is running after stopping the Confluence releated to Synchrony, please kill those processes so that Port will be free. In the next Confluence restart, we will not see such error "Cannot bind to a given address" or "Port is already in use".