Synchrony process killed due to WMIC error
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
The Synchrony process is killed, at random, usually after Confluence is running for a couple of days. As Confluence fails to restart the Synchrony process, Collaborative Editing stops working and the users can only edit pages after Collaborative Editing is disabled in the UI.
Synchrony can only be restarted by restarting Confluence.
Environment
- 6.15.10 - but can affect different Confluence versions after Synchrony was implemented (Confluence 6.0)
- Windows Server
Diagnosis
The Synchrony process suddenly dies, as can be observed in the Synchrony logs:
2020-05-12 03:20:25,853 DEBUG [184:StdOutHandler [E:\Confluence\jre\bin\java.exe]] {"synchrony":{"message":"synchrony.components.watch-pid [warn] PID 3152 has died. Terminating","ns":"synchrony.components.watch-pid","level":"warn"}}
Almost at the same time, Confluence tries to restart Synchrony with no success. If running Windows Server 2008 R2, you may see an error message like the following:
2020-05-12 03:20:26,401 INFO [184:StdOutHandler [E:\Confluence\jre\bin\java.exe]] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] startup Starting Synchrony and enabling Collaborative Editing
2020-05-12 03:20:26,869 ERROR [184:StdOutHandler [E:\Confluence\jre\bin\java.exe]] [atlassian.utils.process.ExternalProcessImpl] notifyAfterFinished Error while processing 'afterFinished' event:
org.zeroturnaround.exec.InvalidExitValueException: Unexpected exit value: -2147217402, allowed exit values: [0], executed command [C:\Windows\System32\Wbem\WMIC.exe, process, where, ProcessId=8144, get, ProcessId], output was 66 bytes:
Node - MELP-CONA01
ERROR:
Description = Out of memory
at org.zeroturnaround.exec.InvalidExitUtil.checkExit(InvalidExitUtil.java:27)
at org.zeroturnaround.exec.WaitForProcess.call(WaitForProcess.java:114)
at org.zeroturnaround.exec.ProcessExecutor.executeNoTimeout(ProcessExecutor.java:938)
at org.zeroturnaround.process.WindowsProcess.isAlive(WindowsProcess.java:78)
at org.zeroturnaround.process.WindowsProcess.taskkill(WindowsProcess.java:115)
at org.zeroturnaround.process.WindowsProcess.destroy(WindowsProcess.java:93)
at org.zeroturnaround.process.AbstractProcess.destroyForcefully(AbstractProcess.java:113)
at org.zeroturnaround.process.AbstractProcess.destroyForcefully(AbstractProcess.java:16)
at org.zeroturnaround.process.ProcessUtil.destroyForcefullyAndWait(ProcessUtil.java:114)
at org.zeroturnaround.process.ProcessUtil.destroyGracefullyOrForcefullyAndWait(ProcessUtil.java:180)
at com.atlassian.confluence.plugins.synchrony.utils.SynchronyProcessUtil.killRunningSynchronyProcess(SynchronyProcessUtil.java:51)
at com.atlassian.confluence.plugins.synchrony.bootstrap.DefaultSynchronyProcessManager.startup(DefaultSynchronyProcessManager.java:628)
at com.atlassian.confluence.plugins.synchrony.bootstrap.DefaultSynchronyProcessManager$SynchronyProcessMonitor.onAfterFinished(DefaultSynchronyProcessManager.java:329)
at com.atlassian.utils.process.ExternalProcessImpl.notifyAfterFinished(ExternalProcessImpl.java:601)
at com.atlassian.utils.process.ExternalProcessImpl.wrapUpProcess(ExternalProcessImpl.java:669)
at com.atlassian.utils.process.ExternalProcessImpl.finish(ExternalProcessImpl.java:192)
at com.atlassian.utils.process.ExternalProcessImpl$AsynchronousOutputHandlerRunnable.maybeFinish(ExternalProcessImpl.java:730)
at com.atlassian.utils.process.ExternalProcessImpl$AsynchronousOutputHandlerRunnable.run(ExternalProcessImpl.java:724)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
The Out of Memory message can be misleading, as Synchrony actually has not run out of memory. Increasing the Synchrony heap allocation doesn't solve the issue.
For newer versions of Windows server, this message may not appear though the PID ### has died. Terminating message will still appear.
Cause
The issue is related to an issue with WMIC.
Workaround
Changing Collaborative Editing Mode to Limited or OFF will prevent the issue.
Confluence 7.5.x or later versions no longer have Limited as an option.
Solution
For Windows Server 2008 R2, please see the link below where a Patch is available:
Customers have also reported success with upgrading to newer versions of Windows Server.