High CPU Usage caused by Page Comparison
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Confluence process is consuming high CPU, reaching sometimes to 100%.
Diagnosis
After collecting and analysing several ThreadDumps, you can observe that the threads consuming high CPU (~90% or higher) are related with class daisy.diff.html, which is involved in Page Comparison
The threads seem stuck in the following method, despite they are in RUNNING status:
"pool-1435-thead-1" #47343 prio=5 os_prio=0 cpu=5101417.11ms elapsed=5785.28s tid=0x0000fffecd228000 nid=0x6f9f runnable [0x0000fffda1b07000]
java.lang.Thread.State: RUNNABLE
at org.eclipse.compare.internal.LCS.findMostProgress(LCS.java:383)
at org.eclipse.compare.internal.LCS.find_middle_snake(LCS.java:329)
at org.eclipse.compare.internal.LCS.lcs_rec(LCS.java:127)
at org.eclipse.compare.internal.LCS.lcs_rec(LCS.java:144)
at org.eclipse.compare.internal.LCS.lcs_rec(LCS.java:144)
...
...
at org.eclipse.compare.internal.LCS.lcs_rec(LCS.java:146)
at org.eclipse.compare.internal.LCS.longestCommonSubsequence(LCS.java:90)
at org.eclipse.compare.rangedifferencer.RangeComparatorLCS.findDifferences(RangeComparatorLCS.java:37)
at org.eclipse.compare.rangedifferencer.RangeDifferencer.findDifferences(RangeDifferencer.java:110)
at org.outerj.daisy.diff.html.ancestor.TextOnlyComparator.getMatchRatio(TextOnlyComparator.java:80)
at org.outerj.daisy.diff.html.dom.TagNode.getMatchRatio(TagNode.java:474)
at org.outerj.daisy.diff.html.TextNodeComparator.markAsDeleted(TextNodeComparator.java:281)
at org.outerj.daisy.diff.html.HTMLDiffer.diff(HTMLDiffer.java:80)
at com.atlassian.confluence.diff.DaisyHtmlDiffer.htmlDiff(DaisyHtmlDiffer.java:266)
at com.atlassian.confluence.diff.DaisyHtmlDiffer.lambda$diff$1(DaisyHtmlDiffer.java:214)
at com.atlassian.confluence.diff.DaisyHtmlDiffer$$Lambda$5182/0x0000001003cb3040.call(Unknown Source)
at java.util.concurrent.FutureTask.run(java.base@11.0.19/FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.19/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.19/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.19/Thread.java:829)Cause
The root cause is unclear, but there are two possibilities that contribute to this issue:
When a user compare two versions of the same page, it creates a request with the following pattern:
/pages/diffpagesbyversion.action?pageId=XXXXXXXXXXX&selectedPageVersions=1&selectedPageVersions=2You can search for this pattern in your confluence logs in order to pinpoint if a specific user(s) has triggered multiple comparisons when the issue started.
- Additionally, as part of Confluence's Scheduled Jobs there is a job called "Send batched notifications" which is responsible for sending email notifications containing all changes to a page or blog post since the last time the job ran.
It is the case, that if a user keeps enabled the option "Show changed content" in their Email Notifications, then multiple comparison can be automatically executed.
Solution
Workaround
To mitigate the impact, please follow below action plan:
Explicitly set the 'diffHtml' timeout as a System Properties in your
setenv.shfile:CATALINA_OPTS="-Dconfluence.html.diff.timeout=60000 ${CATALINA_OPTS}"Disable the "Show Changed Content" feature for every user as stated in KB How to disable Show Changed Content to all users. This will prevent that the automatic processing of those changes could impact your CPU, though it might be that a user enables this option later in time.
- Restart Confluence.