Summary

In some cases, searching a Confluence knowledge base from Jira Service Management (JSM), previously known as Jira Service Desk, can lead to performance problems in Confluence, including high memory consumption, slowness, and unresponsiveness.  

Environment

• Jira Service Management linked to Confluence knowledge base

Diagnosis

Confluence is slow or unresponsive.

Warnings about memory/garbage collection may appear around the incident window:

2021-01-15 16:10:24,610 WARN [alert-dispatch:thread-1] [confluence.alert-log] log 1610745024550 ; WARNING ; JVM ; JVM-1002 ; Garbage collection exceeded time limit ; not-detected ;  ;  ; {"durationInMillis":2096,"windowInMillis":20000,"limitPercent":10,"threadMemoryAllocations":"","threadDump":[]}

Application logs may report warnings about searches from JSM (via the REST endpoint /rest/knowledge-base/1.0/search/) exceeding timing thresholds: 

2021-01-28 16:06:43,844 WARN [http-nio-8090-exec-141] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1611867885719, threadId=132920, threadName='http-nio-8090-exec-141 url:/rest/knowledge-base/1.0/search/TST', userId='testuser', type='web-request', summary='/rest/knowledge-base/1.0/search/TST?type=page&permissionCheckedUser=testuser&queryString=It+was+my&pageSize=3&startIndex=0&highlight=true&xoauth_requestor_id=testuser'}
 -- url: /rest/knowledge-base/1.0/search/TST | traceId: b431c2862ba6951b | userName: testuser

There may be multiple instances of that warning, but with slightly longer search strings each time:

• It+was+my
• It+was+my+und
• It+was+my+understand
• It+was+my+understanding+t
• It+was+my+understanding+that
• It+was+my+understanding+that+there+wou
• It+was+my+understanding+that+there+would+be+n
• It+was+my+understanding+that+there+would+be+no+math
  
  

Confluence access logs may also log similar search requests from that REST endpoint with query strings increasing slightly each time, from a user agent like Apache-HttpClient/4.5.5 (or similar), and occurring within a short timespan. 

Thread dumps (with Thready installed) taken during the incident window may show a high number of HTTP threads associated with the REST endpoint mentioned above, and the following class appears in the stack trace:

com.atlassian.confluence.plugins.confluence_kb_space_blueprint.rest.SearchResource

Cause

There are a large number of incoming search requests coming from JSM. The problem happens because of a low debounce rate in JSM combined with multiple search requests for long string patterns.

In JSM, a search request is fired off when a user stops typing long enough to exceed the debounce threshold (~300ms). When the user continues to add to the search pattern, this can trigger multiple parallel search requests to Confluence in a short amount of time.

Each time Confluence receives a search request, it will perform an index search. Some requests could process more index contents, which means more memory objects would be needed, which leads to more work for garbage collection. With a heavily utilized heap, the JVM can become busy garbage collecting, and when it starts having problems (such as exceeding the time limit or the GC overhead limit), it won't be able to allocate memory efficiently for the results of the index searches and as a result Confluence will slow down or even cease processing other actions until the memory issues are resolved.

In a Data Center configuration, this can also negatively impact performance of the Hazelcast cache such that the other nodes will be unable to retrieve data from the cache, which can slow down the whole cluster as a result.

Workaround

Disable knowledge base searches from JSM:

• In Jira, navigate to the "Manage Apps screen, search for and expand "Service Desk Knowledge Base Plugin" (under "All apps")
• Expand the modules
• Disable the sd-kb-rest-resource module

This disables the REST API methods that the app uses to search Confluence, determine permissions for the articles it returns, and could possibly affect reporting on the effectiveness of the KB articles. Search will be disabled on tickets (agent view), on the sidebar (agent view), and on the customer portal. Jira administrators will still be able to link Confluence spaces to SD projects, but agents and users won't be able to search knowledge base articles from it.