Recent updates and new pages are not searchable right away and the indexing queue has suddenly grown very large

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform Notice: Server and Data Center Only - This article only applies to Atlassian products on the server and data center platforms.

Summary

Intermittently the indexing queue (<base_url>/admin/viewindexqueue.action) may suddenly grow very large, for example ~250,000 new items to process all showing up to the queue at the same time. 

Due to the increased workload in the indexing queue other normal edits do not get indexed right away and users may begin to report that their recently created pages or recent page updates are not searchable. Over time these new pages and updates to existing pages will being to appear in searches but not right after they're made, there will be a significant delay observed.

Diagnosis

Run the following SQL query:

SELECT message,COUNT(4) as OccurenceValue FROM journalentry GROUP BY message ORDER BY OccurenceValue DESC limit 50;

Confirm if all or most of the new entries all come from a single page (or subset of pages) such as in the example below:

$> SELECT message,COUNT(4) as OccurenceValue FROM journalentry GROUP BY message ORDER BY OccurenceValue DESC limit 50;
+--------------------------------------------------+----------------+
| message                                          | OccurenceValue |
+--------------------------------------------------+----------------+
| com.atlassian.confluence.pages.Page-1135451839   |         520266 |
| com.atlassian.confluence.pages.Page-1360182692   |            468 |
....truncated....

Once you have identified the affected page, reload it and check the numbers again. Reloading the page has been known to cause multiple new entries to be made each reload.

Cause

The root cause is unknown precisely but is related to 3rd party plugin content.

Workaround

Alter the page data related to the plugins involved in the page and save it. For example: possibly the Customware Scaffolding and/or Scriptrunner add-label macro are suspected causes of this issue. The intention is to make some small change and save, forcing the page to be re-written and saved.

This stops the content indexing queue to increase on page reload alone and prevents further expansion of the indexing queue by this page. Continue editing the page's various 3rd party plugin sections until a reload of the page no longer causes the queue to increase. 

At that point you can either let the indexing queue process completely to clear it out or you can re-index the index from scratch to clear the queue. This will trigger a full reindex upon its next start and ensures that any recent updates are included in the index and not lost when you clear the indexing queue as a result of performing the re-index from scratch steps. 


Last modified on Jul 10, 2021

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.