Symptoms

After an upgrade to Confluence 5.2 or later, the instance is experiencing performance degradation because of the indexing process.

Diagnosis

The same index thread has been running for more than an hour. In the instance's thread dump, the following stack would be found in a thread being run for more than an hour:

#011at com.atlassian.confluence.search.lucene.tasks.BulkWriteIndexTask.perform(BulkWriteIndexTask.java:44)
#011at com.atlassian.bonnie.LuceneConnection.withWriter(LuceneConnection.java:499)
#011at com.atlassian.confluence.search.lucene.tasks.LuceneConnectionBackedIndexTaskPerformer.perform(LuceneConnectionBackedIndexTaskPerformer.java:20)
#011at com.atlassian.confluence.search.lucene.DefaultConfluenceIndexManager$BatchUpdateAction.perform(DefaultConfluenceIndexManager.java:438)
#011at com.atlassian.bonnie.LuceneConnection.withBatchUpdate(LuceneConnection.java:532)
#011at com.atlassian.confluence.search.lucene.DefaultConfluenceIndexManager.processTasks(DefaultConfluenceIndexManager.java:214)
#011at com.atlassian.confluence.search.lucene.DefaultConfluenceIndexManager.flushQueue(DefaultConfluenceIndexManager.java:166)

Cause

When upgrading to Confluence 5.2 or later versions, an upgrade task is run in order to re-index the user information. User items are added to the indexing queue and processed 1500 items per batch.

If the instance has indexing 25,000 users or more, and then a queue flush is triggered (manually or by a feature), all the content in the queue will be added to a single transaction causing the instance to have its performance degraded.

Features that flush the indexing queue:

1. Manually triggering a flush from the UI
2. Space deletion/removal

Resolution

The processing of the index queue items need to be reverted so that the tasks are processed in batches. Two scenarios:

If any content was created or updated since the flush was triggered (or if you're unsure)

1. Restart Confluence. Note that this will purge all the items which were being processed in the queue

2. Perform a full re-index. This is necessary to get the index into a consistent state

If *no* content was created or updated since the flush was triggered

1. Restart Confluence. Note that this will purge all the items which were being processed in the queue

2. Manually schedule the UserIndexingUpgradeTask (<baseurl>/admin/force-upgrade.action and select 'userIndexingUpgradeTask'). This will re-run the upgrade task but the items will be processed in batches

3. Do not perform any actions which will trigger a flushing of the queue (e.g. space deletion)