Confluence Reindex Stuck at less than 100%
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Problem
Reindexing progress appears to be stuck at a number less than 100%.
Diagnosis
- When rebuilding Confluence's indexes from the scratch:
- Confluence's logs will show the progress of reindexing (check the
atlassian-confluence-index.log
file; in case it doesn't exist, review theatlassian-confluence.log
) - accessing the <Confluence-BaseURL>/json/reindextaskprogress.action endpoint will display the progress of reindexing
- Confluence's logs will show the progress of reindexing (check the
- When rebuilding Confluence's indexes from the UI:
- navigate to Confluence cog icon >> General Configuration >> Content Indexing page to review the progress of reindexing
In all scenarios, the progress tracker may appear to be stuck at a number less than 100% and won't display new information even after sufficiently long time intervals.
Cause
The total count of indexable content is obtained by executing SQL on the Confluence database.
The count of actually indexed content can fall short of that total due to errors/exceptions encountered for specific data during the reindexing process.
The following bugs exists for this behavior:
- CONFSERVER-81283 - Confluence reindexing progress appears to stall at less than 100% when rebuilding index from scratch
- CONFSERVER-81685 - Indexing : SQL Errors and lower percentage completion reported post reindex completion
Workaround
To find out whether reindexing is still in progress (without taking into account whether it finished successfully or was abnormally terminated by say an out-of-memory error), check for the following evidence:
indirect evidence (optional check)
Is the outcome of regularly scheduled incremental/background indexing by one of theCaesium
threads still being reported asABORTED
with the messageFlushing or indexing is already in progress
:atlassian-confluence.log showing scheduled job LuceneIndexFlusher ABORTED because "Flushing or indexing is already in progress"YYYY-MM-DD HH:MM:SS,mmm WARN [Caesium-1-2] [impl.schedule.caesium.JobRunnerWrapper] runJob Scheduled job LuceneIndexFlusher#LuceneIndexFlusher completed unsuccessfully with response JobRunnerResponse[runOutcome=ABORTED,message='Flushing or indexing is already in progress']
and- direct evidence (mandatory check)
Do the latest thread dumps show the presence of threads namedIndexer: N
:- extract thread dumps,
navigate to the directory containing the extracted thread dumps, and search for the pattern "
Indexer: [one or more digits]
":linux sample# listing the extracted thread dump files ➜ ls thread_dump_1.txt thread_dump_2.txt thread_dump_3.txt thread_dump_4.txt thread_dump_5.txt thread_dump_6.txt # grepping the above thread dumps for the presence of "Indexer" threads # -o only match # -h suppress printing of filenames # -a process binary files as text # -i ignore case # --include to only search files that have "thread" in the filename ➜ egrep -ohai --include="thread*" 'Indexer: [[:digit:]]{1,}' | sort -nk2 | uniq Indexer: 1 Indexer: 2 Indexer: 3 Indexer: 4
If the data collected matches the samples listed above, then reindexing is still in progress, because:
Caesium
based incremental/background indexing jobs will cease execution if reindexing is already in progress, andIndexer: N
threads only appear during reindexing attempts.
For situations like these where Indexer: N
are still active, and reindexing appears to be abnormally slow (or appears to have stalled), thread dumps and support zip(s) (from the affected instance) will provide useful information on the state of the Indexer
threads and application processing.
Please gather that data and contact Atlassian Support if additional assistance is needed.
Resolution
Please upgrade Confluence to version 7.19.6 (or above) which contains the fixes for the following bugs:
- CONFSERVER-81283 - Confluence reindexing progress appears to stall at less than 100% when rebuilding index from scratch
- CONFSERVER-81685 - Indexing : SQL Errors and lower percentage completion reported post reindex completion