Problem

Reindexing progress appears to be stuck at a number less than 100%.

Diagnosis

• When rebuilding Confluence's indexes from the scratch:
  • Confluence's logs will show the progress of reindexing (check the atlassian-confluence-index.log file; in case it doesn't exist, review the atlassian-confluence.log)
  • accessing the <Confluence-BaseURL>/json/reindextaskprogress.action endpoint will display the progress of reindexing

• When rebuilding Confluence's indexes from the UI:
  • navigate to Confluence cog icon >> General Configuration >> Content Indexing page to review the progress of reindexing

In all scenarios, the progress tracker may appear to be stuck at a number less than 100% and won't display new information even after sufficiently long time intervals.

Cause

The total count of indexable content is obtained by executing SQL on the Confluence database.
The count of actually indexed content can fall short of that total due to errors/exceptions encountered for specific data during the reindexing process.

The following bugs exists for this behavior:
CONFSERVER-81283 - Getting issue details... STATUS  

CONFSERVER-81685 - Getting issue details... STATUS

Workaround

To find out whether reindexing is still in progress (without taking into account whether it finished successfully or was abnormally terminated by say an out-of-memory error), check for the following evidence:

• indirect evidence (optional check)
  Is the outcome of regularly scheduled incremental/background indexing by one of the Caesium threads still being reported as ABORTED with the message Flushing or indexing is already in progress:

  atlassian-confluence.log showing scheduled job LuceneIndexFlusher ABORTED because "Flushing or indexing is already in progress"

  YYYY-MM-DD HH:MM:SS,mmm WARN [Caesium-1-2] [impl.schedule.caesium.JobRunnerWrapper] runJob Scheduled job LuceneIndexFlusher#LuceneIndexFlusher completed unsuccessfully with response JobRunnerResponse[runOutcome=ABORTED,message='Flushing or indexing is already in progress']

  
  and
  
  

• direct evidence (mandatory check)
  Do the latest thread dumps show the presence of threads named Indexer: N:
  1. extract thread dumps,

  2. navigate to the directory containing the extracted thread dumps, and search for the pattern "Indexer: [one or more digits]":

     linux sample

     # listing the extracted thread dump files
     ➜ ls
     thread_dump_1.txt
     thread_dump_2.txt
     thread_dump_3.txt
     thread_dump_4.txt
     thread_dump_5.txt
     thread_dump_6.txt
     
     # grepping the above thread dumps for the presence of "Indexer" threads
     
     
     
     # -o only match
     # -h suppress printing of filenames
     # -a process binary files as text
     # -i ignore case
     # --include to only search files that have "thread" in the filename
     
     
     
     ➜ egrep -ohai --include="thread*" 'Indexer: [[:digit:]]{1,}' | sort -nk2 | uniq
     Indexer: 1
     Indexer: 2
     Indexer: 3
     Indexer: 4

If the data collected matches the samples listed above, then reindexing is still in progress, because:

• Caesium based incremental/background indexing jobs will cease execution if reindexing is already in progress, and
• Indexer: N threads only appear during reindexing attempts.

For situations like these where Indexer: N are still active, and reindexing appears to be abnormally slow (or appears to have stalled), thread dumps and support zip(s) (from the affected instance) will provide useful information on the state of the Indexer threads and application processing.
Please gather that data and contact Atlassian Support if additional assistance is needed.

Resolution

Please upgrade Confluence to version 7.19.6 (or above) which contains the fixes for the following bugs:

CONFSERVER-81283 - Getting issue details... STATUS  

CONFSERVER-81685 - Getting issue details... STATUS