Index not working due to 'journal state store is corrupt' error

Still need help?

The Atlassian Community is here for you.

Ask the community

Problem

Search doesn't include recent pages, and the recently updated macro stops working.

The following appears in the atlassian-confluence.log:

WARN [http-bio-8090-exec-3] [confluence.impl.journal.DefaultJournalManager] enqueue Newly enqueued entry in journal 
[main_index] has an ID [X] that should have been higher than the journal state store's most-recent-id [Y]. it is 
likely that this node's journal state store is corrupt.

Cause

The journal is used to track changes made in Confluence. This is particularly useful in Data Center installations, in which each node checks this log index roughly every five seconds, and then  update its own index accordingly.

This issue occurs when there is an entry in the journal with an ID higher than what the journal is expecting. This can be due to a corruption of the Confluence index itself. Or, it can be an issue with your database setup.

Alternatively, if you are using an Oracle RAC database, you may be encountering this bug: CONF-35764 - Getting issue details... STATUS

Resolution 1

  1. Shutdown Confluence (on all nodes, if using Data Center)
  2. Move or rename journal directory located at <CONF_HOME_Dir>/
  3. Rebuild content indexes from scratch (on all nodes)
  4. Restart Confluence
  5. Rebuild the index from the Administration menu


Resolution 2: Change the next value for "seq_journal_entry_id" in PostgreSQL

In some cases, especially when migrating from one database to another the next sequence number can be lower than the last ID that is used for the index in the <confluence-home>/journal/main_index file. In that case, we should update the next sequence number to a value that is greater than the number in <confluence-home>/journal/main_index file.

Check the last value for 'seq_journal_entry_id' by using the below SQL query:

SELECT last_value FROM pg_catalog.pg_sequences WHERE sequencename='seq_journal_entry_id';


If the above value is lower than the value in <confluence-home>/journal/main_index file, follow the below steps: 

  1. Use the below query to ALTER the related view. Try to give a value (NEW_VALUE) that is higher than the value <confluence-home>/journal/main_index file. Preferably the value in main_index file +100. 
    ALTER SEQUENCE seq_journal_entry_id RESTART WITH <NEW_VALUE>;
  2. Rebuild the index from the Administration menu.

Resolution 3: Oracle RAC

Follow the workaround from CONF-35764 - Getting issue details... STATUS

  1. Change SEQ_JOURNAL_ENTRY_ID sequence to be ordered: ALTER SEQUENCE SEQ_JOURNAL_ENTRY_ID ORDER;
  2. Rebuild index.

Resolution 4: Postgres Sequence Cache

Postgres does not sequentially create IDs by default, so a sequence needs to be set up in the DB.  Postgres should create IDs 1 at a time to make sure they are sequential.  If your DB was somehow changed to create more than 1 at a time, it can start to create these non-sequentially.

To Confirm:

  1. Enable DEBUG logging in Confluence on the following packages:

    com.atlassian.bonnie.search.extractor
    com.atlassian.confluence.search.lucene
    com.atlassian.confluence.internal.index.AbstractBatchIndexer
    com.atlassian.confluence.impl.journal
  2. Make a few changes in confluence, such as editing several pages.
  3. Check the logs.  You should see each change logged with a journal entry ID:

    2019-07-29 07:40:46,967 DEBUG [http-nio-127.0.0.1-8090-exec-5] [confluence.impl.journal.DefaultJournalManager] enqueue Enqueued JournalEntry: JournalEntry{id=5936925, journalId='JournalIdentifier{journalName=edge_index}', creationDate=Mon Jul 29 07:40:46 MDT 2019, type='ADD_DOCUMENT', message='{"edgeId":"162070548","userKey":"1c0c8294582c611c015847a7c66e0032","targetId":162070548,"date":1564407548478,"typeKey":"page.create"}'}
    2019-07-29 07:40:51,108 DEBUG [http-nio-127.0.0.1-8090-exec-8] [confluence.impl.journal.DefaultJournalManager] enqueue Enqueued JournalEntry: JournalEntry{id=5936863, journalId='JournalIdentifier{journalName=main_index}', creationDate=Mon Jul 29 07:40:51 MDT 2019, type='ADD_CHANGE_DOCUMENT', message='com.atlassian.confluence.pages.Page-162070546'}
  4. In the logs above, you can see the first entry was given an ID of 5936925, and the second, 5936863.  This is not only non-sequential, but a big jump.
  5. Check the sequence in Postgres by exporting the schema/generating a DDL. Or run this command:

    Postgres versions below 10
    SELECT cache_value FROM seq_journal_entry_id;
    Postgres 10 and above
    SELECT cache_size FROM pg_catalog.pg_sequences
     WHERE sequencename='seq_journal_entry_id';
  6. The sequence should look like this:

    CREATE SEQUENCE public.seq_journal_entry_id
    START WITH 1
    INCREMENT BY 1
    NO MINVALUE
    NO MAXVALUE
    CACHE 1;
  7. If the CACHE value is anything other than 1, Postgres may generate IDs in the wrong order.

To Fix:

  1. Shut down Confluence
  2. Run the following query to update the Postgres Sequence:

    ALTER SEQUENCE public.seq_journal_entry_id CACHE 1;
  3. Follow the steps on rebuilding the index from scratch

  4. Re-enable the DEBUG logging mentioned above
  5. Verify that journal IDs are now created in sequential order.
Last modified on Feb 26, 2024

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.