Problem

Search doesn't include recent pages, and the recently updated macro stops working.

The following appears in the atlassian-confluence.log:

WARN [http-bio-8090-exec-3] [confluence.impl.journal.DefaultJournalManager] enqueue Newly enqueued entry in journal 
[main_index] has an ID [X] that should have been higher than the journal state store's most-recent-id [Y]. it is 
likely that this node's journal state store is corrupt.

Cause

The journal is used to track changes made in Confluence. This is particularly useful in Data Center installations, in which each node checks this log index roughly every five seconds, and then  update its own index accordingly.

This issue occurs when there is an entry in the journal with an ID higher than what the journal is expecting. This can be due to a corruption of the Confluence index itself. Or, it can be an issue with your database setup.

Alternatively, if you are using an Oracle RAC database, you may be encountering this bug: https://jirCONFSERVER-35764 - Index updates can get lost on Oracle RACa.atlassian.com/browse/CONFSERVER-35764.

Resolution 1: Change the next value for "seq_journal_entry_id" in PostgreSQL

In some cases, especially when migrating from one database to another the next sequence number can be lower than the last ID that is used for the index in the <confluence-home>/journal/main_index file. In that case, we should update the next sequence number to a value that is greater than the number in <confluence-home>/journal/main_index file.

Check the last value for 'seq_journal_entry_id' by using the below SQL query:

SELECT last_value FROM pg_catalog.pg_sequences WHERE sequencename='seq_journal_entry_id';

If the above value is lower than the value in <confluence-home>/journal/main_index file, follow the below steps: 

1. Access the database and run the following SQL to truncate the journalentry table and start a new sequence.

   TRUNCATE TABLE JOURNALENTRY;

2. Rebuild the index from the Administration menu.

Resolution 2: Oracle RAC

Follow the workaround from CONFSERVER-35764 - Index updates can get lost on Oracle RAC.

1. Change SEQ_JOURNAL_ENTRY_ID sequence to be ordered: ALTER SEQUENCE SEQ_JOURNAL_ENTRY_ID ORDER;
2. Rebuild index.

Resolution 3: Postgres Sequence Cache

Postgres does not sequentially create IDs by default, so a sequence needs to be set up in the DB.  Postgres should create IDs 1 at a time to make sure they are sequential.  If your DB was somehow changed to create more than 1 at a time, it can start to create these non-sequentially.

To Confirm:

1. Enable DEBUG logging in Confluence on the following packages:

   com.atlassian.bonnie.search.extractor
   com.atlassian.confluence.search.lucene
   com.atlassian.confluence.internal.index.AbstractBatchIndexer
   com.atlassian.confluence.impl.journal

2. Make a few changes in confluence, such as editing several pages.

3. Check the logs.  You should see each change logged with a journal entry ID:

   2019-07-29 07:40:46,967 DEBUG [http-nio-127.0.0.1-8090-exec-5] [confluence.impl.journal.DefaultJournalManager] enqueue Enqueued JournalEntry: JournalEntry{id=5936925, journalId='JournalIdentifier{journalName=edge_index}', creationDate=Mon Jul 29 07:40:46 MDT 2019, type='ADD_DOCUMENT', message='{"edgeId":"162070548","userKey":"1c0c8294582c611c015847a7c66e0032","targetId":162070548,"date":1564407548478,"typeKey":"page.create"}'}
   2019-07-29 07:40:51,108 DEBUG [http-nio-127.0.0.1-8090-exec-8] [confluence.impl.journal.DefaultJournalManager] enqueue Enqueued JournalEntry: JournalEntry{id=5936863, journalId='JournalIdentifier{journalName=main_index}', creationDate=Mon Jul 29 07:40:51 MDT 2019, type='ADD_CHANGE_DOCUMENT', message='com.atlassian.confluence.pages.Page-162070546'}

4. In the logs above, you can see the first entry was given an ID of 5936925, and the second, 5936863.  This is not only non-sequential, but a big jump.

5. Check the sequence in Postgres by exporting the schema/generating a DDL. Or run this command:

   Postgres versions below 10

   SELECT cache_value FROM seq_journal_entry_id;

   Postgres 10 and above

   SELECT cache_size FROM pg_catalog.pg_sequences
    WHERE sequencename='seq_journal_entry_id';

6. The sequence should look like this:

   CREATE SEQUENCE public.seq_journal_entry_id
   START WITH 1
   INCREMENT BY 1
   NO MINVALUE
   NO MAXVALUE
   CACHE 1;

7. If the CACHE value is anything other than 1, Postgres may generate IDs in the wrong order.

To Fix:

1. Shut down Confluence

2. Run the following query to update the Postgres Sequence:

   ALTER SEQUENCE public.seq_journal_entry_id CACHE 1;

3. Follow the steps on rebuilding the index from scratch

4. Re-enable the DEBUG logging mentioned above
5. Verify that journal IDs are now created in sequential order.