Lexorank Rebalancing gets stuck with Error "Expected the row last migrated to be in the new bucket"

Related content

  • No related content found

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.

Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible

Symptoms

When rebalancing is running, it does not progress. There is no error thrown in the Lexorank Management page.


Diagnosis

  • Enable DEBUG for the rebalancing package  com.atlassian.greenhopper.service.lexorank.balance.  In the atlassian-jira.log, you'll see
2018-07-10 14:56:10,272 lexorank-executor-thread-0 DEBUG      [service.lexorank.balance.LexoRankBalancer] Locking lexorank balance with lock=com.atlassian.beehive.simple.SingleNodeLock@912d88d
2018-07-10 14:56:10,272 lexorank-executor-thread-0 INFO      [service.lexorank.balance.LexoRankBalancer] Balancing 1 fields
2018-07-10 14:56:10,272 lexorank-executor-thread-0 DEBUG      [service.lexorank.balance.LexoRankBalancer] Balancing field with id[13000]
2018-07-10 14:56:10,272 lexorank-executor-thread-0 DEBUG      [greenhopper.manager.lexorank.LexoRankDaoImpl]  LexoRank Query : SELECT "FIELD_ID","ID","ISSUE_ID","LOCK_HASH","LOCK_TIME","RANK","TYPE" FROM (SELECT "FIELD_ID","ID","ISSUE_ID","LOCK_HASH","LOCK_TIME","RANK","TYPE" FROM "AO_60DB71_LEXORANK" WHERE "FIELD_ID" = ? AND "TYPE" = ? ORDER BY "RANK" ASC) WHERE ROWNUM <= 1 params[13000, 0]
2018-07-10 14:56:10,272 lexorank-executor-thread-0 DEBUG      [greenhopper.manager.lexorank.LexoRankDaoImpl]  LexoRank Query : SELECT "FIELD_ID","ID","ISSUE_ID","LOCK_HASH","LOCK_TIME","RANK","TYPE" FROM (SELECT "FIELD_ID","ID","ISSUE_ID","LOCK_HASH","LOCK_TIME","RANK","TYPE" FROM "AO_60DB71_LEXORANK" WHERE "FIELD_ID" = ? AND "TYPE" = ? ORDER BY "RANK" DESC) WHERE ROWNUM <= 1 params[13000, 2]
2018-07-10 14:56:10,288 lexorank-executor-thread-0 DEBUG      [service.lexorank.balance.LexoRankBalancer] Balancing next rank row for rank field [id=13000]
2018-07-10 14:56:10,288 lexorank-executor-thread-0 DEBUG      [service.lexorank.balance.LexoRankBalanceOperation] Balancing next rank row to migrate from bucket[2] to bucket[0] for rank field[id=13000]
2018-07-10 14:56:10,288 lexorank-executor-thread-0 DEBUG      [greenhopper.manager.lexorank.LexoRankDaoImpl]  LexoRank Query : SELECT "FIELD_ID","ID","ISSUE_ID","LOCK_HASH","LOCK_TIME","RANK","TYPE" FROM (SELECT "FIELD_ID","ID","ISSUE_ID","LOCK_HASH","LOCK_TIME","RANK","TYPE" FROM "AO_60DB71_LEXORANK" WHERE "FIELD_ID" = ? AND "BUCKET" = ? ORDER BY "RANK" ASC) WHERE ROWNUM <= 1 params[13000, 2]
2018-07-10 14:56:10,288 lexorank-executor-thread-0 DEBUG      [greenhopper.manager.lexorank.LexoRankDaoImpl]  LexoRank Query : SELECT "FIELD_ID","ID","ISSUE_ID","LOCK_HASH","LOCK_TIME","RANK","TYPE" FROM (SELECT "FIELD_ID","ID","ISSUE_ID","LOCK_HASH","LOCK_TIME","RANK","TYPE" FROM "AO_60DB71_LEXORANK" WHERE "FIELD_ID" = ? AND "BUCKET" = ? ORDER BY "RANK" DESC) WHERE ROWNUM <= 1 params[13000, 0]
2018-07-10 14:56:10,288 lexorank-executor-thread-0 DEBUG      [service.lexorank.balance.LexoRankBalanceOperation] Fetched rows at balance boundary
2018-07-10 14:56:10,288 lexorank-executor-thread-0 DEBUG      [service.lexorank.balance.LexoRankBalanceOperation] Fetched rows at balance boundary
2018-07-10 14:56:10,288 lexorank-executor-thread-0 DEBUG      [service.lexorank.balance.LexoRankBalanceOperation]                rowToMigrate : LexoRankRow{id=81332, fieldId=13000, issueId=97286, lockHash='null', lockTime=null, bucket=2, rank='2|hztity:', type=ISSUE_RANK_ROW}
2018-07-10 14:56:10,288 lexorank-executor-thread-0 DEBUG      [service.lexorank.balance.LexoRankBalanceOperation]                 rowLastMigrated : LexoRankRow{id=69625, fieldId=13000, issueId=87413, lockHash='null', lockTime=null, bucket=2, rank='2|i0366b:', type=ISSUE_RANK_ROW}
2018-07-10 14:56:10,288 lexorank-executor-thread-0 ERROR      [service.lexorank.balance.LexoRankBalanceOperation] Expected the row last migrated to be in the new bucket [newBucket=0, rowBucket=2]
2018-07-10 14:56:10,288 lexorank-executor-thread-0 DEBUG      [service.lexorank.balance.LexoRankBalancer] Unlocking lexorank balance lock=com.atlassian.beehive.simple.SingleNodeLock@912d88d
2018-07-10 14:56:10,288 LexoRankReindexer:thread-1 ERROR      [service.lexorank.balance.LexoRankBalancer] Indexing for LexoRank re-balancing was interrupted.
2018-07-10 14:56:10,288 lexorank-executor-thread-0 INFO      [service.lexorank.balance.LexoRankBalancer] Balancing of 1 fields completed with 1 errors
2018-07-10 14:56:10,288 lexorank-executor-thread-0 WARN      [service.lexorank.balance.LexoRankScheduledBalanceHandler] Balance for fieldId=13000 returned errors: Errors: {}
Error Messages: [Expected the row last migrated to be in the new bucket]

  • SQL queries to verify the mismatch "BUCKET column" vs "Bucket in RANK", they all should return 0:

    select count(1) from "AO_60DB71_LEXORANK" where "FIELD_ID"=<RANK_FIELD_ID> and "BUCKET"=0 and "RANK" not like '0|%';
select count(1) from "AO_60DB71_LEXORANK" where "FIELD_ID"=<RANK_FIELD_ID> and "BUCKET"=1 and "RANK" not like '1|%';
select count(1) from "AO_60DB71_LEXORANK" where "FIELD_ID"=<RANK_FIELD_ID> and "BUCKET"=2 and "RANK" not like '2|%';

  • In case of a non-zero result of the previous query, in the AO_60DB71_LEXORANK table, you'll see a record as shown below (in this case, rank values like '2|%' are marked under bucket 0, instead of bucket 2);

FIELD_ID,     ID,        ISSUE_ID,     RANK,           TYPE,      BUCKET
13000,        69542,     87373,        2|hztjbb:f,     1,         0
13000,        69389,     87149,        2|hztjbb:p,     1,         0
13000,        69390,     87150,        2|hztjbb:s,     1,         0


Cause

There is currently no known cause why the data gets corrupted with this integrity issue.


Resolution

The resolution would be to fix the Bucket value in AO_60DB71_LEXORANK table.

  1. Generate JIRA backup
  2. Shutdown JIRA

  3. (warning) This is just an example, needs to be modified accordingly. Run the following SQL queries to update the bucket value to follow rank value (following the above example, we'll update the bucket value to 2 for those rank values like '2|%')

    UPDATE "AO_60DB71_LEXORANK" SET "BUCKET"=2 WHERE "FIELD_ID"=<RANK_FIELD_ID> AND "BUCKET"<>2 AND "RANK" like '2|%' ;

  4. Run the following SQL queries to verify that there is no corrupted entry in the AO_60DB71_LEXORANK table

    select count(1) from "AO_60DB71_LEXORANK" where "FIELD_ID"=<RANK_FIELD_ID> and "BUCKET"=0 and "RANK" not like '0|%';
select count(1) from "AO_60DB71_LEXORANK" where "FIELD_ID"=<RANK_FIELD_ID> and "BUCKET"=1 and "RANK" not like '1|%';
select count(1) from "AO_60DB71_LEXORANK" where "FIELD_ID"=<RANK_FIELD_ID> and "BUCKET"=2 and "RANK" not like '2|%';
  5. Trigger back the rebalancing, and wait for sometime and click refresh button. The rebalancing should be progressing.


Last modified on Jul 18, 2018

Was this helpful?

Yes
No
Provide feedback about this article

Related content

  • No related content found
Powered by Confluence and Scroll Viewport.