Purpose

In Agile 6.4 and above, the ranking system has been changed to support cluster-safe rank operations on JIRA Data Center. The new rank system (often called 'LexoRank') is stored in a lexicographical order of strings to determine the ranking of issues.

See also related KB Understand the LexoRank managment page in Jira server

In the course of this article, some specific terms will be used:

• Re-balance job

  This is a plugin job to maintain the integrity of the ranking table. The JIRA Agile plugin should run this job automatically. This is a background job, which means it should not interfere with the user operation.

• Rank bucket

  The new ranking system utilizes 3 'buckets' (0, 1, 2) to move issues around during the re-balance job. The bucket is indicated by the number prefix before the pipeline character ('|') in the Rank field value:

  "0|...." indicates ranking is stored in bucket 0, "1|..." indicates bucket 1, and so on.

Solution

1. Check the database collation and encoding setting, as per Configuring Database Character Encoding.
2. Ensure you are running on the latest Agile version.

3. Run Rank Integrity Checker by accessing https://<jira-base-url>/rest/greenhopper/1.0/lexorank/integrity (perform a GET, this is done if you access the URL in the browser).

   Click here to expand for sample integrity results...

   {

     "reports": [
       {
         "rankFieldId": 15201,
         "rankFieldName": "Global Rank",
         "results": [
           {
             "name": "Marker rows present in table for rank field",
             "description": "Checks if the rank table has been properly initialized for the rank field. A minimum and maximum marker row are expected to be present in the table for the rank field.",
             "passed": true,
             "fatal": true
           },
           {
             "name": "Marker rows correctness check",
             "description": "Checks whether the marker rows for a rank field have the expected rank value.",
             "passed": true,
             "fatal": true
           },
           {
             "name": "Marker rows in valid bucket check.",
             "description": "Checks if the marker rows of a rank field are in valid bucket(s).",
             "passed": true,
             "fatal": true
           },
           {
             "name": "Rank out of bounds check",
             "description": "Checks if there is a rank value for the rank field that is out of bounds.",
             "passed": true,
             "fatal": true
           },
           {
             "name": "Duplicate ranks check",
             "description": "Checks if there are any duplicate rank values for a rank field.",
             "passed": false,
             "failureReason": "Detected duplicate ranks for rank field 15201 : {0|11t9b4:=2, 0|11sqnc:=2, 0|11suow:=2, 0|11tm0w:=2, 0|11thk8:=2, 0|11spog:=2, 0|11tes0:=2, 0|11sx00:=2, 0|11t2qg:=2, 0|11sqx4:=2, 0|11tde8:=2, 0|11t9zc:=2, 0|11sxcw:=2, 0|11ssi0:=2, 0|11teqo:=2, 0|11tlm0:=2, 0|11t140:=2, 0|11tbp4:=2, 0|11tg5s:=2, 0|11srhs:=2, 0|11t03s:=2, 0|11t4bc:=2, 0|11taow:=2, 0|11td48:=2}",
             "fatal": false
           },
           {
             "name": "Issue ranks different from marker ranks check",
             "description": "Checks if there are any issue ranks that have the same rank as the maximum or minimum rank.",
             "passed": true,
             "fatal": true
           },
           {
             "name": "Issue rows in valid bucket check.",
             "description": "Checks if the issue rows of a rank field are in a valid bucket. Issue rows should be in one of the buckets the marker rows are in.",
             "passed": true,
             "fatal": false
           },
           {
             "name": "Balance status check",
             "description": "Checks if there is a correct balance entry is present or absent for the rank field.",
             "passed": true,
             "fatal": false
           }
         ]
       }

4. Depending on which test is failing, check with the KB article section down below.

5. If you are in doubt, please follow sending information to support.

Enable Additional Logging for LexoRank

By default, the logging of ranking system is disabled. To get detailed logging of what the system is doing add a new logger to JIRA's Logging and Profiling page

1. Logging for balancing

   level: DEBUG
   package: com.atlassian.greenhopper.service.lexorank.balance

2. Logging for ranking

   level: DEBUG
   package: com.atlassian.greenhopper.service.lexorank
3. Logging for ranking DB operations
   level: DEBUG
   package: com.atlassian.greenhopper.manager.lexorank
4. Logging for statistics: every minute logs the activity and performance of the ranking system
   level: DEBUG
   package: com.atlassian.greenhopper.service.lexorank.LexoRankStatisticsAgent

List of Lexorank errors

See related KB List of all known Agile Lexorank error

FAQ

How to check if a re-balance is scheduled?

When a re-balance is scheduled, the AO_60DB71_LEXORANKBALANCER table in the database should contain an entry for each Rank field.

The REBALANCE_TIME column will contain the time when the operation is scheduled to start in Epoch time (aka Unix time).

How to check if a re-balance is running?

Accessing https://<jira-base-url>/rest/greenhopper/1.0/lexorank/balance through the browser will return information on the status of the balancer. This will show you whether or not it's running, and also a % status and the distribution of the buckets.

  "lexoRankBalancingServiceStatus": {
    "balancingDisabled": false,
    "balanceHandlerRunning": false
  },
  "lexoRankBalancerStatus": {
    "balancerLocked": false,
    "perFieldStatus": [
      {
        "fieldName": "Rank",
        "fieldId": 10105,
        "numRankedIssues": 1500,
        "percentComplete": 100,
        "distribution": [
          1500,
          0,
          0
        ]
      }
    ]
  }
}

A re-balance will be running when the following conditions are met:

1. There is an entry in the AO_60DB71_LEXORANKBALANCER table in the database, scheduled for a time which has already passed.

2. The query below should return two rows. They must be on different buckets during a rebalance, on one the following bucket combinations: 0 and 1; 1 and 2; 2 and 0

   Once a rebalance operation is completed, this query will return two ranks on the same bucket (e.g. "1|000000:" and "1|zzzzzz:").

SELECT "RANK" FROM "AO_60DB71_LEXORANK" WHERE "TYPE" <> 1 ORDER BY "RANK";

When is a re-balance job is triggered or required? 

1. When the ranking value splits in 2 different buckets.

2. The number of characters on the lengthiest rank field on the table is what defines the need for a rebalance operation, and here are the thresholds:

   1. 128 characters: a rebalance operation is scheduled to occur in 12 hours;

   2. 160 characters: a rebalance operation is immediately started;

   3. Issues with > 254 characters RANK can't be ranked.

How to check lengthiest rank field:

SELECT "FIELD_ID", max(length("RANK")) AS max_rank_string_length FROM "AO_60DB71_LEXORANK" GROUP BY "FIELD_ID";

Check issues with long lexorank

SELECT l."FIELD_ID",l."ISSUE_ID",j.project,j.issuenum, length(l."RANK") AS rank_length FROM "AO_60DB71_LEXORANK" l join jiraissue j on l."ISSUE_ID"=j.ID where length(l."RANK") > 250;

What happens if we stop JIRA while a re-balance is happening?

Whenever the JIRA Agile plugin is stopped, JIRA is stopped or a foreground re-index is triggered, the LexoRank scheduler job will stop any balance job that is running. Then the actual plugin scheduler job is stopped by the plugin system. Once the plugin is re-enabled, the balance job will start again and pick up from where it left. It will check the AO_60DB71_LEXORANKBALANCER table again and see that there are still entries for balance job to be run and start going through them 1 by 1. Once a balance job is completed, the entry in the table will be removed.

Can issue be ranked while re-balancing is running? 

Before: 7.2.8

If max RANK length exceed 200 characters: a rebalance operation is immediately started and all ranking operations are disabled for given field. 

From JIRA 7.2.8 to 8.8.x: 

If max RANK length doesn't exceed 254 characters: all issues can be ranked during re-balancing. For issues which target RANK length exceeds 254 characters, ranking operation will not be possible until re-balancing normalises the RANK length. 

See related: 

• JIRA Software Data Center Lexorank Indexing Lag
• JSWSERVER-15712 - Getting issue details... STATUS

From JIRA 8.9.0 and upwards:

• If rank length is between 128 and 159 characters: a rebalance is scheduled for the next 12 hours.
• If rank length is between 160 and 253 characters: a rebalance starts immediately.
• If rank length equal to, or above 254 characters: a rebalance starts immediately, it is still possible to rank issues, but any ranking operations yielding values with more than 254 characters will still fail.

What happens when issue rank exceed 254 characters

For issues which future allocated RANK length exceeds 254 characters during ranking operation, that operation will not be possible until RANK length gets back to normal. 

Eg: IssueA has RANK 254 and IssueB - 30, if you try to position IssueB after IssueA that will fail. Positioning IssueA before IssueB will work. 

In case of operation failed, you will get the following error in the logs:

[c.a.g.customfield.lexorank.LexoRankCFType] gh.api.rank.error.lexorank.fieldlength.exceeded.norebalance
 
or
 
2018-02-12 14:36:21,726 NodeReindexServiceThread:thread-1 WARN      [c.a.g.customfield.lexorank.LexoRankCFType] Unable to retrieve rank for field [10600] and issue [2341873]
2018-02-12 14:36:21,726 NodeReindexServiceThread:thread-1 WARN      [c.a.g.customfield.lexorank.LexoRankCFType] gh.api.rank.error.lexorank.fieldlength.exceeded.norebalance

Can JIRA Agile be updated while a re-balance is happening?

Yes. However, we highly recommend taking a backup from the JIRA instance prior to updating the plugin. Additionally, updating JIRA Agile might not necessarily fix existing inconsistencies in the LexoRank system in case it is already corrupted. In such cases, please report a support ticket providing the information requested on the Sending Information to Support section.