Troubleshooting New Ranking System Issues

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform Notice: Server and Data Center Only - This article only applies to Atlassian products on the server and data center platforms.

Purpose

In Agile 6.4 and above, the ranking system has been changed to support cluster-safe rank operations on JIRA Data Center. The new rank system (often called 'LexoRank') is stored in a lexicographical order of strings to determine the ranking of issues.

See also related KB Understand the LexoRank managment page in Jira server

In the course of this article, some specific terms will be used:

  • Re-balance job

    This is a plugin job to maintain the integrity of the ranking table. The JIRA Agile plugin should run this job automatically. This is a background job, which means it should not interfere with the user operation.

  • Rank bucket

    The new ranking system utilizes 3 'buckets' (0, 1, 2) to move issues around during the re-balance job. The bucket is indicated by the number prefix before the pipeline character ('|') in the Rank field value:

    "0|...." indicates ranking is stored in bucket 0"1|..." indicates bucket 1, and so on.

Solution

  1. Check the database collation and encoding setting, as per Configuring Database Character Encoding.
  2. Ensure you are running on the latest Agile version.
  3. Run Rank Integrity Checker by accessing https://<jira-base-url>/rest/greenhopper/1.0/lexorank/integrity (perform a GET, this is done if you access the URL in the browser).

    Click here to expand for sample integrity results...

    {

      "reports": [
        {
          "rankFieldId": 15201,
          "rankFieldName": "Global Rank",
          "results": [
            {
              "name": "Marker rows present in table for rank field",
              "description": "Checks if the rank table has been properly initialized for the rank field. A minimum and maximum marker row are expected to be present in the table for the rank field.",
              "passed": true,
              "fatal": true
            },
            {
              "name": "Marker rows correctness check",
              "description": "Checks whether the marker rows for a rank field have the expected rank value.",
              "passed": true,
              "fatal": true
            },
            {
              "name": "Marker rows in valid bucket check.",
              "description": "Checks if the marker rows of a rank field are in valid bucket(s).",
              "passed": true,
              "fatal": true
            },
            {
              "name": "Rank out of bounds check",
              "description": "Checks if there is a rank value for the rank field that is out of bounds.",
              "passed": true,
              "fatal": true
            },
            {
              "name": "Duplicate ranks check",
              "description": "Checks if there are any duplicate rank values for a rank field.",
              "passed": false,
              "failureReason": "Detected duplicate ranks for rank field 15201 : {0|11t9b4:=2, 0|11sqnc:=2, 0|11suow:=2, 0|11tm0w:=2, 0|11thk8:=2, 0|11spog:=2, 0|11tes0:=2, 0|11sx00:=2, 0|11t2qg:=2, 0|11sqx4:=2, 0|11tde8:=2, 0|11t9zc:=2, 0|11sxcw:=2, 0|11ssi0:=2, 0|11teqo:=2, 0|11tlm0:=2, 0|11t140:=2, 0|11tbp4:=2, 0|11tg5s:=2, 0|11srhs:=2, 0|11t03s:=2, 0|11t4bc:=2, 0|11taow:=2, 0|11td48:=2}",
              "fatal": false
            },
            {
              "name": "Issue ranks different from marker ranks check",
              "description": "Checks if there are any issue ranks that have the same rank as the maximum or minimum rank.",
              "passed": true,
              "fatal": true
            },
            {
              "name": "Issue rows in valid bucket check.",
              "description": "Checks if the issue rows of a rank field are in a valid bucket. Issue rows should be in one of the buckets the marker rows are in.",
              "passed": true,
              "fatal": false
            },
            {
              "name": "Balance status check",
              "description": "Checks if there is a correct balance entry is present or absent for the rank field.",
              "passed": true,
              "fatal": false
            }
          ]
        }
  4. Depending on which test is failing, check with the KB article section down below.

  5. If you are in doubt, please follow sending information to support.

Sending Information to Support

Please raise a case at https://support.atlassian.com and provide the following information:

  1. Generate JIRA support zip.
  2. The result of rank integrity checks.
  3. An XML backup of your database (Administration > System > Backup System). You can anonymise if you wish by going as per our Anonymising JIRA Data page.
  4. If attaching a backup is not possible, please provide us with the result of the following SQL (preferably in a CSV file format).

    SELECT * FROM "AO_60DB71_LEXORANK" ORDER BY "RANK";

    (info) Depending upon the DBMS used, the tablename may be different. For example AO_60DB71_LEXORANK instead of "AO_60DB71_LEXORANK".

Enable Additional Logging for LexoRank

By default, the logging of ranking system is disabled. To get detailed logging of what the system is doing add a new logger to JIRA's Logging and Profiling page

  1. Logging for balancing

    level: DEBUG
    package: com.atlassian.greenhopper.service.lexorank.balance
  2. Logging for ranking

    level: DEBUG
    package: com.atlassian.greenhopper.service.lexorank
  3. Logging for ranking DB operations
    level: DEBUG
    package: com.atlassian.greenhopper.manager.lexorank
  4. Logging for statistics: every minute logs the activity and performance of the ranking system
    level: DEBUG
    package: com.atlassian.greenhopper.service.lexorank.LexoRankStatisticsAgent

List of Lexorank errors

See related KB List of all known Agile Lexorank error

FAQ

How to check if a re-balance is scheduled?

When a re-balance is scheduled, the AO_60DB71_LEXORANKBALANCER table in the database should contain an entry for each Rank field.

The REBALANCE_TIME column will contain the time when the operation is scheduled to start in Epoch time (aka Unix time).

How to check if a re-balance is running?

Accessing https://<jira-base-url>/rest/greenhopper/1.0/lexorank/balance through the browser will return information on the status of the balancer. This will show you whether or not it's running, and also a % status and the distribution of the buckets.

Click here to expand for sample integrity results...

{

  "lexoRankBalancingServiceStatus": {
    "balancingDisabled": false,
    "balanceHandlerRunning": false
  },
  "lexoRankBalancerStatus": {
    "balancerLocked": false,
    "perFieldStatus": [
      {
        "fieldName": "Rank",
        "fieldId": 10105,
        "numRankedIssues": 1500,
        "percentComplete": 100,
        "distribution": [
          1500,
          0,
          0
        ]
      }
    ]
  }
}

A re-balance will be running when the following conditions are met:

  1. There is an entry in the AO_60DB71_LEXORANKBALANCER table in the database, scheduled for a time which has already passed.
  2. The query below should return two rows. They must be on different buckets during a rebalance, on one the following bucket combinations: 0 and 1; 1 and 2; 2 and 0

    Once a rebalance operation is completed, this query will return two ranks on the same bucket (e.g. "1|000000:" and "1|zzzzzz:").

SELECT "RANK" FROM "AO_60DB71_LEXORANK" WHERE "TYPE" <> 1 ORDER BY "RANK";

When is a re-balance job is triggered or required? 

  1. When the the ranking value splits in 2 different buckets.
  2. The number of characters on the lengthiest rank field on the table is what defines the need for a rebalance operation, and here are the thresholds:

    1. 50 characters: a rebalance operation is scheduled to occur in 12 hours;

    2. 100 characters: a rebalance operation is immediately started;

    3. Issues with > 254 characters RANK can't be ranked.

How to check lengthiest rank field:

SELECT "FIELD_ID", max(length("RANK")) AS max_rank_string_length FROM "AO_60DB71_LEXORANK" GROUP BY "FIELD_ID";

Check issues with long lexorank

SELECT l."FIELD_ID",l."ISSUE_ID",j.project,j.issuenum, length(l."RANK") AS rank_length FROM "AO_60DB71_LEXORANK" l join jiraissue j on l."ISSUE_ID"=j.ID where length(l."RANK") > 250;

What happens if we stop JIRA while a re-balance is happening?

Whenever the JIRA Agile plugin is stopped, JIRA is stopped or a foreground re-index is triggered, the LexoRank scheduler job will stop any balance job that is running. Then the actual plugin scheduler job is stopped by the plugin system. Once the plugin is re-enabled, the balance job will start again and pick up from where it left. It will check the AO_60DB71_LEXORANKBALANCER table again and see that there are still entries for balance job to be run and start going through them 1 by 1. Once a balance job is completed, the entry in the table will be removed.

Can issue be ranked while re-balancing is running? 

Before: 7.2.8

If max RANK length exceed 200 characters: a rebalance operation is immediately started and all ranking operations are disabled for given field. 

JIRA 7.2.8+

If max RANK length doesn't exceed 254 characters: all issues can be ranked during re-balancing. For issues which target RANK length exceeds 254 characters, ranking operation will not be possible until re-balancing normalises the RANK length. 

See related: 

What happens when issue rank exceed 254 characters

tip/resting Created with Sketch.

JIRA 7.2.8+ only

For issues which future allocated RANK length exceeds 254 characters during ranking operation, that operation will not be possible until RANK length gets back to normal. 

Eg: IssueA has RANK 254 and IssueB - 30, if you try to position IssueB after IssueA that will fail. Positioning IssueA before IssueB will work. 

In case of operation failed, you will get the following error in the logs:

[c.a.g.customfield.lexorank.LexoRankCFType] gh.api.rank.error.lexorank.fieldlength.exceeded.norebalance
 
or
 
2018-02-12 14:36:21,726 NodeReindexServiceThread:thread-1 WARN      [c.a.g.customfield.lexorank.LexoRankCFType] Unable to retrieve rank for field [10600] and issue [2341873]
2018-02-12 14:36:21,726 NodeReindexServiceThread:thread-1 WARN      [c.a.g.customfield.lexorank.LexoRankCFType] gh.api.rank.error.lexorank.fieldlength.exceeded.norebalance


Can JIRA Agile be updated while a re-balance is happening?

Yes. However, we highly recommend taking a backup from the JIRA instance prior to updating the plugin. Additionally, updating JIRA Agile might not necessarily fix existing inconsistencies in the LexoRank system in case it is already corrupted. In such cases, please report a support ticket providing the information requested on the Sending Information to Support section.

DescriptionIn Agile 6.4 and above, the ranking system has been changed to support cluster-safe rank operations on JIRA Data Center. The new rank system (often called 'LexoRank') is stored in a lexicographical order of strings to determine the ranking of issues.
ProductJira
PlatformServer
Last modified on Nov 14, 2018

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.