Remove abandoned or offline nodes in JIRA Data Center

Still need help?

The Atlassian Community is here for you.

Ask the community

The nodes that show as active but not alive (that is with no heartbeat) or offline can clutter up your cluster information. In Jira 8.9 we improved the way cluster information is displayed however, you still needed to manually remove stale or offline nodes from the cluster.

That behavior changed in Jira 8.10 and later where stale nodes are automatically removed. (tick)

Pre-Jira 8.9Jira 8.9 and later


Removing an offline node 

If you run Jira 7.13 and later you do not need to stop or restart all running nodes to remove an offline one. It takes a few seconds for the running nodes to respond to changes introduced with REST or using SQL directly on the database. 

Remove offline nodes using REST (for Jira 8.1 and later)

  1. Make the following request for Jira to return a list of all nodes:

GET /rest/api/2/cluster/nodes

Example response...
[
 {
  "nodeId": "node3",
  "state": "ACTIVE",
  "lastStateChangeTimestamp": 1551080754927,
  "ip": "localhost",
  "cacheListenerPort": 40001,
  "nodeBuildNumber": 801000,
  "nodeVersion": "8.1.0-SNAPSHOT",
  "alive": true
 },{
  "nodeId": "node1",
  "state": "OFFLINE",
  "lastStateChangeTimestamp": 1551080840469,
  "ip": "localhost",
  "cacheListenerPort": 40001,
  "nodeBuildNumber": 801000,
  "nodeVersion": "8.1.0-SNAPSHOT",
  "alive": false
 }
]

2. If you have nodes showing as deleted, delete each offline node from the cluster
DELETE /rest/api/2/cluster/node/{your_offline_node_Id}

3. Double check that the offline node is now deleted and does not show up

GET /rest/api/2/cluster/nodes

For more on these REST API calls, see Jira REST API

Remove offline nodes with database queries

  1.  (optional) Double-check the number of offline nodes by running the command: 
select NODE_ID from  clusternode where NODE_STATE ='offline'; 

2. Copy the node_id  of the offline node and run the following command to delete it:

delete from clusternode where node_id=<id from first query>;


Removing a stale node with no heartbeat

Nodes might show up active but in fact, have no heartbeat if they have been killed abruptly. We advise stopping Jira gracefully to avoid that situation. 

How do I stop Jira gracefully?

Before you remove a VM or AWS instance from the Data Center cluster:

  1. Shut down Jira by using the stop-jira.sh script in <JIRA_INSTALL>/bin or stopping its service.
  2. Then kill the AWS instance or power off the VM. 
If you want to remove a stale node you need to stop Jira on all cluster nodes first. 

You can either remove a stale node using REST API (available for Jira 8.1 and later) or do it manually. 


Remove stale nodes using REST (for Jira 8.1 and later)

  1. Make the following request for Jira to return a list of all nodes:
    GET /rest/api/2/cluster/nodes

    Example response...
    [
     {
      "nodeId": "node3",
      "state": "ACTIVE",
      "lastStateChangeTimestamp": 1551080754927,
      "ip": "localhost",
      "cacheListenerPort": 40001,
      "nodeBuildNumber": 801000,
      "nodeVersion": "8.1.0-SNAPSHOT",
      "alive": true
     },{
      "nodeId": "node1",
      "state": "ACTIVE",
      "lastStateChangeTimestamp": 1551080840469,
      "ip": "localhost",
      "cacheListenerPort": 40001,
      "nodeBuildNumber": 801000,
      "nodeVersion": "8.1.0-SNAPSHOT",
      "alive": false
     }
    ]
  2. Copy the nodeId's of the nodes that are "ACTIVE" and "alive"=false. 
  3. Ping the nodes to make sure they are not running in your environment and will not back to the cluster in the future.
  4. Make the following request to move the stale node to the offline state
    PUT /rest/api/2/cluster/node/{your_stale_node_Id}/offline
  5. Run GET /rest/api/2/cluster/nodes to make sure the node is offline. 
  6. Delete the stale node from the cluster:
    DELETE /rest/api/2/cluster/node/{your_stale_node_Id}
  7. Double check that the stale node is now deleted and does not show up:
    GET /rest/api/2/cluster/nodes

    For more on these REST API calls, see Jira REST API

Remove old nodes with database queries

  1. Check the database tables and find all rows related to old nodes:

    select * from clusternode;
    select * from clusternodeheartbeat;
  2. Copy the node_id of the stale node and run the command:

    delete from clusternode where node_id = '<node_id>';
    delete from clusternodeheartbeat where node_id = '<node_id>';
  3. Clean old replication records:

    // validate if this is necessary 
    select count(id) from replicatedindexoperation where node_id = '<node_id>';
    // delete
    delete from replicatedindexoperation where node_id = '<node_id>';


    Related tickets

Last modified on Dec 30, 2020

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.