Managing internode latency in Jira Data Center

This insight is shown when your Jira Data Center cluster's internode latency exceeds 10ms.

What is internode latency?

Internode latency measures the communication speed between nodes in your Jira Data Center cluster. It is measured by the node.latency.statistics metric, which represents the time in milliseconds it takes to send a cache invalidation message to other nodes through RMI (Remote Method Invocation). This metric provides a basic measure of internode communication latency within your Jira Data Center cluster.

How does internode latency affect performance?

High internode latency can impact several critical aspects of your Jira Data Center performance:

• Cache replication: Slower synchronization of cached data across nodes

• Document-based replication (DBR): Delayed consistency of index data between nodes

• Cluster consistency: Increased time for changes to propagate across the cluster

• Overall system performance: Potential slowdowns in user operations and background tasks

For more details on how internode latency affects performance, read Interpreting cross-product metrics for in-product diagnostics.

While your cluster can still function with higher latency, it may lead to degraded performance and inconsistent user experiences across nodes.

What's the recommendation?

You should aim to reduce the internode latency to below 10ms to ensure optimal performance and consistency across your Jira Data Center cluster. If you're unable to achieve this, you should at least try to keep it below 50ms to avoid severe performance degradation.

Here are actions you can take:

• Measure and verify internode latency

• Optimize network connectivity

• Monitor and tune JVM settings

• Review and adjust cluster configuration

• Engage Atlassian Support for persistent issues

Measuring and verifying internode latency

TWhen viewing this recommendation from the dashboard, you can use basic network tools to measure the current internode latency across your cluster. For detailed instructions on how to use these tools, refer to live monitoring Using the JMX Interface.

To measure internode latency:

1. Open a terminal or command prompt on one of your Jira nodes.

2. Use the ping command to estimate network latency:

   ping -c 10 <other_node_ip_or_hostname>

   Replace <other_node_ip_or_hostname> with the IP address or hostname of another node in your cluster.

3. Use netcat to confirm network connectivity:

   netcat -zv <other_node_ip_or_hostname> <listening_port>

   Replace <listening_port> with the port used for internode communication (typically 40001 by default).

4. Repeat these tests from each node to every other node in your cluster.

To find the correct <other_node_ip_or_hostname> and <listening_port>:

• Check the cluster.properties file on each node for ehcache.listener.hostName and ehcache.listener.port arguments.

• If these aren't defined, Jira Data Center sets them automatically. You can verify the current values using this SQL query:

  SELECT * FROM clusternode;

Note: These tests only confirm immediate latency/connectivity and may not identify intermittent network issues. Engage your network administration team for more comprehensive monitoring.

Optimizing network connectivity

If you've identified high internode latency, consider these steps:

1. Review your network infrastructure for potential bottlenecks

2. Ensure all nodes are on the same local network, if possible

3. Check for any network devices that might be introducing latency

4. Consider upgrading network hardware if necessary

5. Engage your network administration team to verify and improve the network infrastructure

Monitoring and tuning JVM settings

JVM settings can impact internode communication. Consider reviewing and adjusting:

• Garbage collection settings

• Heap size

• Thread pool configurations

To adjust JVM settings:

1. Locate the setenv.sh (Linux/Mac) or setenv.bat (Windows) file in your Jira installation directory.

2. Edit the file to include or modify JVM arguments.

3. Restart Jira after making changes.

Learn more about monitoring Jira with Prometheus and Grafana

Additional factors affecting internode latency

While network latency is the primary factor, other elements can contribute to increased internode latency:

• Large volumes of data being transferred due to frequent cache invalidations or index updates

• High CPU utilization on individual nodes

• Disk I/O bottlenecks affecting response times

These factors are typically less common but may need investigation in persistent high-latency scenarios.

Adjusting cluster configuration

In some cases, you may need to adjust your cluster configuration:

• Review load balancer settings

• Check Ehcache configuration

• Optimize the number of nodes

• Verify cluster node specifications

Learn more on Jira data center performance.

Engaging Atlassian Support

If you've tried the above recommendations and still experience high internode latency, contact Atlassian Support. When creating a support ticket, include:

• Support Zip from all nodes in your cluster

• Results of your ping and netcat tests

• Any feedback or analysis from your network administration team

• JVM garbage collection logs

• Recent changes to your Jira configuration or infrastructure