Managing Lucene index write latency in Jira Data Center

The lucene index write latency measures the average time taken for write operations to the local disk by the index writer. This metric is crucial for understanding the performance of data indexing and retrieval operations across your Jira Data Center cluster.

How does high Home local write latency affect Jira Data Center performance?

High lucene index write latency can significantly impact your Jira Data Center instance in several ways:

• Slower indexing of new or updated issues

• Delayed search results and reduced search performance

• Increased response times for operations involving data writes

• Potential inconsistencies in data across cluster nodes

• Overall degradation of system responsiveness

It's important to note that this metric is influenced not only by disk performance but also by the volume of updated documents. Sporadic high latency times may not always indicate disk performance issues.

What's the recommendation?

You should aim to keep the lucene index write latency below 50ms for optimal performance. If you're consistently seeing latencies above this threshold, consider the following actions:

• Monitor disk performance

• Check open files limit

• Review system configuration

• Optimize Lucene index configuration

Monitor disk performance

Regular monitoring of disk performance can help you identify patterns and potential issues before they become critical.

Steps to implement monitoring:

1. Set up JMX monitoring:

   • Enable JMX monitoring on all Jira Data Center nodes.

   • Configure monitoring tools to collect disk performance metrics, including write latency.

2. Analyze performance patterns:

   • Look for consistent high latency periods.

   • Identify any correlation with specific times or activities (e.g., bulk issue updates, reindexing operations).

3. Set up alerts:

   • Configure alerts for when write latency exceeds acceptable thresholds.

   • Ensure alerts are actionable and reach the appropriate team members.

Learn how to set up live monitoring using the JMX interface in Jira Data Center.

Check open files limit

A system approaching its open file limit can impact write performance and overall system stability.

Steps to check and adjust open files limit:

1. Check current open file usage:

   • Run the command: lsof +L1 > open_files.txt

   • Analyze the output to see how many files are currently open.

2. Verify system limits:

   • Check the current open files limit: ulimit -n

   • Review the system-wide limit in /etc/security/limits.conf

3. Adjust limits if necessary:

   • Increase the open files limit if it's close to the current usage.

   • Ensure the new limit is applied to the Jira service user.

4. Monitor and maintain:

   • Regularly check open file usage as part of your maintenance routine.

   • Set up alerts for when open file usage approaches the configured limit.

For more information on managing open file limits, see Health check:Open Files Limit.

Review system configuration

System-level configurations and software can interfere with Lucene write operations, impacting performance.

Steps to review system configuration:

1. Engage system administrators:

   • Involve your system administration team in the review process.

   • Provide them with the observed latency data and any patterns you've identified.

2. Review installed software:

   • Check for any recently installed or updated software that might be impacting disk performance.

   • Pay particular attention to antivirus software, backup solutions, and disk encryption tools.

3. Analyze system resources:

   • Review CPU, memory, and I/O usage across the cluster.

   • Look for any resource bottlenecks that might be impacting write performance.

4. Check for OS-level constraints:

   • Review any OS-level I/O scheduling or throttling mechanisms.

   • Ensure that Jira's processes are given appropriate priority.

More details on common causes for Jira Server crashes and performance issues.

Optimize Lucene index configuration

Proper configuration of Lucene indexing can help reduce write latency and improve overall system performance.

Steps to optimize Lucene index configuration:

1. Review current index settings:

   • Check the current index size and fragmentation level.

   • Review the frequency of index optimization operations.

2. Adjust index optimization schedule:

   • Configure index optimization to run during off-peak hours.

   • Consider more frequent, smaller optimization runs instead of less frequent, larger ones.

3. Fine-tune indexing parameters:

   • Adjust the index.queue.size parameter to balance between write performance and memory usage.

   • Consider enabling index.use.snappy.compression for better compression of index files.

4. Monitor and iterate:

   • Regularly review indexing performance after making changes.

   • Adjust settings based on observed performance and system load.

More on how to increase the speed of full reindex in Jira Server and Data Center

Additional considerations:

• Remember that the lucene index write latency metric is updated only when Lucene persists in the latest index changes. Sporadic high latency reports may not always indicate persistent performance issues.

• Consider the impact of cluster-wide operations (e.g., reindexing) on local write latency across all nodes.

• Regularly review and optimize Jira's database performance, as it can indirectly impact indexing performance.