Enterprise Support

This article explains how to resolve errors that appear in status health checks for Lucene and OpenSearch.

If the node has just started and the index is not ready, or if the index becomes inconsistent while the node is running, the status health check returns a 500 HTTP code and {"state":"ERROR"}. This response tells the load balancer not to route traffic to that node (Data Center only).

On this page:

What is search index consistency?
Resolving error state for Lucene
Resolving error state for OpenSearch
Disabling index consistency checking

You should also see this message in the logs:

WARN [c.a.j.issue.index.IndexConsistencyUtils] Index consistency check failed for index 'Issue': expectedCount=20; actualCount=5; tolerancePercentage=10, tolerance=10
INFO [c.a.j.search.lucene.LuceneConsistencyChecker] Index consistency check finished. indexConsistent: false; timeToCheckIndexConsistency: 76 ms
WARN [c.a.jira.servlet.ApplicationStateResolverImpl] The search index is inconsistent. This node will report its status as ERROR.

What is search index consistency?

Jira performs consistency checks to make sure the search index matches the current state of issues in the database. If Jira finds any inconsistencies, it reports an ERROR status to prevent serving incorrect search results.

The consistency check compares:

The number of issues in the database
The number of documents in the search index
The state of the index relative to recent changes

Resolve status health check returning 500 with error state

Below you’ll find guidance on how to solve this issue for both Lucene and OpenSearch search platforms.

Resolving error state for Lucene search platform

If your Jira instance is using Lucene as the search platform and the status health check returns a 500 with an ERROR state, start by waiting. This is often a temporary state. Jira will attempt to recover the index from a snapshot. If recovery is successful, the health check will return 200.

If the status does not change after a significant period (the duration depends on your instance size and configuration), review how the index is distributed between Jira nodes in the Jira Data Center search indexing documentation.

To resolve the issue:

Follow the steps in the Index Replication Jira Data Center troubleshooting guide.
If the problem persists, you can temporarily disable index consistency checking while you address the underlying issue.

Resolving error state for OpenSearch search platform

If your Jira instance is using OpenSearch as the search platform:

Check network connectivity:
1. Make sure the network connection between Jira and the OpenSearch cluster is stable.
2. Check firewall rules and monitor network latency.
Check OpenSearch cluster health:
1. Go to System, then Troubleshooting and support tools and verify that your OpenSearch cluster is healthy.
2. Make sure all nodes are accessible and working as expected.
3. Check OpenSearch logs for any errors or warnings

Verify index replay service:

Check if OpenSearch uses the index replay service to maintain consistency.
Go to System, then Scheduler administration to make sure the service is running as expected.
Review logs for any replay job failures.

Gather information about the index summary or, alternatively, use the script below instead of index_os.sh.

Show the script

The script gathers /index/summary response and displays it in a more visualised way. There are three modes:

Real-time mode with live monitoring every 10 seconds: ./index_os.sh
Manual mode that collects data to files for later analysis: ./index_os.sh -m
Analysis mode that analyzes historical trends from collected data: ./index_os.sh -a

#!/bin/bash
#
# A script to curl against IndexSummary REST API and parse the data to a readable format
# OpenSearch Enhanced Version
#
# Created: 2025-09-02
# Version: 2.0
#-----------
#
# Pre-requisite:
# - JQ and CURL
# - Command "index_os.sh -a" requires Bash 4+ (analysis mode only)
#   * If you have Bash 4+, run: bash index_os.sh -a
#   * To install Bash 4+: brew install bash (macOS) or apt-get install bash (Linux)
#   * Other modes (-m, default) work with any bash version
#
#-----------
# IMPORTANT !!!
# Update the node URL in nodeurl.conf
# Content example:
# node1=http://localhost:47134/j7134
# node2=http://localhost:47135/j7134
# node3=http://localhost:47136/j7134
# node4=
# 
# Once added, please save the file
#-----------

#----------
# Script starts
#----------
ERR=0
MAN=0
ANALYZ=0

while getopts ":am" opt; do
	case $opt in
  	a)
		if [ -z "${BASH_VERSINFO}" ] || [ -z "${BASH_VERSINFO[0]}" ] || [ ${BASH_VERSINFO[0]} -lt 4 ]; then echo "Option -a requires Bash version >= 4"; exit 1; fi
		echo "Analysing OpenSearch Index Summary data..." >&2
		ANALYZ=1
    	;;
    m)
    	echo "Manual OpenSearch Index Summary gathering..." >&2
    	MAN=1
    	;;
    \?)
    	echo "Invalid option: -$OPTARG" >&2
    	exit 1
    	;;
	esac
done

declare -A reportTime nodeId countInDatabase countInIndex countInArchive replicationQueues queueSize externalPlatformQueues extQueueSize
declare -a nodeurl
declare f i

#----------
# Check if JQ and CURL is installed
#----------
if ! which curl >/dev/null; then echo -e " You do not have \"curl\" installed (or not found in PATH)."; ERR=1;
fi

if ! which jq >/dev/null; then echo -e " You do not have \"jq\" installed (or not found in PATH). Please run ./index_os.sh -m"; ERR=1;
fi

if [ "$ERR" = "1" ]; then exit 1;
fi

# Ask for Jira admin credentials
if [ "$ANALYZ" != "1" ]; then
	echo "Jira Administrator login credentials:"
	read -p 'Username: ' uservar
	read -p 'Password: ' -s passvar
	echo
fi

#----------
# Check URL
#----------
getNodeUrlfromConfig (){
	while IFS="=" read -r value var; do
		if [ -z "$var" ]; then
			echo ""
		else
			node+=("$var")
		fi
	done < nodeurl.conf
	nodeurlCheck
}
nodeurlCheck (){
	# Checks for node[0] if its empty
	if [ -z "${node[0]}" ]; then
		echo "Please update the node URL for each nodes in the cluster by editing nodeurl.conf"
		exit
	elif [ "$ANALYZ" != "1" ]; then
		echo "The script will check the OpenSearch indexes count on below nodes:"
		for i in "${!node[@]}" ;do	echo ${node[$i]}; done
	else
		echo "Starting OpenSearch analyzing"
	fi
}

#----------
# Generating Index summary for OpenSearch
#----------
indexStoreNoJq () {
	echo " "
	echo Time: `date '+%Y-%m-%d %H:%M:%S'`
	for i in "${!node[@]}"
	do
		content="$(curl -u $uservar:$passvar -s "${node[$i]}/rest/api/2/index/summary")"
		echo "$content" >> os_indexsummary.node$i.txt
	done
}

indexSumOnScreen () {
	# Variables for cluster-wide external platform data
	extQueueNames=""
	extQueueSizes=""
	extDataCollected=false
	
	echo "=== NODE SUMMARY ==="
	printf "%-12s %-20s %-20s %-20s %-20s\n" "NODE" "countInDatabase" "countInIndex" "countInArchive" "queueSize"
	
	for i in "${!node[@]}"
	do
			## Run the curl against all nodes
		    content="$(curl -u $uservar:$passvar -s "${node[$i]}/rest/api/2/index/summary")"
		    echo "$content" > temp.out

		    ## Check if response is valid JSON first
		    if ! cat temp.out | jq empty 2>/dev/null; then
		        echo "Warning: Node ${node[$i]} returned invalid JSON, skipping..."
		        continue
		    fi

		    ## Store the data to var using jq
		    nodeId=`cat temp.out | jq -r '.nodeId // "N/A"' 2>/dev/null || echo "ERROR"`
		    countInDatabase=`cat temp.out | jq '.issueIndex .countInDatabase' 2>/dev/null || echo "0"`
		    countInIndex=`cat temp.out | jq '.issueIndex .countInIndex' 2>/dev/null || echo "0"`
		    countInArchive=`cat temp.out | jq '.issueIndex .countInArchive' 2>/dev/null || echo "0"`
		    
		    # Get replication queue info - exactly like original index.sh
		    replQueueNames=`cat temp.out | jq '.replicationQueues | keys_unsorted[]' 2>/dev/null || echo ""`
		    replQueueSizes=`cat temp.out | jq '.replicationQueues | .[].queueSize' 2>/dev/null || echo ""`
		    
		    # Collect external platform data only once (cluster-wide)
		    if [ "$extDataCollected" = false ]; then
		    	extQueueNames=`cat temp.out | jq -r '.externalPlatformIndexReplay | keys_unsorted[]? // "none"' 2>/dev/null | tr '\n' ',' | sed 's/,$//' || echo "none"`
		    	extQueueSizes=`cat temp.out | jq '.externalPlatformIndexReplay | .[].queueSize' 2>/dev/null | tr '\n' ',' | sed 's/,$//' || echo "0"`
		    	extDataCollected=true
		    fi

		    ## Index Summary - Main stats (like original index.sh)
			printf "%-12s %-20s %-20d %-20d %-20s %-20s\n" "$nodeId" "$countInDatabase" "$countInIndex" "$countInArchive" " " " "
			
			## Replication Queues - display exactly like original (unquoted variables expand as multiple args)
			if [ -n "$replQueueNames" ]; then
				printf "%-12s %-20s %-20s %-20s %-20s %-20s\n" " " " " " " " " $replQueueNames
				printf "%-12s %-20s %-20s %-20s %-20s %-20s\n" " " " " " " " " $replQueueSizes
			fi
	done 
	
	echo ""
	echo "=== CLUSTER EXTERNAL PLATFORM QUEUES ==="
	if [ "$extQueueNames" != "none" ] && [ -n "$extQueueNames" ]; then
		printf "%-30s %-10s\n" "Queue Name" "Size"
		echo "----------------------------------------"
		
		# Convert comma-separated back to arrays for proper pairing
		IFS=',' read -ra QUEUE_NAMES <<< "$extQueueNames"
		IFS=',' read -ra QUEUE_SIZES <<< "$extQueueSizes"
		
		# Display each queue with its size
		for i in "${!QUEUE_NAMES[@]}"; do
			queueName="${QUEUE_NAMES[$i]}"
			queueSize="${QUEUE_SIZES[$i]:-0}"
			printf "%-30s %-10s\n" "$queueName" "$queueSize"
		done
	else
		echo "No external platform queues found"
	fi
	echo "-----"
}

indexReadPerNode () {
	### Print header
	echo '===== ANALYSIS RESULTS ====='
	printf "%-12s %-30s %-15s %-15s %-15s %-30s\n" "NODE" "reportTime" "DB" "IDX" "ARC" "Replication Queues"
	echo "--------------------------------------------------------------------------------------------------"
	
	# Track external platform data (cluster-wide, only show once)
	extPlatformShown=false
	clusterExtQueues=""
	clusterExtSizes=""
	
	### Printing information stored
	for i in "${!node[@]}" #file content
	do
		for (( fn=0; fn<$f; fn++ )); do #file number count in repeatStart
			# Skip if no valid data for this data point
			if [ "${nodeId[$fn,$i]}" = "N/A" ] || [ -z "${nodeId[$fn,$i]}" ]; then
				continue
			fi
			
			# Prepare replication queue display
			if [ "${replicationQueues[$fn,$i]}" != "none" ] && [ -n "${replicationQueues[$fn,$i]}" ]; then
				replDisplay="${replicationQueues[$fn,$i]} (${queueSize[$fn,$i]})"
			else
				replDisplay="none"
			fi
			
			## Main data line with replication queues
			printf "%-12s %-30s %-15s %-15s %-15s %-30s\n" "${nodeId[$fn,$i]}" "${reportTime[$fn,$i]}" "${countInDatabase[$fn,$i]}" "${countInIndex[$fn,$i]}" "${countInArchive[$fn,$i]}" "$replDisplay"
			
			# Collect external platform data only once
			if [ "$extPlatformShown" = false ] && [ "${externalPlatformQueues[$fn,$i]}" != "none" ] && [ -n "${externalPlatformQueues[$fn,$i]}" ]; then
				clusterExtQueues="${externalPlatformQueues[$fn,$i]}"
				clusterExtSizes="${extQueueSize[$fn,$i]}"
				extPlatformShown=true
			fi
		done
	done
	
	# Show cluster external platform data once at the end
	echo ""
	echo "=== CLUSTER EXTERNAL PLATFORM QUEUES (from analysis) ==="
	if [ -n "$clusterExtQueues" ] && [ "$clusterExtQueues" != "none" ]; then
		printf "%-30s %-10s\n" "Queue Name" "Size"
		echo "----------------------------------------"
		
		# Convert comma-separated back to arrays for proper pairing
		IFS=',' read -ra QUEUE_NAMES <<< "$clusterExtQueues"
		IFS=',' read -ra QUEUE_SIZES <<< "$clusterExtSizes"
		
		# Display each queue with its size
		for i in "${!QUEUE_NAMES[@]}"; do
			queueName="${QUEUE_NAMES[$i]}"
			queueSize="${QUEUE_SIZES[$i]:-0}"
			printf "%-30s %-10s\n" "$queueName" "$queueSize"
		done
	else
		echo "No external platform queues found in analysis data"
	fi
}

storingIndexPerNode () {
for i in "${!node[@]}" #file content
	do
		#### Check if JSON is valid first
		if ! jq empty <<< "${node[$i]}" 2>/dev/null; then
			echo "Warning: Invalid JSON in data point $f for node $i, skipping..."
			continue
		fi
		
		#### Storing data with error handling
	    reportTime[$f,$i]=`jq '.reportTime' <<< "${node[$i]}" 2>/dev/null || echo "N/A"`
	    nodeId[$f,$i]=`jq -r '.nodeId // "N/A"' <<< "${node[$i]}" 2>/dev/null || echo "N/A"`
	    countInDatabase[$f,$i]=`jq '.issueIndex .countInDatabase' <<< "${node[$i]}" 2>/dev/null || echo "0"`
	    countInIndex[$f,$i]=`jq '.issueIndex .countInIndex' <<< "${node[$i]}" 2>/dev/null || echo "0"`
	    countInArchive[$f,$i]=`jq '.issueIndex .countInArchive' <<< "${node[$i]}" 2>/dev/null || echo "0"`
	    replicationQueues[$f,$i]=`jq -r '.replicationQueues | keys_unsorted[]? // "none"' <<< "${node[$i]}" 2>/dev/null | tr '\n' ',' | sed 's/,$//' || echo "none"`
		queueSize[$f,$i]=`jq '.replicationQueues | .[].queueSize' <<< "${node[$i]}" 2>/dev/null | tr '\n' ',' | sed 's/,$//' || echo "0"`	
		externalPlatformQueues[$f,$i]=`jq -r '.externalPlatformIndexReplay | keys_unsorted[]? // "none"' <<< "${node[$i]}" 2>/dev/null | tr '\n' ',' | sed 's/,$//' || echo "none"`
		extQueueSize[$f,$i]=`jq '.externalPlatformIndexReplay | .[].queueSize' <<< "${node[$i]}" 2>/dev/null | tr '\n' ',' | sed 's/,$//' || echo "0"`
	done
}

#----------
# Run in sequence
#----------
repeatStart () {
	if [ "$MAN" = "1" ]; then 
		for i in $(seq 10); do
			indexStoreNoJq
			sleep 10;
		done
	elif [ "$ANALYZ" = "1" ]; then
		f=0
		for filez in os_indexsummary.*.txt; do
			echo "Processing $filez"
			IFS=$'\n' read -d '' -r -a node < $filez
			storingIndexPerNode
			f=$((f+1))
		done
		indexReadPerNode
	else
		## loop for 10 times every 10 seconds
		for i in $(seq 10); do
		echo " " 
		echo Time: `date '+%Y-%m-%d %H:%M:%S'`
		indexSumOnScreen
		sleep 10;
		done
	fi
}

#----------
# Run Script
#----------
getNodeUrlfromConfig
repeatStart
# cleanup temp
rm -f temp.out

Perform a full reindex: go to Administration, then System, then Indexing, and then select Full re-index to rebuild the entire search index.

Workarounds

Disabling index consistency checking

Only use this workaround as a temporary solution. Disabling consistency checks can hide actual index issues.

You can temporarily turn off index consistency checks in Jira:

Set the system property: -Dcom.atlassian.jira.status.index.check=false.
Follow the steps in the Setting properties and options on startup documentation.
Restart Jira.

After the restart, Jira won’t show the indexing status in the health check.

Enterprise Support

Products

Jira Software

Jira Service Management

Jira Work Management

Confluence

Bitbucket

Resources

Documentation

Community

Marketplace

Billing and licensing

Troubleshoot search index consistency checks

Jira Data Center Troubleshooting

On this page

In this section

Still need help?

What is search index consistency?

Resolve status health check returning 500 with error state

Resolving error state for Lucene search platform

Workarounds

Disabling index consistency checking

In this section

Page

Viewport

Confluence

Versions

Troubleshoot search index consistency checks

Jira Data Center Troubleshooting

On this page

In this section

Related content

Still need help?

What is search index consistency?

Resolve status health check returning 500 with error state

Resolving error state for Lucene search platform

Workarounds

Disabling index consistency checking

In this section

Related content