Elasticsearch indexing and search is not available due to "IndexShardRecoveryException[failed to fetch index version after copying it over]"

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform Notice: Server and Data Center Only. This article only applies to Atlassian products on the server and data center platforms.

Problem

When performing a search in Bitbucket Server, the following message is returned in the search box for the user:

  • "Search is not available now"

Also the indexing on new activities is not indexed successfully.

The following appears in the bitbucket_search.log:

[WARN ][indices.cluster          ] [bitbucket_bundled] [[bitbucket-search-v1][4]] marking and sending shard failed due to [failed recovery]
[bitbucket-search-v1][[bitbucket-search-v1][4]] IndexShardRecoveryException[failed to fetch index version after copying it over]; nested: IndexShardRecoveryException[shard allocated for local recovery (post api), should exist, but doesn't, current files: [_21.si, write.lock, _fr.fdt, _21.nvd, _fr.nvd, _fr_Lucene50_0.tim, _fr_Lucene54_0.dvd, _21.nvm, _21_Lucene50_0.doc, _21.fdt, _fr.si, _21_Lucene50_0.tim, _21.fnm, _fr.fdx, _fr_Lucene54_0.dvm, _21_Lucene54_0.dvm, _fr.fnm, _21.fdx, _fr_Lucene50_0.tip, _fr_Lucene50_0.pos, _21_Lucene50_0.pos, _21_Lucene54_0.dvd, _fr_Lucene50_0.doc, _21_Lucene50_0.tip, _fr.nvm]]; nested: IndexNotFoundException[no segments* file found in store(default(mmapfs(/var/atlassian/application-data/bitbucket/shared/search/data/bitbucket_search/nodes/0/indices/bitbucket-search-v1/4/index),niofs(/var/atlassian/application-data/bitbucket/shared/search/data/bitbucket_search/nodes/0/indices/bitbucket-search-v1/4/index))): files: [_21.fdt, _21.fdx, _21.fnm, _21.nvd, _21.nvm, _21.si, _21_Lucene50_0.doc, _21_Lucene50_0.pos, _21_Lucene50_0.tim, _21_Lucene50_0.tip, _21_Lucene54_0.dvd, _21_Lucene54_0.dvm, _fr.fdt, _fr.fdx, _fr.fnm, _fr.nvd, _fr.nvm, _fr.si, _fr_Lucene50_0.doc, _fr_Lucene50_0.pos, _fr_Lucene50_0.tim, _fr_Lucene50_0.tip, _fr_Lucene54_0.dvd, _fr_Lucene54_0.dvm, write.lock]];
	at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:224)

The following appears in the atlassian-bitbucket.log:

ERROR [search-indexing:thread-1]  c.a.b.s.i.i.e.DefaultIndexEventWorker Indexing - Failed for project <project_name> (id: 489) with error: Elasticsearch returned 503 (service unavailable) for the request
com.atlassian.bitbucket.search.indexer.exceptions.ServiceUnavailableException: Elasticsearch returned 503 (service unavailable) for the request
	at com.atlassian.bitbucket.search.indexer.internal.util.ElasticsearchUtil.lambda$executeHandleUnavailable$0(ElasticsearchUtil.java:42) ~[bitbucket-search-index-2.1.4.jar:na]
	at rx.internal.operators.OperatorMap$1.onNext(OperatorMap.java:54) [bitbucket-search-index-2.1.4.jar:na]

Cause

There may be various causes for this, the following ones have been experienced in the past:

  • moving the indexes (located in the <$BITBUCKET_HOME>/shared/search/data/bitbucket_search) from one server to another one without using the Bitbucket Server Backup Client

Resolution

Trigger a reindex of the Elasticsearch indexes by running the following command:

curl -u <admin-user>:<admin-password>  -X POST -H 'Content-Type: application/json' -H 'Accept: application/json' <bitbucket-url>/rest/indexing/latest/sync

(warning) While the indexes will be rebuilt, only partial results will be available. The search feature will be available immediately.

Last modified on Oct 26, 2016

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.