Rebuild the Search server indexes for Bitbucket Server

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.

Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible

Please note:

For Bitbucket 7.20 and below, Elasticsearch was bundled. Starting with 7.21, OpenSearch is the bundled search server.

Reindex a specific repository:

If you need to reindex a specific repository, please see KB Reindex a specific repository for Code Search

Summary

When attempting a code search in Bitbucket, the "Search is currently unavailable" error is displayed in the user interface.

Other errors may be seen in the logs as a response to the search failure. In some cases, this can lead to high CPU for the bundled Search server Java process.

Environment

  • Bitbucket 5.x, 6.x, and 7.x Server/DC

Diagnosis

Errors in Log Files

There are many different messages to show that the Search server index is corrupt, somehow. Some of the errors seen so far are outlined below:

Errors in the atlassian-bitbucket.log:

404 errors

Unexpected response code from Elasticsearch: 404
com.atlassian.bitbucket.search.indexer.exceptions.IndexException: Unexpected error occurred when requesting index state document (id=43) the HTTP Status code is: 404

A 404 status code indicates that the target resource could not be found.


503 errors

ERROR [http-nio-7990-exec-16] X182486 <session id> <username> <ip address> "POST /rest/search/latest/search HTTP/1.1" c.a.b.s.internal.rest.SearchResource Unexpected response code from Elasticsearch: 503

A 503 status code indicates that the target resource is unavailable.


400 errors

ERROR [http-nio-7990-exec-2746] user *1U2QK8x795x992313x2 c6fz54 10.10.10.5,10.10.10.13 "POST /rest/search/latest/search HTTP/1.1" c.a.b.i.s.search.rest.SearchResource Unexpected response code from Elasticsearch: 400

A 400 is a Bad Request and is typically associated with a misconfigured instance.


You are not permitted to access this resource

ERROR [https-jsse-nio-8443-exec-26] n0200293 <session id> <username> <ip address> "POST /rest/search/latest/search HTTP/1.1" c.a.b.i.s.search.rest.SearchResource Failed to process search request 'RestSearchRequest{entities={code=PagingInfo{start=0, limit=0}}, limits=Limits{primary=25, secondary=10}, query='<search keyword>'}' - error: You are not permitted to access this resource


Errors may also appear in the bitbucket_search.log:

IndexxNotFoundException - The index does not exist.

[bitbucket-search] IndexNotFoundException[no such index]


Unable to find a field mapper for fieldFoundException - Incorrect mapping

Caused by: ElasticsearchException[Unable to find a field mapper for field [quickSearchProjectName.length]. No 'missing' value defined.]



Failed to load metadata - 
Unable to read the index data. A sign of corruption.

[2020-12-02T10:03:52,001][ERROR][o.e.b.Bootstrap          ] [bitbucket_bundled]Exception
org.elasticsearch.ElasticsearchException: failed to load metadata
...
Failed to load metadata Caused by: java.io.IOException: failed to find metadata for existing index bitbucket-index-state [location: IAdbCfIHT4yFBHhlsSk5VQ, generation: 5]


IndexShardRecoveryException - Unable to find or read the Search server shard files. A sign of corruption.

[2016-11-23 16:10:27,080][WARN ][indices.cluster          ] [bitbucket_bundled] [[bitbucket-search-v1][3]] marking and sending shard failed due to [failed recovery]
[bitbucket-search-v1][[bitbucket-search-v1][3]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: NoSuchFileException[/var/atlassian/application-data/bitbucket/shared/search/data/bitbucket_search/nodes/0/indices/bitbucket-search-v1/3/translog/translog-7.tlog];
        at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:250)
...
Caused by: [bitbucket-search-v1][[bitbucket-search-v1][3]] EngineCreationFailureException[failed to create engine]; nested: NoSuchFileException[/var/atlassian/application-data/bitbucket/shared/search/data/bitbucket_search/nodes/0/indices/bitbucket-search-v1/3/translog/translog-7.tlog];
        at org.elasticsearch.index.engine.InternalEngine.< int>(InternalEngine.java:155)
...
Caused by: java.nio.file.NoSuchFileException: /var/atlassian/application-data/bitbucket/shared/search/data/bitbucket_search/nodes/0/indices/bitbucket-search-v1/3/translog/translog-7.tlog
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
... 


IndexShardRecoveryException[failed to fetch index version after copying it over];  Unable to fetch indexes. A sign of corruption. The possible cause could be moving indexes(located in the <$BITBUCKET_HOME>/shared/search/data/bitbucket_search) from one server to another manually, without using Backup Client

From bitbucket_search.log,

[WARN ][indices.cluster          ] [bitbucket_bundled] [[bitbucket-search-v1][4]] marking and sending shard failed due to [failed recovery]
[bitbucket-search-v1][[bitbucket-search-v1][4]] IndexShardRecoveryException[failed to fetch index version after copying it over]; nested: IndexShardRecoveryException[shard allocated for local recovery (post api), should exist, but doesn't, current files: [_21.si, write.lock, _fr.fdt, _21.nvd, _fr.nvd, _fr_Lucene50_0.tim, _fr_Lucene54_0.dvd, _21.nvm, _21_Lucene50_0.doc, _21.fdt, _fr.si, _21_Lucene50_0.tim, _21.fnm, _fr.fdx, _fr_Lucene54_0.dvm, _21_Lucene54_0.dvm, _fr.fnm, _21.fdx, _fr_Lucene50_0.tip, _fr_Lucene50_0.pos, _21_Lucene50_0.pos, _21_Lucene54_0.dvd, _fr_Lucene50_0.doc, _21_Lucene50_0.tip, _fr.nvm]]; nested: IndexNotFoundException[no segments* file found in store(default(mmapfs(/var/atlassian/application-data/bitbucket/shared/search/data/bitbucket_search/nodes/0/indices/bitbucket-search-v1/4/index),niofs(/var/atlassian/application-data/bitbucket/shared/search/data/bitbucket_search/nodes/0/indices/bitbucket-search-v1/4/index))): files: [_21.fdt, _21.fdx, _21.fnm, _21.nvd, _21.nvm, _21.si, _21_Lucene50_0.doc, _21_Lucene50_0.pos, _21_Lucene50_0.tim, _21_Lucene50_0.tip, _21_Lucene54_0.dvd, _21_Lucene54_0.dvm, _fr.fdt, _fr.fdx, _fr.fnm, _fr.nvd, _fr.nvm, _fr.si, _fr_Lucene50_0.doc, _fr_Lucene50_0.pos, _fr_Lucene50_0.tim, _fr_Lucene50_0.tip, _fr_Lucene54_0.dvd, _fr_Lucene54_0.dvm, write.lock]];
	at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:224)

From atlassian-bitbucket.log,

ERROR [search-indexing:thread-1]  c.a.b.s.i.i.e.DefaultIndexEventWorker Indexing - Failed for project <project_name> (id: 489) with error: Elasticsearch returned 503 (service unavailable) for the request
com.atlassian.bitbucket.search.indexer.exceptions.ServiceUnavailableException: Elasticsearch returned 503 (service unavailable) for the request
	at com.atlassian.bitbucket.search.indexer.internal.util.ElasticsearchUtil.lambda$executeHandleUnavailable$0(ElasticsearchUtil.java:42) ~[bitbucket-search-index-2.1.4.jar:na]
	at rx.internal.operators.OperatorMap$1.onNext(OperatorMap.java:54) [bitbucket-search-index-2.1.4.jar:na]

Check the content on the filesystem

List the search index files to see if the index has been corrupted or deleted, with one of the following two commands:

tree -L 8 <bitbucket-home>/shared/search/data

(info) The tree command, if installed, lists contents of directories in a tree format. The -L argument displays the max depth of the tree.

If tree command is not available, you can use the following command:

ls -R <bitbucket-home>/shared/search/data

(info) The ls command lists directory content and the -R argument makes it recursive which lists the sub-directories contents.

The purpose of these commands is to list the whole structure of the data directory and its sub-directories and files. You can choose any of the commands above. These should return a large number of index files. Below is sample output after running the tree command:

...
_0_1.liv  _1.si   _4.si              _e_Lucene50_0.tim  _e.si   _g.si
_0.cfe    _2.cfe  _e.fdt             _e_Lucene50_0.tip  _f.cfe  segments_1
_0.cfs    _2.cfs  _e.fdx             _e_Lucene54_0.dvd  _f.cfs  write.lock
_0.si     _2.si   _e.fnm             _e_Lucene54_0.dvm  _f.si
_1.cfe    _4.cfe  _e_Lucene50_0.doc  _e.nvd             _g.cfe
_1.cfs    _4.cfs  _e_Lucene50_0.pos  _e.nvm             _g.cfs
...

The set of data retrieved by the commands above should be far larger than this. If this directory is empty or very few files exist, the index likely needs to be rebuilt.

Cause

  • The search index in <bitbucket-home>/shared/search/data has been deleted or corrupted and needs to be rebuilt.
  • The repositories didn’t get fully indexed. Some possible reasons this might have happened are:
    • Search server issue
    • Temporary issue caused by high CPU utilization
    • Search server might have been down while the file was committed.

Solution

If there is an issue with the Indexing, you will see some of these errors in the atlassian-bitbucket.log or in the bitbucket_search.log. In case that happens, we recommend rebuilding the Search server index:

(Recommended) Resolution #1 - trigger a reindex

The Bitbucket Server REST API provides a way to restart indexing by making an HTTP request. This can be done when running a Bitbucket instance using the following curl command to make a POST to the /rest/indexing/latest/sync REST endpoint:

curl -u <admin-user> -X POST -H 'Content-Type: application/json' -H 'Accept: application/json' <bitbucket-url>/rest/indexing/latest/sync

To check the index status: 

curl -u <admin-user> -X GET -H 'Content-Type: application/json' -H 'Accept: application/json' <bitbucket-url>/rest/indexing/latest/status


This will trigger a re-index of all repositories. Bitbucket will be available with all functionality other than search of unindexed portions of your code.  The time it takes to index depends on how much indexable content you have. This will be the amount of code contained in files under 512 KB.

Resolution #2 - Delete the Index

This resolution requires downtime to stop the application since it tries to remove the content in the filesystem and fix the issue by attempting to rebuild the indexes after starting Bitbucket. If the "Resolution #1" option does not resolve the problem, perform the following steps:

For Bitbucket Server using bundled Search server:

  1. Stop Bitbucket Server
  2. Create a backup of <Bitbucket-home>/shared/search/data/nodes directory
  3. Delete the contents of <Bitbucket-home>/shared/search/data/nodes directory
  4. Start Bitbucket Server

For Bitbucket Server and Data Center using an external Search server instance:

  1. Stop Bitbucket Server
  2. Stop the Search server
  3. Create a backup of <SEARCH-SERVER_HOME>/data/local/nodes directory in your Search server
  4. Delete the contents of <SEARCH-SERVER_HOME>/data/local/nodes directory
  5. Start the Search server
  6. Start Bitbucket Server


Last modified on Feb 6, 2024

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.