Summary

When performing a code search, no results are returned or you are prompted that search is not available.

Diagnosis

1. For bundled Elasticsearch, open the bitbucket_search.log located in BITBUCKET_HOME/shared/log. 
2. For remote Elasticsearch, refer to the logging location specified during your install.
3. Search for all shards failed.

2021-02-27T14:28:50,191][WARN ][r.suppressed             ] [bitbucket_bundled]path: /bitbucket-project/project/_search, params: {index=bitbucket-project, type=project}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:296) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:259) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:100) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$1(InitialSearchPhase.java:208) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:187) [elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.8.6.jar:6.8.6]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151] 

Causes

There are multiple potential causes that have been identified for this issue.

Cause 1:  Stuck or corrupted index

As noted on our KB Bitbucket Server Code Search is unable to find results from some or all repositories, if there are failures to index a repository, the max retry count can be reached and the repository will not attempt a future index. (The max retry has been reached and is stored in the index itself) The failure must first be identified as you can encounter the issue again.

Cause 2: Disk space

Low disk space can trigger a failure to search and write to the index. If the data store for the shard allocation reaches a max of 95 percent, Elasticsearch will force a read only index block. This is due to the flood_stage watermark being exceeded. For details, see Elasticsearch 6.8 disk allocator:

Property: cluster.routing.allocation.disk.watermark.flood_stage

Solutions

Solution 1: Corrupted Index

To resolve this problem you first need to delete the Bitbucket Server index on the Elasticsearch server and then restart indexing in Bitbucket Server.

Step 1. Delete Elasticsearch index

The Elasticsearch REST API provides a way to delete an index using an HTTP request. 

• If you are using a Bitbucket Server version before 6.0 you can use the following curl command:

curl -u bitbucket -k -X DELETE <Elasticsearch-URL>/bitbucket-search-v1

• If you are using Bitbucket Server 6.0 or newer run all of the following commands:

1. curl -u bitbucket -k -X DELETE <Elasticsearch-URL>/bitbucket-search
2. curl -u bitbucket -k -X DELETE <Elasticsearch-URL>/bitbucket-index-state
3. curl -u bitbucket -k -X DELETE <Elasticsearch-URL>/bitbucket-index-version
4. curl -u bitbucket -k -X DELETE <Elasticsearch-URL>/bitbucket-project
5. curl -u bitbucket -k -X DELETE <Elasticsearch-URL>/bitbucket-repository

Replace the <Elasticsearch-URL> with the URL where Elasticsearch is running. For example, in the case of the bundled Elasticsearch with Bitbucket Server, the URL would be http://localhost:7992 and these commands need to be run where the Bitbucket Server is hosted. Here, 7992 is the default port used by Elasticsearch when bundled with Bitbucket Server. If you are using your own installation of Elasticsearch (for instance because you're using Bitbucket Data Center), be sure to change the URL accordingly.

You will be prompted for a password when you send the request. You can retrieve the password to use from the buckler.yml file by copying the value of the auth.basic.password property contained in that file.

• If you are on a Bitbucket server instance, this file can be found at <BitbucketHome>/shared/search/buckler/buckler.yml.
• If you are on a Bitbucket Data Center instance, the file can be found in the remote ElasticSearch node at the location elasticsearch/config/buckler/buckler.yml.

If the request is successful, no output will be returned by the command.

Step 2. Restart indexing

The Bitbucket Server REST API provides a way to restart indexing by making an HTTP request. You can use the following curl command to make such a request:

curl -u <username> -X POST -v -H 'Content-Type: application/json' -H 'Accept: application/json' <BaseUrl>/rest/indexing/latest/sync

where <username> is the username of a Bitbucket Server user with administrative privileges, and <BaseUrl> is the base URL of your Bitbucket Server or Data Center instance. 

If the request is successful, no output will be returned by the command.

The indexing process will now begin. Depending on the number of repositories this can take a significant amount of time, but at some stage, you should be able to find code from all of your repositories.

Solution 2: Disk space

Increase disk space.