Problem

Bitbucket Server and Elasticsearch have been set up but code search does not return any search results from some or all repositories although they should have been indexed.

The following error is found in the <BitbucketHome>/log/atlassian-bitbucket.log file:

2016-10-31 15:26:32,044 ERROR [search-indexing:thread-1]  c.a.b.s.i.i.e.DefaultIndexEventWorker Indexing - Failed for repository <projectname>/<reponame> (id: <repoid>) with error: Indexing for repository with id <repoid> failed and has reached maximum retries (5)
com.atlassian.bitbucket.search.indexer.exceptions.UnrecoverableIndexException: Indexing for repository with id <repoid> failed and has reached maximum retries (5)
	at com.atlassian.bitbucket.search.indexer.internal.DefaultIndexService.lambda$updateIndexStateForError$12(DefaultIndexService.java:354) ~[bitbucket-search-index-3.0.0.jar:na]
	at rx.internal.operators.OperatorMap$1.onNext(OperatorMap.java:54) [bitbucket-search-index-3.0.0.jar:na]
[...]

Cause

By default the indexing process of Bitbucket Server, which populates the Elasticsearch index, only tries four more times if an error occurs during indexing. Even if the underlying root cause (for instance problems with the git executable) has been resolved, the indexing process will never try indexing a repository that has reached the maximum retry count again because the number of retries are stored in the index as well.

Before proceeding with the resolution described here you will first need to ensure that the underlying problem that is causing indexing to fail is resolved. If you need help with this process, contact Atlassian Support via support.atlassian.com. Once the root cause has been found and the problem has been resolved, follow the steps below to reset the index so that indexing can be restarted.

Resolution

To resolve this problem you first need to delete the Bitbucket Server index on the Elasticsearch server and then restart indexing in Bitbucket Server.

Step 1. Delete Elasticsearch index

The Elasticsearch REST API provides a way to delete an index using an HTTP request. 

• If you are using a Bitbucket Server version before 6.0 you can use the following curl command:

curl -u bitbucket -k -X DELETE <Elasticsearch-URL>/bitbucket-search-v1

• If you are using Bitbucket Server 6.0 or newer run all of the following commands:

1. curl -u bitbucket -k -X DELETE <Elasticsearch-URL>/bitbucket-search
2. curl -u bitbucket -k -X DELETE <Elasticsearch-URL>/bitbucket-index-state
3. curl -u bitbucket -k -X DELETE <Elasticsearch-URL>/bitbucket-index-version
4. curl -u bitbucket -k -X DELETE <Elasticsearch-URL>/bitbucket-project
5. curl -u bitbucket -k -X DELETE <Elasticsearch-URL>/bitbucket-repository

Replace the <Elasticsearch-URL> with the URL where Elasticsearch is running. For example, in the case of the bundled Elasticsearch with Bitbucket Server, the URL would be http://localhost:7992 and these commands need to be run where the Bitbucket Server is hosted. Here, 7992 is the default port used by Elasticsearch when bundled with Bitbucket Server. If you are using your own installation of Elasticsearch (for instance because you're using Bitbucket Data Center), be sure to change the URL accordingly.

You will be prompted for a password when you send the request. You can retrieve the password to use from the buckler.yml file by copying the value of the auth.basic.password property contained in that file.

• If you are on a Bitbucket server instance, this file can be found at <BitbucketHome>/shared/search/buckler/buckler.yml.
• If you are on a Bitbucket Data Center instance, the file can be found in the remote ElasticSearch node at the location elasticsearch/config/buckler/buckler.yml.

If the request is successful, no output will be returned by the command.

Step 2. Restart indexing

The Bitbucket Server REST API provides a way to restart indexing by making an HTTP request. You can use the following curl command to make such a request:

curl -u <username> -X POST -v -H 'Content-Type: application/json' -H 'Accept: application/json' <BaseUrl>/rest/indexing/latest/sync

where <username> is the username of a Bitbucket Server user with administrative privileges, and <BaseUrl> is the base URL of your Bitbucket Server or Data Center instance. 

If the request is successful, no output will be returned by the command.

The indexing process will now begin. Depending on the number of repositories this can take a significant amount of time, but at some stage, you should be able to find code from all of your repositories.