Problem

Elasticsearch logs report the following error when starting up:

[2020-01-17T02:23:03,966][WARN ][o.e.i.e.Engine           ] [bitbucket_bundled] [bitbucket-search][2]failed engine [failed to recover from translog]
org.elasticsearch.index.engine.EngineException: failed to recover from translog
	at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslogInternal(InternalEngine.java:445) ~[elasticsearch-6.5.3.jar:6.5.3]
....

Background and Diagnosis

Each Elasticsearch shard copy also writes operations into its transaction log known as the translog. In the event of a crash, recent operations that have been acknowledged but not yet included are recovered from the translog when the shard recovers. The data in the translog is only persisted to disk when the translog is fsynced and committed. In the event of a hardware failure or an operating system crash or a JVM crash or a shard failure, any data written since the previous translog commit will be lost. In some cases, even a bad drive or user error can cause the translog to become corrupted. When this corruption is detected by Elasticsearch due to mismatching checksums, Elasticsearch will fail the shard and refuse to allocate that copy of the data to the node, recovering from a replica if available.

In such cases, elasticsearch-translog tool can be used to recover data that is currently contained in the translog.

Resolution

1. Stop Bitbucket Server and embedded Elasticsearch (see Warning above)
2. Go to <bitbucket-home>/shared/search/data/nodes/0/indices folder
3. Run <bitbucket-install>/elasticsearch/bin/elasticsearch-translog truncate -d <bitbucket-home>/shared/search/data/nodes/0/indices/<index-folder>/translog/
   for each of the indices
    If the Elasticsearch error log shows the index folder as part of the error, you can delete translog of just that one folder.
4. Restart Bitbucket Server

Error failed to recover from translog during Elasticsearch startup