Bundled Elasticsearch throws node validation exception after Bitbucket upgrade
Platform Notice: Server and Data Center Only. This article only applies to Atlassian products on the server and data center platforms.
Problem
After upgrading Bitbucket, the bundled Elasticsearch instance fails to start. In addition, the following error is found in the <BitbucketHome>/log/search/bitbucket_search.log file:
[2020-02-24T06:15:46,142][ERROR][o.e.b.Bootstrap ] [bitbucket_bundled]node validation exception
Cluster name [bitbucket_search] subdirectory exists in data paths [<BITBUCKET_HOME>/shared/search/data/bitbucket_search]. All data under these paths must be moved up one directory to paths [<BITBUCKET_HOME>/shared/search/data]Cause
This bitbucket_search directory may exist because an older Bitbucket version (Bitbucket 5.4.x or less) used the following path to store the Elasticsearch indexes:
$BITBUCKET_HOME/shared/search/data/bitbucket_search/nodes
From Bitbucket 5.15.x onward, it stores the Elasticsearch indexes in the following path:
$BITBUCKET_HOME/shared/search/data/nodes
In the following ticket, we upgraded the version of the bundled Elasticsearch instance to 6.8.6 - which contains a change from Elasticsearch which prevents the node from starting up if there's a cluster name in the provided data path:
BSERV-12187 - Getting issue details... STATUS
Resolution
The following bug ticket tracks our progress towards automatically applying the below steps in later versions of Bitbucket:
BSERV-12206 - Getting issue details... STATUS
To resolve this problem, the older $BITBUCKET_HOME/shared/search/data/bitbucket_search directory needs to be fully removed.
Once this directory has been removed, run the appropriate Bitbucket start-up script for your environment in order to attempt to start the bundled Elasticsearch instance.
This should not have any impact on the integrity of the indexes stored in $BITBUCKET_HOME/shared/search/data/nodes, which is the current location where the bundled Elasticsearch instance stores its indexes.