Bitbucket Server: Elasticsearch not starting up after an upgrade.
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
The inbuilt Elasticsearch of Bitbucket server not starting up after a major version upgrade. The search functionality remains unavailable.
Environment
Bitbucket Server or Bitbucket Data Center with a single node(with inbuilt elasticsearch)
Diagnosis
You could see similar lines in atlassian.bitbucket.log
2022-03-08 15:00:09,311 WARN [Caesium-1-1] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to Elasticsearch during index synchronisation.This could indicate any connection issues with Elasticsearch. So to confirm if the issue is what we are discussing here, you will need to look at the bitbucket_search log as well.
You could see similar lines in the bitbucket_search log around the time of the first start-up after the upgrade of the instance.
[2022-03-08 12:22:09,921][INFO ][node ] [bitbucket_bundled] version[2.3.1], pid[28507], build[bd98092/2016-04-04T12:25:05Z]
[2022-03-08 12:22:09,922][INFO ][node ] [bitbucket_bundled] initializing ...
....
[2022-03-08 12:22:12,081][ERROR][bootstrap ] Exception
java.lang.IllegalArgumentException: Unknown Discovery type [single-node]Cause
The version indicated in the bitbucket_search log snippet(2.3.1) of the Diagnosis part is a much older version of Elasticsearch. So first it would be better to understand the default Elasticsearch versions that comes bundled with Bitbucket server.
| Bitbucket version | Elasticsearch version |
|---|---|
| 7.16 and above(till 7.21) | Elasticsearch 7.10.2 |
| 7.1.0 and above | Elasticsearch 7.5.2 |
| 6.10.17 (or any later 6.10.x version) | Elasticsearch 6.8.22 |
| 6.9.2 and 6.10.1 and above | Elasticsearch 6.8.6 |
| 6.10.0, 6.9.0, 6.9.1 | Elasticsearch 6.5.3 |
If Bitbucket is starting an older version of Elasticsearch than the one mentioned in the table, then there must have been an issue during the upgrade and the elasticsearch component was not upgraded. If you could send the logs to Atlassian support team, they could confirm this.
Solution
To fix this, either you have to reinstall the same version of Bitbucket or do a minor version upgrade. In both the cases, make sure that both Bitbucket and Elasticsearch processes are completely stopped before the upgrade.
If the issue still persists, please get in touch with the Atlassian support team.