After upgrading Bitbucket Server, bundled Elasticsearch has been crashing while indexing repositories
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
After upgrading Bitbucket Server, the bundled Elasticsearch has been systematically crashing when attempting to index repositories
Environment
This issue may affect multiple versions of Bitbucket Server.
For example, it has been seen after the upgrade from version 6.10.0 to 7.6.0.
Diagnosis
In atlassian-bitbucket.log you may find errors similar to the following:
2021-03-01 01:23:45,678 ERROR [search-indexing:thread-1] c.a.b.i.s.i.e.DefaultIndexEventWorker Indexing - Failed for project PROJO1 (id: 1001) with error: Index response returned status code 429. Response: IndexResponse{statusCode=429, content={
"error": {
"root_cause": [
{
"type": "circuit_breaking_exception",
"reason": "[parent] Data too large, data for [<http_request>] would be [1034946678/987mb], which is larger than the limit of [986061209/940.3mb], real usage: [1034946568/987mb], new bytes reserved: [110/110b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=110/110b, accounting=41161613/39.2mb]",
"bytes_wanted": 1034946678,
"bytes_limit": 986061209,
"durability": "PERMANENT"
}
],
"type": "circuit_breaking_exception",
"reason": "[parent] Data too large, data for [<http_request>] would be [1034946678/987mb], which is larger than the limit of [986061209/940.3mb], real usage: [1034946568/987mb], new bytes reserved: [110/110b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=110/110b, accounting=41161613/39.2mb]",
"bytes_wanted": 1034946678,
"bytes_limit": 986061209,
"durability": "PERMANENT"
},
"status": 429
}}
com.atlassian.bitbucket.internal.search.indexing.exceptions.IndexException: Index response returned status code 429. Response: IndexResponse{statusCode=429, content={
"error": {
"root_cause": [
{
"type": "circuit_breaking_exception",
"reason": "[parent] Data too large, data for [<http_request>] would be [1034946678/987mb], which is larger than the limit of [986061209/940.3mb], real usage: [1034946568/987mb], new bytes reserved: [110/110b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=110/110b, accounting=41161613/39.2mb]",
"bytes_wanted": 1034946678,
"bytes_limit": 986061209,
"durability": "PERMANENT"
}
],
"type": "circuit_breaking_exception",
"reason": "[parent] Data too large, data for [<http_request>] would be [1034946678/987mb], which is larger than the limit of [986061209/940.3mb], real usage: [1034946568/987mb], new bytes reserved: [110/110b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=110/110b, accounting=41161613/39.2mb]",
"bytes_wanted": 1034946678,
"bytes_limit": 986061209,
"durability": "PERMANENT"
},
"status": 429
}}
Cause
Bundled Elasticsearch is running with insufficient JVM heap.
If Elasticsearch was previously configured to allow a larger heap, that custom configuration has been overwritten by the new Bitbucket version.
Solution
- Verify what was the maximum heap size defined for Elasticsearch before the upgrade:
- Navigate to $BITBUCKET_HOME/search/config
- Open a pre-upgrade copy of the jvm.options file.
If you don't have it, you may need to restore it from pre-upgrade backup into a location outside of $BITBUCKET_HOME and open it there.
Find the previously used heap settings defined in the 2 separate lines.
One of them should start with -Xms and the other with -Xmx - eg.-Xms4g -Xmx4gEdit the current $BITBUCKET_HOME/search/config/jvm.options file and find the equivalent configuration lines, with the default values of
1g:-Xms1g -Xmx1gReplace both lines with those you found in the pre-upgrade copy of this file.
- Save the modified jvm.options file
- Restart your Bitbucket Server, so the bundled Elasticsearch also gets restarted, and the new configuration is applied.
- Confirm that the errors mentioned above do not occur any longer.