Elasticsearch reports HTTP status 403 in bitbucket logs
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
With Elasticsearch version 6 or higher, code search may stop indexing with index failures.
Environment
Elasticsearch version 6 or higher
- Bitbucket 6.0 or higher
Diagnosis
The following error will be in atlassian-bitbucket.log during an index write operation:
com.atlassian.bitbucket.internal.search.indexing.exceptions.IndexException: Index response returned status code 403. Response: IndexResponse{statusCode=403, content={
"error": {
"root_cause": [
{
"type": "cluster_block_exception",
"reason": "blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"
}
],
"type": "cluster_block_exception",
"reason": "blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"
},
"status": 403
}}And one of the following may be visible in the Elasticsearch logs:
[o.e.c.r.a.DiskThresholdMonitor] [bitbucket_bundled]high disk watermark [85%] exceeded on [MqF6riY3TcyaM46vc7KWfg][bitbucket_bundled][C:\Atlassian\ApplicationData\Bitbucket\shared\search\data\nodes\0] free: 5.7gb[5.1%], shards will be relocated away from this node2020-02-04T00:00:22,306][INFO ][o.e.c.r.a.DiskThresholdMonitor] [bitbucket_bundled]low disk watermark [85%] exceeded on [1f1m2yepQ5Sti64cWwD6ZQ][bitbucket_bundled][/var/atlassian/application-data/bitbucket/shared/search/data/nodes/0] free: 17.6gb[11.8%], replicas will not be assigned to this node
[2020-02-04T00:00:52,348][INFO ][o.e.c.r.a.DiskThresholdMonitor] [bitbucket_bundled]low disk watermark [85%] exceeded on [1f1mq3e5Q5Sti6pcWwD6ZQ][bitbucket_bundled][/var/atlassian/application-data/bitbucket/shared/search/data/nodes/0] free: 17.5gb[11.7%], replicas will not be assigned to this nodeCause
When Elasticsearch detects disk is running low on space, it puts itself into read-only mode. This decision is based on the percentage of free space on the disk, so on large volumes and disks, this error can manifest even if several gigabytes of disk space is free.
Elasticsearch sets a default of 85% of disk usage (i.e. at least 15% of disk space must be free)
Therefore, even though you may notice that your current free disk space is above the required free disk space, the indices could have been locked if it was running a search on a low disk space and a solution needs to be applied to unlock the indices.
Solution
The bundled Elasticsearch does not accept requests from outside localhost. The below curl commands must be run from the same server that is hosting the Bitbucket application
Solution #1
Free up disk space to ensure at least 15% of the disk is free. Once enough space is available, unlock the indices by issuing a command such as the following:
ES_HOST=localhost
ES_PORT=7992
curl -u elasticsearchusername:elasticsearchpassword -XPUT -H "Content-Type: application/json" http://$ES_HOST:$ES_PORT/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'
Be sure to replace the elasticsearchusername and elasticsearchpassword fields above with the actual credentials you have configured for Elasticsearch in $BITBUCKET_HOME/shared/search/buckler/buckler.yml
Server instances by default have the bundled Elasticsearch. For remote Elasticsearch instances, please update ES_HOST and ES_PORT as appropriate.
Solution #2
Set the cluster.routing.allocation.disk.watermark.flood_stage value to a lower free percentage (or a fixed value). Here's an example of setting the value to a fixed amount of disk space:
Create a JSON file, say es.json, that contains the following:
{
"transient": {
"cluster.routing.allocation.disk.watermark.low": "50gb",
"cluster.routing.allocation.disk.watermark.high": "20gb",
"cluster.routing.allocation.disk.watermark.flood_stage": "5gb"
}
}Then, run the following command to change threshold values:
ED_HOST=localhost
ES_PORT=7992
JSON_FILE=es.json
curl -u elasticsearchusername:elasticsearchpassword -XPUT -H "Content-Type: application/json" http://$BITBUCKET_HOST:$ES_PORT/_cluster/settings -d@$JSON_FILEOnce done, run the following command to unlock the indices:
ES_HOST=localhost
ES_PORT=7992
curl -u elasticsearchusername:elasticsearchpassword -XPUT -H "Content-Type: application/json" http://$BITBUCKET_HOST:$ES_PORT/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'In case you are using Windows to run the command syntax is the following:
ES_HOST=localhost
ES_PORT=7992
curl -u elasticsearchusername:elasticsearchpassword -XPUT -H "Content-Type: application/json" http://$BITBUCKET_HOST:$ES_PORT/_all/_settings -d "{\"index.blocks.read_only_allow_delete\": null}" Be sure to replace the elasticsearchusername and elasticsearchpassword fields above with the actual credentials you have configured for Elasticsearch in $BITBUCKET_HOME/shared/search/buckler/buckler.yml
Server instances by default have the bundled Elasticsearch. For remote Elasticsearch instances, please update ES_HOST and ES_PORT as appropriate.