How to increase Pagination in Elasticsearch since every search is limited to 1000 results
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Purpose
In this article we will go through the steps to increase the max_result_window of Elasticsearch.
Problem
- When searching a word which happens to present in more than 1000 files in all of Bitbucket repositories, results can be seen for only the first 1000. Subsequent results are not fetched and we are instead presented with the following alert:
- Clicking on the "Try again" button does not help and clicking on close will present a new error at the bottom of the page:
- The following appears in the
bitbucket_search.log
[2018-12-11 12:53:43,310][DEBUG][action.search] [bitbucket_bundled] [bitbucket-search-v1][4], node[L8MlmuIgS_6ZQaT34fxrQg], [P], v[2], s[STARTED], a[id=vGhgCsYaSe-eROu1iuuqoQ]: Failed to execute [org.elasticsearch.action.search.SearchRequest@7f46c5bd] lastShard [true]
RemoteTransportException[[bitbucket_bundled][local[1]][indices:data/read/search[phase/query]]]; nested: QueryPhaseExecutionException[Result window is too large, from + size must be less than or equal to: [1000] but was [1010]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level parameter.];
Caused by: QueryPhaseExecutionException[Result window is too large, from + size must be less than or equal to: [1000] but was [1010]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level parameter.]Cause
This is because of the Pagination settings which has set the max_result_window to only 1000 results. Read more about this in their official documentation.
Workaround
- We can update the max_result_window to a number high enough to accommodate the search results.
- Atlassian does not support manipulation of Elasticsearch directly. Therefore please perform this on a test instance first.
- Increasing to index.max_result_window value to a higher number may impact the performance of the system.
Increase max_result_window to 2000
curl -u bitbucket -X PUT "<SEARCH_SERVER_URL>/bitbucket-search/_settings" -d '{ "index" : { "max_result_window" : 2000 } }'- You will need to enter the bitbucket user's password.
- For bundled Elasticsearch instances, this is located in:
$BITBUCKET_HOME/search/buckler/buckler.ymlfile. - For remote Elasticsearch instances, this is located in: <E
lasticsearch config>/buckler/buckler.yml.See Install and configure a remote Elasticsearch server for further details.
- For bundled Elasticsearch instances, this is located in:
- If the value is updated you should see the following response:
Enter host password for user 'bitbucket':
{"acknowledged":true}
Last modified on Dec 4, 2023
Powered by Confluence and Scroll Viewport.