Elasticsearch index stuck with Unable to enqueue event warning message
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Elasticsearch fails and requires a rebuild before it functions again. This might take the form of having to trigger the rebuilding of the index manually or restarting Bitbucket which can trigger a re-index.
Environment
Bitbucket Data Center
Diagnosis
The following is seen in the atlassian-bitbucket.log file
XXXX-XX-XX 00:31:20,694 ERROR [AtlassianEvent::thread-7] <string> <request_ID> <string> <ip> <protocol> - git-receive-pack '/project/repo.git' c.a.b.i.s.i.event.EventHandler Unable to enqueue event for processing: [RepositoryContentModifiedIndexEvent{repositoryId=XXXXX}] Event is discarded and will not be retried.
XXXX-XX-XX 00:31:52,695 WARN [Caesium-1-4] c.a.b.i.s.i.s.DefaultIndexEventPublisher Retry 2/5 - Unable to enqueue event: [RepositoryModifiedIndexEvent{repositoryId=XXXXX}]
XXXX-XX-XX 00:32:52,696 WARN [Caesium-1-4] c.a.b.i.s.i.s.DefaultIndexEventPublisher Retry 3/5 - Unable to enqueue event: [RepositoryModifiedIndexEvent{repositoryId=XXXXX}]
XXXX-XX-XX 00:33:52,696 WARN [Caesium-1-4] c.a.b.i.s.i.s.DefaultIndexEventPublisher Retry 4/5 - Unable to enqueue event: [RepositoryModifiedIndexEvent{repositoryId=XXXXX}]
XXXX-XX-XX 00:34:01,758 WARN [Caesium-1-4] c.a.b.i.s.indexing.util.ElderScroll Unable to remove the scroll token previously reserved. This is not strictly speaking a serious error, but has resource utilisation consequences for Elasticsearch. And should therefore not be allowed to happen often.
XXXX-XX-XX 00:34:01,759 ERROR [Caesium-1-4] c.a.b.i.s.i.s.StreamingComparator Unable to perform job due to an unexpected error
com.atlassian.bitbucket.internal.search.indexing.exceptions.IndexException: The request to Elasticsearch failed with HTTP status code: 404Cause
There is a queue per Data Center cluster node with a 50k limit.
If this queue is not cleared fast enough no other indexing events can be enqueued.
This seems to lead to Index corruption which requires the index to be rebuilt.
Solution
- Restart Bitbucket. If a difference is detected between the HEAD commit on master and what has been indexed the index will be rebuilt.
- Rebuild the index manually Rebuild the Search server indexes for Bitbucket Server
- If feasible add a node to the cluster
Set this property to a value greater than the default in bitbucket.properties:
plugin.search.indexing.event.queue.size=70000
We recommend increasing it in increments of 20K. For large instances, we have seen good results doubling it to 100K. There's no rule of thumb for this as it depends on the content and size of repositories.