Summary

Elasticsearch fails to index content from Bitbucket.

Environment

• Bitbucket Data Center 7.7.1 with one node only
• Bundled Elasticsearch 7.5

Diagnosis

This error is found on the search logs:

[2020-01-06T11:05:13,248][WARN ][o.e.m.j.JvmGcMonitorService] [bitbucket_bundled][gc][6577647] overhead, spent [1.5s] collecting in the last [2.3s]
[2020-01-06T11:05:17,287][WARN ][o.e.m.j.JvmGcMonitorService] [bitbucket_bundled][gc][6577649] overhead, spent [2.7s] collecting in the last [3s]
[2020-01-06T11:06:27,699][WARN ][o.e.m.j.JvmGcMonitorService] [bitbucket_bundled][gc][young][6577718][7632] duration [2.2s], collections [1]/[2.3s], total [2.2s]/[5.7m], memory [2.7gb]->[1.8gb]/[3.8gb], all_pools {[young] [969.4mb]->[1.5mb]/[1gb]}{[survivor] [89mb]->[136.4mb]/[136.5mb]}{[old] [1.7gb]->[1.7gb]/[2.6gb]}

Cause

This issue is caused because Elasticsearch heap memory is not properly sized to the size of the content available to index. The messages above show that the JVM is running continuous garbage collection cycles cleaning objects from memory, due to the small heap size allocated. During those GC cycles, the JVM will pause making Elasticsearch not responsive to index content.

Solution

Increase JVM heap memory available for Elasticsearch.

1. For the bundled version of Elasticsearch edit the <BITBUCKET_HOME>/search/config/jvm.option and change the following properties from:

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms1g
-Xmx1g

To:

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms2g
-Xmx2g

2. Restart Bitbucket.

 We recommend increasing the heap in 1GB increments until the overhead messages are not shown anymore on the search logs.