c.a.s.i.h.OptimisticOutOfMemoryHandler OutOfMemoryError in Bitbucket Server
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
At least 2 nodes are failing to server git clone requests and have logs that indicate an OutOfMemory issue.
2 or more nodes are throwing OOMs and have to be rebooted and re-added to the cluster.
Environment
- encountered at 6.10.4
Diagnosis
2020-10-29 21:55:59,518 WARN [hz.hazelcast.cached.thread-26] c.a.s.i.h.OptimisticOutOfMemoryHandler OutOfMemoryError occurred attempting to continue operating as normal
java.lang.OutOfMemoryError: Java heap space
- heap dump contains a lot of unprocessed packet related object (look for NIO)
- For additional diagnosis enable JMX and gather MBean statistics
- from the Bitbucket host machine issue
free -mas well assar -n DEVto gather memory and networking statistics
Cause
Slow network has caused cluster congestion and caused an OOM at the hazelcast level.
Solution
Remediation is needed at the Network level after confirming network latency during these same periods of OOM at the JVM level.