Problem

Hundreds of WARN messages like the following are in the atlassian-bitbucket-yyyy-mm-dd.log: 

2018-04-18 07:18:00,882 WARN  [cached12]  c.h.s.i.o.impl.Invocation [10.64.98.221]:5701 [stash-prod-cluster] [3.5.2-atlassian-37] Asking if operation execution has been started: com.hazelcast.spi.impl.operationservice.impl.IsStillRunningService$InvokeIsStillRunningOperationRunnable@336f67ee

These can lead to OutOfMemory ERRORS that can bring down nodes. 

Cause

This is caused by Hazelcast bug 5468 which is described in Hazelcast Cluster members going out of memory due to huge number of “IsStillRunningService” objects. Bitbucket 4.14.x uses 3.5.2 of the Hazelcast library which is based on Hazelcast 3.5 and contains Hazelcast bug 5468. This bug has been fixed in Hazelcast 3.7.x. 

Resolution

Upgrade to Bitbucket 5.x versions which use Hazelcast 3.7.4.