Stash stops with HazelcastInstanceNotActiveException
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Problem
Stash stops and the following appears in the atlassian-stash.log:
2015-04-21 11:00:00,000 DEBUG [Navlink Plugin Executor:thread-5] c.a.p.n.c.m.s.CachingLinkedApplicationCapabilitiesImpl Failed to refresh linked application capabilities cache
com.atlassian.cache.CacheException: com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at com.atlassian.cache.memory.DelegatingCachedReference.get(DelegatingCachedReference.java:60) ~[atlassian-cache-memory-2.6.10.jar:na]
at com.atlassian.cache.hazelcast.HazelcastHybridCachedReference.get(HazelcastHybridCachedReference.java:79) ~[atlassian-cache-hazelcast-2.6.10.jar:na]
at com.atlassian.applinks.core.DefaultReadOnlyApplicationLinkService.getApplicationLinks(DefaultReadOnlyApplicationLinkService.java:128) ~[na:na]
at com.atlassian.plugins.navlink.consumer.menu.services.CapabilitiesCacheLoader.getKeys(CapabilitiesCacheLoader.java:39) ~[atlassian-nav-links-plugin-3.3.15_1425959230000.jar:na]
at com.atlassian.failurecache.ExpirationDateBasedCacheImpl.updateCacheKeys(ExpirationDateBasedCacheImpl.java:86) ~[atlassian-failure-cache-plugin-0.15.jar:na]
at com.atlassian.failurecache.ExpirationDateBasedCacheImpl.refresh(ExpirationDateBasedCacheImpl.java:78) ~[atlassian-failure-cache-plugin-0.15.jar:na]
at com.atlassian.plugins.navlink.consumer.menu.services.CachingLinkedApplicationCapabilitiesImpl.refreshCache(CachingLinkedApplicationCapabilitiesImpl.java:90) [atlassian-nav-links-plugin-3.3.15_1425959230000.jar:na]
at com.atlassian.plugins.navlink.consumer.menu.services.CachingLinkedApplicationCapabilitiesImpl.run(CachingLinkedApplicationCapabilitiesImpl.java:70) [atlassian-nav-links-plugin-3.3.15_1425959230000.jar:na]
at com.atlassian.sal.core.executor.ThreadLocalDelegateRunnable.run(ThreadLocalDelegateRunnable.java:38) ~[sal-core-2.13.3.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_25]
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) ~[na:1.7.0_25]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) ~[na:1.7.0_25]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) ~[na:1.7.0_25]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.7.0_25]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_25]
at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]
... 21 frames trimmed
Caused by: com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!Cause
When Hazelcast, the library Bitbucket Server uses for caching and many other things, detects an OutOfMemoryError it automatically shuts down.
Upon observing a lot of Full GC operations the heap size may need to be increased. The high memory consumption may also be caused by add-ons which don't manage their memory well.
Resolution
Upgrade to Bitbucket Server 3.10+
In Bitbucket Server 3.10, the behavior was improved around Hazelcast and OutOfMemoryErrors - BSERV-7420 - Getting issue details... STATUS
Increase the heap size
setenv and environment variable changes in Bitbucket Server 5.0+
Starting with Bitbucket Server 5.0, setenv.sh and setenv.bat have been removed. The options that were set in this file can now be set via environment variables. Where to set the environment variable depends on which Operating System you're running on.
Linux
When using the atlbitbucket service on Linux, the environment variables are ignored. You must set the parameters in _start-webapp.sh (or start-bitbucket.sh). These values will be read when the service starts.
As an example, to set JVM_SUPPORT_RECOMMENDED_ARGS, you would add this line to the file:
JVM_SUPPORT_RECOMMENDED_ARGS=-XX:+HeapDumpOnOutOfMemoryErrorWindows
Set the parameter as an environment variable for the user running Bitbucket Server. For example, if you want to set JVM_SUPPORT_RECOMMENDED_ARGS, create it as an environment variable and assign the appropriate value to it. When Bitbucket Server starts using the startup scripts or service, it will pick up and apply this value.
Linux
Windows Service
Windows (starting from .bat file)
If having any 3rd party add-ons installed that could be causing this:
- Disable the add-on for a while.
- Monitor the Bitbucket Server instance, checking if the issue persists.
Some add-ons may require more memory to be used, so increasing the heap space may help in this case.
If the above doesn't help and the issue needs to be further investigated generate a Heap Dump as per the instructions in How to debug Out of Memory Heap Space and open a support ticket at https://support.atlassian.com.