Stash stops with HazelcastInstanceNotActiveException
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Problem
Stash stops and the following appears in the atlassian-stash.log
:
2015-04-21 11:00:00,000 DEBUG [Navlink Plugin Executor:thread-5] c.a.p.n.c.m.s.CachingLinkedApplicationCapabilitiesImpl Failed to refresh linked application capabilities cache
com.atlassian.cache.CacheException: com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at com.atlassian.cache.memory.DelegatingCachedReference.get(DelegatingCachedReference.java:60) ~[atlassian-cache-memory-2.6.10.jar:na]
at com.atlassian.cache.hazelcast.HazelcastHybridCachedReference.get(HazelcastHybridCachedReference.java:79) ~[atlassian-cache-hazelcast-2.6.10.jar:na]
at com.atlassian.applinks.core.DefaultReadOnlyApplicationLinkService.getApplicationLinks(DefaultReadOnlyApplicationLinkService.java:128) ~[na:na]
at com.atlassian.plugins.navlink.consumer.menu.services.CapabilitiesCacheLoader.getKeys(CapabilitiesCacheLoader.java:39) ~[atlassian-nav-links-plugin-3.3.15_1425959230000.jar:na]
at com.atlassian.failurecache.ExpirationDateBasedCacheImpl.updateCacheKeys(ExpirationDateBasedCacheImpl.java:86) ~[atlassian-failure-cache-plugin-0.15.jar:na]
at com.atlassian.failurecache.ExpirationDateBasedCacheImpl.refresh(ExpirationDateBasedCacheImpl.java:78) ~[atlassian-failure-cache-plugin-0.15.jar:na]
at com.atlassian.plugins.navlink.consumer.menu.services.CachingLinkedApplicationCapabilitiesImpl.refreshCache(CachingLinkedApplicationCapabilitiesImpl.java:90) [atlassian-nav-links-plugin-3.3.15_1425959230000.jar:na]
at com.atlassian.plugins.navlink.consumer.menu.services.CachingLinkedApplicationCapabilitiesImpl.run(CachingLinkedApplicationCapabilitiesImpl.java:70) [atlassian-nav-links-plugin-3.3.15_1425959230000.jar:na]
at com.atlassian.sal.core.executor.ThreadLocalDelegateRunnable.run(ThreadLocalDelegateRunnable.java:38) ~[sal-core-2.13.3.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_25]
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) ~[na:1.7.0_25]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) ~[na:1.7.0_25]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) ~[na:1.7.0_25]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.7.0_25]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_25]
at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]
... 21 frames trimmed
Caused by: com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
Cause
When Hazelcast, the library Bitbucket Server uses for caching and many other things, detects an OutOfMemoryError it automatically shuts down.
Upon observing a lot of Full GC operations the heap size may need to be increased. The high memory consumption may also be caused by add-ons which don't manage their memory well.
Resolution
Upgrade to Bitbucket Server 3.10+
In Bitbucket Server 3.10, the behavior was improved around Hazelcast and OutOfMemoryErrors - BSERV-7420 - Improving handling of OOME in certain contexts
Increase the heap size
setenv and environment variable changes in Bitbucket Server 5.0+
Starting with Bitbucket Server 5.0, setenv.sh
and setenv.bat
have been removed. The options that were set in this file can now be set via environment variables. Where to set the environment variable depends on which Operating System you're running on.
Linux
When using the atlbitbucket
service on Linux, the environment variables are ignored. You must set the parameters in _start-webapp.sh (or start-bitbucket.sh)
. These values will be read when the service starts.
As an example, to set JVM_SUPPORT_RECOMMENDED_ARGS
, you would add this line to the file:
JVM_SUPPORT_RECOMMENDED_ARGS=-XX:+HeapDumpOnOutOfMemoryError
Windows
Set the parameter as an environment variable for the user running Bitbucket Server. For example, if you want to set JVM_SUPPORT_RECOMMENDED_ARGS
, create it as an environment variable and assign the appropriate value to it. When Bitbucket Server starts using the startup scripts or service, it will pick up and apply this value.
Linux
Edit <bitbucket-install>/bin/setenv.sh
and modify the JVM_MAXIMUM_MEMORY value to be larger.
Windows Service
- Identify the name of the service that Bitbucket Server is installed as in Windows (
Control Panel > Administrative Tools > Services
): - Open the command window from
Start >> Run >> type in 'cmd' >> Enter
cd
to thebin
directory of your BITBUCKET installation directory.Run:
tomcat8w //ES//%SERVICENAME%
- Click on the
Java
tab to see the list of current start-up options. Modify the
-Xmx
value to be larger.
Windows (starting from .bat file)
Edit <bitbucket-install>/bin/setenv.bat
and modify the JVM_MAXIMUM_MEMORY value to be larger.
If having any 3rd party add-ons installed that could be causing this:
- Disable the add-on for a while.
- Monitor the Bitbucket Server instance, checking if the issue persists.
Some add-ons may require more memory to be used, so increasing the heap space may help in this case.
If the above doesn't help and the issue needs to be further investigated generate a Heap Dump as per the instructions in How to debug Out of Memory Heap Space and open a support ticket at https://support.atlassian.com.