Bamboo Specs scans causes server to run out of resources
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Bamboo Repository Stored Java Specs scans running in Docker may consume all available memory and cause the Bamboo server to crash due to running out of resources. As Java specs is processed by Maven in a different JVM, a misconfigured specs project can consume resources that are beyond what is setup for Bamboo.
Diagnosis
The following stacktrace can be found in the server logs:
java.lang.OutOfMemoryError
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at com.atlassian.bamboo.configuration.external.RepositoryStoredSpecsServiceImpl.generateBambooYamlsFromSpecs(RepositoryStoredSpecsServiceImpl.java:608)
at com.atlassian.bamboo.configuration.external.RepositoryStoredSpecsServiceImpl.processSpecs(RepositoryStoredSpecsServiceImpl.java:492)
at com.atlassian.bamboo.configuration.external.RepositoryStoredSpecsServiceImpl.runBambooSpecs(RepositoryStoredSpecsServiceImpl.java:297)
at com.atlassian.bamboo.configuration.external.RssDetectionServiceImpl.runRssDetection(RssDetectionServiceImpl.java:267)
at com.atlassian.bamboo.configuration.external.detection.RssDetectionRunnable.run(RssDetectionRunnable.java:42)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at com.atlassian.bamboo.utils.BambooRunnables$1.run(BambooRunnables.java:48)
at com.atlassian.bamboo.security.ImpersonationHelper.runWith(ImpersonationHelper.java:26)
at com.atlassian.bamboo.security.ImpersonationHelper.runWithSystemAuthority(ImpersonationHelper.java:17)
at com.atlassian.bamboo.security.ImpersonationHelper$1.run(ImpersonationHelper.java:41)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError
at java.lang.AbstractStringBuilder.hugeCapacity(AbstractStringBuilder.java:161)
at java.lang.AbstractStringBuilder.newCapacity(AbstractStringBuilder.java:155)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:125)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:541)
at java.lang.StringBuilder.append(StringBuilder.java:175)
at java.lang.StringBuilder.append(StringBuilder.java:76)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:488)
at java.lang.StringBuilder.append(StringBuilder.java:166)
at com.spotify.docker.client.DefaultLogStream.readFully(DefaultLogStream.java:78)
at com.atlassian.bamboo.configuration.external.RepositoryStoredSpecsServiceImpl.runSpecsProcessingInDocker(RepositoryStoredSpecsServiceImpl.java:872)
at com.atlassian.bamboo.configuration.external.RepositoryStoredSpecsServiceImpl.lambda$runSpecsWithDocker$9(RepositoryStoredSpecsServiceImpl.java:839)
... 8 moreCause
Bamboo offers the option to process specs in a docker container (Administration > Security > Repository Stored Specs security settings > Process Bamboo Specs in Docker). When limits are not set in Docker, the container may take up all available resources while running the specs code.
Solution
The limits should be set in /etc/systemd/system/docker_limit.slice. See the example below
cat /etc/systemd/system/docker_limit.slice
[Unit] Description=Slice that limits docker resources Before=slices.target [Slice] CPUAccounting=true CPUQuota=90% MemoryAccounting=true MemoryHigh=1G MemoryMax=1.2GMemoryLimit(cgroupv1) may be required instead ofMemoryMax(cgroupv2) depending on the cgroup version in your OS: Resource control unit settings
cat /etc/docker/daemon.json
{ "exec-opts": ["native.cgroupdriver=systemd"], "cgroup-parent": "docker_limit.slice" }
Alternatively, you can set a limit to Maven's JVM heap size by adding a .mvn/jvm.config to the Java specs project:
-Xmx128m -Xms128m
Please follow and vote on the feature improvement request below
-
BAM-21187 - Add UI option to limit memory consumption for RSS containers in Bamboo Gathering Interest