OutOfMemory Errors in Bamboo Elastic Agent
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Problem
Bamboo builds start failing when executed on elastic agents and the following exception is observed in the Bamboo System Error Details page (accessible through the Bamboo UI):
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.io.UnixFileSystem.resolve(UnixFileSystem.java:108)
at java.io.File.<init>(File.java:367)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1256)
...
Diagnosis
Environment
- An elastic agent is an agent that runs in the Amazon Elastic Compute Cloud (EC2). An elastic agent process runs in an elastic instance of an elastic image.
Diagnostic Steps
- If the build process is running out of memory while executing a Maven or Ant task, please check the "Bamboo builds fail due to java.lang.OutOfMemoryError" knowledge base article as it describes how to workaround the problem and allocate more memory to both
ANT_OPTS
andMAVEN_OPTS
. - If you are not running one of the tasks mentioned above, but the elastic agent is still running out of memory, please continue to read.
Cause
The default memory setting for the elastic agent is 256MB. It is necessary to adjust the settings to run a bigger elastic agent with sufficient memory.
Resolution
Third-party apps have also be known to increase the memory requirements for Elastic Agents. Artifactory for Bamboo in particular is one that requires more memory than the default configuration when it is loading. If you are looking to reduce the memory requirements for your elastic agents, rather than increase the limits, removing unneeded third-party apps can reduce the overhead.
For Bamboo 8.0.0 and later versions
If you use a custom Elastic Image that has been created before Bamboo 8.0, you must recreate it with the new version of the atlassian-bamboo-agent-elastic-installer.jar
on it in order to use the Elastic Agent supervisor.
Elastic Agent startup scripts, provided in the atlassian-bamboo-elastic-image, use a dedicated variable WRAPPER_PROPERTIES
that can carry one or multiple properties and is passed to the Elastic Agent startup command. The WRAPPER_PROPERTIES
variable can be set up in an Elastic Image startup script in Bamboo to increase the initial and maximum Java heap size:
Go to Bamboo administration >> Overview >> Elastic Bamboo >> Image configurations page. From there edit the desired elastic image and add the following script to the Instance startup script field:
Linux:
sudo su -c "echo \"export WRAPPER_PROPERTIES='\"-Dwrapper.java.maxmemory=2048\" \"-Dwrapper.java.initmemory=512\"'\" >> /etc/profile.d/bamboo.sh"
Windows:
Option 1:
powershell.exe -Command "[System.Environment]::SetEnvironmentVariable(\"WRAPPER_PROPERTIES\", '\"-Dwrapper.java.maxmemory=2048\" \"-Dwrapper.java.initmemory=512\"', 'Machine')"
The WRAPPER_PROPERTIES variable can also be used to pass other properties to the agent, please see the Elastic Agent supervisor documentation for more details.
For Bamboo versions below 8.0.0
It is possible to change the heap size of the elastic agent using the Instance startup script option located on the Bamboo administration >> Overview >> Elastic Bamboo >> Image configurations
page. From there edit the desired elastic image and add the following script to the Instance startup script field:
The example below will bump the Maximum Heap size by replacing the "-Xmx256m" entries with "-Xmx512m". You can extend the commands to add more Java Heap memory options such as modifying the initial heap size "-Xms32m"
Linux:
sed -i 's@.*MARKER.*@sed -i "s/-Xmx256m/-Xmx512m/" $startupScript@' /opt/bamboo-elastic-agent/bin/bamboo-elastic-agent
Windows:
Option 1:
If you have created a custom Windows AMI from scratch without using one of our stock images as a base, our start-up scripts will likely not execute unless you configured it in exactly the same way as our stock images in the Task Scheduler. If startup scripts do not execute, this workaround will not work and you should refer to Option #2.
sed -i "s/-Dimagefiles/-Dbamboo.agent.syncOnly -Dimagefiles/" C:\opt\bamboo-elastic-agent\bin\bamboo-elastic-agent.bat
sed -i 's@cd %%OLD_PWD%%@sed -i \'s/-Xmx256m/-Xmx512m/\' \%%TEMP\%%\\\\bamboo-agent-bootstrap-1.bat\n\%%TEMP\%%\\\\bamboo-agent-bootstrap-1.bat ^>^>bamboo-elastic-agent.out 2^>\^&1\ncd %%OLD_PWD%%@' C:\opt\bamboo-elastic-agent\bin\bamboo-elastic-agent.bat
Option 2:
The below option will override heap sizing for any Java processes that are run after the instance starts up so may not be ideal if you're using anything Java in your builds (e.g. Gradle or Maven)
setx -m _JAVA_OPTIONS "-Xms256m -Xmx512m"
This will increase the maximum elastic agent memory to 512MB. It is possible to adjust the -Xmx
setting to allocate more memory to the agent, if necessary.