OutOfMemory Errors in Bamboo elastic agent
Platform Notice: Server and Data Center Only. This article only applies to Atlassian products on the server and data center platforms.
Bamboo builds start failing when executed on elastic agents and the following exception is observed in the Bamboo System Error Details page (accessible through the Bamboo UI):
java.lang.OutOfMemoryError: GC overhead limit exceeded at java.io.UnixFileSystem.resolve(UnixFileSystem.java:108) at java.io.File.<init>(File.java:367) at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1256) ...
- An elastic agent is an agent that runs in the Amazon Elastic Compute Cloud (EC2). An elastic agent process runs in an elastic instance of an elastic image.
- If the build process is running out of memory while executing a Maven or Ant task, please check the "Bamboo builds fail due to java.lang.OutOfMemoryError" knowledge base article as it describes how to workaround the problem and allocate more memory to both
- If you are not running one of the tasks mentioned above, but the elastic agent is still running out of memory, please continue to read.
The default memory setting for the elastic agent is 256MB. It is necessary to adjust the settings to run a bigger elastic agent with sufficient memory.
Third-party apps have also be known to increase the memory requirements for Elastic Agents. Artifactory for Bamboo in particular is one that requires more memory than the default configuration when it is loading. If you are looking to reduce the memory requirements for your elastic agents, rather than increase the limits, removing unneeded third-party apps can reduce the overhead.
For Bamboo 8.0.0 and later versions
If you use a custom Elastic Image that has been created before Bamboo 8.0, you must recreate it with the new version of the
atlassian-bamboo-agent-elastic-installer.jar on it in order to use the Elastic Agent supervisor.
Elastic Agent startup scripts, provided in the atlassian-bamboo-elastic-image, use a dedicated variable
WRAPPER_PROPERTIES that can carry one or multiple properties and is passed to the Elastic Agent startup command. The
WRAPPER_PROPERTIES variable can be set up in an Elastic Image startup script in Bamboo to increase the initial and maximum Java heap size:
Go to Bamboo administration >> Overview >> Elastic Bamboo >> Image configurations page. From there edit the desired elastic image and add the following script to the Instance startup script field:
sudo su -c "echo \"export WRAPPER_PROPERTIES='\"-Dwrapper.java.maxmemory=2048\" \"-Dwrapper.java.initmemory=512\"'\" >> /etc/profile.d/bamboo.sh"
powershell.exe -Command "[System.Environment]::SetEnvironmentVariable(\"WRAPPER_PROPERTIES\", '\"-Dwrapper.java.maxmemory=2048\" \"-Dwrapper.java.initmemory=512\"', 'Machine')"
The WRAPPER_PROPERTIES variable can also be used to pass other properties to the agent, please see the Elastic Agent supervisor documentation for more details.
For Bamboo versions below 8.0.0
It is possible to change the heap size of the elastic agent using the Instance startup script option located on the
Bamboo administration >> Overview >> Elastic Bamboo >> Image configurations page. From there edit the desired elastic image and add the following script to the Instance startup script field:
The example below will bump the Maximum Heap size by replacing the "-Xmx256m" entries with "-Xmx512m". You can extend the commands to add more Java Heap memory options such as modifying the initial heap size "-Xms32m"
sed -i 's@.*MARKER.*@sed -i "s/-Xmx256m/-Xmx512m/" $startupScript@' /opt/bamboo-elastic-agent/bin/bamboo-elastic-agent
If you have created a custom Windows AMI from scratch without using one of our stock images as a base, our start-up scripts will likely not execute unless you configured it in exactly the same way as our stock images in the Task Scheduler. If startup scripts do not execute, this workaround will not work and you should refer to Option #2.
sed -i "s/-Dimagefiles/-Dbamboo.agent.syncOnly -Dimagefiles/" C:\opt\bamboo-elastic-agent\bin\bamboo-elastic-agent.bat sed -i 's@cd %%OLD_PWD%%@sed -i \'s/-Xmx256m/-Xmx512m/\' \%%TEMP\%%\\\\bamboo-agent-bootstrap-1.bat\n\%%TEMP\%%\\\\bamboo-agent-bootstrap-1.bat ^>^>bamboo-elastic-agent.out 2^>\^&1\ncd %%OLD_PWD%%@' C:\opt\bamboo-elastic-agent\bin\bamboo-elastic-agent.bat
The below option will override heap sizing for any Java processes that are run after the instance starts up so may not be ideal if you're using anything Java in your builds (e.g. Gradle or Maven)
setx -m _JAVA_OPTIONS "-Xms256m -Xmx512m"
This will increase the maximum elastic agent memory to 512MB. It is possible to adjust the
-Xmx setting to allocate more memory to the agent, if necessary.