Summary

Bamboo builds fail semi-randomly and you see messages like this in the server logs:

Build XXX had to be cancelled: it was marked as queued but was not present in the queue for (at least) the past 720 seconds.

Cause

There are multiple causes of this problem, each with a different solution.

Cause A: Multiple builds triggered at the same time in versions older than 6.2.2

Post-commit triggers are superior to polling because they reduce the load on the build server and source control.  However, if multiple commits occur rapidly this can cause a race condition. In this case, there may be "ghost" builds that result in the "not present in the queue for 720 seconds" messages.  BAM-17890 - Getting issue details... STATUS

Cause B: Conflict with Group Agent Plugin and Bamboo 5.7.x

There is an incompatibility between Bamboo 5.7.x. and the Group Agent plugin version 1.6.  If you are affected by this you will see errors when adding builds to the queue in your logs:

2014-11-21 09:41:59,766 INFO [3-DelayedChangeDetectionThread:pool-7-thread-3] [ChainExecutionManagerImpl] Build BCPRELEASE8-PAR-JOB1-331 has been dispatched
2014-11-21 09:41:59,767 INFO [AtlassianEvent::0-BAM::EVENTS:pool-3-thread-2] [BuildQueueManagerImpl] Attempting to queue BCP Release 8 - Parent - Run build #331 (BCPRELEASE8-PAR-JOB1-331).
2014-11-21 09:41:59,768 INFO [AtlassianEvent::0-BAM::EVENTS:pool-3-thread-2] [ExecutionPhaseServiceImpl] BCP Release 8 - Parent - Run build #331 (BCPRELEASE8-PAR-JOB1-331) queued
2014-11-21 09:41:59,802 ERROR [AtlassianEvent::0-BAM::EVENTS:pool-3-thread-2] [AsynchronousAbleEventDispatcher] There was an exception thrown trying to dispatch event '[com.atlassian.bamboo.v2.build.events.BuildTriggeredEvent[source=com.atlassian.bamboo.build.DefaultBuildExecutionManager@74793ec9], SingleParameterMethodListenerInvoker{method=public void com.atlassian.bamboo.v2.build.queue.BuildTriggeredListener.handleEvent(com.atlassian.bamboo.v2.build.events.BuildTriggeredEvent), listener=com.atlassian.bamboo.v2.build.queue.BuildTriggeredListener@2370e095}]' from the invoker 'java.lang.RuntimeException: com.edwardawebb.bamboo.groupagent.filter.GroupAgentFilter.filter(Lcom/atlassian/bamboo/v2/build/CommonContext;Ljava/util/Collection;Lcom/atlassian/bamboo/v2/build/agent/capability/MinimalRequirementSet;)Ljava/util/Collection;'

Cause C: Default setting for Bamboo versions older than 5.0

In versions of Bamboo prior to version 5.0 the default value of bamboo.send.file.names.to.agent is set to true which sometimes results in problems adding builds to the queue.  This setting only affects Bamboo versions less than 5.0 and is already deprecated.

Cause D: The agent may have the wrong file/folder permissions and ownership

You may find a java.lang.IllegalStateException: Agent working directory is not a directory message in the Agent logs <bamboo-agent-home>/atlassian-bamboo-agent.log. That error means the Bamboo agent cannot write to a specific location within its work structure. This is an indicator of wrong file permissions and/or incorrect ownership.

This is a very common problem and happens when the Agent is inadvertently started with a user other than the regular Bamboo Agent user (E.g. root)

2021-09-23 19:30:31,212 ERROR [0-BAM::agent-1.mydomain.com::Agent:pool-3-thread-1] [BuildAgentControllerImpl] Unknown exception occurred on 'agent-1.mydomain.com'. Agent will attempt to recover its normal operation...
io.atlassian.util.concurrent.LazyReference$InitializationException: java.lang.IllegalStateException: Agent working directory is not a directory
	at io.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:156)
	at io.atlassian.util.concurrent.LazyReference.get(LazyReference.java:116)
	at com.atlassian.bamboo.v2.build.agent.remote.workspaces.RemoteAgentWorkspaceManagerImpl.registerWorkspace(RemoteAgentWorkspaceManagerImpl.java:183)
	at com.atlassian.bamboo.v2.build.agent.remote.RemoteBuildAgent.onContextReceived(RemoteBuildAgent.java:134)
	at com.atlassian.bamboo.v2.build.agent.BuildAgentControllerImpl.waitAndPerformBuild(BuildAgentControllerImpl.java:111)
	at com.atlassian.bamboo.v2.build.agent.DefaultBuildAgent.lambda$start$0(DefaultBuildAgent.java:110)
	at com.atlassian.bamboo.utils.BambooRunnables$1.run(BambooRunnables.java:48)
	at com.atlassian.bamboo.security.ImpersonationHelper.runWith(ImpersonationHelper.java:26)
	at com.atlassian.bamboo.security.ImpersonationHelper.runWithSystemAuthority(ImpersonationHelper.java:17)
	at com.atlassian.bamboo.security.ImpersonationHelper$1.run(ImpersonationHelper.java:41)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Agent working directory is not a directory
	at com.atlassian.bamboo.v2.build.agent.remote.workspaces.RemoteAgentWorkspaceManagerImpl$1.create(RemoteAgentWorkspaceManagerImpl.java:72)
	at com.atlassian.bamboo.v2.build.agent.remote.workspaces.RemoteAgentWorkspaceManagerImpl$1.create(RemoteAgentWorkspaceManagerImpl.java:64)
	at io.atlassian.util.concurrent.LazyReference$Sync.run(LazyReference.java:332)
	at io.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:150)
	... 10 more

Cause E: No space left on device

We had reports from customers that this issue might happen when the Agent has not enough free space left on the <bamboo-agent-home> location.

Solution

Solution A: Add a 30 second quiet period to the linked repository associated with the build plan.

1. Edit the settings for your linked repository
2. Expand the Advanced section
3. Check the box for "Enable quiet period" and enter "30" in the text field
4. Save the changes

For more detailed instructions for all linked repository types available to Bamboo see Linking to source code repositories.

Solution B: Upgrade the Group Agent Plugin to version 1.7 or greater

1. Go to Administration > Add-ons
2. Find "bamboo-group-agent" in the list of installed plugins

3. Click "Update" and wait for the process to complete

4. Restart Bamboo for the plugin changes to take effect

For full details see Installing Marketplace apps.

Solution C: Change bamboo.send.file.names.to.agent to false

Apply this solution only if that property is set to true. By default, it is already set to false.

1. Edit the file $BAMBOO_INSTALL/conf/wrapper.conf

2. Find the lines that start with wrapper.java.additional and add another line, incrementing the number that follows.  Example:

   wrapper.java.additional.1=-Dorg.eclipse.jetty.xml.XmlParser.Validating=false
   wrapper.java.additional.2=-XX:MaxPermSize=256m
   wrapper.java.additional.3=-Djava.awt.headless=true
   wrapper.java.additional.4=-Djava.io.tmpdir=%WINDIR%/Temp

3. Add another line beginning with wrapper.java.additional. plus the next number in the sequence, followed by =-Dbamboo.send.file.names.to.agent=false}.  Using the example above this would look like this: 

   wrapper.java.additional.5=-Dbamboo.send.file.names.to.agent=false

Solution D: Investigate and fix the agent's file/folder permissions and ownership

1. Stop the agent

2. In Linux, run the following command as root to list the permissions of the <bamboo-agent-home> folders and files. In Windows, you will need to investigate the permissions manually or use PowerShell's Get-Acl/Get-ChildItem

   For directories

   find <bamboo-agent-home> -maxdepth 2 \( -type f -o -type d \) -ls

3. Based on the output, fix the permissions and ownership of the files and start the Agent

Solution E: Cleanup some disk space or extend the <bamboo-agent-home> filesystem

1. Stop the agent
2. Extend or cleanup the <bamboo-agent-home> filesystem
3. Start the agent