Summary

Bamboo builds fail semi-randomly and you see messages like this in the server logs:

Build XXX had to be cancelled: it was marked as queued but was not present in the queue for (at least) the past 720 seconds.

Cause

There are multiple causes of this problem, each with a different solution.

Cause A: Multiple builds triggered at the same time in versions older than 6.2.2

Post-commit triggers are superior to polling because they reduce the load on the build server and source control.  However, if multiple commits occur rapidly this can cause a race condition. In this case, there may be "ghost" builds that result in the "not present in the queue for 720 seconds" messages.  BAM-17890 - Getting issue details... STATUS

Cause B: Conflict with Group Agent Plugin and Bamboo 5.7.x

There is an incompatibility between Bamboo 5.7.x. and the Group Agent plugin version 1.6.  If you are affected by this you will see errors when adding builds to the queue in your logs:

2014-11-21 09:41:59,766 INFO [3-DelayedChangeDetectionThread:pool-7-thread-3] [ChainExecutionManagerImpl] Build BCPRELEASE8-PAR-JOB1-331 has been dispatched
2014-11-21 09:41:59,767 INFO [AtlassianEvent::0-BAM::EVENTS:pool-3-thread-2] [BuildQueueManagerImpl] Attempting to queue BCP Release 8 - Parent - Run build #331 (BCPRELEASE8-PAR-JOB1-331).
2014-11-21 09:41:59,768 INFO [AtlassianEvent::0-BAM::EVENTS:pool-3-thread-2] [ExecutionPhaseServiceImpl] BCP Release 8 - Parent - Run build #331 (BCPRELEASE8-PAR-JOB1-331) queued
2014-11-21 09:41:59,802 ERROR [AtlassianEvent::0-BAM::EVENTS:pool-3-thread-2] [AsynchronousAbleEventDispatcher] There was an exception thrown trying to dispatch event '[com.atlassian.bamboo.v2.build.events.BuildTriggeredEvent[source=com.atlassian.bamboo.build.DefaultBuildExecutionManager@74793ec9], SingleParameterMethodListenerInvoker{method=public void com.atlassian.bamboo.v2.build.queue.BuildTriggeredListener.handleEvent(com.atlassian.bamboo.v2.build.events.BuildTriggeredEvent), listener=com.atlassian.bamboo.v2.build.queue.BuildTriggeredListener@2370e095}]' from the invoker 'java.lang.RuntimeException: com.edwardawebb.bamboo.groupagent.filter.GroupAgentFilter.filter(Lcom/atlassian/bamboo/v2/build/CommonContext;Ljava/util/Collection;Lcom/atlassian/bamboo/v2/build/agent/capability/MinimalRequirementSet;)Ljava/util/Collection;'

Cause C: Default setting for Bamboo versions older than 5.0

In versions of Bamboo prior to version 5.0 the default value of bamboo.send.file.names.to.agent is set to true which sometimes results in problems adding builds to the queue.  This setting only affects Bamboo versions less than 5.0 and is already deprecated.

Cause D: The agent may have the wrong file/folder permissions and ownership

You may find a java.lang.IllegalStateException: Agent working directory is not a directory message in the Agent logs <bamboo-agent-home>/atlassian-bamboo-agent.log. That error means the Bamboo agent cannot write to a specific location within its work structure. This is an indicator of wrong file permissions and/or incorrect ownership.

This is a very common problem and happens when the Agent is inadvertently started with a user other than the regular Bamboo Agent user (E.g. root)

2021-09-23 19:30:31,212 ERROR [0-BAM::agent-1.mydomain.com::Agent:pool-3-thread-1] [BuildAgentControllerImpl] Unknown exception occurred on 'agent-1.mydomain.com'. Agent will attempt to recover its normal operation...
io.atlassian.util.concurrent.LazyReference$InitializationException: java.lang.IllegalStateException: Agent working directory is not a directory
	at io.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:156)
	at io.atlassian.util.concurrent.LazyReference.get(LazyReference.java:116)
	at com.atlassian.bamboo.v2.build.agent.remote.workspaces.RemoteAgentWorkspaceManagerImpl.registerWorkspace(RemoteAgentWorkspaceManagerImpl.java:183)
	at com.atlassian.bamboo.v2.build.agent.remote.RemoteBuildAgent.onContextReceived(RemoteBuildAgent.java:134)
	at com.atlassian.bamboo.v2.build.agent.BuildAgentControllerImpl.waitAndPerformBuild(BuildAgentControllerImpl.java:111)
	at com.atlassian.bamboo.v2.build.agent.DefaultBuildAgent.lambda$start$0(DefaultBuildAgent.java:110)
	at com.atlassian.bamboo.utils.BambooRunnables$1.run(BambooRunnables.java:48)
	at com.atlassian.bamboo.security.ImpersonationHelper.runWith(ImpersonationHelper.java:26)
	at com.atlassian.bamboo.security.ImpersonationHelper.runWithSystemAuthority(ImpersonationHelper.java:17)
	at com.atlassian.bamboo.security.ImpersonationHelper$1.run(ImpersonationHelper.java:41)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Agent working directory is not a directory
	at com.atlassian.bamboo.v2.build.agent.remote.workspaces.RemoteAgentWorkspaceManagerImpl$1.create(RemoteAgentWorkspaceManagerImpl.java:72)
	at com.atlassian.bamboo.v2.build.agent.remote.workspaces.RemoteAgentWorkspaceManagerImpl$1.create(RemoteAgentWorkspaceManagerImpl.java:64)
	at io.atlassian.util.concurrent.LazyReference$Sync.run(LazyReference.java:332)
	at io.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:150)
	... 10 more

Cause E: No space left on device

We had reports from customers that this issue might happen when the Agent has not enough free space left on the <bamboo-agent-home> location.

Cause F: Too many full Agents recalculations in parallel

Recalculation messages are printed in a timeframe very close to the "720s" events. These recalculations of type "AgentAssignmentsUpdatedEvent" are called when there is an update that will effect all Agents. For example, disabling/enabling all agents or updating capability sets. As no Agents will pickup from the Queue when it gets rewritten that may cause the "720s" if it lasts for too long. This is better described on BAM-25145 - Getting issue details... STATUS .

2023-10-03 21:30:52,125 INFO [AtlassianEvent::0-BAM::EVENTS:pool-2-thread-17] [BuildQueueManagerImpl$1] Recalculation #4 started
2023-10-03 21:32:54,959 INFO [AtlassianEvent::0-BAM::EVENTS:pool-2-thread-17] [BuildQueueManagerImpl$1] Recalculation #4 took 2.047 min

2023-10-03 21:33:03,418 INFO [AtlassianEvent::0-BAM::EVENTS:pool-2-thread-64] [BuildQueueManagerImpl$1] Recalculation #5 started
2023-10-03 21:35:17,558 INFO [AtlassianEvent::0-BAM::EVENTS:pool-2-thread-64] [BuildQueueManagerImpl$1] Recalculation #5 took 2.236 min

2023-10-03 21:35:17,559 INFO [AtlassianEvent::0-BAM::EVENTS:pool-2-thread-10] [BuildQueueManagerImpl$1] Recalculation #6 started
2023-10-03 21:37:44,734 INFO [AtlassianEvent::0-BAM::EVENTS:pool-2-thread-10] [BuildQueueManagerImpl$1] Recalculation #6 took 2.453 min

2023-10-03 21:37:44,735 INFO [AtlassianEvent::0-BAM::EVENTS:pool-2-thread-35] [BuildQueueManagerImpl$1] Recalculation #7 started
2023-10-03 21:40:25,668 INFO [AtlassianEvent::0-BAM::EVENTS:pool-2-thread-35] [BuildQueueManagerImpl$1] Recalculation #7 took 2.682 min

Solution

Solution A: Add a 30 second quiet period to the linked repository associated with the build plan.

1. Edit the settings for your linked repository
2. Expand the Advanced section
3. Check the box for "Enable quiet period" and enter "30" in the text field
4. Save the changes

For more detailed instructions for all linked repository types available to Bamboo see Linking to source code repositories.

Solution B: Upgrade the Group Agent Plugin to version 1.7 or greater

1. Go to Administration > Add-ons
2. Find "bamboo-group-agent" in the list of installed plugins

3. Click "Update" and wait for the process to complete

4. Restart Bamboo for the plugin changes to take effect

For full details see Installing Marketplace apps.

Solution C: Change bamboo.send.file.names.to.agent to false

Apply this solution only if that property is set to true. By default, it is already set to false.

1. Edit the file $BAMBOO_INSTALL/conf/wrapper.conf

2. Find the lines that start with wrapper.java.additional and add another line, incrementing the number that follows.  Example:

   wrapper.java.additional.1=-Dorg.eclipse.jetty.xml.XmlParser.Validating=false
   wrapper.java.additional.2=-XX:MaxPermSize=256m
   wrapper.java.additional.3=-Djava.awt.headless=true
   wrapper.java.additional.4=-Djava.io.tmpdir=%WINDIR%/Temp

3. Add another line beginning with wrapper.java.additional. plus the next number in the sequence, followed by =-Dbamboo.send.file.names.to.agent=false}.  Using the example above this would look like this: 

   wrapper.java.additional.5=-Dbamboo.send.file.names.to.agent=false

Solution D: Investigate and fix the agent's file/folder permissions and ownership

1. Stop the agent

2. In Linux, run the following command as root to list the permissions of the <bamboo-agent-home> folders and files. In Windows, you will need to investigate the permissions manually or use PowerShell's Get-Acl/Get-ChildItem

   For directories

   find <bamboo-agent-home> -maxdepth 2 \( -type f -o -type d \) -ls

3. Based on the output, fix the permissions and ownership of the files and start the Agent

Solution E: Cleanup some disk space or extend the <bamboo-agent-home> filesystem

1. Stop the agent
2. Extend or cleanup the <bamboo-agent-home> filesystem
3. Start the agent

Solution F: There are too many Agents on your Bamboo system

In cases where Bamboo has no proper Agent cleanup routines – mostly on very busy systems – this may lead to thousands, unused Agents left in the Database, forcing Bamboo to recalculate the Queue much more often.

1. Reduce the number of Agents if possible (data hygiene for inactive Agents). You can run some SQL statements to find unused Agents:
   • Generating Bamboo Agents counts from the database
2. Increase the number of ORPHANED_BUILD_MONITOR_JOB_SCHEDULER_REACTION_DELAY_MULTIPLIER. For that you'd have to add a new -Dbamboo.orphaned.build.monitor.reaction.delay.multiplier system property to Bamboo and with a value between 3 and 20. The multiplier is applied to the following formula:
   • heartbeatTimeoutSeconds (600s) + ( orphaned.build.monitor.reaction.delay.multiplier (2) * heartbeatTimeout (60s) ) = 720 seconds
3. Ultimately, upgrade Bamboo to one of the releases listed as fixed in BAM-25145 - Getting issue details... STATUS