Builds cancelled due to marked as queued but not present in the queue for 720 seconds
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the server and data center platforms.
Summary
Bamboo builds fail semi-randomly and you see messages like this in the server logs:
Build XXX had to be cancelled: it was marked as queued but was not present in the queue for (at least) the past 720 seconds.
Cause
There are multiple causes of this problem, each with a different solution.
Cause A: Multiple builds triggered at the same time in versions older than 6.2.2
Post-commit triggers are superior to polling because they reduce the load on the build server and source control. However, if multiple commits occur rapidly this can cause a race condition. In this case, there may be "ghost" builds that result in the "not present in the queue for 720 seconds" messages. - BAM-17890Getting issue details... STATUS
Cause B: Conflict with Group Agent Plugin and Bamboo 5.7.x
There is an incompatibility between Bamboo 5.7.x. and the Group Agent plugin version 1.6. If you are affected by this you will see errors when adding builds to the queue in your logs:
2014-11-21 09:41:59,766 INFO [3-DelayedChangeDetectionThread:pool-7-thread-3] [ChainExecutionManagerImpl] Build BCPRELEASE8-PAR-JOB1-331 has been dispatched
2014-11-21 09:41:59,767 INFO [AtlassianEvent::0-BAM::EVENTS:pool-3-thread-2] [BuildQueueManagerImpl] Attempting to queue BCP Release 8 - Parent - Run build #331 (BCPRELEASE8-PAR-JOB1-331).
2014-11-21 09:41:59,768 INFO [AtlassianEvent::0-BAM::EVENTS:pool-3-thread-2] [ExecutionPhaseServiceImpl] BCP Release 8 - Parent - Run build #331 (BCPRELEASE8-PAR-JOB1-331) queued
2014-11-21 09:41:59,802 ERROR [AtlassianEvent::0-BAM::EVENTS:pool-3-thread-2] [AsynchronousAbleEventDispatcher] There was an exception thrown trying to dispatch event '[com.atlassian.bamboo.v2.build.events.BuildTriggeredEvent[source=com.atlassian.bamboo.build.DefaultBuildExecutionManager@74793ec9], SingleParameterMethodListenerInvoker{method=public void com.atlassian.bamboo.v2.build.queue.BuildTriggeredListener.handleEvent(com.atlassian.bamboo.v2.build.events.BuildTriggeredEvent), listener=com.atlassian.bamboo.v2.build.queue.BuildTriggeredListener@2370e095}]' from the invoker 'java.lang.RuntimeException: com.edwardawebb.bamboo.groupagent.filter.GroupAgentFilter.filter(Lcom/atlassian/bamboo/v2/build/CommonContext;Ljava/util/Collection;Lcom/atlassian/bamboo/v2/build/agent/capability/MinimalRequirementSet;)Ljava/util/Collection;'
Cause C: Default setting for Bamboo versions older than 5.0
In versions of Bamboo prior to version 5.0 the default value of bamboo.send.file.names.to.agent
is set to true which sometimes results in problems adding builds to the queue. This setting only affects Bamboo versions less than 5.0 and is already deprecated.
Cause D: The agent may have the wrong file/folder permissions and ownership
You may find a java.lang.IllegalStateException: Agent working directory is not a directory
message in the Agent logs <bamboo-agent-home>/atlassian-bamboo-agent.log
. That error means the Bamboo agent cannot write to a specific location within its work structure. This is an indicator of wrong file permissions and/or incorrect ownership.
This is a very common problem and happens when the Agent is inadvertently started with a user other than the regular Bamboo Agent user (E.g. root
)
2021-09-23 19:30:31,212 ERROR [0-BAM::agent-1.mydomain.com::Agent:pool-3-thread-1] [BuildAgentControllerImpl] Unknown exception occurred on 'agent-1.mydomain.com'. Agent will attempt to recover its normal operation...
io.atlassian.util.concurrent.LazyReference$InitializationException: java.lang.IllegalStateException: Agent working directory is not a directory
at io.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:156)
at io.atlassian.util.concurrent.LazyReference.get(LazyReference.java:116)
at com.atlassian.bamboo.v2.build.agent.remote.workspaces.RemoteAgentWorkspaceManagerImpl.registerWorkspace(RemoteAgentWorkspaceManagerImpl.java:183)
at com.atlassian.bamboo.v2.build.agent.remote.RemoteBuildAgent.onContextReceived(RemoteBuildAgent.java:134)
at com.atlassian.bamboo.v2.build.agent.BuildAgentControllerImpl.waitAndPerformBuild(BuildAgentControllerImpl.java:111)
at com.atlassian.bamboo.v2.build.agent.DefaultBuildAgent.lambda$start$0(DefaultBuildAgent.java:110)
at com.atlassian.bamboo.utils.BambooRunnables$1.run(BambooRunnables.java:48)
at com.atlassian.bamboo.security.ImpersonationHelper.runWith(ImpersonationHelper.java:26)
at com.atlassian.bamboo.security.ImpersonationHelper.runWithSystemAuthority(ImpersonationHelper.java:17)
at com.atlassian.bamboo.security.ImpersonationHelper$1.run(ImpersonationHelper.java:41)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Agent working directory is not a directory
at com.atlassian.bamboo.v2.build.agent.remote.workspaces.RemoteAgentWorkspaceManagerImpl$1.create(RemoteAgentWorkspaceManagerImpl.java:72)
at com.atlassian.bamboo.v2.build.agent.remote.workspaces.RemoteAgentWorkspaceManagerImpl$1.create(RemoteAgentWorkspaceManagerImpl.java:64)
at io.atlassian.util.concurrent.LazyReference$Sync.run(LazyReference.java:332)
at io.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:150)
... 10 more
Cause E: No space left on device
We had reports from customers that this issue might happen when the Agent has not enough free space left on the <bamboo-agent-home>
location.
Solution
Solution A: Add a 30 second quiet period to the linked repository associated with the build plan.
- Edit the settings for your linked repository
- Expand the Advanced section
- Check the box for "Enable quiet period" and enter "30" in the text field
- Save the changes
For more detailed instructions for all linked repository types available to Bamboo see Linking to source code repositories.
Solution B: Upgrade the Group Agent Plugin to version 1.7 or greater
- Go to Administration > Add-ons
- Find "bamboo-group-agent" in the list of installed plugins
Click "Update" and wait for the process to complete
- Restart Bamboo for the plugin changes to take effect
For full details see Installing Marketplace apps.
Solution C: Change bamboo.send.file.names.to.agent to false
Apply this solution only if that property is set to true. By default, it is already set to false.
- Edit the file $
BAMBOO_INSTALL/conf/wrapper.conf
Find the lines that start with
wrapper.java.additional
and add another line, incrementing the number that follows. Example:wrapper.java.additional.1=-Dorg.eclipse.jetty.xml.XmlParser.Validating=false wrapper.java.additional.2=-XX:MaxPermSize=256m wrapper.java.additional.3=-Djava.awt.headless=true wrapper.java.additional.4=-Djava.io.tmpdir=%WINDIR%/Temp
Add another line beginning with
wrapper.java.additional.
plus the next number in the sequence, followed by=-Dbamboo.send.file.names.to.agent=false
}. Using the example above this would look like this:wrapper.java.additional.5=-Dbamboo.send.file.names.to.agent=false
Solution D: Investigate and fix the agent's file/folder permissions and ownership
- Stop the agent
In Linux, run the following command as root to list the permissions of the <bamboo-agent-home> folders and files. In Windows, you will need to investigate the permissions manually or use PowerShell's
Get-Acl/Get-ChildItem
For directoriesfind <bamboo-agent-home> -maxdepth 2 \( -type f -o -type d \) -ls
Based on the output, fix the permissions and ownership of the files and start the Agent
Solution E: Cleanup some disk space or extend the <bamboo-agent-home> filesystem
- Stop the agent
- Extend or cleanup the <bamboo-agent-home> filesystem
- Start the agent