Summary

We have a task to checkout source code from bitbucket. When we run the plan, it fails with the below error in the build log.

simple	03-Aug-2017 09:00:02	Build MYPLAN - Continuous Integration (develop) - feature-MYPLAN-421 - Build sources #2 (PROJ-PLANNAME-JOB1-2) started building on agent bamboo-win-1 (3)
simple	03-Aug-2017 09:00:02	Remote agent on host bamboo-windows-agent
error	03-Aug-2017 09:00:02	Error occurred while executing the build for MYPLAN - Continuous Integration (develop) - feature-MYPLAN-421 - Build sources #2 (AA-PLANNAME-JOB1-2) : Plugin com.atlassian.bamboo.plugins.stash.atlassian-bamboo-plugin-stash: bbserver not available

You may also find that plugins/apps (and thus tasks) other than Bitbucket or Source code checkout are failing with a similar error about it not being available. It is not just specific to the Bitbucket plugin.

Diagnosis

Cause #1: Multiple remote agent processes running on the same home directory

Check if there are multiple remote agent processes running on the same $BAMBOO_AGENT_HOME directory:

If the $BAMBOO_AGENT_HOME directory is on a local drive, you can simply run a ps aux | grep agent and review the output to see if there are multiple agent processes started on the same directory. In a normal setup, there should be at most two processes related to the Bamboo remote agent on the same home directory (-Dbamboo.home=): the wrapper and the Java process itself.

If the $BAMBOO_AGENT_HOME is on a shared/NFS drive, it's possible that the multiple processes have been started on different servers that have access to that mount. In this case, it is more difficult to track down as you'll need to check every server that has access to it and can't simply run ps on a single server.

Another indicator of this happening and way to diagnose it, is to check the $BAMBOO_AGENT_HOME/atlassian-bamboo-agent.log (*not* the files in the /logs directory as they lack an important indicator) and look for interleaved messages from different JVM IDs. E.g.

INFO   | jvm 4    | 2020/12/04 11:02:49 | 2020-12-04 11:02:49,634 INFO [0-BAM::agenthost::Agent:pool-3-thread-1] [BuildAgentControllerImpl] Agent 111111111 checking build queue for executables...
INFO   | jvm 3    | 2020/12/04 11:02:49 | 2020-12-04 11:02:49,633 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [RemoteEventRebroadcasterMessageListener] ExecutableQueueUpdate: addToQueue, agents known to be affected: []
INFO   | jvm 3    | 2020/12/04 11:02:49 | 2020-12-04 11:02:49,634 INFO [0-BAM::agenthost::Agent:pool-3-thread-1] [BuildAgentControllerImpl] Agent 111111111 checking build queue for executables...
INFO   | jvm 4    | 2020/12/04 11:02:49 | 2020-12-04 11:02:49,634 INFO [remoteEventRebroadcasterMessageListenerContainer-1] [RemoteEventRebroadcasterMessageListener] ExecutableQueueUpdate: addToQueue, agents known to be affected: []

Notice how JVM 3 messages are interleaved between log messages from JVM 4, this is clear proof that two agent JVMs are concurrently running and logging to the same file.

Cause #2: Incorrect Java version:

Run java -version in a command prompt from the remote agent and check if it lists the version that is compatible for Bamboo agent to run and if Bamboo remote agent is configured to use the same. At the time of writing, Bamboo agents are only compatible with Java 8.

Cause

Cause #1: Multiple remote agent processes running on the same home directory

Multiple remote agent processes have been started from the same home directory. This means that the startup of the subsequent agent corrupts the plugin cache of the first agent, while it still remains online and continues to occasionally get builds.

Cause #2: Incorrect Java version:

The Remote Agent did not start properly, as it runs a non-supported version of Java.

• Note, in versions of Bamboo 6.10 and above this is far less likely as the agent will perform validation of the Java version and refuse to start if it is incorrect.

Resolution

Cause #1: Multiple remote agent processes running on the same home directory

If you find multiple agent processes that share the same (it's OK to run multiple agents on the same server, as long as they are configured to different homes) home directory, please kill all the processes once there are no builds running and start the agent again. You should also review how you start the agent as it's not normal for a second agent process to be able to start under normal circumstances as the default startup scripts come with PID checking and suggests something may have been customized.

Cause #2: Incorrect Java version:

Reinstall the remote agent by following the below steps.

1. Stop the agent
2. Install the supported version of Java for that version of Bamboo.
3. Backup the $BAMBOO_AGENT_HOME/bamboo-agent.cfg.xml and $BAMBOO_AGENT_HOME/conf/wrapper.conf (if customizations to these need to be preserved)
4. Delete the remote agent home (to make sure everything is created again) 
5. Install the agent once again.
6. Restore the configuration from Step 3.
7. Run the same build in the same Remote agent.