Atlassian strives to provide high quality and evolving products to our customers. With this in mind, we've made the decision to discontinue our Bamboo Cloud offering on 31 January 2017. Learn more

Troubleshooting Remote Agents

Technical Overview of Remote Agent Connectivity Mechanism

The traffic between agent and server is bi-directional and it is important to understand that the connection is initiated from the agent (sent from the agent to the server). Responses are sent from the server to the agent.
The server has two addresses (URL:port) to connect with a remote agent:

  1. Bamboo Base URL, usually http(s)://<host>:8085/<bamboo-app-context>
  2. Bamboo Broker URL, usually tcp://<host>:54663 if not defined otherwise

Both ports need to be open in the firewall.

To establish connections to the server the remote agent selects two available local ports, one for each connection. The communication between the remote agent and server is done through the two protocols, HTTP and JMS. The HTTP protocol is used for the initial ping to the server and bootstrapping. Artifacts are also transferred using HTTP. Other communications and transfers of data are done through the Apache ActiveMQ (AMQ) based JMS protocol.

Diagnosing Remote Agent Connectivity
  1. Check that the remote agent server is reachable from the Bamboo server and vice versa. You can check this using ping, e.g.:
    1. From the remote agent: "ping bamboo_server_host"
    2. From the Bamboo server: "ping remote_agent_host"
  2. Check that the client is able to connect to the server on the defined ports by using Telnet. If you are running Bamboo on port 8085 (default) and the Broker URL listens on port 54663 (default), you should try to run the following commands from the remote agent:

    1. "telnet bamboo_server_host 8085"

    2. "telnet bamboo_server_host 54663"

  3. Check that your Broker URL is not pointing to localhost as it should be pointing to the Bamboo's server IP address. Please refer to: Remote agent points to incorrect hostname instead of Bamboo's server IP

  4. Check that port binding is not an issue by searching the logs for a message indicating that the defined port numbers are already in use. This mostly happens when two instances of Bamboo are running on same server – each defined to use the same port. We have seen this issue where the production and development instance are installed together each listening on port 54663. If this is the case:
    1. Shut down one of the instances

    2. Edit the BAMBOO_HOME/bamboo.cfg.xml file and update the port in the following two lines (or make the change from Administration > General Configuration page if you are using Bamboo 5.X+)

      <property name="bamboo.jms.broker.client.uri">failover:(tcp://bamboo_server_host:54663?wireFormat.maxInactivityDuration=300000)?maxReconnectAttempts=10&amp;initialReconnectDelay=15000</property>
      <property name="bamboo.jms.broker.uri">tcp://0.0.0.0:54663?wireFormat.maxInactivityDuration=300000</property>
    3. You will need to restart your Bamboo server for the changes to take effect

  5. In cases where you have only a single Bamboo instance yet are still getting a port binding error logged, the netstat command will come in handy. You can check, for example, using the command "netstat -an | findstr 54663" on Windows or "netstat -an | grep 54663" on a Unix based OS.

Troubleshooting remote agent connectivity problems

This section will give you some tips on troubleshooting the most common connectivity problems with remote agents

Remote Agent Registration

Comment out the following packages and make sure that DEBUG logging is enabled for all of them from the <bamboo-install>/atlassian-bamboo/WEB-INF/classes/log4j.properties file. The debug level on the remote agent can be done in the same way. In case you don't have log4j.properties file on the remote agent, you can copy the one from the Bamboo server to the agent home:

log4j.category.com.atlassian.bamboo.agent=DEBUG
log4j.category.com.atlassian.bamboo.v2.build.agent=DEBUG
log4j.category.com.atlassian.bamboo.buildqueue.manager.RemoteAgentManagerImpl=DEBUG

Next, restart the Bamboo server and remote agents. Useful debug traces will be produced that show the agent-server communication in detail which Atlassian Support can use to investigate further.

Network Timeouts

If your agent is dropping out due to errors similar to:

java.net.ConnectException: Connection timed out
Firewall/Router

Ensure that you don't have a firewall/router which might be closing network connections due to inactivity.

Low Network Throughput

The error can also occur, whilst sending large messages over slow network link:

This problem arises because the ActiveMQ message broker continuously checks for activity on a socket, and whilst building a large message in the TCP socket buffer, the inactivity monitor will time out

As the document suggests, edit your BAMBOO_HOME/bamboo.cfg.xml file and update the following section – set the maxInactivityDuration value to a number greater than 300000 or set it to 0 to disable it all together:

<property name="bamboo.jms.broker.client.uri">failover:(tcp://192.168.35.128:54663?wireFormat.maxInactivityDuration=300000)?maxReconnectAttempts=10&amp;initialReconnectDelay=15000</property>

Remote Agent Communication with the Server

The first step is determine at which stage the failure is occurring. For example, is the issue that the remote agent can't be started or that the artifact can't be transferred? In which case the problem is coming from the HTTP. Otherwise, a JMS issue.

Debugging HTTP 

Base URL

The remote agent uses the base URL stored in its configuration to access Bamboo and not the base URL defined in Bamboo. Check agent's log for a line similar to the following: 

Agent bootstrap using baseUrl: http://localhost:8085/bamboo/agentServer

If you notice that the base URL is wrong, you can update this in BAMBOO_HOME/bamboo-cfg.xml. You might also want to consider using an IP address instead.

Ports

The very first step the agent takes is to retrieve the Bamboo server's unique fingerprint. Is the Bamboo server's port (as defined in the agent's configuration) accessible from the agent? This is how this step is logged on the agent:

Requesting fingerprintRequestUrl: http://localhost:8085/bamboo/agentServer/GetFingerprint.action?hostName=atlassian_support

Investigate by trying to do this manually from the remote agent:

wget -S --max-redirect=0 http://bamboo_server_host:8085/bamboo/agentServer/GetFingerprint.action?hostName=atlassian-support

Or:

curl -D - http://bamboo_server_host:8085/bamboo/agentServer/GetFingerprint.action?hostName=atlassian-support

Both commands should return a HTTP/1.1 200 OK response and Bamboo's Agents page should display a similar message:

"Aug 31, 2011 6:10:59 AM A remote agent is loading on atlassian-support (10.199.238.14).".

If either don't and the Agents page isn't similar then there must be a configuration issue and you will have to check the network and/or the configuration file. Contact Support at this time so that they can further investigate the application log files. 

Domain Name System (DNS)

If the agent is not able to ping the server with a similar error this is most likely due to the fact that the DNS server cannot be resolved:

javax.naming.CommunicationException: Failed to connect to server localhost:8085

Update the DNS records to include the appropriate server name & IP used in the configuration file. If updating the DNS server is impossible, similar results can be achieved by updating the hosts file on the Bamboo server itself (you must have local administrator rights on the server). Common file location based on OS:

OS Location
Linux /etc/hosts
Windows C:\Windows\System32\drivers\etc\hosts

The entry must contain an IP address and corresponding server name:  

<IP of server>    <host name of server>
192.168.1.123     atlassian.com
Proxy 

If utilizing a proxy, attempt to bypass it as a possible solution. In order to accomplish this you'll need to need to add another Tomcat connector without the proxy configuration. If using an EC2 agent note that after connecting through the secure tunnel, the agent will try to resolve the server identified by the value of the proxyName attribute in the Tomcat connector (see https://confluence.atlassian.com/kb/integrating-apache-http-server-reverse-proxy-with-bamboo-753894403.html). Bypass this functionality by Configuring Bamboo on start-up with the -Dbamboo.ec2.agent.endpoint JVM parameter (make sure that there is a corresponding connector defined for the port), e.g.

setenv.sh
 JVM_SUPPORT_RECOMMENDED_ARGS="-Dbamboo.ec2.agent.endpoint=http://localhost:8085"

If there are artifact transfer problems (rejections, timeouts, etc.):

  • Is HTTP/1.0 compatibility turned on? 
  • Are there any configuration options turned on related to chunked transfer encoding?

Your proxy server needs to support HTTP/1.1:  Agent upload of artifacts fails to set Content-Length in HTTP/1.1

Debugging JMS

Corruption

If you are getting JMS related errors in the logs of Bamboo or the remote agent, it is a good idea to purge the following folders from BAMBOO_HOME and restart both server and agent(s):

  • jms-store
  • caches
  • index
  • temp
Load

Check the load on the Bamboo server, DB server, and agent servers. Also check the number of agents (local and remote including elastic) and how busy they are. If the load is very high and you aren't on the latest version consider an application upgrade as AMQ performance updates are common.

Contact Support if you continue experiencing issues. Attach the following data in order to expedite the investigation:

  • Bamboo server and remote agent logs after adding the line log4j.category.org.apache.activemq=DEBUG to your atlassian-bamboo/WEB-INF/classes/log4j.properties file and restarting the server. The debug level on the remote agent can be done in the same way. In case you don't have the log4j.properties file on the remote agent, you can simply copy the one from the Bamboo server to the agent home directory.
  • Network traffic dumps between agent and the server, preferably, one from the agent and one form the server. Refer to Capturing HTTP traffic using Wireshark or Fiddler

Timeout has expired for waiting for tunnel to instance

If you are getting a timeout error when starting the tunnel, such as:

Timeout has expired for waiting for tunnel to instance [i-HASH], terminating [Tunnel ##.##.##.##:1234 - (##.##.##.##:26224) - 0.0.0.0:54663] Error: Connection refused

please check that the tunnel port (26224 in the example above) is open and accessible.

To troubleshoot:

  1. Run the following from the Bamboo host:

    1. telnet elastic-agent-ip-here 26224

    2. nmap -p 26224 elastic-agent-ip-here

    3. netstat -nat | grep LISTEN

  2. Are there any security settings in EC2 that may have blocked that port?

Open the ports as needed depending on the output of the above. Contact Support if the issue persists.

Elastic agents remain in Pending and do not start after upgrading Bamboo

If you have recently updated Bamboo, and your Elastic Agents are timing out with SSL Handshake Exceptions, please see Elastic agents remain in Pending and do not connect after upgrading Bamboo.

Was this helpful?

Thanks for your feedback!

Why was this unhelpful?

Have a question about this article?

See questions about this article

Powered by Confluence and Scroll Viewport