We, Atlassians, strive to provide high quality and evolving products to our customers. With this in mind, we've made the decision to discontinue our Bamboo Cloud offering by 31 January 2017. Learn more

Troubleshooting Remote Agents

Technical Overview of Remote Agent Connectivity Mechanism

The traffic between agent and server is bi-directional, however initial connections are only sent from agent to server. Responses are sent from server to agent.
The server has 2 urls / ports to connect with a remote agent.

  1. Bamboo base url, usually http(s)://<host>:8085/<bamboo-app-context>
  2. Bamboo broker url, usually tcp://<host>:54663 if not defined otherwise

Both ports need to be open in the firewall. The direction of the initial connection is from agent to server.

To establish connections to the server the remote agent selects 2 available local ports, one for each connection. The Communication between the remote agent and server is therefore done through the two protocols http and jms. The http is for the initial ping to the server and bootstrapping. Artifacts are also transferred through the http. Other communication and transfer of data is done through the Apache AMQ based jms protocol.

Diagnosing Remote Agent Connectivity
  1. It is also a good idea to check and ensure that the Remote agent server is reachable from the Bamboo server and vice versa. You can check this using ping. An example is something like "ping your_host"
  2. Telnet to check whether the client is able to connect to the server on a particular port. If you are running Bamboo on port 8085, and the broker URL listens port 54663, you should try to run the following commands from the remote agent: "telnet bamboo_server_host 8085" and "telnet bamboo_server_host 54663"

  3. Check that your Broker URL is not pointing to localhost when it should be pointing to the Bamboo's server IP. Please refer to: Remote agent points to incorrect hostname instead of Bamboo's server IP

  4. You might sometimes be facing issues with the port binding where the logs complain about the port numbers already in use. This mostly happens when two instances of Bamboo are running on same server each using the same port. We have seen this issue where the production and development instance are installed together each listening on port 54663. If this is the case, you can shut down one of the instances, edit the HOME/bamboo.cfg.xml file and edit the port in the following two lines (or make the change from the Administration -> General Configuration page if you are using Bamboo 5.x or later). You will need to restart your Bamboo server for the changes to take effect

    <property name="bamboo.jms.broker.client.uri">failover:(tcp://bamboo_server_host:54663?wireFormat.maxInactivityDuration=300000)?maxReconnectAttempts=10&amp;initialReconnectDelay=15000</property>
    <property name="bamboo.jms.broker.uri">tcp://0.0.0.0:54663?wireFormat.maxInactivityDuration=300000</property>

    In cases where you have only a single Bamboo server running and still getting a binding port error, the netstat command will come in handy. You can check for example using the command "netstat -an | findstr 54663" for Windows OR "netstat -an | grep 54663" for a Unix based OS

Troubleshooting remote agent connectivity problems

This section will give you some tips on troubleshooting the most common connectivity problems with remote agents

Remote Agent Registration

Comment out the following packages and make sure that DEBUG logging is enabled for all of them from the <bamboo-install>/atlassian-bamboo/WEB-INF/classes/log4j.properties file. The debug level on the remote agent can be done in the same way. In case you don't have log4j.properties file on the remote agent, you can copy the one from the Bamboo server to the agent home:

log4j.category.com.atlassian.bamboo.agent=DEBUG
log4j.category.com.atlassian.bamboo.v2.build.agent=DEBUG
log4j.category.com.atlassian.bamboo.buildqueue.manager.RemoteAgentManagerImpl=DEBUG

Next, restart the Bamboo server and remote agents. Useful debug traces will be produced that show the agent-server communication in detail which Atlassian Support can use to investigate further.

Network Timeouts

If your agent is dropping out due to errors similar to:

java.net.ConnectException: Connection timed out
Firewall/Router

Ensure that you don't have a firewall/router which might be closing network connections due to inactivity.

Low Network Throughput

The error can also occur, whilst sending large messages over slow network link:

This problem arises because the ActiveMQ message broker continuously checks for activity on a socket, and whilst building a large message in the TCP socket buffer, the inactivity monitor will time out

As the document suggests, edit your <Bamboo-Home>/bamboo.cfg.xml file to figure out and edit the following section:

<property name="bamboo.jms.broker.client.uri">failover:(tcp://192.168.35.128:54663?wireFormat.maxInactivityDuration=300000)?maxReconnectAttempts=10&amp;initialReconnectDelay=15000</property>

setting the maxInactivityDuration=300000 value to a greater number or 0 to disable it all together.

Remote Agent Communication with the Server

The first thing to know is the stage you are facing the issue.. Is it that the remote agent can not be started or that artifact can not be transferred? In which case the problem is coming from the HTTP. Otherwise, a JMS issue.

Debugging HTTP Issue
  • Base URL - The remote agent uses the base url stored in its configuration to access Bamboo and not the BaseURL defined in Bamboo. Check agent's log for the line: Agent bootstrap using baseUrl: http://localhost:8085/bamboo/agentServer If in the process you noticed that the base URL is wrong, you can update this appropriately from the HOME/bamboo-cfg.xml. You might also want to consider the use of the IP address
  • Ports: is the Bamboo's server port (as defined in agent's configuration) accessible from the agent? Since the very first action of the agent is to get fingerprint from the server, (log line:Requesting fingerprintRequestUrl: http://localhost:8085/bamboo/agentServer/GetFingerprint.action?hostName=atlassian_support) let's do it from the remote agent's host:

    wget -S --max-redirect=0 http://bamboo_server_host:8085/bamboo/agentServer/GetFingerprint.action?hostName=atlassian-support
    
    or
    
    curl -D - http://bamboo_server_host:8085/bamboo/agentServer/GetFingerprint.action?hostName=atlassian-support

    This should return a HTTP/1.1 200 OK response in both cases - if you didn't, then there most be a configuration issue and you will have to check the network and/or the configuration file.. If it did return HTTP/1.1 200 OK, check if Bamboo's Agents page shows

    "Aug 31, 2011 6:10:59 AM A remote agent is loading on atlassian-support (10.199.238.14).".

    If it does not, then you might probably be connected to the wrong bamboo server or there are some more root causes in the logs. You can contact support with the logs.

  • If the agent is not able to ping the server with the errors similar to:

    javax.naming.CommunicationException: Failed to connect to server localhost:8085

    This is most likely due to the fact that the DNS can not be resolved. Update the DNS records to include the appropriate server name & IP used in the configuration file. If updating of the DNS server used is not possible, this can be achieved by manually adding the following entry, replacing the IP & server name as appropriate to the below file on the server hosting Bamboo 
    Linux:/etc/hosts
    Windows:C:\Windows\System32\drivers\etc\hosts

    <IP of server>    <host name of server>
    192.168.1.123     atlassian.com

    (lightbulb)You need to have local administrator rights on the machine hosting Bamboo to make these changes.

  • Proxy: If you have a proxy server involved, try to bypass them by connecting to the server directly. Please note that if there are artifact transfer problems (rejections or timeouts): Does it have HTTP 1.0 compatibility turned on? Does it have any configuration options related to chunked transfer encoding turned on? Your proxy server needs to support HTTP/1.1. See Agent upload of artifacts fails to set Content-Length in HTTP/1.1

Debugging JMS Issue
  • Corruption:If you are getting jms related errors in the logs of Bamboo or the remote agent, it is a good idea to purge the following folders; jms-store, cachesindex and temp from the Bamboo home and restart both server and agents.
  • Load: Check the load on the server, dbms and agent machines. Also check the number of your agents (local,remote or elastic) and how busy they are? If the load is very high, then an upgrade your Bamboo server to the latest version is recommended as most likely the AMQ has been updated.

Should the above not helpful to resolve, contact support attaching the following:

  • The Bamboo server and remote agent logs after adding the line log4j.category.org.apache.activemq=DEBUG to your atlassian-bamboo/WEB-INF/classes/log4j.properties file and restarting the server. The debug level on the remote agent can be done in the same way. In case you dont have log4j.properties file in the remote agent, you can copy one from the Bamboo server to the agent home.
  • If possible, try getting the network traffic dumps between agent and the server, preferably one from the agent and one form the server. Refer to Capturing HTTP traffic using Wireshark or Fiddler

Was this helpful?

Thanks for your feedback!

Why was this unhelpful?

Have a question about this article?

See questions about this article

Powered by Confluence and Scroll Viewport