Troubleshooting Remote Agents
Technical Overview of Remote Agent Connectivity Mechanism
The traffic between agent and server is bi-directional and it is important to understand that the connection is initiated from the agent (sent from the agent to the server). Responses are sent from the server to the agent.
The server has two addresses (URL:port) to connect with a remote agent:
- Bamboo Base URL, usually http(s)://<host>:8085/<bamboo-app-context>
- Bamboo Broker URL, usually tcp://<host>:54663 if not defined otherwise
Both ports need to be open in the firewall.
To establish connections to the server the remote agent selects two available local ports, one for each connection. The communication between the remote agent and server is done through the two protocols, HTTP and JMS. The HTTP protocol is used for the initial ping to the server and bootstrapping. Artifacts are also transferred using HTTP. Other communications and transfers of data are done through the Apache ActiveMQ (AMQ) based JMS protocol.
Diagnosing Remote Agent Connectivity
- Check that the remote agent server is reachable from the Bamboo server and vice versa. You can check this using ping, e.g.:
- From the remote agent:
- From the Bamboo server: "
- From the remote agent:
Check that the client is able to connect to the server on the defined ports by using Telnet. If you are running Bamboo on port 8085 (default) and the Broker URL listens on port 54663 (default), you should try to run the following commands from the remote agent:
telnet bamboo_server_host 8085"
"telnet bamboo_server_host 54663"
Check that your Broker URL is not pointing to
localhostas it should be pointing to the Bamboo's server IP address. Please refer to: Remote agent points to incorrect hostname instead of Bamboo's server IP
- Check that port binding is not an issue by searching the logs for a message indicating that the defined port numbers are already in use. This mostly happens when two instances of Bamboo are running on same server – each defined to use the same port. We have seen this issue where the production and development instance are installed together each listening on port 54663. If this is the case:
Shut down one of the instances
BAMBOO_HOME/bamboo.cfg.xmlfile and update the port in the following two lines (or make the change from Administration > General Configuration page if you are using Bamboo 5.X+)
You will need to restart your Bamboo server for the changes to take effect
- In cases where you have only a single Bamboo instance yet are still getting a port binding error logged, the netstat command will come in handy. You can check, for example, using the command
"netstat -an | findstr 54663"on Windows or
"netstat -an | grep 54663"on a Unix based OS.
Troubleshooting remote agent connectivity problems
This section will give you some tips on troubleshooting the most common connectivity problems with remote agents
Remote Agent Registration
Comment out the following packages and make sure that DEBUG logging is enabled for all of them from the <bamboo-install>/atlassian-bamboo/WEB-INF/classes/log4j.properties file. The debug level on the remote agent can be done in the same way. In case you don't have log4j.properties file on the remote agent, you can copy the one from the Bamboo server to the agent home:
Next, restart the Bamboo server and remote agents. Useful debug traces will be produced that show the agent-server communication in detail which Atlassian Support can use to investigate further.
If your agent is dropping out due to errors similar to:
Ensure that you don't have a firewall/router which might be closing network connections due to inactivity.
Low Network Throughput
The error can also occur, whilst sending large messages over slow network link:
This problem arises because the ActiveMQ message broker continuously checks for activity on a socket, and whilst building a large message in the TCP socket buffer, the inactivity monitor will time out
As the document suggests, edit your
BAMBOO_HOME/bamboo.cfg.xml file and update the following section – set the
maxInactivityDuration value to a number greater than 300000 or set it to 0 to disable it all together:
Remote Agent Communication with the Server
The first step is determine at which stage the failure is occurring. For example, is the issue that the remote agent can't be started or that the artifact can't be transferred? In which case the problem is coming from the HTTP. Otherwise, a JMS issue.
The remote agent uses the base URL stored in its configuration to access Bamboo and not the base URL defined in Bamboo. Check agent's log for a line similar to the following:
If you notice that the base URL is wrong, you can update this in
BAMBOO_HOME/bamboo-cfg.xml. You might also want to consider using an IP address instead.
The very first step the agent takes is to retrieve the Bamboo server's unique fingerprint. Is the Bamboo server's port (as defined in the agent's configuration) accessible from the agent? This is how this step is logged on the agent:
Investigate by trying to do this manually from the remote agent:
Both commands should return a
HTTP/1.1 200 OK response and Bamboo's Agents page should display a similar message:
If either don't and the Agents page isn't similar then there must be a configuration issue and you will have to check the network and/or the configuration file. Contact Support at this time so that they can further investigate the application log files.
Domain Name System (DNS)
If the agent is not able to ping the server with a similar error this is most likely due to the fact that the DNS server cannot be resolved:
Update the DNS records to include the appropriate server name & IP used in the configuration file. If updating the DNS server is impossible, similar results can be achieved by updating the
hosts file on the Bamboo server itself (you must have local administrator rights on the server). Common file location based on OS:
The entry must contain an IP address and corresponding server name:
If utilizing a proxy, attempt to bypass it as a possible solution. In order to accomplish this you'll need to need to add another Tomcat connector without the proxy configuration. If using an EC2 agent note that after connecting through the secure tunnel, the agent will try to resolve the server identified by the value of the
proxyName attribute in the Tomcat connector (see https://confluence.atlassian.com/kb/integrating-apache-http-server-reverse-proxy-with-bamboo-753894403.html). Bypass this functionality by Configuring Bamboo on start-up with the
-Dbamboo.ec2.agent.endpoint JVM parameter (make sure that there is a corresponding connector defined for the port), e.g.
If there are artifact transfer problems (rejections, timeouts, etc.):
- Is HTTP/1.0 compatibility turned on?
- Are there any configuration options turned on related to chunked transfer encoding?
Your proxy server needs to support HTTP/1.1: Agent upload of artifacts fails to set Content-Length in HTTP/1.1
If you are getting JMS related errors in the logs of Bamboo or the remote agent, it is a good idea to purge the following folders from
BAMBOO_HOME and restart both server and agent(s):
Check the load on the Bamboo server, DB server, and agent servers. Also check the number of agents (local and remote including elastic) and how busy they are. If the load is very high and you aren't on the latest version consider an application upgrade as AMQ performance updates are common.
Contact Support if you continue experiencing issues. Attach the following data in order to expedite the investigation:
- Bamboo server and remote agent logs after adding the line log4j.category.org.apache.activemq=DEBUG to your
atlassian-bamboo/WEB-INF/classes/log4j.propertiesfile and restarting the server. The debug level on the remote agent can be done in the same way. In case you don't have the
log4j.propertiesfile on the remote agent, you can simply copy the one from the Bamboo server to the agent home directory.
- Network traffic dumps between agent and the server, preferably, one from the agent and one form the server. Refer to Capturing HTTP traffic using Wireshark or Fiddler
Timeout has expired for waiting for tunnel to instance
If you are getting a timeout error when starting the tunnel, such as:
please check that the tunnel port (26224 in the example above) is open and accessible.
Run the following from the Bamboo host:
telnet elastic-agent-ip-here 26224
nmap -p 26224 elastic-agent-ip-here
netstat -nat | grep LISTEN
- Are there any security settings in EC2 that may have blocked that port?
Open the ports as needed depending on the output of the above. Contact Support if the issue persists.
Elastic agents remain in Pending and do not start after upgrading Bamboo
If you have recently updated Bamboo, and your Elastic Agents are timing out with SSL Handshake Exceptions, please see Elastic agents remain in Pending and do not connect after upgrading Bamboo.
Was this helpful?
Thanks for your feedback!