Troubleshooting Remote Agents
Technical Overview of Remote Agent Connectivity Mechanism
The traffic between agent and server is bi-directional, however initial connections are only sent from agent to server. Responses are sent from server to agent.
The server has 2 urls / ports to connect with a remote agent.
- Bamboo base url, usually http(s)://<host>:8085/<bamboo-app-context>
- Bamboo broker url, usually tcp://<host>:54663 if not defined otherwise
Both ports need to be open in the firewall. The direction of the initial connection is from agent to server.
To establish connections to the server the remote agent selects 2 available local ports, one for each connection. The Communication between the remote agent and server is therefore done through the two protocols http and jms. The http is for the initial ping to the server and bootstrapping. Artifacts are also transferred through the http. Other communication and transfer of data is done through the Apache AMQ based jms protocol.
Diagnosing Remote Agent Connectivity
- It is also a good idea to check and ensure that the Remote agent server is reachable from the Bamboo server and vice versa. You can check this using ping. An example is something like
Telnet to check whether the client is able to connect to the server on a particular port. If you are running Bamboo on port 8085, and the broker URL listens port 54663, you should try to run the following commands from the remote agent: "
telnet bamboo_server_host 8085"and
"telnet bamboo_server_host 54663"
Check that your Broker URL is not pointing to
localhostwhen it should be pointing to the Bamboo's server IP. Please refer to: Remote agent points to incorrect hostname instead of Bamboo's server IP
You might sometimes be facing issues with the port binding where the logs complain about the port numbers already in use. This mostly happens when two instances of Bamboo are running on same server each using the same port. We have seen this issue where the production and development instance are installed together each listening on port 54663. If this is the case, you can shut down one of the instances, edit the
HOME/bamboo.cfg.xmlfile and edit the port in the following two lines (or make the change from the Administration -> General Configuration page if you are using Bamboo 5.x or later). You will need to restart your Bamboo server for the changes to take effect
In cases where you have only a single Bamboo server running and still getting a binding port error, the netstat command will come in handy. You can check for example using the command
"netstat -an | findstr 54663"for Windows OR
"netstat -an | grep 54663"for a Unix based OS
Troubleshooting remote agent connectivity problems
This section will give you some tips on troubleshooting the most common connectivity problems with remote agents
Remote Agent Registration
Comment out the following packages and make sure that DEBUG logging is enabled for all of them from the <bamboo-install>/atlassian-bamboo/WEB-INF/classes/log4j.properties file. The debug level on the remote agent can be done in the same way. In case you don't have log4j.properties file on the remote agent, you can copy the one from the Bamboo server to the agent home:
Next, restart the Bamboo server and remote agents. Useful debug traces will be produced that show the agent-server communication in detail which Atlassian Support can use to investigate further.
If your agent is dropping out due to errors similar to:
Ensure that you don't have a firewall/router which might be closing network connections due to inactivity.
Low Network Throughput
The error can also occur, whilst sending large messages over slow network link:
This problem arises because the ActiveMQ message broker continuously checks for activity on a socket, and whilst building a large message in the TCP socket buffer, the inactivity monitor will time out
As the document suggests, edit your <Bamboo-Home>/bamboo.cfg.xml file to figure out and edit the following section:
maxInactivityDuration=300000 value to a greater number or 0 to disable it all together.
Remote Agent Communication with the Server
The first thing to know is the stage you are facing the issue.. Is it that the remote agent can not be started or that artifact can not be transferred? In which case the problem is coming from the HTTP. Otherwise, a JMS issue.
Debugging HTTP Issue
- Base URL - The remote agent uses the base url stored in its configuration to access Bamboo and not the BaseURL defined in Bamboo. Check agent's log for the line: Agent bootstrap using baseUrl:
HOME/bamboo-cfg.xml. You might also want to consider the use of the IP address
If in the process you noticed that the base URL is wrong, you can update this appropriately from the
Ports: is the Bamboo's server port (as defined in agent's configuration) accessible from the agent? Since the very first action of the agent is to get fingerprint from the server, (log line:Requesting fingerprintRequestUrl:_support) let's do it from the remote agent's host:
This should return a HTTP/1.1 200 OK response in both cases - if you didn't, then there most be a configuration issue and you will have to check the network and/or the configuration file.. If it did return HTTP/1.1 200 OK, check if Bamboo's Agents page shows
If it does not, then you might probably be connected to the wrong bamboo server or there are some more root causes in the logs. You can contact support with the logs.
If the agent is not able to ping the server with the errors similar to:
This is most likely due to the fact that the DNS can not be resolved. Update the DNS records to include the appropriate server name & IP used in the configuration file. If updating of the DNS server used is not possible, this can be achieved by manually adding the following entry, replacing the IP & server name as appropriate to the below file on the server hosting Bamboo
You need to have local administrator rights on the machine hosting Bamboo to make these changes.
- Proxy: If you have a proxy server involved, try to bypass them by connecting to the server directly. Please note that if there are artifact transfer problems (rejections or timeouts): Does it have HTTP 1.0 compatibility turned on? Does it have any configuration options related to chunked transfer encoding turned on? Your proxy server needs to support HTTP/1.1. See Agent upload of artifacts fails to set Content-Length in HTTP/1.1
Debugging JMS Issue
- Corruption:If you are getting jms related errors in the logs of Bamboo or the remote agent, it is a good idea to purge the following folders;
tempfrom the Bamboo home and restart both server and agents.
- Load: Check the load on the server, dbms and agent machines. Also check the number of your agents (local,remote or elastic) and how busy they are? If the load is very high, then an upgrade your Bamboo server to the latest version is recommended as most likely the AMQ has been updated.
Should the above not helpful to resolve, contact support attaching the following:
- The Bamboo server and remote agent logs after adding the line log4j.category.org.apache.activemq=DEBUG to your
atlassian-bamboo/WEB-INF/classes/log4j.propertiesfile and restarting the server. The debug level on the remote agent can be done in the same way. In case you dont have log4j.properties file in the remote agent, you can copy one from the Bamboo server to the agent home.
- If possible, try getting the network traffic dumps between agent and the server, preferably one from the agent and one form the server. Refer to Capturing HTTP traffic using Wireshark or Fiddler
Was this helpful?
Thanks for your feedback!