Bamboo Documentation

   Bamboo 5.9

   Bamboo 5.8.x

   Bamboo 5.7.x
   Bamboo 5.6.x
   More... 

 

 

Bamboo Knowledge Base Resources

Skip to end of metadata
Go to start of metadata

Technical Overview of Remote Agent Connectivity Mechanism

The traffic between agent and server is bi-directional, however initial connections are only sent from agent to server. Responses are sent from server to agent.
The server has 2 urls / ports to connect with a remote agent.

  1. bamboo base url, usually http(s)://<host>:8085/<bamboo-app-context>
  2. bamboo broker url, usually tcp://<host>:54663 if not defined otherwise

Both ports need to be open in the firewall. The direction of the initial connection is from agent to server.

To establish connections to the server the remote agent selects 2 available local ports, one for each connection. The Communication between the remote agent and server is therefore done through the two protocols http and jms. The http is for the initial ping to the server and bootstrapping. Artifacts are also transferred through the http. Other communication and transfer of data is done through the Apache AMQ based jms protocol.

Diagnosing Remote Agent Connectivity
  1. It is also a good idea to check and ensure that the Remote agent server is reachable from the Bamboo server and vice versa. You can check this using ping. An example is something like "ping your_host"
  2. Telnet to check whether the client is able to connect to the server on a particular port. If you are running Bamboo on port 8085, and the broker URL listens port 54663, you should try to run the following commands from the remote agent: "telnet bamboo_server_host 8085" and "telnet bamboo_server_host 54663"

  3. Check that your Broker URL is not pointing to localhost when it should be pointing to the Bamboo's server IP. Please refer to: Remote agent points to localhost and not to Bamboo's server IP

  4. You might sometimes be facing issues with the port binding where the logs complain about the port numbers already in use. This mostly happens when two instances of Bamboo are running on same server each using the same port. We have seen this issue where the production and development instance are installed together each listening on port 54663. If this is the case, you can shutddown one of the instances, edit the HOME/bamboo.cfg.xml file and edit the port in the following two lines (or make the change from the Administration -> General Configuration page if you are using Bamboo 5.x or later). You will need to restart your Bamboo server for the changes to take effect

    In cases where you have only a single Bamboo server running and still getting a binding port error, the netstat command will come in handy. You can check for example using the command "netstat -an | findstr 54663" for Windows OR "netstat -an | grep 54663" for a Unix based OS

 

Troubleshooting remote agent connectivity problems

This section will give you some tips on troubleshooting the most common connectivity problems with remote agents

Remote Agent Registration

Comment out the following packages and make sure that DEBUG logging is enabled for all of them from the <bamboo-install>/atlassian-bamboo/WEB-INF/classes/log4j.properties file. The debug level on the remote agent can be done ins same way. In case you don't have log4j.properties file in the remote agent, you can copy one from the Bamboo server to the agent home:

Next, restart Bamboo server and remote agents, and useful debug traces will be produced that show the agent-server communication in detail. Please use these logs to support your inquiries with Atlassian Support.

Network Timeouts

If your agent is dropping out due to errors similar to:

Firewall/Router

Ensure that you don't have a firewall/router which might be closing network connections due to inactivity.

Low Network Throughput

The error can also occur, whilst sending large messages over slow network link:

This problem arises because the ActiveMQ message broker continuously checks for activity on a socket, and whilst building a large message in the TCP socket buffer, the inactivity monitor will time out

As the document suggests, edit your <Bamboo-Home>/bamboo.cfg.xml file to figure out and edit the following section:

setting the maxInactivityDuration=300000 value to a greater number or 0 to disable it all together.

 

Remote Agent Communication with the Server

The first thing to know is the stage you are facing the issue.. Is it that the remote agent can not be started or that artifact can not be transferred? In which case the problem is coming from the HTTP. Otherwise, a JMS issue.

Debugging HTTP Issue
  • Base URL - The remote agent uses the base url stored in its configuration to access Bamboo and not the BaseURL defined in Bamboo. Check agent's log for the line: Agent bootstrap using baseUrl: http://localhost:8085/bamboo/agentServer If in the process you noticed that the base URL is wrong, you can update this appropriately from the HOME/bamboo-cfg.xml. You might also want to consider the use of the IP address
  • Ports: is the Bamboo's server port (as defined in agent's configuration) accessible from the agent? Since the very first action of the agent is to get fingerprint from the server, (log line:Requesting fingerprintRequestUrl: http://localhost:8085/bamboo/agentServer/GetFingerprint.action?hostName=atlassian_support) let's do it from the remote agent's host:

    This should return a HTTP/1.1 200 OK response in both cases - if you didn't, then there most be a configuration issue and you will have to check the network and/or the configuration file.. If it did return HTTP/1.1 200 OK, check if Bamboo's Agents page shows

    If it does not, then you might probably be connected to the wrong bamboo server or there are some more root causes in the logs. You can contact support with the logs.

  • If the agent is not able to ping the server with the errors similar to:

    This is most likely due to the fact that the DNS can not be resolved. Update the DNS records to include the appropriate server name & IP used in the configuration file. If updating of the DNS server used is not possible, this can be achieved by manually adding the following entry, replacing the IP & server name as appropriate to the below file on the server hosting Bamboo 
    Linux:/etc/hosts
    Windows:C:\Windows\System32\drivers\etc\hosts

    (lightbulb)You need to have local administrator rights on the machine hosting Bamboo to make these changes.

  • Proxy: If you have a proxy server involved, try to bypass them by connecting to the server directly. Please note that if there are artifact transfer problems (rejections or timeouts): Does it have HTTP 1.0 compatibility turned on? Does it have any configuration options related to chunked transfer encoding turned on? Your proxy server needs to support HTTP/1.1. See Agent upload of artifacts fails to set Content-Length in HTTP/1.1

 

Debugging JMS Issue
  • Corruption:If you are getting jms related errors in the logs of Bamboo or the remote agent, it is a good idea to purge the following folders; jms-store, cachesindex and temp from the Bamboo home and restart both server and agents.
  • Load: Check the load on the server, dbms and agent machines. Also check the number of your agents (local,remote or elastic) and how busy they are? If the load is very high, then an upgrade your Bamboo server to the latest version is recommended as most likely the AMQ has been updated.

Should the above not helpful to resolve, contact support attaching the following:

  • The Bamboo server and remote agent logs after adding the line log4j.category.org.apache.activemq=DEBUG to your atlassian-bamboo/WEB-INF/classes/log4j.properties file and restarting the server. The debug level on the remote agent can be done ins same way. In case you dont have log4j.properties file in the remote agent, you can copy one from the Bamboo server to the agent home.
  • If possible, try getting the network traffic dumps between agent and the server, preferably one from the agent and one form the server. Refer to Capturing HTTP traffic using Wireshark or Fiddler
Page: Error when running agent java.net.SocketException Address already in use Page: Remote agent does not restart after server outage Page: Bamboo 3.3 Remote Agent doesn't start due to incompatible plugin Page: All Remote agents went offline and cannot connect - No space left on device Page: Remote Agents fail to start due to missing files Page: Agent Startup Fails With Message "The system cannot find the file specified. (0x2)" Page: Errors when attempting to install remote agent for bamboo with service wrapper Page: Remote Agent disconnecting after 10 minutes Page: Remote Agent fails to start up: Error creating bean with name jmsMessageConverterTarget Page: Bamboo remote agent fails to start - javax.servlet.ServletContext Page: Remote Agent fails to launch as it does not have a no-args constructor Page: Remote Agent fails to start up: HTTP status code 404 received in response to fingerprint request Page: Downloading Remote Agent jar hangs Page: Bamboo crash while downloading the remote agent Page: All Bamboo Remote agents cannot connect after an upgrade Page: Installed remote agents failing to start due to corrupt files Page: When starting up the agent, it keeps looping infinitely Page: Agent upload of artifacts fails to set Content-Length in HTTP/1.1 Page: Remote agents are unable to validate the failover and connect to the server Page: Remote agents does not build or deploy immediately Page: A remote agent is loading on Bamboo UI never finishes Page: Remote agent points to localhost and not to Bamboo's server IP

9 Comments

  1. You may want to mention to Linux users that incoming TCP port 26224 is required for the agent to run.  We build a custom fedora based image and didn't realize that the firewall port needed to be opened.  Fedora has a firewall by default while Ubuntu and Amazon Linux for instance do not.

     

    Edit: we did notice that the requirement is mentioned for the windows install: Creating a custom elastic image - Windows

  2. Anyone ever see the NetworkAnalyser tool fail?  No matter what URL I provide, it's always successful even if it's complete jibberish:

    % java NetworkAnalyser http://blahblahdlbhal
    Testing connectivity to Bamboo Server: http://blahblahdlbhal
    Trying to establish connection with remote server: http://blahblahdlbhal
    Connection established!
    Testing connectivity to Bamboo Server: http:∫//blahblahdlbhal:54663
    Trying to establish connection with remote server: http:∫//blahblahdlbhal:54663
    Connection established!

     

     

    1. No, never! Nice tool...

      Though it does fail when you use another/wrong protocol:

      Error parsing Bamboo Server URL.java.net.MalformedURLException: unknown protocol: tcp

  3. I noticed that my logs are saying:

    Agent bootstrap using baseUrl: http://bamboo-host-server:8085/agentServer/ - which is not the correct URL (it appears to be the default).

    So I suspect that I need to change my bambo-cfg.xml file. However, I cannot find this file. Are the docs correct up above?

    Any help would be appreciated.

    1. Hi Ken,

       

      Can you navigate to the Administration >> System >> System information page and check for the Bamboo home ? The location there contains the bamboo-cfg.xml file which you can edit. Also check that you have the base url of bamboo correctly set.

  4. Hi

    Looking at my bamboo.cfg.xml file, I don't see what to change:

    enguser@DC1-Alt-Bamboo:/opt/BambooData$ more bamboo.cfg.xml
    <?xml version="1.0" encoding="UTF-8"?>

    <application-configuration>
    <setupStep>complete</setupStep>
    <setupType>custom</setupType>
    <buildNumber>4010</buildNumber>
    <properties>
    <property name="bamboo.artifacts.directory">${bambooHome}/artifacts</property>
    <property name="bamboo.config.directory">${bambooHome}/xml-data/configuration</property>
    <property name="bamboo.jms.broker.client.uri">failover:(tcp://DC1-Alt-Bamboo:54663?wireFormat.maxInactivityDuration=300000)?initialReconnect
    Delay=15000&amp;maxReconnectAttempts=10</property>
    <property name="bamboo.jms.broker.uri">tcp://0.0.0.0:54663?wireFormat.maxInactivityDuration=300000</property>
    <property name="bamboo.project.directory">${bambooHome}/xml-data/builds</property>
    <property name="buildWorkingDir">${bambooHome}/xml-data/build-dir</property>
    <property name="daily.backup.dir">${bambooHome}/backups</property>
    <property name="hibernate.c3p0.acquire_increment">1</property>
    <property name="hibernate.c3p0.idle_test_period">100</property>
    <property name="hibernate.c3p0.max_size">100</property>
    <property name="hibernate.c3p0.max_statements">0</property>
    <property name="hibernate.c3p0.min_size">0</property>
    <property name="hibernate.c3p0.timeout">30</property>
    <property name="hibernate.connection.driver_class">net.sourceforge.jtds.jdbc.Driver</property>
    <property name="hibernate.connection.password">pcChNnQdZFm9&gt;7v</property>
    <property name="hibernate.connection.url">jdbc:jtds:sqlserver://10.10.10.2:1433/atlassianbamboodb</property>
    <property name="hibernate.connection.username">jira.sql</property>
    <property name="hibernate.dialect">net.sf.hibernate.dialect.SQLServerIntlDialect</property>
    <property name="hibernate.setup">true</property>
    <property name="license.string">AAABSw0ODAoPeNpdkEtvwjAQhO/+FZZ6Dkp4NIBkqeSBmgqSlLQqh16MWaglx47sJC3/vk4g6uPgg 2c1387s3VpznLEae3PsecvZbDl2cRi
    94LHrTVAEhmle1VxJEtDyoBTOtTqBMVah4n2JQ1WWoBmnA hegW9AobcoD6Oz0akAb4rko1EA7QkRrIB3W8VzHW6BQyZqyOqUlkFgclcRvSp4HOd5SLgh0+ujT6 g+05Wcu7I+pEsUtFU1PJ
    ScqDKBDn25kjbwFUusGkAXIGiSVDOKviuvLEGDaBXAXg2fDGUgD8ZH3v HyXreOiSLJ0tUGZPlPJzXXT6pbAltaV0r2Iijgl9jnj+cybTH10vUISkcD3Ayfwn/aOv48enfvne f5v48ulgr5
    9mG238S5M7MLbyPo3SfSbO6T8adJIwUtewxHljWYf1MCfC9uC3xfRnmcwLAIURntEC CRVvfrgqgc3sJefPuwPH1cCFE1ebZr9PeijFvbW7nfRNrEDydHAX02gc</property>
    <property name="lucene.index.dir">${bambooHome}/index</property>
    <property name="serverId">BH9B-JVHH-SXOE-2E93</property>
    <property name="webwork.multipart.saveDir">${bambooHome}/temp</property>
    </properties>
    </application-configuration>

     

    I'm looking to change: Check agent's log for the line: Agent bootstrap using baseUrl: http://localhost:8085/bamboo/agentServer If in the process you noticed that the base URL is wrong, you can update this appropriately from the HOME/bamboo-cfg.xml

  5. If you use a different path, and /or different listener port than 8085,  run the bamboo agent installer with your settings:
    java -Dbamboo.home=/data/bamboo-agent-home -jar atlassian-bamboo-agent-installer-5.5.1.jar http://bamboo.example.com:8088/agentServer/

    you'll see this error:
    I/O exception (java.net.ConnectException) caught when processing request: Connection refused


    Modify /data/bamboo-agent-home/conf/wrapper.conf
    wrapper.app.parameter.2=http://bamboo.example.com:8088/agentServer/

  6. Hi,

    I am trying to install a Remote Agent by running this command on Amazon WebServices  

    java -jar atlassian-bamboo-agent-installer-5.5.1.jar http://bamboo.Domain.xxx/agentServer/


    But I keep getting I/O Exception (java.net.ConnectException_ caught when processing request: Connection timed out: connect INFO | jvm 90.....[WrapperSimpleAppMain] [HttpMethodDirector] Retrying request

    Has anyone seen this before? Any help is appreciated.

    Thanks!

    Amar.,

  7. Hi all,

    my experience with EC2 elastic agents, is that they connect first to the Bamboo Server on port 54663 - and then attempt to connect to the Base URL via the tunnel to the Bamboo Server on 54663.

    What that means is that the Bamboo Server sends a connection request to itself at the Base URL (because of the tunnelling).

    So - if like me, you have an ELB, Nginx, or some other reverse proxy in-front of Bamboo, then the Bamboo Server will be attempting to connect to that machine.

    My workaround is to add a hosts file (/etc/hosts) alias for 127.0.0.1 matching the Base URL.

    eg.

    127.0.0.1  localhost bamboo.example.com

     

    The trick with this, is that the localhost will need to be listening on the same port as the reverse proxy.  Additionally if you have HTTPS configured on the reverse proxy, then the localhost listener will also need to be providing HTTPS.

    This may assist those with crazier HTTP/HTTPS routing via forward proxies, etc. common in enterprise environments.