Elastic agents remain in Pending and do not connect
Problem
Elastic Agents are stuck in Pending and connections timeout
The following appears in bamboo-elastic-agent.out:
2016-11-03 14:21:04,054 INFO [tunnelserver:26224-1-thread-3] [LocalAcceptor] Accepted a local connection on port 46593, tunnel age: 0s
Exiting due to fatal exception.
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:992)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
...
Caused by: java.io.EOFException: SSL peer shut down incorrectly
...
The following appears in atlassian-bamboo.log
:
2016-11-03 00:56:28,855 ERROR [tunnel:54663-4527] [TerminateOnTimeoutTunnelStatusTracker] Timeout has expired for waiting for tunnel to instance [i-<instance id>], terminating [Tunnel <ip-address>:4527 - (<ip-address>:26224) - 0.0.0.0:54663] Error: Connection timed out (Connection timed out)
2016-11-03 00:56:28,855 INFO [elastic-pool-3-thread-4] [InstanceTerminator] Bamboo is requesting that EC2 instance i-<instance id> be shut down.
Cause
Traffic between the Bamboo Server and Elastic agents is tunneled. HTTP traffic from the Elastic Agent arrives on the Bamboo Server via the tunnel and it is routed by the tunnel client to the Base URL (Bamboo Administration >> Overview >> General Configuration >> Base URL
) of the Bamboo Server. If the Base URL of the instance cannot be reached from the Bamboo Server (e.g. the address is a load balancer or reverse proxy and the Bamboo Server has no outbound access), the HTTP connection cannot be established. You can bypass this by changing the endpoint Bamboo uses by using the bamboo.ec2.agent.endpoint
JVM argument.
Since this file is in the installation folder, an upgrade requires that the changes to this file be replayed in the new installation. For example, if you had configured your EC2 agents to use http://
localhost:TOMCAT_HTTP_PORT (e.g. http://localhost:8085)
instead of https://bamboo.mydomain.com
to communicate with the Bamboo server, and setenv.sh
was not copied to your new Bamboo installation directory, the setting will have been lost.
Resolution
- Update Bamboo startup to include the -Dbamboo.ec2.agent.endpoint=http://localhost:TOMCAT_HTTP_PORT JVM parameter. This can be added to the JVM_SUPPORT_RECOMMENDED_ARGS= in the
<Bamboo installation folder>/bin/setenv.sh
or setenv.bat file or added to the service definition if running on Windows as a system service. Please see the following documentation for configuring Bamboo start-up arguments: Configuring your system properties.- Be sure to replace
TOMCAT_HTTP_PORT
with the http port configured in your<bamboo-install>/conf/server.xml
. - If a context path has been configured in
<bamboo-install>/conf/server.xml,
it also needs to be added to the URL.
- Be sure to replace
- Restart Bamboo
Any changes made to the setenv.sh or setenv.bat files will need to be lost following an upgrade and changes will need to be redone.