Problem

Elastic Agents are stuck in Pending and connections timeout

The following appears in bamboo-elastic-agent.out:

2016-11-03 14:21:04,054 INFO [tunnelserver:26224-1-thread-3] [LocalAcceptor] Accepted a local connection on port 46593, tunnel age: 0s
Exiting due to fatal exception.
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:992)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
...
Caused by: java.io.EOFException: SSL peer shut down incorrectly
...

The following appears in atlassian-bamboo.log:

2016-11-03 00:56:28,855 ERROR [tunnel:54663-4527] [TerminateOnTimeoutTunnelStatusTracker] Timeout has expired for waiting for tunnel to instance [i-<instance id>], terminating [Tunnel <ip-address>:4527 - (<ip-address>:26224) - 0.0.0.0:54663] Error: Connection timed out (Connection timed out)
2016-11-03 00:56:28,855 INFO [elastic-pool-3-thread-4] [InstanceTerminator] Bamboo is requesting that EC2 instance i-<instance id> be shut down.

Cause

Traffic between the Bamboo Server and Elastic agents is tunneled. HTTP traffic from the Elastic Agent arrives on the Bamboo Server via the tunnel and it is routed by the tunnel client to the Base URL (Bamboo Administration >> Overview >> General Configuration >> Base URL) of the Bamboo Server. If the Base URL of the instance cannot be reached from the Bamboo Server (e.g. the address is a load balancer or reverse proxy and the Bamboo Server has no outbound access), the HTTP connection cannot be established. You can bypass this by changing the endpoint Bamboo uses by using the bamboo.ec2.agent.endpoint JVM argument.

Since this file is in the installation folder, an upgrade requires that the changes to this file be replayed in the new installation. For example, if you had configured your EC2 agents to use http://localhost:TOMCAT_HTTP_PORT (e.g. http://localhost:8085) instead of https://bamboo.mydomain.com to communicate with the Bamboo server, and setenv.sh was not copied to your new Bamboo installation directory, the setting will have been lost. 

Resolution

1. Update Bamboo startup to include the -Dbamboo.ec2.agent.endpoint=http://localhost:TOMCAT_HTTP_PORT JVM parameter. This can be added to the JVM_SUPPORT_RECOMMENDED_ARGS= in the <Bamboo installation folder>/bin/setenv.sh or setenv.bat file or added to the service definition if running on Windows as a system service. Please see the following documentation for configuring Bamboo start-up arguments: Configuring your system properties.
   1. Be sure to replace TOMCAT_HTTP_PORT with the http port configured in your <bamboo-install>/conf/server.xml.
   2. If a context path has been configured in <bamboo-install>/conf/server.xml, it also needs to be added to the URL.
2. Restart Bamboo

    Any changes made to the setenv.sh or setenv.bat files will need to be lost following an upgrade and changes will need to be redone.