Fail to start Bamboo Elastic agent in Custom Windows Image: Network is unreachable or Connection timed out
Problem
After customizing a Windows Elastic AMI, the following appears in the C:\Users\Bamboo\bamboo-elastic-agent.out log file which suggests a connection issue when connecting to the EC2 Metadata URL: http://169.254.169.254/latest/meta-data/
Exception in thread "main" java.net.SocketException: Network is unreachable: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:79)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
...
at com.atlassian.aws.utils.URLFetcherImpl.fetchData(URLFetcherImpl.java:26)
at com.atlassian.aws.utils.URLFetcherUtils.fetchData(URLFetcherUtils.java:48)
at com.atlassian.aws.utils.URLFetcherUtils.fetchString(URLFetcherUtils.java:40)
at com.atlassian.aws.ec2.Ec2UtilsThin.getUserData(Ec2UtilsThin.java:49)
at com.atlassian.bamboo.agent.elastic.installer.ElasticAgentInstaller.<init>(ElasticAgentInstaller.java:47)
at com.atlassian.bamboo.agent.elastic.installer.ElasticAgentInstaller.main(ElasticAgentInstaller.java:166)Or:
Exception in thread "main" java.net.ConnectException: Connection timed out: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:79)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
...
at com.atlassian.aws.utils.URLFetcherImpl.fetchData(URLFetcherImpl.java:26)
at com.atlassian.aws.utils.URLFetcherUtils.fetchData(URLFetcherUtils.java:48)
at com.atlassian.aws.utils.URLFetcherUtils.fetchString(URLFetcherUtils.java:40)
at com.atlassian.aws.ec2.Ec2UtilsThin.getUserData(Ec2UtilsThin.java:49)
at com.atlassian.bamboo.agent.elastic.installer.ElasticAgentInstaller.<init>(ElasticAgentInstaller.java:47)
at com.atlassian.bamboo.agent.elastic.installer.ElasticAgentInstaller.main(ElasticAgentInstaller.java:166)Diagnosis
RDP into the instance and confirm if the URL http://169.254.169.254/latest/meta-data/ can be accessed.
Cause
The elastic agent fail to connect to AWS Instance Metadata due to incorrect route tables. Routes from the original subnet this AMI was launched into must have been snapshot when creating the new AMI and those routes are not applicable for the new VPC/AZ/Subnet. This can occur when the custom AMI is not prepared with (or without for Windows Server 2016) Sysprep via EC2 Launch.
Resolution
Windows Server 2016:
- Make sure the EC2 instance stops instead of terminates upon shutdown so AMI can be created after instance shutdown: EC2 Console > Right click EC2 instance > Instance Settings > Change Shutdown Behaviour: Stop
- Run the application EC2LaunchSettings
- Click "Shutdown without Sysprep"
- Once the instance is in the Stopped state, create a new AMI from it.
Windows Server 2012 & 2014:
Run the script file C:\ProgramData\Amazon\EC2-Windows\Launch\Module\Scripts\Add-Routes.ps1 or InitializeInstance.ps1 to fix the routing and test or run Sysprep to initialize the AMI correctly: