Bamboo elastic agents fail to start when there's a mismatch in the elastic agent instance id
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Bamboo elastic agents fail to start with the following errors showing up inside the <bamboo-home>/logs/atlassian-bamboo.log file:
2021-11-11 13:15:18,120 WARN [ActiveMQ Session Task-114] [RemoteInvocationTraceInterceptor] Processing of BambooJmsInvokerServiceExporter remote call resulted in fatal exception: com.atlassian.bamboo.buildqueue.manager.RemotedRemoteAgentManager.registerAgent
java.lang.IllegalArgumentException: Elastic agent instance id i-08fff415d32b2d397 is not matching instance id on server
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:144)
at com.atlassian.bamboo.buildqueue.manager.RemoteAgentManagerImpl$2.visitElastic(RemoteAgentManagerImpl.java:185)
at com.atlassian.bamboo.v2.build.agent.ElasticAgentDefinitionImpl.accept(ElasticAgentDefinitionImpl.java:99)
at com.atlassian.bamboo.buildqueue.manager.RemoteAgentManagerImpl.registerReturningAgent(RemoteAgentManagerImpl.java:170)
at com.atlassian.bamboo.buildqueue.manager.RemoteAgentManagerImpl.registerAgent(RemoteAgentManagerImpl.java:123)
at com.atlassian.bamboo.buildqueue.manager.RemoteAgentManagerImpl.registerAgent(RemoteAgentManagerImpl.java:112)
at sun.reflect.GeneratedMethodAccessor2733.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.remoting.support.RemoteInvocationTraceInterceptor.invoke(RemoteInvocationTraceInterceptor.java:80)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy271.registerAgent(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.remoting.support.RemoteInvocation.invoke(RemoteInvocation.java:215)
at org.springframework.remoting.support.DefaultRemoteInvocationExecutor.invoke(DefaultRemoteInvocationExecutor.java:39)
at org.springframework.remoting.support.RemoteInvocationBasedExporter.invoke(RemoteInvocationBasedExporter.java:78)
at org.springframework.remoting.support.RemoteInvocationBasedExporter.invokeAndCreateResult(RemoteInvocationBasedExporter.java:114)
at org.springframework.jms.remoting.JmsInvokerServiceExporter.onMessage(JmsInvokerServiceExporter.java:106)
at com.atlassian.bamboo.remoting.BambooJmsInvokerServiceExporter.onMessage(BambooJmsInvokerServiceExporter.java:17)
at com.atlassian.bamboo.spring.MessageListenerUtils$1.lambda$onMessage$0(MessageListenerUtils.java:25)
at com.atlassian.bamboo.spring.MessageListenerUtils.wrapWithServerFingerprintCheck(MessageListenerUtils.java:64)
at com.atlassian.bamboo.spring.MessageListenerUtils.access$000(MessageListenerUtils.java:17)
at com.atlassian.bamboo.spring.MessageListenerUtils$1.onMessage(MessageListenerUtils.java:26)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:736)
at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:696)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:674)
at org.springframework.jms.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:645)
at org.springframework.jms.listener.SimpleMessageListenerContainer.processMessage(SimpleMessageListenerContainer.java:345)
at org.springframework.jms.listener.SimpleMessageListenerContainer.lambda$createListenerConsumer$2(SimpleMessageListenerContainer.java:322)
at org.apache.activemq.ActiveMQMessageConsumer.dispatch(ActiveMQMessageConsumer.java:1435)
at org.apache.activemq.ActiveMQSessionExecutor.dispatch(ActiveMQSessionExecutor.java:131)
at org.apache.activemq.ActiveMQSessionExecutor.iterate(ActiveMQSessionExecutor.java:202)
at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)Environment
- Bamboo 8.0 or higher.
- Custom elastic image.
Diagnosis
When a Bamboo elastic agent starts up and registers against Bamboo a new entry is added to the queue table (inside the Bamboo database) containing information about said agent, such as the agent id (queue_id), elastic agent instance id (elastic_instance_id), the date & time when it was created, whether it's enabled or disabled and what elastic image configuration it's using among other details. The important things to remember here are the agent id and elastic agent instance id. Starting from Bamboo 8.0 a new mechanism was introduced that allows elastic agents to reconnect to Bamboo in case they loose connection to it.
The first step to diagnosing this issue is to check whether you're using a custom elastic image. Elastic agents created from default (stock) elastic images that ship with Bamboo do not exhibit this problem. If you are using a custom elastic image you should ssh into the elastic instance from the elastic agent that fails to startup (following the steps inside your elastic agent page in Bamboo) and locate the /home/bamboo/bamboo-agent-home/bamboo-agent.cfg.xml file. Check if there's an agent id declared inside the xml file. If there is, then you're likely affected by the issue described in this kb article.
Cause
If you hardcoded the agent id inside your custom elastic image configuration Bamboo will think that you're trying to reconnect an existing elastic agent every time you try to start a new elastic agent because of the mechanism mentioned in the previous section that allows agents to try and reconnect.
While registering an elastic agent Bamboo goes to the queue table inside the database to look for information about the agent. If the agent id there matches the one from the agent you're trying to start up (because you hardcoded the agent id) it will try to compare the elastic agent instance id that you have in the database with the one from the new elastic agent. In this case the agent ids will match but the elastic agent instance ids will not and the agent will fail to startup. This is expected since elastic agent instance ids are dynamically assigned every time you start a new elastic agent.
This wasn't a problem before Bamboo 8.0 since elastic agents wouldn't attempt to reconnect but it is now because the elastic agent instance ids will never match.
Solution
Modify your custom elastic image and remove the /home/bamboo/bamboo-agent-home/bamboo-agent.cfg.xml file and re-snapshot the image then use it.