Elastic Agents can't decode user-data

Still need help?

The Atlassian Community is here for you.

Ask the community

Summary

Elastic Agent is not started on the Elastic Instance, because it's unable to decode user data passed by Bamboo to the EC2. The following error is present in /tmp/testIfStartedFromBamboo.log file on the Elastic Instance:

Wed Feb 16 12:34:19 UTC 2022
Exception in thread "main" com.fasterxml.jackson.core.JsonParseException: Unexpected character ('�' (code 65533 / 0xfffd)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (String)"�XG��ȑ�+��ʞ����{ƒ $�w$��
                                              �[;�ؕ��@V֫�YY�����(�S�R�~|�2����æ����i��鴥���k^�iY�l�~=�q�%���}��K
                                                                                                              �s�D5y?�K���6=���1�<�/�:/���P$�ti�%d�%���]����s�����eV�ߓs����r�ϩ3r2���ۇ�oSڤgh���o1ߡ����DG�L
      #� �4B�k|�����į��O�߳�;�S�-�b8�\I�1�aE�$v���';i�O}KN�3�����5�i/�w�J��c4͐
                                                                             �I��p��$z�p�h�I��{�!L"��E$!	�$��!���{@ewr���?���ʯ_dN��������>_����Ω�v]���$��������v���̻~J�tZދ�K�i\��Y���xڄ�R�R{&�黬����HJ�~�$���Y<`���]�qF�}�֘����1��U��o
                                         �~
�^��~��k����5IR�]���a��{l)BX���"[truncated 2443 chars]; line: 1, column: 2]
	at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1804)
	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:693)
	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:591)
	at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1902)
	at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:757)
	at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:4141)
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4000)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3004)
	at com.atlassian.aws.utils.JsonUtils.fromJson(JsonUtils.java:43)
	at com.atlassian.aws.ec2.Ec2UtilsThin.getUserData(Ec2UtilsThin.java:49)
	at com.atlassian.bamboo.agent.elastic.metadata.RetrieveMetaData.main(RetrieveMetaData.java:24)
Marker /root/.startedFromBamboo not found. This instance was started manually.
/opt/bamboo-elastic-agent/bin/testIfStartedFromBamboo.sh: 17: exit: Illegal number: -1

On the Agent startup logs you can also observe the following message:

Fri Feb 16 12:35:02 UTC 2022 - This instance has not been started by Bamboo. The agent will not be started.

Environment

Bamboo using Elastic Agents.

Diagnosis

On the Elastic Agent, create the following Java code:

import com.atlassian.aws.utils.JsonUtils;
import com.atlassian.aws.utils.URLFetcherUtils;
import com.atlassian.bamboo.agent.elastic.ElasticAgentUserData;
import com.atlassian.bamboo.agent.elastic.ElasticAgentUserDataImpl;

import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.SortedMap;
import java.util.TreeMap;

public class getUserData {
    private static <T> T fetchData(final Class<T> aClass) throws IOException {
        String USER_DATA_URL = "http://169.254.169.254/latest/user-data";
        boolean shouldAttemptDecompression = true;
        return JsonUtils.fromJson(URLFetcherUtils.fetchString(USER_DATA_URL, StandardCharsets.UTF_8, shouldAttemptDecompression), aClass);
    }

    public static void main(final String[] args) throws IOException {
        final ElasticAgentUserData agentUserData = fetchData(ElasticAgentUserDataImpl.class);
        final SortedMap<String, String> userData = new TreeMap<>(agentUserData.getMetaData());
        if (args.length == 0) {
            userData.forEach((_key, _value) -> System.out.println(_key + "=" + _value));
        }
        else {
            String value = userData.get(args[0]);
            System.out.println(userData.get(args[0]) + "=" + value);
        }
    }
}

Then run it as root. A failed Server x Agent JDK combination will result in the following exception:

# java -cp /opt/bamboo-elastic-agent/bin/atlassian-bamboo-agent-elastic-installer.jar getUserData.java
Exception in thread "main" com.fasterxml.jackson.databind.JsonMappingException: Could not deserialize keystore
 at [Source: (String)"{"baseURL":"https://bamboo.mydomain.net/agentServer/","startupTimeoutSeconds":600,"keyManagerAlgorithmName":"SunX509","trustManagerAlgorithmName":"PKIX","metaData":{"bamboo.version.specific.data.location":"s3://bamboo-agent-release-ap-se2/9.1.0-rc2/4049d092bc97001cafb7b38f7465fa9e2ce987e1","bamboo.server.fingerprint":"-3826544862034641465","aws.startedFromBamboo":"true","startupScript_0":""[truncated 4731 chars]; line: 1, column: 5135] (through reference chain: com.atlassian.bamboo.agent.elastic.ElasticAgentUserDataImpl["keyStore"])
	at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:392)
	at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:351)
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapAndThrow(BeanDeserializerBase.java:1821)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:315)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:176)
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:322)
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4674)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3629)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3597)
	at com.atlassian.aws.utils.JsonUtils.fromJson(JsonUtils.java:43)
	at getUserData.fetchData(getUserData.java:15)
	at getUserData.main(getUserData.java:19)
Caused by: com.fasterxml.jackson.core.JsonParseException: Could not deserialize keystore
 at [Source: (String)"{"baseURL":"https://bamboo.mydomain.net/agentServer/","startupTimeoutSeconds":600,"keyManagerAlgorithmName":"SunX509","trustManagerAlgorithmName":"PKIX","metaData":{"bamboo.version.specific.data.location":"s3://bamboo-agent-release-ap-se2/9.1.0-rc2/4049d092bc97001cafb7b38f7465fa9e2ce987e1","bamboo.server.fingerprint":"-3826544862034641465","aws.startedFromBamboo":"true","startupScript_0":""[truncated 4731 chars]; line: 1, column: 5135]
	at com.atlassian.aws.utils.JsonUtils$KeyStoreDeserialiser.deserialize(JsonUtils.java:107)
	at com.atlassian.aws.utils.JsonUtils$KeyStoreDeserialiser.deserialize(JsonUtils.java:82)
	at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:138)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:313)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:176)
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:322)
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4674)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3629)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3597)
	at com.atlassian.aws.utils.JsonUtils.fromJson(JsonUtils.java:43)
	at getUserData.fetchData(getUserData.java:15)
	at getUserData.main(getUserData.java:19)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at jdk.compiler/com.sun.tools.javac.launcher.Main.execute(Main.java:404)
	at jdk.compiler/com.sun.tools.javac.launcher.Main.run(Main.java:179)
	at jdk.compiler/com.sun.tools.javac.launcher.Main.main(Main.java:119)
Caused by: java.io.IOException: Integrity check failed: java.security.NoSuchAlgorithmException: Algorithm HmacPBESHA256 not available
	at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2167)
	at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:222)
	at java.base/java.security.KeyStore.load(KeyStore.java:1479)
	at com.atlassian.aws.utils.JsonUtils$KeyStoreDeserialiser.deserialize(JsonUtils.java:101)
	... 18 more
Caused by: java.security.NoSuchAlgorithmException: Algorithm HmacPBESHA256 not available
	at java.base/javax.crypto.Mac.getInstance(Mac.java:191)
	at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2145)
	... 21 more

A successful Server x Agent JDK combination will print the expected user data:

# java -cp /opt/bamboo-elastic-agent/bin/atlassian-bamboo-agent-elastic-installer.jar getUserData.java
aws.agentAssemblyLocation=bamboo-agent-release-ap-se2/9.1.0-rc2/5d2d537148a7<...>215248b24f4c11c145e9d72
aws.bambooServerVersion=9.1.0-rc2
aws.startedFromBamboo=true
bamboo.instance.specific.data.locations=
bamboo.server.fingerprint=-3826544862034641465
bamboo.version.specific.data.location=s3://bamboo-agent-release-ap-se2/9.1.0-rc2/4049d092bc9700<...>e2ce987e1
startupScript_0=

Cause

User data is compressed on the Server and then decompressed on the Elastic Instance using Java’s built-in compression mechanism. If the Java version on the Bamboo server is significantly newer than the Java version on the Elastic Image, the decompression process may fail. Check this JDK bug for more details.

Sample malfunctioning combinations:

Server

Agent

JDK 11.0.14.1+1

JDK 1.8.0_202

JDK 11.0.14.1+1

JDK 1.8.0_275

JDK 11.0.17JDK 1.8.0_312

JDK 11.0.17+8

JDK-11.0.11+9

Sample working combinations:

Server

Agent

JDK 1.8.0_202

JDK 11.0.14.1

JDK 1.8.0_262

JDK 1.8.0_202

JDK 1.8.0_262

JDK 11.0.14.1

JDK 1.8.0_322

JDK 1.8.0_202

JDK 1.8.0_322

JDK 11.0.14.1

JDK 11.0.9.1+1

JDK 1.8.0_202

JDK 11.0.9.1+1

JDK 11.0.14.1

JDK 11.0.14.1

JDK 1.8.0_312

JDK 11.0.14.1+1

JDK 11.0.14.1

JDK 11.0.17+8JDK 11.0.15+10

Solution

Upgrade the Java version installed on your Elastic Image, or downgrade the Java version used by the Bamboo server. Ideally, we recommended using the same JDK version on both Bamboo Server and Agents to avoid any future compatibility issues.

Last modified on Mar 8, 2023

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.