Summary

Elastic agents are failing to bootstrap showing the following error message: The AWS Access Key Id you provided does not exist in our records or AWS authentication requires a valid Date or x-amz-date header

Environment

Bamboo Elastic Agents based on version 3.8 or earlier

Diagnosis

Bamboo Elastic Agents based on image version 3.8 or earlier will fail to connect to Atlassian's AWS S3 bucket during its bootstrap with one of the following errors which can be found in the $HOME/bamboo-elastic-agent.out log:

Exception in thread "main" AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: , AWS Error Code: InvalidAccessKeyId, AWS Error Message: The AWS Access Key Id you provided does not exist in our records., S3 Extended Request ID: 
	at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:644\)
	at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:338\)

Exception in thread "main" com.amazonaws.services.s3.model.AmazonS3Exception: AWS authentication requires a valid Date or x-amz-date header (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: ), S3 Extended Request ID: 

Cause

Bamboo Elastic Agent versions 3.8 and earlier will send an Authorization header whilst connecting to the S3 bucket which is no longer valid:

2021-12-01 02:19:42,621 DEBUG [main] [wire] >> "Authorization: AWS ..."

Solution

Update the base Elastic Agent image in your AMI. Though the Elastic Agent will update itself to the same level as the Bamboo Server during bootstrap, the base image won't.

You can refer to the following KB to know how to create a custom Bamboo Elastic Agent:

• Creating a custom elastic image - Atlassian Documentation

TL;DR: this is the part that you should be interested in:

If you want to find out the AMI IDs for a version of Bamboo you don't have running or you're starting an image from scratch and you need the image baseline:

1. Open https://packages.atlassian.com/maven/repository/public/com/atlassian/bamboo/atlassian-bamboo/
2. On the resulting directory page, select the link that represents the version of Bamboo you are currently running. For example, if you are running Bamboo 5.9.7, select on the 5.9.7 link. Another directory page opens, listing a .pom and some additional checksum files.
    Do not select a version number link that contains 'mX', 'rcX' or 'betaX' (where 'X' is a number), since these relate to publicly available developmental releases of Bamboo.
3. Open the atlassian-bamboo-x.x.x.pom file (where x.x.x) is your version of Bamboo). The image version/baseline is stored in the elastic-image.version tag. For example, this value is "4.2" for Bamboo version 5.9.4.
4. Open https://packages.atlassian.com/maven/repository/public/com/atlassian/bamboo/atlassian-bamboo-elastic-image/
   Select the image version/baseline you found in the elastic-image.version tag.

On the resulting directory page, the file with AMI extension contains all stock image AMI IDs.

As an example, by following the guide above, if you are running Bamboo 7.2.3, the equivalent Elastic Agent base image will be 6.20:

% grep "<elastic-image.version>" atlassian-bamboo-7.2.3.pom
    <elastic-image.version>6.20</elastic-image.version>

Using an image greater than the recommended one can possibly work with older Bamboo versions, but there is no guarantee.