Unable to load AWS credentials from any provider in the chain

Still need help?

The Atlassian Community is here for you.

Ask the community

Platform Notice: Server and Data Center Only. This article only applies to Atlassian products on the server and data center platforms.

Summary

Unable to set up an environment with an outbound proxy to both:

  • Make a connection to the Atlassian Marketplace through the outbound proxy.
  • Use the "nonProxyHosts" JVM argument to connect to an AWS Elasticsearch service without proxying the request.

See here for additional information on using an outbound proxy with Bitbucket Server/Data Center.

Environment

  • Bitbucket Server/Data Center
  • AWS Elasticsearch
  • Outgoing proxy
  • Using IAM roles for authentication between servers

Diagnosis

The following shows up in the logs when you are able to connect to the Marketplace but not Elasticsearch:

2020-11-02 14:43:41,299 ERROR [Caesium-1-2]  c.a.b.i.s.i.IndexingSynchronizationService An error was encountered while checking or creating the mapping in Elasticsearch
com.atlassian.bitbucket.internal.search.indexing.exceptions.IndexException: Unable to check whether a valid mapping exists in Elasticsearch
	at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.lambda$isMappingPresent$9(IndexingSynchronizationService.java:268)
	at io.atlassian.fugue.Either$Left.fold(Either.java:478)
	at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.isMappingPresent(IndexingSynchronizationService.java:267)
	at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeMapping(IndexingSynchronizationService.java:109)
	at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeStores(IndexingSynchronizationService.java:82)
	at com.atlassian.bitbucket.internal.search.indexing.jobs.StartupChecksJob.run(StartupChecksJob.java:80)
	at com.atlassian.bitbucket.internal.search.common.cluster.ClusterJobRunner.runJob(ClusterJobRunner.java:82)
	at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:153)
	at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:118)
	at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:97)
	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:443)
	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJob(CaesiumSchedulerService.java:438)
	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJobWithRecoveryGuard(CaesiumSchedulerService.java:462)
	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:390)
	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$1.consume(CaesiumSchedulerService.java:285)
	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$1.consume(CaesiumSchedulerService.java:282)
	at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:65)
	at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:59)
	at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:34)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
	at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:74)
	at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:43)
	at rx.Observable.unsafeSubscribe(Observable.java:10327)
	at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
	at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
	at rx.Observable.subscribe(Observable.java:10423)
	at rx.Observable.subscribe(Observable.java:10390)
	at rx.Observable.subscribe(Observable.java:10271)
	at com.atlassian.bitbucket.internal.search.indexing.util.Observables.consume(Observables.java:64)
	at com.atlassian.bitbucket.internal.search.indexing.util.Observables.consumeSingle(Observables.java:92)
	... 18 common frames omitted
Caused by: com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain
	at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:117)
	at vc.inreach.aws.request.AWSSigner.getSignedHeaders(AWSSigner.java:91)
	at vc.inreach.aws.request.AWSSigningRequestInterceptor.process(AWSSigningRequestInterceptor.java:29)
	at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:133)
	at org.apache.http.impl.nio.client.MainClientExec.prepareRequest(MainClientExec.java:520)
	at org.apache.http.impl.nio.client.MainClientExec.prepare(MainClientExec.java:146)
	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:124)
	at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141)
	at com.atlassian.elasticsearch.client.apache.httpclient.InstrumentedNHttpClientBuilder$1.execute(InstrumentedNHttpClientBuilder.java:93)
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:74)
	at com.atlassian.elasticsearch.client.apache.httpclient.ApacheRequestExecutor.execute(ApacheRequestExecutor.java:132)
	at com.atlassian.elasticsearch.client.internal.InternalClient.execute(InternalClient.java:29)
	at com.atlassian.elasticsearch.client.Client.execute(Client.java:38)
	at com.atlassian.bitbucket.internal.search.client.DefaultConfigurableElasticsearchClient.lambda$execute$1(DefaultConfigurableElasticsearchClient.java:34)
	at java.util.Optional.map(Optional.java:215)
	at com.atlassian.bitbucket.internal.search.client.DefaultConfigurableElasticsearchClient.execute(DefaultConfigurableElasticsearchClient.java:34)
	at com.atlassian.bitbucket.internal.search.indexing.administration.DefaultIndexAdministrationService.codeSearchMappingExists(DefaultIndexAdministrationService.java:55)
	at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.isMappingPresent(IndexingSynchronizationService.java:266)
	... 17 common frames omitted

Cause

There is a metadata IP address that Bitbucket needs to be able to hit in addition to the Elasticsearch URL, as per this AWS guide.

This error indicates that an outbound proxy or some other network-level restriction is preventing access to this endpoint.

Solution

Ensure that Bitbucket is able to send traffic to the IP address 169.254.169.254, and add this IP address to the -Dhttp.nonProxyHosts list if using an outbound proxy.

Last modified on Dec 18, 2020

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.