Unable to load AWS credentials from any provider in the chain
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Unable to set up an environment with an outbound proxy to both:
- Make a connection to the Atlassian Marketplace through the outbound proxy.
- Use the "nonProxyHosts" JVM argument to connect to an AWS Elasticsearch service without proxying the request.
See here for additional information on using an outbound proxy with Bitbucket Server/Data Center.
Environment
- Bitbucket Server/Data Center
- AWS Elasticsearch
- Outgoing proxy
- Using IAM roles for authentication between servers
Diagnosis
The following shows up in the logs when you are able to connect to the Marketplace but not Elasticsearch:
2020-11-02 14:43:41,299 ERROR [Caesium-1-2] c.a.b.i.s.i.IndexingSynchronizationService An error was encountered while checking or creating the mapping in Elasticsearch
com.atlassian.bitbucket.internal.search.indexing.exceptions.IndexException: Unable to check whether a valid mapping exists in Elasticsearch
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.lambda$isMappingPresent$9(IndexingSynchronizationService.java:268)
at io.atlassian.fugue.Either$Left.fold(Either.java:478)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.isMappingPresent(IndexingSynchronizationService.java:267)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeMapping(IndexingSynchronizationService.java:109)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.synchronizeStores(IndexingSynchronizationService.java:82)
at com.atlassian.bitbucket.internal.search.indexing.jobs.StartupChecksJob.run(StartupChecksJob.java:80)
at com.atlassian.bitbucket.internal.search.common.cluster.ClusterJobRunner.runJob(ClusterJobRunner.java:82)
at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:153)
at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:118)
at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:97)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:443)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJob(CaesiumSchedulerService.java:438)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJobWithRecoveryGuard(CaesiumSchedulerService.java:462)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:390)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$1.consume(CaesiumSchedulerService.java:285)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$1.consume(CaesiumSchedulerService.java:282)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:65)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:59)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:34)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:74)
at rx.internal.operators.OnSubscribeToObservableFuture$ToObservableFuture.call(OnSubscribeToObservableFuture.java:43)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
at rx.Observable.subscribe(Observable.java:10423)
at rx.Observable.subscribe(Observable.java:10390)
at rx.Observable.subscribe(Observable.java:10271)
at com.atlassian.bitbucket.internal.search.indexing.util.Observables.consume(Observables.java:64)
at com.atlassian.bitbucket.internal.search.indexing.util.Observables.consumeSingle(Observables.java:92)
... 18 common frames omitted
Caused by: com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain
at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:117)
at vc.inreach.aws.request.AWSSigner.getSignedHeaders(AWSSigner.java:91)
at vc.inreach.aws.request.AWSSigningRequestInterceptor.process(AWSSigningRequestInterceptor.java:29)
at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:133)
at org.apache.http.impl.nio.client.MainClientExec.prepareRequest(MainClientExec.java:520)
at org.apache.http.impl.nio.client.MainClientExec.prepare(MainClientExec.java:146)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:124)
at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141)
at com.atlassian.elasticsearch.client.apache.httpclient.InstrumentedNHttpClientBuilder$1.execute(InstrumentedNHttpClientBuilder.java:93)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:74)
at com.atlassian.elasticsearch.client.apache.httpclient.ApacheRequestExecutor.execute(ApacheRequestExecutor.java:132)
at com.atlassian.elasticsearch.client.internal.InternalClient.execute(InternalClient.java:29)
at com.atlassian.elasticsearch.client.Client.execute(Client.java:38)
at com.atlassian.bitbucket.internal.search.client.DefaultConfigurableElasticsearchClient.lambda$execute$1(DefaultConfigurableElasticsearchClient.java:34)
at java.util.Optional.map(Optional.java:215)
at com.atlassian.bitbucket.internal.search.client.DefaultConfigurableElasticsearchClient.execute(DefaultConfigurableElasticsearchClient.java:34)
at com.atlassian.bitbucket.internal.search.indexing.administration.DefaultIndexAdministrationService.codeSearchMappingExists(DefaultIndexAdministrationService.java:55)
at com.atlassian.bitbucket.internal.search.indexing.IndexingSynchronizationService.isMappingPresent(IndexingSynchronizationService.java:266)
... 17 common frames omittedCause
There is a metadata IP address that Bitbucket needs to be able to hit in addition to the Elasticsearch URL, as per this AWS guide.
This error indicates that an outbound proxy or some other network-level restriction is preventing access to this endpoint.
Solution
Ensure that Bitbucket is able to send traffic to the IP address 169.254.169.254, and add this IP address to the -Dhttp.nonProxyHosts list if using an outbound proxy.
Last modified on Dec 18, 2020
Powered by Confluence and Scroll Viewport.