I'm trying to use Spring Cloud's AWS SQS in a project I'm working on. At this time, I'm only running the application locally on my dev machine. Thus, what I want is to connect to the SQS on AWS without having to deploy my app to an EC2 instance.
However, it seems like the AWS SDK used in Spring Cloud's AWS package will attempt to authenticate through metadata and wants to resolve 169.254.169.254/latest/meta-data/instance-id. Since I'm still the running the app locally, the endpoint cannot be resolved and an error is thrown:
2019-12-29 16:38:27.420 WARN 22462 --- [ restartedMain] com.amazonaws.util.EC2MetadataUtils : Unable to retrieve the requested metadata (/latest/meta-data/instance-id). Failed to connect to service endpoint:
com.amazonaws.SdkClientException: Failed to connect to service endpoint:
at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100) ~[aws-java-sdk-core-1.11.699.jar:na]
at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70) ~[aws-java-sdk-core-1.11.699.jar:na]
at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75) ~[aws-java-sdk-core-1.11.699.jar:na]
at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:62) ~[aws-java-sdk-core-1.11.699.jar:na]
at com.amazonaws.util.EC2MetadataUtils.getItems(EC2MetadataUtils.java:400) ~[aws-java-sdk-core-1.11.699.jar:na]
at com.amazonaws.util.EC2MetadataUtils.getData(EC2MetadataUtils.java:369) ~[aws-java-sdk-core-1.11.699.jar:na]
at org.springframework.cloud.aws.context.support.env.AwsCloudEnvironmentCheckUtils.isRunningOnCloudEnvironment(AwsCloudEnvironmentCheckUtils.java:38) ~[spring-cloud-aws-context-2.2.1.RELEASE.jar:2.2.1.RELEASE]
at org.springframework.cloud.aws.context.annotation.OnAwsCloudEnvironmentCondition.matches(OnAwsCloudEnvironmentCondition.java:37) ~[spring-cloud-aws-context-2.2.1.RELEASE.jar:2.2.1.RELEASE]
at org.springframework.context.annotation.ConditionEvaluator.shouldSkip(ConditionEvaluator.java:108) ~[spring-context-5.2.2.RELEASE.jar:5.2.2.RELEASE]
at org.springframework.context.annotation.ConfigurationClassParser.processConfigurationClass(ConfigurationClassParser.java:221) ~[spring-context-5.2.2.RELEASE.jar:5.2.2.RELEASE]
at org.springframework.context.annotation.ConfigurationClassParser.processImports(ConfigurationClassParser.java:587) ~[spring-context-5.2.2.RELEASE.jar:5.2.2.RELEASE]
...
I've tried to explicitly supply an SQS endpoint in my bean, but it still attempts to connect to 169.254.169.254 resulting in the error above:
public AmazonSQSAsync sqsClient() {
EndpointConfiguration endpointConfig = new AwsClientBuilder.EndpointConfiguration(
"sqs.us-east-1.amazonaws.com",
"us-east-1"
);
return AmazonSQSAsyncClientBuilder
.standard()
.withEndpointConfiguration(endpointConfig)
.withCredentials(new AWSStaticCredentialsProvider(new DefaultAWSCredentialsProviderChain().getCredentials()))
.build();
}
Although I'm not certain about this, I suspect that the error is occurring because I'm running the app locally on my dev machine and so it couldn't resolve to the endpoint. But I'm not entirely sure about this too because I'm running other AWS services in the same app using the AWS SDK and I don't have this error.
I've the following dependencies in my pom.xml and it appears that having any one of them will cause the error to occur. That's, I don't even have to create the beans to have that error. Adding these dependencies will immediately, for some reason, causes the SDK to attempt to resolve that endpoint and fail with that error.
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-aws</artifactId>
<version>2.2.1.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-aws-messaging</artifactId>
<version>2.2.1.RELEASE</version>
</dependency>
What else should I do to fix the error that it failed to connect to service endpoint?
I have received below errors when running Spring Boot Project for AWS SNS
i.InstanceMetadataServiceResourceFetcher : Fail to retrieve token
com.amazonaws.util.EC2MetadataUtils : Unable to retrieve the requested metadata (/latest/dynamic/instance-identity/document). Failed to connect to service endpoint:
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'stackResourceRegistryFactoryBean' defined in class path resource [org/springframework/cloud/aws/autoconfigure/context/ContextStackAutoConfiguration.class]: Unsatisfied dependency expressed through method 'stackResourceRegistryFactoryBean' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'amazonCloudFormation' defined in class path resource [org/springframework/cloud/aws/autoconfigure/context/ContextStackAutoConfiguration.class]: Invocation of init method failed; nested exception is java.lang.IllegalStateException: There is no EC2 meta data available, because the application is not running in the EC2 environment. Region detection is only possible if the application is running on a EC2 instance
Error creating bean with name 'org.springframework.cloud.aws.core.env.ResourceIdResolver.BEAN_NAME': Invocation of init method failed; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'stackResourceRegistryFactoryBean' defined in class path resource [org/springframework/cloud/aws/autoconfigure/context/ContextStackAutoConfiguration.class]: Unsatisfied dependency expressed through method 'stackResourceRegistryFactoryBean' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'amazonCloudFormation' defined in class path resource [org/springframework/cloud/aws/autoconfigure/context/ContextStackAutoConfiguration.class]: Invocation of init method failed; nested exception is java.lang.IllegalStateException: There is no EC2 meta data available, because the application is not running in the EC2 environment. Region detection is only possible if the application is running on a EC2 instance
Resolution applied:
Add below code snippet to SpringBootApplication & Click on Run Button in the IDE
#SpringBootApplication(
exclude = {
org.springframework.cloud.aws.autoconfigure.context.ContextInstanceDataAutoConfiguration.class,
org.springframework.cloud.aws.autoconfigure.context.ContextStackAutoConfiguration.class,
org.springframework.cloud.aws.autoconfigure.context.ContextRegionProviderAutoConfiguration.class
}
)
I know many people have already explained what's issue and how to skip this error. But, all the above method didn't work for me. Finally after lot of search I set following aws configuration property and it worked for me.
cloud.aws.region.use-default-aws-region-chain=true
cloud.aws.stack.auto=false
cloud.aws.credentials.access-key=
cloud.aws.credentials.secret-key=
cloud.aws.region.auto=false
cloud.aws.region.static=
cloud.aws.stack=false
logging.level.com.amazonaws.util.EC2MetadataUtils=error
logging.level.com.amazonaws.internal.InstanceMetadataServiceResourceFetcher=error
The following configuration worked for me:
cloud:
aws:
region:
auto: false
static: eu-west-1
use-default-aws-region-chain: true
stack: false
stack.auto: false
logging:
level:
com:
amazonaws:
internal:
InstanceMetadataServiceResourceFetcher: error
util:
EC2MetadataUtils: error
When using the AWS Java SDK outside of an EC2 instance, you can disable loading credentials or regions from EC2 Metadata instance service with a System property or an Environment variable.
System Property : com.amazonaws.sdk.disableEc2Metadata
Environment variable : AWS_EC2_METADATA_DISABLED
However, you will still get a warning at application startup, it tells you the EC2Metadata is disabled by configuration : "EC2 Instance Metadata Service is disabled"
2020-03-27 18:10:42.483 WARN 71123 --- [main] com.amazonaws.util.EC2MetadataUtils : Unable to retrieve the requested metadata (/latest/meta-data/instance-id). EC2 Instance Metadata Service is disabled
com.amazonaws.AmazonClientException: EC2 Instance Metadata Service is disabled
at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:65)
at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66)
at com.amazonaws.util.EC2MetadataUtils.getItems(EC2MetadataUtils.java:402)
at com.amazonaws.util.EC2MetadataUtils.getData(EC2MetadataUtils.java:371)
at org.springframework.cloud.aws.context.support.env.AwsCloudEnvironmentCheckUtils.isRunningOnCloudEnvironment(AwsCloudEnvironmentCheckUtils.java:38)
...
Fix the problem adding in the application.yaml file:
logging:
level:
com:
amazonaws:
util:
EC2MetadataUtils: error
As far as I understand its trying to check if your application is running on cloud environment, from the 5th last line of the stacktrace
org.springframework.cloud.aws.context.support.env.AwsCloudEnvironmentCheckUtils.isRunningOnCloudEnvironment(AwsCloudEnvironmentCheckUtils.java:38)
I checked out the the repo and here is the code snippet -
public static boolean isRunningOnCloudEnvironment() {
if (isCloudEnvironment == null) {
try {
isCloudEnvironment = EC2MetadataUtils
.getData(EC2_METADATA_ROOT + "/instance-id", 1) != null;
}
catch (AmazonClientException e) {
isCloudEnvironment = false;
}
}
return isCloudEnvironment;
}
As per the snippet it tried to fetch instance metadata and if it fails it return false, which means you are not on the Cloud Environment. However this code is expected to Catch AmazonClientException but the stacktrace you shared is throwing SdkClientException exception, which this part of the snippet can't catch and hence can't return false that you are not on a cloud environment.
Seems like some issue with this build of the library, As per the stacktrace you are using 2.2.1.RELEASE1 which was released 9 days back only. Could you please try to use one of the older version and then report if you are still facing the same issue.
Or as a temporary fix you can try tunnelling to your VPC's entry server. You can use sshuttle for tunnelling purpose if you want. This will mostly make that IP reachable from local machine as well.
I was getting same kind of error when running s3 bucket in spring boot i solved using different location used while creating bucket:
return AmazonS3ClientBuilder
.standard()
.withRegion("us-east-1") //This is the code i added to fix
.withCredentials(new AWSStaticCredentialsProvider(awsCredentials))
.build();
Set cloud.aws.region.auto property to false in Spring application properties (or via environment variables etc.).
Define cloud.aws.region.static property.
This will tell Spring not to use automatic detection based EC2 meta data. Some details.
You may need to configure credentials similarly, you can find related info in the same doc.
The issue is with the spring-cloud version Hoxton.SR8. Replace your spring-cloud version in your pom.xml with Finchley.SR1.
I also faced similar error but not from the Spring cloud. When i added below configuration in application.properties, error went away.
logging.level.com.amazonaws.util.EC2MetadataUtils=error
logging.level.com.amazonaws.internal.InstanceMetadataServiceResourceFetcher=error
But i'm facing some other issues while opening s3a path. Then i printed the stacktrace of the exception and then i got below error
Caused by: com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [BasicAWSCredentialsProvider: Access key or secret key is null, WebIdentityTokenCredentialsProvider: You must specify a value for roleArn and roleSessionName, EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), com.amazonaws.auth.InstanceProfileCredentialsProvider#7700439a: Failed to connect to service endpoint: ]
Already credentials are there /.aws folder but not sure why application still giving the error. Fixed by creating environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_KEY and AWS_SESSION_TOKEN
This was coming from an internal dependency at my company, so I couldn't get any of the previously posted solutions to work. What worked for me was to simply set in my environment variables AWS_REGION = 'us-east-1'.
Related
I am trying to use Spring Boot Actuator to restart the spring to update its newest properties.
I followed this tutorial:
https://javapointers.com/spring/spring-boot/how-to-restart-spring-boot-automatically-using-actuator/
So what I did is just to add the properties
management.endpoint.restart.enabled=true
management.endpoints.web.exposure.include=restart,health
and call the endpoint
{host}/actuator/restart
I tested it on my local using the embedded tomcat server, and it works.
But when I tried it in jboss/wildfly, because my client are using it for their production server, it doesn't work.
This is the error that I got:
org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.springframework.boot.autoconfigure.web.servlet.error.ErrorMvcAutoConfiguration': Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.boot.context.properties.ConfigurationPropertiesBindException: Error creating bean with name 'server-org.springframework.boot.autoconfigure.web.ServerProperties': Could not bind properties to 'ServerProperties' : prefix=server, ignoreInvalidFields=false, ignoreUnknownFields=true; nested exception is java.lang.IllegalStateException: org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext#3aa3a991 has been closed already
Please let me know if there is something else that I need to provide.
Maybe You could take a look at Spring Cloud Config to solve properly your problem : Spring Cloud Config
It allow you to externalize properties, manage them and expose them to clients (like your application).
And it has some benefits like 'No restart needed'
I am upgrading an application to Spring Boot 2.1.3 (from 1.5.x), and I am facing an issue at startup time. Below block can't be bound properly :
spring:
zipkin:
enabled: true
base-url: http://myZipkinServer
sender:
type: web
I am getting this error :
***************************
APPLICATION FAILED TO START
***************************
Description:
Failed to bind properties under 'spring.zipkin.sender.type' to org.springframework.cloud.sleuth.zipkin2.sender.ZipkinSenderProperties$SenderType:
Property: spring.zipkin.sender.type
Value: web
Origin: class path resource [application.yml]:68:13
Reason: 0
Action:
Update your application's configuration
A bit before I am getting a WARN log announcing the issue :
Exception encountered during context initialization - cancelling refresh attempt: org.springframework.boot.context.properties.ConfigurationPropertiesBindException: Error creating bean with name 'spring.zipkin.sender-org.springframework.cloud.sleuth.zipkin2.sender.ZipkinSenderProperties': Could not bind properties to 'ZipkinSenderProperties' : prefix=spring.zipkin.sender, ignoreInvalidFields=false, ignoreUnknownFields=true; nested exception is org.springframework.boot.context.properties.bind.BindException: Failed to bind properties under 'spring.zipkin.sender.type' to org.springframework.cloud.sleuth.zipkin2.sender.ZipkinSenderProperties$SenderType
I am trying to follow in debug, and I end up pretty deep in Spring Boot internals
in org.springframework.boot.context.properties.bind.Binder .
I have a similar app with more or less same version for which it works just fine. I am trying to find a difference, compare the execution flows, but not finding anything obvious.
In IntelliJ, I get the auto-completion so I know my yaml is formatted properly : the "web" value is proposed to me.
Any idea of how to investigate this kind of issue ?
Alright, so after a few hours struggling, I made some progress, and now the app starts - even though the root cause of the issue is not fully clear to me at this time.
Below are my findings :
one strange thing I noticed : if I change the sender.type from web to rabbit, then the application starts with no error.
I also found this Spring Boot issue report, very similar to mine, that was pointing at a JDK bug. And indeed, upgrading from jdk1.8.0_25 to jdk1.8.0_201 .
Finally, I also found that if I was using jdk1.8.0_25 and wasn't providing the sender.type at all, then the app was also starting with no issue.
For some reason, in the other app that I have and that works, I am able to use jdk1.8.0_25 and sender.type: web
If anyone has a methodology to figure out this kind of issue quickly, don't hesitate to add it in the comment or edit this answer.
When using org.springframework.cloud:spring-cloud-gcp-starter-trace:1.0.0.RELEASE and running my integration tests locally I get this error message:
org.springframework.beans.factory.BeanCreationException: Error
creating bean with name 'stackdriverSender' defined in class path
resource
[org/springframework/cloud/gcp/autoconfigure/trace/StackdriverTraceAutoConfiguration.class]:
Bean instantiation via factory method failed; nested exception is
org.springframework.beans.BeanInstantiationException: Failed to
instantiate [zipkin2.reporter.Sender]: Factory method
'stackdriverSender' threw exception; nested exception is
java.io.IOException: The Application Default Credentials are not
available. They are available if running in Google Compute Engine.
Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS
must be defined pointing to a file defining the credentials. See
https://developers.google.com/accounts/docs/application-default-credentials
for more information.
This is totally understandable since this environment variable is not present locally and I don't want to use Sleuth/Stackdriver tracing when running the tests. I've looked in the reference documentation but I can only seem to find information on how to disable Sleuth for specific integration points such as RxJava, RestTemplate etc. But how can I disable Sleuth entirely?
I've tried setting spring.sleuth.enabled=false but this doesn't seem to make any difference.
HowTo: Disable all tests for a Webflux Sleuth implementation with Zipkin
If, you have these dependencies:
implementation 'org.springframework.cloud:spring-cloud-starter-sleuth'
implementation 'org.springframework.cloud:spring-cloud-sleuth-zipkin'
Then, fully disable Sleuth/Zipkin for your integration tests by adding an application.yml to your test/resources folder with:
spring:
sleuth: # All sleuth features disabled for integration tests
enabled: false
reactor:
enabled: false
web:
enabled: false
async:
enabled: false
client:
template:
enabled: false
zipkin: # All zipkin features disabled for integration tests
enabled: false
I actually found the answer by looking at the source for the StackdriverTraceAutoConfiguration class. The way to solve it if using GCP is to set spring.cloud.gcp.trace.enabled=false. This disables tracing for all integration points.
You can disable tracing, logging and set a fake project id as follow:
spring.cloud.gcp.project-id=fake-project-id
spring.cloud.gcp.logging.enabled=false
spring.cloud.gcp.trace.enabled=false
same code with spring boot version 1.3.8.RELEASE and cloud version 'Brixton.SR' works right.
but '1.5.2.RELEASE' and 'Dalston.RC1' not work.
exception
HTTP Status 500 - Request processing failed; nested exception is java.lang.RuntimeException: com.netflix.client.ClientException: Load balancer does not have available server for client: math-service
OK, you can add the following to your application.properties, then it works.
feign.hystrix.enabled=true
See Dalston Release note and Disable HystrixCommands For FeignClients By Default, Feign clients no longer wrap methods in Hystrix commands by default . You must have Hystrix on the classpath and also set feign.hystrix.enabled=true to have Feign automatically wrap methods in Hystrix commands.
I have a small Spring Boot app, using Spring Cloud AWS (1.0.0.RELEASE) to access SQS queue. It is beeing deployed on an EC2 instance with Instance Profile set. It appears that AWS side of things is working, as I can access both relevant metadata links: iam/info and iam/security-credentials/role-name, and they do contain correct information. Just to be sure, I've used aws cmdline utility (aws sqs list-queues) and it does work, so I guess setup is ok. However, when the app starts, it reads application.properties (which contains line cloud.aws.credentials.instanceProfile=true) then drops following warning: com.amazonaws.util.EC2MetadataUtils: Unable to retrieve the requested metadata and finally throws following exception:
Caused by: com.amazonaws.AmazonServiceException: The security token included in the request is invalid. (Service: AmazonSQS; Status Code: 403; Error Code: InvalidClientTokenId; Request ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1071)
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:719)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:454)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:294)
at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2291)
at com.amazonaws.services.sqs.AmazonSQSClient.getQueueUrl(AmazonSQSClient.java:516)
at com.amazonaws.services.sqs.buffered.AmazonSQSBufferedAsyncClient.getQueueUrl(AmazonSQSBufferedAsyncClient.java:278)
at org.springframework.cloud.aws.messaging.support.destination.DynamicQueueUrlDestinationResolver.resolveDestination(DynamicQueueUrlDestinationResolver.java:78)
at org.springframework.cloud.aws.messaging.support.destination.DynamicQueueUrlDestinationResolver.resolveDestination(DynamicQueueUrlDestinationResolver.java:37)
at org.springframework.messaging.core.CachingDestinationResolverProxy.resolveDestination(CachingDestinationResolverProxy.java:88)
at org.springframework.cloud.aws.messaging.listener.AbstractMessageListenerContainer.start(AbstractMessageListenerContainer.java:295)
at org.springframework.cloud.aws.messaging.listener.SimpleMessageListenerContainer.start(SimpleMessageListenerContainer.java:38)
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:173)
... 17 common frames omitted
...which means that for some reason Spring Cloud AWS is not picking up on Instance Profile credentials. I've enabled debug log level on com.amazonaws.request and it appears that request is sent without access key and secret key.
DEBUG --- com.amazonaws.request : Sending Request: POST https://sqs.eu-west-1.amazonaws.com / Parameters: (Action: GetQueueUrl, Version: 2012-11-05, QueueName: xxxxxxxxxxxxx, ) Headers: (User-Agent: aws-sdk-java/1.9.3 Linux/3.14.35-28.38.amzn1.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.45-b02/1.8.0_45 AmazonSQSBufferedAsyncClient/1.9.3, )
Anybody has any idea what am I missing or at least any hints how to further debug this?
EDIT: After going through spring-cloud-aws code a bit, I've kinda moved forward. Configuration file application.properties bundled with jar had some text value for accessKey and secretKey. My customized application.properties haven't got those properties and that probably caused spring to use values in bundled file as defaults. I've included them with empty values, which changed the exception to com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain. It appears that AWS SDK is configured with DefaultProviderChain, yet it still fails to pick up instance profile credentials.
The solution to this problem comes from two distinct facts.
Instance profile credentials are going to be used only and only if application.properties has instanceProfile property set to true and accessKey set to null (ContextCredentialsAutoConfiguration).
Even if you will provide your custom application.properties file, Spring is going to read application.properties file bundled with app jar (if it does exist). If that's the case, properties from both files will sum up to create an execution enviroment. I suspect that bundled file is parsed first, then custom second, overriding any property present in bundled file.
In my case, bundled application.properties had accessKey and secretKey placeholders (with phony values) which were filled out by developer whenever he wanted some testing outside of EC2 enviroment. That made accessKey not null and therefore, excluded instance profile path. I just removed the application.properties file from jar and that solved the problem.
cloud:
aws:
credentials:
accessKey:
secretKey:
instanceProfile: true
useDefaultAwsCredentialsChain: true
This would do the trick, if you were using the latest (2.X.X) Spring AWS Cloud.