Develop Reference

Spring Boot with Cloud GCP Firestore, got error with Google Compute Engine GCP Credential, How to disable the error logs

In our springboot project, we use only firestore, no other GCP services included
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-gcp-starter-data-firestore</artifactId>
<version>1.2.7.RELEASE</version>
</dependency>
but we always got error with GCP Credidential, I want to disable that track, how to do it?
java.io.IOException: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
In the application property we put as:
spring
cloud:
gcp:
firestore:
enabled: true
project-id: my-firestore-project-id
credentials:
location: classpath:my-firestore-authentication-file-xxxx.json
I already tried to disabled as:
spring.cloud.gcp.logging.enabled=false
spring.cloud.gcp.trace.enabled=false
and still error, the error wont stop the application but it's disturb and how to disable that?

Error while creating S3 Client in Java

I'm having a problem.
I have a Java project that works fine (written by someone else), I cloned it to my computer (from GitHub) and opened it in Eclipse. When I'm trying to run it, I keep getting an error when the script tries to create a s3 client.
This is the code (this.client = AmazonS3Client instance):
ClientConfiguration clientConfiguration = new ClientConfiguration();
ClientConfiguration.setMaxConnections(10);
clientConfiguration.setConnectionTimeout(120000);
clientConfiguration.setSocketTimeout(120000);
clientConfiguration.setMaxErrorRetry(5);
this.client = new AmazonS3Client(new InstanceProfileCredentialsProvider(), clientConfiguration);
This is the error:
Exception in thread "main" java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/ObjectMapper
at com.amazonaws.internal.config.InternalConfig.<clinit>(InternalConfig.java:43)
at com.amazonaws.internal.config.InternalConfig$Factory.<clinit>(InternalConfig.java:304)
at com.amazonaws.util.VersionInfoUtils.userAgent(VersionInfoUtils.java:142)
at com.amazonaws.util.VersionInfoUtils.initializeUserAgent(VersionInfoUtils.java:137)
at com.amazonaws.util.VersionInfoUtils.getUserAgent(VersionInfoUtils.java:100)
at com.amazonaws.ClientConfiguration.<clinit>(ClientConfiguration.java:64)
I tried also "this.client = AmazonS3ClientBuilder.defaultClient();" with a "Amazon3Client" instance, but it didn't work as well - same problem, error when trying to define ClientConfiguration...
I'm using aws-java-sdk-1.11.160.jar (tried before with 1.11.111 and didn't work, so I thought maybe upgrading the jar would help... it didn't)
I'm on this a couple of days now, and I can't seem to find the problem...
Any help would be great... Thank you!

You miss the jackson-databind jar in your classpath.
If you use maven, you can add the following dependency to solve this error:
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version><your version></version>
</dependency>

This code is written to grab the AWS secret and access key from the EC2 instance profile. When you clone it locally you'll need to provide a different way to authenticate to the AWS API.
http://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html
When constructing the client this code is using the option for a IAM instance profile
new InstanceProfileCredentialsProvider(),
Credentials provider implementation that loads credentials from the Amazon EC2 Instance Metadata Service.
http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/InstanceProfileCredentialsProvider.html

AWS Instance Profile doesn't work with Spring Cloud AWS

I have a small Spring Boot app, using Spring Cloud AWS (1.0.0.RELEASE) to access SQS queue. It is beeing deployed on an EC2 instance with Instance Profile set. It appears that AWS side of things is working, as I can access both relevant metadata links: iam/info and iam/security-credentials/role-name, and they do contain correct information. Just to be sure, I've used aws cmdline utility (aws sqs list-queues) and it does work, so I guess setup is ok. However, when the app starts, it reads application.properties (which contains line cloud.aws.credentials.instanceProfile=true) then drops following warning: com.amazonaws.util.EC2MetadataUtils: Unable to retrieve the requested metadata and finally throws following exception:
Caused by: com.amazonaws.AmazonServiceException: The security token included in the request is invalid. (Service: AmazonSQS; Status Code: 403; Error Code: InvalidClientTokenId; Request ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1071)
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:719)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:454)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:294)
at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2291)
at com.amazonaws.services.sqs.AmazonSQSClient.getQueueUrl(AmazonSQSClient.java:516)
at com.amazonaws.services.sqs.buffered.AmazonSQSBufferedAsyncClient.getQueueUrl(AmazonSQSBufferedAsyncClient.java:278)
at org.springframework.cloud.aws.messaging.support.destination.DynamicQueueUrlDestinationResolver.resolveDestination(DynamicQueueUrlDestinationResolver.java:78)
at org.springframework.cloud.aws.messaging.support.destination.DynamicQueueUrlDestinationResolver.resolveDestination(DynamicQueueUrlDestinationResolver.java:37)
at org.springframework.messaging.core.CachingDestinationResolverProxy.resolveDestination(CachingDestinationResolverProxy.java:88)
at org.springframework.cloud.aws.messaging.listener.AbstractMessageListenerContainer.start(AbstractMessageListenerContainer.java:295)
at org.springframework.cloud.aws.messaging.listener.SimpleMessageListenerContainer.start(SimpleMessageListenerContainer.java:38)
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:173)
... 17 common frames omitted
...which means that for some reason Spring Cloud AWS is not picking up on Instance Profile credentials. I've enabled debug log level on com.amazonaws.request and it appears that request is sent without access key and secret key.
DEBUG --- com.amazonaws.request : Sending Request: POST https://sqs.eu-west-1.amazonaws.com / Parameters: (Action: GetQueueUrl, Version: 2012-11-05, QueueName: xxxxxxxxxxxxx, ) Headers: (User-Agent: aws-sdk-java/1.9.3 Linux/3.14.35-28.38.amzn1.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.45-b02/1.8.0_45 AmazonSQSBufferedAsyncClient/1.9.3, )
Anybody has any idea what am I missing or at least any hints how to further debug this?
EDIT: After going through spring-cloud-aws code a bit, I've kinda moved forward. Configuration file application.properties bundled with jar had some text value for accessKey and secretKey. My customized application.properties haven't got those properties and that probably caused spring to use values in bundled file as defaults. I've included them with empty values, which changed the exception to com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain. It appears that AWS SDK is configured with DefaultProviderChain, yet it still fails to pick up instance profile credentials.

The solution to this problem comes from two distinct facts.
Instance profile credentials are going to be used only and only if application.properties has instanceProfile property set to true and accessKey set to null (ContextCredentialsAutoConfiguration).
Even if you will provide your custom application.properties file, Spring is going to read application.properties file bundled with app jar (if it does exist). If that's the case, properties from both files will sum up to create an execution enviroment. I suspect that bundled file is parsed first, then custom second, overriding any property present in bundled file.
In my case, bundled application.properties had accessKey and secretKey placeholders (with phony values) which were filled out by developer whenever he wanted some testing outside of EC2 enviroment. That made accessKey not null and therefore, excluded instance profile path. I just removed the application.properties file from jar and that solved the problem.

cloud:
aws:
credentials:
accessKey:
secretKey:
instanceProfile: true
useDefaultAwsCredentialsChain: true
This would do the trick, if you were using the latest (2.X.X) Spring AWS Cloud.

Access EC2 Instance using Basic Credentials

I'm trying to use the Java SDK to programmatically stop an EC2 instance. I'm very new the AWS api, but what I'm trying to do should be very basic:
BasicAWSCredentials credentials = new BasicAWSCredentials(ACCESS_KEY, SECRET_KEY);
AmazonEC2Client ec2 = new AmazonEC2Client(credentials); //ERROR caused by this line
I've seen several examples of this, for example here. I know the better way to do this is by using IAM roles, but I'm just trying to get something to work first. The line that creates the AmazonEC2 throws this error
java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/ObjectMapper at com.amazonaws.util.json.Jackson.
I can't figure out why it's throwing a class definition not found error.

I figured out the issue. Apparently the AWS library requires three more dependencies in fasterxml.jackson.core. Because I was using Ivy to configure these, I had to add these manually

ServiceBusConfiguration.configureWithConnectionString can't work, error "The key 'SharedAccessKeyName' is not valid for this connection string"

I'm using newest Azure java sdk(version 0.6), but I found I have problems when I use service bus configuration "configureWithConnectionString" function.
Here is my code below,
//get config
string connectionString = "Endpoint=sb://testservicebusnamespace.servicebus.windows.net/;SharedAccessKeyName=MySharedAccessKey;SharedAccessKey=<this is secret not show>";
config = new Configuration();
ServiceBusConfiguration.configureWithConnectionString(null, config, connectionString);
//create service
service = ServiceBusService.create(config);
And I get connection String from my service bus namespace portal in Azure. But When I run this code, it throws an exception "The key 'SharedAccessKeyName' is not valid for this connection string".
I don't know what's the problem, because I get the connection from Azure portal, and I've checked its content(the SharedAccessKeyName, SharedAccessKey), they are right.
So could anyone help me? Is it my problem or this SDK needs update(because I've heard portal already uses SAS, but SDK still uses ACS to authenticate)?
Thanks very much.

It looks like Java Azure SDK(0.6.0) still uses ACS to authenticate. See this github issue comments:
the current SDK works with service bus namespace with ACS auth mode, but not new created servicebus namespace which use SAS token auth mode. we will investigate this issue
and
One workaround is to create the service bus namespace via PowerShell.
This seems to enable the ACS auth mode by default. Command is:
New-AzureSBNameSpace -Name MyNameSpace -Location "West Europe"
Take a note of the "DefaultKey" you get after running the command. You
need to use that with the ServiceBusConfiguration and I'm not sure if
that is available via the Azure management portal. Use "owner" as the
authenticationName parameter to the configureWithWrapAuthentication.

I'm adding this answer, because I thought I was doomed. But found a new library.
This dependency (which 0.5.0 was latest version when I wrote this post)......gives you the issue.
<dependency>
<groupId>com.microsoft.windowsazure</groupId>
<artifactId>microsoft-azure-api-servicebus</artifactId>
<version>0.5.0</version>
</dependency>
Microsoft slightly altered the groupid. Here is a dependency that seems much more update to date.
<dependency>
<groupId>com.microsoft.azure</groupId>
<artifactId>azure-servicebus</artifactId>
<version>0.9.7</version>
</dependency>
Maven link here : https://search.maven.org/#artifactdetails%7Ccom.microsoft.azure%7Cazure-servicebus%7C0.9.7%7Cjar
So its an old question. But hopefully this points somebody in the right direction.