AWS S3 Java: doesObjectExist results in 403: FORBIDDEN - java

I'm having trouble with my Java program using the AWS SDK to interact with an S3 bucket.
This is the code I use to create an S3 client:
public S3StorageManager(S3Config config) throws StorageException {
BasicAWSCredentials credentials = new BasicAWSCredentials(myAccessKey(), mySecretKey());
AWSStaticCredentialsProvider provider = new AWSStaticCredentialsProvider(credentials);
this.s3Client = AmazonS3ClientBuilder
.standard()
.withCredentials(provider)
.withRegion(myRegion)
.build();
When I try to download a file, before starting the download I check wether the file exists or not with:
s3Client.doesObjectExists(bucketName, objectName);
This is where I get 403: FORBIDDEN.
The weird thing is this problem is raised only when I try to perform an object existence check before performing uploads in the same session.
In other words, after initializing the s3Client:
- if I first try to check if an object exists, it raises the FORBIDDEN problem;
- if I first perform file upload, it works fine and after that any object existence check works fine as well;
Here is my stacktrace:
com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Reques
t ID: A23BB805491E411F)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588) ~[aws-java-sdk-core-1.
11.128.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258) ~[aws-java-sdk-core-1.11
.128.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030) ~[aws-java-sdk-core-1.11.128
.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742) ~[aws-java-sdk-core-1.11.128.jar:
?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716) ~[aws-java-sdk-core-1.11.1
28.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) ~[aws-java-sdk-core-1.11.128.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) ~[aws-java-sdk-core-1.11.128.jar
:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) ~[aws-java-sdk-core-1.1
1.128.jar:?]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) ~[aws-java-sdk-core-1.11.128.jar:?]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4169) ~[aws-java-sdk-s3-1.11.128.jar:?]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4116) ~[aws-java-sdk-s3-1.11.128.jar:?]
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1237) ~[aws-java-sdk-s3-1.11.128.jar:?]
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1213) ~[aws-java-sdk-s3-1.11.128.jar:?]
at com.amazonaws.services.s3.AmazonS3Client.doesObjectExist(AmazonS3Client.java:1272) ~[aws-java-sdk-s3-1.11.128.jar:?]
Another weird thing is that all these problems started when I moved my Java program an EC2 remote machine.
If I execute it on my local machine, the S3 interaction works fine.
However I don't think the problem depends on the IAM roles, since I use the AWSStaticCredentialsProvider.

Your credentials may be correct, but you will still get FORBIDDEN if you do not set the correct IAM polices. To check for objects in s3 you need the following:
{
"Version":"2012-10-17",
"Statement":[
{
"Effect":"Allow",
"Action":[
"s3:ListBucket"
],
"Resource":["arn:aws:s3:::examplebucket"]
},
{
"Effect":"Allow",
"Action":[
"s3:GetObject"
],
"Resource":["arn:aws:s3:::examplebucket/*"]
}
]
}

Make sure the date time is set correctly on the machine you are making the request from, otherwise you will get a 403.

You need an action "ListBucket" for your bucket but not for the file in your bucket like:
{
"Action": [
"s3:ListBucket"
],
"Resource": "arn:aws:s3:::bucketName",
"Effect": "Allow"
}

I really look like an IAM policy issue.
What is your user's policies on your local machine vs what is your IAM role with which policy(ies)?
For your EC2 instance, when you create it, create a role with "AmazonS3FullAccess" policy, if it solves the problem you'll remove the useless rights.

Related

Transfer google drive file ownership using google java api

I have uploaded some test files to my google drive using their website and my gmail account (x#gmail.com). The files are in "My Drive" and not shared drive. I am then trying to write a google drive java api and oauth authorization code flow based program to transfer the ownership of these files to my another gmail account(y#gmail.com). i.e both the accounts belong to #gmail.com.
I am logged in to my application using x#gmail.com using authorization code workflow and my app is trying to access Google APIs with my access token. I am able to create a new permission on test files to make y#gmail.com a "writer" of the file. But when I try to make "y" an owner, I get error message. This is despite of x being the owner and being the one whose access token is used.
How do I transfer the ownership to another account?
Message changePerms(#PathVariable(name = "fileId") String fileId, #PathVariable(name = "userEmail") String newOwnerEmail) throws Exception {
Drive drive = new Drive.Builder(HTTP_TRANSPORT, JSON_FACTORY, getOauthCredentials())
.setApplicationName("my-app").build();
Permission newOwnerPermission = new Permission();
newOwnerPermission.setEmailAddress(newOwnerEmail);
newOwnerPermission.setType("user");
newOwnerPermission.setRole("owner"); // Works if this is "writer" so my credentials and auth is working
drive.permissions().create(fileId,newOwnerPermission).setTransferOwnership(true).execute();
}
Error :
POST https://www.googleapis.com/drive/v3/files/19wvNc4TmXhODJa7Iy3Qq97UAD8fJrVg/permissions?transferOwnership=true
{
"code": 400,
"errors": [
{
"domain": "global",
"message": "Bad Request. User message: \"You can't change the owner of this item.\"",
"reason": "invalidSharingRequest"
}
],
I think you can't transfer ownership files if they have been uploaded to your Drive. There was a mention of this on the help page, but that one line was removed earlier this year (though I can't say I know why as it appears that nothing has changed).
The current version of the help page is here, but as you can see using the Wayback Machine that only one month ago the following line was still present:
You can only transfer ownership of Google files and folders.
So I would say that this is probably the reason.

Google service account authorization error

I'm trying to use Google APIs with a service account.
Originally I used a normal project account with credentials and I managed to query the APIs. To use a service account, I modified the code to skip the browser auth flow and only use the service account credentials. It seems that the same request that used to work with the normal project credentials now give an error without clarifying much in the message. It goes as follows:
directoryService = new GoogleService().buildDirectoryService();
Users result = directoryService.users().list()
.setCustomer("my_customer")
.setOrderBy("email")
.execute();
public Directory buildDirectoryService() throws GeneralSecurityException, IOException {
final NetHttpTransport HTTP_TRANSPORT = GoogleNetHttpTransport.newTrustedTransport();
GoogleCredential credential = GoogleCredential
.fromStream(new FileInputStream("src/main/resources/newest_service_account_creds.json"))
.createScoped(SCOPES);
return new Directory.Builder(HTTP_TRANSPORT, JSON_FACTORY, credential)
.setApplicationName(APPLICATION_NAME)
.setHttpRequestInitializer(credential).build();
}
Unfortunately the error message is not saying much:
Exception in thread "main" com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request
{
"code" : 400,
"errors" : [ {
"domain" : "global",
"message" : "Invalid Input",
"reason" : "invalid"
} ],
"message" : "Invalid Input"
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1108)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at GetDrivePermissions.getFiles(GetDrivePermissions.java:35)
at Main.main(Main.java:7)
I repeated the process two times to make sure I didn't miss anything, including creating a new project, enabling the APIs, adding a service account, adding and enabling domain-wide delegation and obtaining a credentials json file, also adding access to the app in App Access Control in the admin UI. But calling
directoryService.users().list().setCustomer("my_customer").setOrderBy("email").execute();
service still returns the above error message.
The error message refers to "Invalid input", but the request used to work before, with normal application credentials. So the problem must lay not in how I call the Directory service but with my authentication, the only difference between the working and the non-working version.
Since this worked with using the project credentials, there must be something wrong with how I use the service account, but I can't find the error even after repeating the whole process two times. Based on the guides I found about service accounts, this is supposed to work. The credential object is created and it looks proper.
What can I look into next?
Thnaks for any insight in advance.
I changed this part back to use the application's own credentials. I have another call that uses the service account credentials and that works, so this is a workaround for now.

Google Play Developer API - 400 Invalid Value - InAppPurchases

My question is similar to this one. However, I am using the API Java Client Library with a service account, making calls to the API from my server.
My code is following this guide, which is very simple. However, I can't seem to get an appropriate error for my request. This is how I build my AndroidPublisher:
val credential = GoogleCredential.fromStream(FileInputStream(
"/path/to/json"
)).createScoped(Collections.singleton(AndroidPublisherScopes.ANDROIDPUBLISHER))
androidPublisher = AndroidPublisher.Builder(httpTransport, JSON_FACTORY, credential)
.setApplicationName(packageName)
.build()
Where the JSON is generated from the Developer Console, under Service Accounts. This is how I make my request:
androidPublisher.purchases().subscriptions().get(packageName, "valid-sku", "invalid-token").execute()
My subscription ID is valid but my token is invalid. I expect an error such as "invalid token" in the response. However, what I get is:
com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request
{
"code" : 400,
"errors" : [ {
"domain" : "global",
"message" : "Invalid Value",
"reason" : "invalid"
} ],
"message" : "Invalid Value"
}
Is that a generic error because of the invalid token or is it an authentication issue? If it an authentication issue, how do I solve it? If it is an invalid token issue, how am I supposed to know?
Some more information:
I get the same error when trying to make that call from the API Explorer as well (this time using a Client ID and API Key instead of Service Account).
I have not delegated domain-wide access to the service account. Do I have to for some reason?
I can successfully make other calls to the API such as inappproducts.list
From my experiences, if you have HTTP 400 error with Invalid Value then that purchase or subscription is FRAUD.
You can check out Order Id part of those purchases. Probably in the format of XXXXXXXXXXXX.XXXXXXXXXXXX which is wrong and should be GPA.XXXX.XXXXX.XXXXX.XXX
I don't really count the X char number. I just added to show the logic.
In my case the problem was that I was calling:
purchases.products.get
Instead of:
purchases.subscriptions.get
So, the reason that happened was just because the purchaseToken I was using was wrong.
I did not expect that to be the reason as I thought that in the case of an invalid token, I would receive a "token invalid" error (or something similar). As it turns out, the responses given by Google are pretty inconsistent (a 404 could also be given for an invalid token).
Scratched my head for a few hours, ALL my parameters were correct, and then well.. I realized that I was barking up the wrong tree (endpoint)
https://androidpublisher.googleapis.com/androidpublisher/v3/applications/{packageName}/purchases/subscriptions/{subscriptionId}/tokens/{token}
is not this
https://androidpublisher.googleapis.com/androidpublisher/v3/applications/{packageName}/purchases/products/{productId}/tokens/{token}
/purchases/subscriptions/.. vs /purchases/products/..
For all those who run into this problem, 99% of you need to publish the application for internal testers.
Follow this guide: https://support.google.com/googleplay/android-developer/answer/6062777?hl=en

S3: using a token in a header as a dumbed-down auth token

I want to grant read access to an S3 bucket if the request has a token attached to it. My AWS policy looks like this:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "PublicReadGetObject",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::some_bucket/*",
"Condition": {
"StringEquals": {
"aws:principaltype": "authorized_user"
}
}
}
]
}
And I'm trying to write links to it like so (it's in Java):
fileInfo.add('https://s3.amazonaws.com/some_bucket/' + file + '?aws:principaltype=authorized_user');
But I'm just getting a standard access denied message:
<Code>AccessDenied</Code>
What am I doing wrong? Or is there a better way to achieve a pseudo-public access setup?
You can't just make up condition keys.
They are not equivalent to headers -- the fact that some of them happen to correspond to headers is for convenience rather than necessity.
When I see questions like this, I become suspicious that the motivation is primarily one of wanting to avoid learning how to just do things the customary way, for whatever reason. If you're in an environment where it's easy enough to inject a request header, then why not just generate an Authorization header with a matching signsture?
There is no capability like you envision, in S3.
If you were to configure CloudFront in front of the bucket, with an origin access identity, it would be possible to generate a pre-signed URL with a wildcard, so one pre-signed URL could represent any or all of a bucket, e.g. allow https://example.com/cats/*.jpg which would allow access to JPEG file with an S3 object key prefix of cats/. CloudFront validates the wildcard signed URL, then fetches the object from S3 using its origin access identity. (S3 signed URLs do not support wildcards; CloudFront signed URLs do).
This seems like a straightforward solution.
Alternately, if your application has a web server, send the request to that server, and if the desired "token" matches, then sign a URL and return a 302 redirect, sending the browser to get the object from S3. I have a couple of systems that do this, one of which looks up the user's session cookie contents and compares it with object metadata, allowing the signed redirect to be generated if there's a match (it can allow users to access objects based on their user id, user class id, or permission set id in session matching what was stored with the object; if access is denied, it returns 403; if the object doesn't exist, it lies and still returns 403 so that the bucket can't be scanned, for security reasons). This -- with or without the session validation, but simply the presence of a token -- could also be implemented with a lambda function behind API gateway... which even supports tokens natively ("API keys.")

Google Drive service account returns 403 usageLimits

I'm trying to write an AppEngine app that writes a Google Document to Google Drive, puts in a specific set of folders and sets access rights. I have this working with the old DocsList API but since that just got deprecated I decided to update my code (and I had some additional functions to add anyway).
Problem I'm facing is this: When I use a service account and try to impersonate a specific user I get a 403 with usageLimits even though I have not used up any of my quota.
Here is the code I'm using:
GoogleCredential credentials = new GoogleCredential.Builder()
.setTransport(HTTP_TRANSPORT)
.setJsonFactory(JSON_FACTORY)
.setServiceAccountId("xxxxxxxxxxgserviceaccount.com")
.setServiceAccountScopes(DriveScopes.DRIVE)
.setServiceAccountPrivateKeyFromP12File(
new java.io.File("xxxx-privatekey.p12"))
.setServiceAccountUser("user#xxxxxx.org").build();
I than use these credentials to initiate my Drive object:
Drive d = Drive.builder(httpTransport, jsonFactory)
.setHttpRequestInitializer(credentials)
.setJsonHttpRequestInitializer(new JsonHttpRequestInitializer() {
#Override
public void initialize(JsonHttpRequest request) {
DriveRequest driveRequest = (DriveRequest) request;
driveRequest.setPrettyPrint(true);
}
}).setApplicationName("MYAPPNAME").build();
BTW: I've tried using new Drive(....) but that just won't work, no matter what I try. Keeps throwing errors that internal methods are not found!
Back to this issue:
When I than use 'd' to call something like .files().get("SOMEFILEID").execute() I get a 403
{ "code" : 403,
"errors" : [ {
"domain" : "usageLimits",
"message" : "Daily Limit Exceeded. Please sign up",
"reason" : "dailyLimitExceededUnreg",
"extendedHelp" : "https://code.google.com/apis/console"
} ],
"message" : "Daily Limit Exceeded. Please sign up"
}
I can't figure out why this doesn't work. I've look online all day but can't find a suitable answer. Some help is very much appreciated.
So pinoyyid's answer did help, although it wasn't the definitive answer.
I ended up solving it like this:
I took my AppID (xxxxx.apps.googleusercontent.com) and added it to my CPanel https://www.google.com/a/cpanel/[[YOURDOMAIN]]/ManageOauthClients
with these scopes:
https://www.googleapis.com/auth/drive
https://www.googleapis.com/auth/drive.file
https://www.googleapis.com/auth/userinfo.email
https://www.googleapis.com/auth/userinfo.profile
After that I had could do an authenticated request to the Drive environment. Running into a stack of different issues now but the 403 got solved.
I usually get a 403 when the API call was missing the Authorization http header. Trace the http and look at the headers. The rationale for the "quota" message is that without an Auth header, you are anonymous, and the quota for anonymous use is zero.
You might also check that your app is registered for both of the Drive APIs, as I've heard that that can cause the same problem.
On the internal method issue, that sounds like you're using incompatible library versions. What worked best for me was to delete all of the libraries I had downloaded with the sample code, then use the Google Eclipse plugin to "Google/Add Google APIs..." to download all of the latest versions.

Categories

Resources