I've been playing with Amazon S3 presigned URLs all night attempting to PUT a file. I generate the presigned URL in java code.
AWSCredentials credentials = new BasicAWSCredentials( accessKey, secretKey );
client = new AmazonS3Client( credentials );
GeneratePresignedUrlRequest request = new GeneratePresignedUrlRequest( bucketName, "myfilename", HttpMethod.PUT);
request.setExpiration( new Date( System.currentTimeMillis() + (120 * 60 * 1000) ));
return client.generatePresignedUrl( request ).toString();
I then want to use the generated, presigned URL to PUT a file using curl.
curl -v -H "content-type:image/jpg" -T mypicture.jpg https://mybucket.s3.amazonaws.com/myfilename?Expires=1334126943&AWSAccessKeyId=<accessKey>&Signature=<generatedSignature>
I assumed that, like a GET, this would work on a bucket which is not public (that's the point of presigned, right?) Well, I got access denied on every attempt. Finally out of frustration I changed the permission of the bucket to allow EVERYONE to write. Of course, then the presigned URL worked. I quickly removed the EVERYONE permission from the bucket. Now, I don't have permission to delete the item that was uploaded into my bucket by my own self-pre-signed URL. I see now that I probably should have put a x-amz-acl header on what I uploaded. I suspect I'll create several more undelete-able objects before I get that right.
This leads to a few questions:
How can I upload with curl using PUT and a generated presigned URL?
How can I delete the uploaded file and the bucket I created to test it with?
The end goal is that a mobile phone will use this presigned URL to PUT images. I'm trying to get it going in curl as a proof of concept.
Update: I asked a question on the amazon forums. If an answer is provided there I'll put it as an answer here.
This is indeed a bit puzzling, I consider it to be a bug in the AWS SDK for Java (see below) - but first and foremost, the following curl command will upload your file as such (assuming an updated pre-signed URL of course):
curl -v -T mypicture.jpg https://mybucket.s3.amazonaws.com/myfilename?Expires=1334126943&AWSAccessKeyId=<accessKey>&Signature=<generatedSignature>
That is, I've excluded the Content type header, which yields application/octet-stream (or binary/octet-stream) as a result, which is obviously not desired; thus, further digging had been order.
Background / Analysis
Pre-signed URLs for PUT (and DELETE as well as HEAD) requests to Amazon S3 are known to work in principle, not the least evidenced in related questions on this site (see e.g. my answer to Upload to s3 with curl using pre-signed URL (getting 403)).
The facilitated Query String Request Authentication Alternative is documented to use the following pseudo-grammar that illustrates the query string request authentication method:
StringToSign = HTTP-VERB + "\n" +
Content-MD5 + "\n" +
Content-Type + "\n" +
Expires + "\n" +
CanonicalizedAmzHeaders +
CanonicalizedResource;
It does include the Content-Type header, and (as you already discovered) this has been the missing piece in some documented cases, see e.g. the AWS team response to GetPreSignedURL with PUT request, yielding a working pre-signed URL once added.
This is easy to achieve with the AWS SDK for .NET indeed, which provides the convenience method GetPreSignedUrlRequest.WithContentType to do just that:
Sets the ContentType property for this request. This property defaults
to "binary/octet-stream", but if you require something else you can
set this property.
Accordingly, extending the respective sample Upload an Object Using Pre-Signed URL - AWS SDK for .NET as follows yields a working pre-signed URL with content type, that can be uploaded via curl as expected (i.e. exactly as you attempted to):
// ...
GetPreSignedUrlRequest request = new GetPreSignedUrlRequest();
// ...
request.WithContentType("image/jpg");
// ...
Now, one would like to extend the semantically identical sample Upload an Object Using Pre-Signed URL - AWS SDK for Java in a similar fashion, but (as you've discovered already as well), there is no dedicated method to achieve this. This might just be a lacking convenience method though and could be achievable via addRequestParameter() or setResponseHeaders() eventually, e.g.:
// ...
request.setExpiration( new Date( System.currentTimeMillis() + (120 * 60 * 1000) ));
request.addRequestParameter("content-type", "image/jpg");
return client.generatePresignedUrl( request ).toString();
// ...
However, both method's documentation suggests other purposes, and it doesn't work indeed, i.e. they always yield the identical signature, no matter which content type is set like so (if any).
Debugging further into the SDKs reveals, that both provide a semantically similar core method to calculate the query string authentication according to the pseudo-grammar referenced above, see buildSigningString() for .NET and makeS3CanonicalString() for Java.
But the respective code in the Java version to Add all interesting headers to a list, then sort them, where "Interesting" is defined as Content-MD5, Content-Type, Date, and x-amz- is never executed in fact, because there is indeed no method to provide these headers somehow, which are only available for class DefaultRequest and not class GeneratePresignedUrlRequest used to initialize the former, which is used as input for calculating the signature in turn, see protected method createRequest().
Interestingly/Notably, the two methods to calculate the query string authentication in .NET vs. Java compose their input from an almost inverse combination of header vs. parameter sources on the call stack, which could hint on the cause of the Java bug, but obviously that might as well be just difficult to decipher, i.e. the internal architecture could differ significantly of course.
Preliminary Conclusion
There are two angles to this:
The AWS SDK for Java is definitely lacking the convenience method for setting the content type, which might be a comparatively rare, but nonetheless obvious use case accounted for in other AWS SDKs accordingly - this is surprising, given its widespread use in AWS related backend services.
Regardless, there seems to be something fishy with the way the Query String Request Authentication is implemented in comparison to the .NET version for example - again this is surprising, given it is a core functionality, however, this is still within the S3 model/namespace and thus might only be required by the respective uses cases above.
In conclusion, the only reasonable way to resolve this would be an updated SDK, so a bug report is in order - obviously one could as well duplicate/extend the SDK functionality to account for this special case separately (ideally in a way allowing to submit a pull request for the aws-sdk-for-java project), but getting this right in a compatible and maintainable way seems to be a bit tricky, thus is likely best done by the SDK maintainers themselves.
Ran into this problem as well. We're already tracking when the file is uploaded on the backend, so our work around was to set the content type after the client uploads the file using the Rails app with a call to copy_from.
Related
I am trying to implement Plaid using the sample code provided on the Java Quickstart [sandbox] and am getting issues when I show the Plaid Dialog (javascript). I am able to successfully get a link_token, but I'm never able to show the dialog. It spins for a brief second, then shows me:
oauth uri does not contain a valid oauth_state_id query parameter. Request ID: DBoT92FCo8AORay
I have tried this with an empty redirectUri, as well as "http://localhost:8080/plaid_test.html", which is registered in my developer account.
I am a bit stuck and hoping someone can direct me in the right direction. I've tested with both versions 9.10.0 and the latest (11.9.0).
Curiously, I am able to get the Java Quickstart working directly, but ONLY if I leave the .env PLAID_REDIRECT_URI blank. If I put localhost in there, it fails when trying to get the link token.
Does anyone have any suggestions on how to overcome this setup issue?
Thank you!
I got this error (oauth uri does not contain a valid oauth_state_id query parameter) while creating a new test application in Plaid's Sandbox environment.
Important note: My application does not use OAuth.
The problem turned out to be, in my configuration parameters being passed to usePlaidLink, I was including a receivedRedirectUri key-value pair. Removing that key-value pair entirely resolved the issue for me.
In other words, my React component looked something like:
import { usePlaidLink } from 'react-plaid-link';
function PlaidLink(props) {
const onSuccess = React.useCallback((public_token, metadata) => {
// ...
});
const config = {
token: props.linkToken,
receivedRedirectUri: window.location.href,
onSuccess,
};
const { open, ready } = usePlaidLink(config);
// ...
}
Removing the line with the receivedRedirectUri was the solution for me, getting me past the oauth uri does not contain a valid oauth_state_id query parameter error, and getting the Plaid Link UI to appear in my app successfully.
This Plaid article, which has a number of mentions of "OAuth state ID" (as mentioned in the error message), helped point me toward this solution.
The issue may be the location you are trying to use -- unless you have manually modified the ports or other code used by the Quickstart, you should use http://localhost:3000/ as the PLAID_REDIRECT_URI (make sure to add this to your Dashboard as an allowed redirect URI). When I tried this just now on the Java quickstart (non-Docker version) it worked fine.
I am trying to retrieve a list of projects from the OpenStack API, and would like to use pagination in order to retrieve n projects at a time.
In the OpenStack documentation, it states that I can append "/?limit=n" to the URL and up to n results will be fetched accordingly.
However, when executing the GET request to the URL as follows:
https://identity-3.eu-de-1.cloud.sap/v3/auth/projects/?limit=1
I still get ALL projects. I can't seem to understand what I am missing.
NOTE: the request itself works and returns results as needed, but simply ignores the limit parameter (this is not an authentication issue).
I think it does not all OpenStack API provide limit parameter
In keystone API doc, there is no limit parameter in Request parameter descriptions for /v3/auth/projects API
keystone-project-API-doc
Other services like cinder volume list, it provides limit parameter in doc
cinder-volume-API-doc
I am not being able to access a video stored in my Amazon s3 bucket through my Cloudfront distribution.
I have set the distribution to require signed URLs or signed cookies(generated by "self" user) and configured everything according to the tutorials on Amazon website (generated the key pair for the "self" user, transformed it to .der format using openssl, etc).
Then I followed the steps in this tutorial: http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/CFPrivateDistJavaDevelopment.html
I was then able to generate a "canned" signed URL to access the content and it worked, I pasted the URL in chrome and it started playing the video.
Here is the code:
String signedUrlCanned = CloudFrontService.signUrlCanned(
resourceURL,
certificateId,
derPrivateKey,
ServiceUtils.parseIso8601Date("2018-11-14T22:20:00.000Z")
);
But then I decided to create a custom policy signed URL, following the steps in the tutorial and even using the exact same values for the parameters as I used for the canned URL, and it is not working... Here is the code:
String policy = CloudFrontService.buildPolicyForSignedUrl(
null,
ServiceUtils.parseIso8601Date("2018-11-14T22:20:00.000Z"),
null,
null
);
String signedUrl = CloudFrontService.signUrl(
resourceURL,
certificateId,
derPrivateKey,
policy
);
Keep in mind that "resourceURL", "certificateId" and "derPrivateKey" are the same parameters used for the canned URL, and they worked in that case.
The parameters remain unchanged the entire time since I checked that at debug time, and the 3 Strings(signedUrlCanned, policy, signedUrl) share the same scope(they are generated consecutively inside the same method) and there is no other code in between that could change the values of the parameters.
Here is the resulting URL for the custom policy signed URL:
http://d1eanqhguto8v8.cloudfront.net/Fox/Dexter.mp4?Policy=eyJTdGF0ZW1lbnQiOiBbeyJSZXNvdXJjZSI6IioiLCJDb25kaXRpb24iOnsiRGF0ZUxlc3NUaGFuIjp7IkFXUzpFcG9jaFRpbWUiOjE1NDIyMzQwMDB9LCJJcEFkZHJlc3MiOnsiQVdTOlNvdXJjZUlwIjoiMC4wLjAuMC8wIn19fV19&Signature=D9sHG6i9GTZRUGwKZYhmz3xKOQsjEWHJPQTCNywzdX9r~F0yEb58ljBAqRyXbHNgmmGuRppS6s0VkiRcyYi8q~DRDoXLtnp~MBbnnBKbF0Qy3xcx59LF6mXC6lXSou7jqz68y~d0VOoLvnFQl~OR5eSTWRHUO7X42~k3qdIiRH~eqzBwAaV9XnbJcLQ2DEDiW13~sfZJXnRsO6hENSk-aGcWeoF80KoccJ2-nFt0ZpndIFI3V~YXRv~Z3VKKS2ek1MW8SE2xdsOdqXAMkJC2X2maQn~MBzczXBuqEO4qKt42FmZI496TckMWzG-pVs~w-EjIWE2EEOzuXxopav8q~Q__&Key-Pair-Id=APKAIS7ACQJH7KT4YWLQ
You can access it and see the error message, it says that the Key-Pair-Id is missing, which is clearly not true since you can see it's there.
Any suggestions?
Edit: I just noticed that the error actually is a 403 "Access Denied" from Cloudfront. When I open the chrome console, under the Network tab I can see 2 GETs, one for the video that throws the 403 error and the other one tries to get something called "favicon.ico" that i've no idea what could be, and that one throws the "Key-Pair-Id is missing" error. One of my coworkers could access the video and still got the favicon.ico error, so I guess that error doesn't matter since it is not interfering with the video download.
From a Web API, I receive the following information about an Amazon S3 Bucket I am allowed to upload a File to:
s3_bucket (the Bucket name)
s3_key (the Bucket key)
s3_policy (the Bucket policy)
s3_signature the Bucket signature)
Because I am not the owner of the Bucket, I am provided with the s3_policy and s3_signature values, which, according to the AWS Upload Examples, can be used to authenticate a Put request to a Bucket.
However, in AWS's official Java SDK I'm using, I can't seem to find a way to perform this authentication. My code:
PutObjectRequest putObjectRequest = new PutObjectRequest(s3_bucket, s3_key, fileToUpload);
s3Client.putObject(putObjectRequest);
I do understand that I need to use the s3_signature and s3_policy I'm given at some point, but how do I do so to authenticate my PutObjectRequest?
Thanks in advance,
CrushedPixel
I don't think you're going to use the SDK for this operation. It's possible that the SDK will do what you need at this step, but it seems unlikely, since the SDK would typically take the access key and secret as arguments, and generate the signature, rather than accepting the signature as an argument.
What you describe is an upload policy document, not a bucket policy. That policy, the signature, and your file, would all go into an HTTP POST (not PUT) request of type multipart/form-data -- a form post -- as shown in the documentation page you cited. All you should need is an HTTP user agent.
You'd also need to craft the rest of the form, including all of the other fields in the policy, which you should be able to access by base64-decoding it.
The form also requires the AWSAccessKeyId, which looks something like "AKIAWTFBBQEXAMPLE", which is -- maybe -- what you are calling the "s3_key," although in S3 terminology, the "(object) key" refers to the path and filename.
This seems like an odd set of parameters to receive from a web API, particularly if they are expecting you to generate the form yourself.
I have been trying for months to get access to a certain api (which has almost no documentation) to work using signpost. The api has oauth 2.0 authentication. The problem is that I have never used oauth before. But I have spent a long time reseaching so I think I have a functional understanding of how it works. I thought that using the handy singpost api it wouldn't be too much trouble to hack through it, but alas I have encountered a wall. The api docs are here:
https://btcjam.com/faq/api
It gives three URLs that are needed for the oauth authentication, which I am writing as java here for consistency with some code below:
String Authorization= "https://btcjam.com/oauth/authorize";
String Token ="https://btcjam.com/oauth/token";
String Applications = "https://btcjam.com/oauth/applications";
I have an application with a name, key, and secret. I also have set my callback URL to be the localhost, i.e.
http://localhost:3000/users/auth/btcjam/callback.
Now, as I am reading the signpost docs, it tells me that in order to request an access token, I need to do something like the following:
OAuthProvider provider = new DefaultOAuthProvider(
REQUEST_TOKEN_ENDPOINT_URL, ACCESS_TOKEN_ENDPOINT_URL,
AUTHORIZE_WEBSITE_URL);
String url = provider.retrieveRequestToken(consumer, CALLBACK_URL);
However, I am unsure exactly what to put for the URL's in these various spots, and I am getting errors. The problem is that The names of the URLs required above do not correspond to the URLs given. The "authorization" and "callback" URLs seem to match up nicely, but I am not sure how the URLs "REQUEST_TOKEN_ENDPOINT_URL" and "ACCESS_TOKEN_ENDPOINT_URL" required in the signpost docs correspond to the URLs given by the api docs on the serverI am trying to access. Of course, there are only two possible permutations, but when I try them both I get two different errors:
"Authorization failed (server replied with a 401). This can happen if the consumer key was not correct or the signatures did not match."
"Communication with the service provider failed: URLDecoder: Illegal hex characters in escape (%) pattern - For input string: " 1""
Could someone please help explain what might be going on here? Am I very close to getting this to work or do I have to take a bunch of steps back?
Any help is much appreciated.
Thanks,
Paul