We are trying to use aws S3 for storing files. We created a simple REST API in JAVA to upload and retrieve a file.
Clients requesting to update files use our REST API's which provide a presigned url to either PUT/GET the file. We are using AWS SDK for java to generate the pre signed urls.
We need to add some custom metadata to the files when they are being updated on S3. As we dont control the upload to S3 itself, is there a way we can add this information while we are generating the pre signed url? It wont be good to have the clients to provide this information as a part of their request headers.
We stumbled upon the same issue today and were trying to use
// does not work
request.putCustomRequestHeader(Headers.S3_USER_METADATA_PREFIX + "foo", "bar");
which unfortunately does not really work, it adds the metadata but the caller of the presigned url has to still provide the metadata using request headers which is something the client should not have to do.
Finally we found that using GeneratePresignedUrlRequest#addRequestParameter does the job marvellously:
GeneratePresignedUrlRequest request = new GeneratePresignedUrlRequest("bucket", "yourFile.ending");
request.addRequestParameter(Headers.S3_USER_METADATA_PREFIX + "foo", "bar");
// import com.amazonaws.services.s3.Headers; needed
The presigned url then looks something like
https://bucket.s3.region.amazonaws.com/yourFile.ending?x-amz-meta-foo=bar&X-Amz-Security-Token=...
The metadata can be clearly seen in the url, using Postman to PUT to that file using upload-file in the body creates the file with the correct metadata in the bucket and it is not possible for the client to change the meta-data because that would make the signature no longer match the request.
The only not-so-pretty part about this is having to specify the internal aws header prefix for user metadata.
Related
We have some Java services that upload files to an S3 compatible storage and generate Presigned URLs.
Other Java services receive such URLs and work on the files. We need the content length of this file, without loading the whole object from the body. Is this possible? And if yes, how?
We upload via
com.amazonaws.services.s3.AmazonS3.putObject()
and create URLs with
com.amazonaws.services.s3.AmazonS3.generatePresignedUrl()
When I check the Http Headers of a file using my S3 Browser, I can see the correct Content-Length entry.
But in our services we use
OkHttpClient.newCall(
new Builder().get()
.url(url).build()
).execute();
And in those Response objects there is no Content-Length.
If the response is streamed (HTTP/1.1 chunked) or compressed then it won't have a content length to read from. You can probably use a HEAD request to get the headers without the body and check first.
https://stackoverflow.com/a/38673237/1542667
I am currently storing text on the cloud using
bucket.create(blobName, "hello".getBytes());
and blob name looks like this 1/1/1674/2020-06-02/9998-2-202062
The requirement is to generate pdf file from the data of multiple blobs. We can do this on our backend in java by getting the content from blob path like this
Blob blob=bucket.get("1/1/1674/2020-06-02/9998-2-202062");
new String(blob.getContent())
But we don't want to increase the load on the server by downloading the content first on the server and then send it to the front-end. So we are sending the signURL on the front-end so we can get the content at the front-end(angular 8) by using that URL and create the pdf. We are creating singurl this way.
Blob blob=bucket.get("1/1/1674/2020-06-02/9998-2-202062");
URL url=blob.signUrl(1, TimeUnit.HOURS,SignUrlOption.signWith(
ServiceAccountCredentials.fromStream(new FileInputStream(jsonfilePath))));
But the issue on front-end is whenever we click the URL on any browser it downloads the file with the blob-name
9998-2-202062 instead of just showing the content.
Is there any way so we can read the data from that cloud URL instead of downloading the content by chrome automatically?
And we do not change the browsers setting because it can not be done on the client machine?
The signUrl method only creates a link to the specified resource, you have to implement a way to read it, join the rest of the files you want to add and process them to create your PDF on the client side.
That said, for example you can create a javascript file and use the signed URL in a fetch function to gather all the text from the blobs, I found this example on another stackoverflow answer that might help you:
Promise.all([
fetch(signedURL1).then(x => x.text()),
fetch(signedURL2).then(x => x.text())
]).then(([sampleResp, sample2Resp]) => {
console.log(sampleResp);
console.log(sample2Resp);
});
Replace the signedURL1 and signedURL2 for the actual signed URLs that you created.
Once that you have the content of your files, you just have to create the PDF, I found a library named jsPDF that could be of your interest.
My requirement is that, I want to upload a file into AWS S3 bucket through Spring REST api. This api should first generate a Cloudfront url and upload the file using the url generated and finally in the response I should get the complete filepath : cloudfront-url/filename.
Please let me know the steps and configuration required to achieve this.
Thank you.
If the server uploads the file from client to s3, it will create unnecessary overhead on the server. Better approach is the return pre-signed URL to client (browser) and it directly uploads to S3.
See details on how to do it here.
If that bucket is configured for cloudfront then the cloudfront url can be used directly.
See details here for setting up cloudfront for s3.
I am using aws sdk, in Java, to generate signed object urls from a given bucket. These objects are images for which I want to generated a signed url using following code.
URL url = amazonS3Client.generatePresignedUrl(generatePresignedUrlRequest);
Following is an example of generated url
In the urls there is AWS3AccessKeyId and signature that will be visible to end users and developers. Is there a chance for someone to generate PUT, DELTE, etc operations on objects using the information in the url. Is it safe to provide these urls to users who I don't know ?
Background
I like to share images and other object urls with people, who will use them in their websites. For example,a url to display an image in a website, or url from where user can download a file.
No, they will not be able to use these URLs to do anything other than what you allow. So if you generate a pre-signed url to get a particular S3 object, they will ONLY be able to use it to get that S3 object. They can't reverse-engineer the url to give themselves access to other buckets or objects.
I wanted to know whether it is possible to perform S3 multipart upload using pre-signed Url using AWS java SDKs.
Unfortunately i am not able to get any clear answer for it, not sure whether it is supported or not.
Thanks in advance !!!
Have you had a chance to examine the documentation? You will want to presign a create multipart upload request. A related issue includes this comment https://github.com/aws/aws-sdk-java-v2/issues/203#issuecomment-663631708 which shows code from the relevant integration test.