I want to upload an object to Amazon versioned bucket (using Java AWS SDK) and set a custom version to this object (goal is to set the same version to all objects, uploaded at once)
PutObjectResult por = amazonS3Client.putObject(...);
por.setVersionId("custom_version");
So, is it the right way to set a version to the uploaded object?
Does this code lead to 2 separate requests to Amazon?
What if Internet is broken while por.setVersionId(..) is being called?
Why por.setVersionId(..) does not throw an exception such as SdkClientException if this method really is trying to set a version ID on Amazon server?
setVersionId would be something the SDK library itself uses to populate the versionId returned by the service when the object is created, so that you can retrieve it if you want to know what it is.
Version IDs in S3 are system-generated opaque strings that uniquely identify a specific version of an object. You can't assign them.
The documentation uses some unfortunate examples like "111111" and "222222," which do not resemble real version-ids. There's a better example further down the page, where you'll find this:
Unique version IDs are randomly generated, Unicode, UTF-8 encoded, URL-ready, opaque strings that are at most 1024 bytes long. An example version ID is 3/L4kqtJlcpXroDTDmJ+rmSpXd3dIbrHY+MTRCxf3vjVBH40Nr8X8gdRQBpUMLUo.
Only Amazon S3 generates version IDs.
They cannot be edited.
You don't get an error here because all this method does is set the versionId inside the PutObjectResult object in local memory after the upload has finished. It succeeds, but serves no purpose.
To store user-defined metadata with objects, such as your release/version-id, you'd need to use object metadata (x-amz-meta-*) or the new object tagging feature in S3.
Related
Hi I want to copy an existing object to same path in AWS S3 and I am getting following exception
This copy request is illegal because it is trying to copy an object to itself without changing the object's metadata, storage class, website redirect location or encryption attributes
I am using Apache camel S3, how can i resolve this. After searching, i found there is a request header which we can use to replace the existing file but it is not working
// multiple other attempts also present, I am not sure which header will work
exchange.`in`.headers[AWS2S3Constants.METADATA] = mutableMapOf(
"x-amz-metadata-directive" to "REPLACE",
"x-amz-meta-directive" to "REPLACE",
"metadata-directive" to "REPLACE",
"MetadataDirective" to "REPLACE"
)
I have logged in the request.
Sending Request: DefaultSdkHttpFullRequest(httpMethod=PUT, protocol=https, host=, port=443, encodedPath=, headers=[amz-sdk-invocation-id, User-Agent, x-amz-copy-source, x-amz-meta-directive, x-amz-meta-metadata-directive, x-amz-meta-MetadataDirective, x-amz-meta-x-amz-metadata-directive], queryParameters=[])
But it is not working. how can i copy an existing object to same path without getting this error.
I end up changing the filename by suffixing it with a timestamp. Now I am no longer getting the error.
But I think, there should be some way to copy existing objects via API, since I am able to do same via aws-cli
Did you look at the copyObject operation in the Apache Camel AWS2 S3 component? https://camel.apache.org/components/3.17.x/aws2-s3-component.html#_s3_producer_operation_examples
There is an example related to headers needed to make the Copy Object operation works.
$sourceBucket = '*** Your Source Bucket Name ***';
$sourceKeyname = '*** Your Source Object Key ***';
$targetBucket = '*** Your Target Bucket Name ***';
$s3 = new S3Client([
'version' => 'latest',
'region' => 'us-east-1'
]);
// Copy an object.
$s3->copyObject([
'Bucket' => $targetBucket,
'Key' => "{$sourceKeyname}-copy",
'CopySource' => "{$sourceBucket}/{$sourceKeyname}",
]);
I am trying to upload a file in amazon S3 storage using java sdk, enabling a explicit expiration date for the object using objectMetadata. when I run the program it uploads to S3 and sets the expiration date to object metadata as expected but eventually it doesn't seem to be deleted after the expiration date. I am not sure where i am doing wrong. Below is the code snippet i used to set the object metadata.
PutObjectRequest putObjectRequest = new PutObjectRequest(bucketName, key, file);
ObjectMetadata objectMetadata = new ObjectMetadata();
objectMetadata.setHttpExpiresDate(new DateTime().plusDays(1).toDate());
putObjectRequest.setMetadata(objectMetadata);
return s3.putObject(putObjectRequest);
I have been going through some of the Amazon documentation https://docs.aws.amazon.com/AmazonS3/latest/dev/manage-lifecycle-using-java.html which says to set the BucketLifeCycle Configuration rule. I am not sure whether if i apply this rule will it be applied to all folders and objects under this bucket or only the objects i upload using this rule through my java program?
Please suggest, Thanks in advance !
Following the documentation you can’t directly set expiration date for particular object.
To solve this problem you can:
Define lifecycle rule for a bucket(remove bucket with objects after number of days)
Define lifecycle rule for bucket to remove objects with specific tag or prefix after numbers of days
To creating a rules use documentation:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/how-to-set-lifecycle-configuration-intro.html
My goal is to get the object metadata without downloading the object/blob using the java api(https://cloud.google.com/storage/docs/json_api/v1/objects#resource).
I am able to do it with JSON API but unable to find the equivalent for the java API.
storage.objects.get can return an object's data or metadata. For the Java client, one of the storage.get methods can be used to retrieve the metadata (as opposed to storage.reader which can read the data).
Once you have the result, you can get the MD5 from BlobInfo.getMd5 method.
I'm migrating my GAE app from the deprecated File API to Google Cloud Storage Client Library.
I used to persist the blobKey, but since there is partial support for it (as specified here) from now on I'll have to persist the object name.
Unfortunately the object name that comes from the GCS looks more or less like this
/gs/bucketname/819892hjd81dh19gf872g8211
as you can see, it also contains the bucket name
Here's the issue, every time I need to get the file for further processing (or to serve it in a servlet) I need to create an instance of GcsFileName(bucketName, objectName) which gives me something like
/bucketName/gs/bucketName/akahsdjahslagfasgfjkasd
which (of course) doesn't work.
so. my question is:
- how can I generate a GcsFileName form the objectName?
UPDATE
I tried using the objectName as BlobKey. But it just doesn't work :(
InputStream is = new BlobstoreInputStream(blobstoreService.createGsBlobKey("/gs/bucketName/akahsdjahslagfasgfjkasd"));
I got the usual answer
BlobstoreInputStream received an invalid blob key
How do I get the file using the ObjectName???
If you have persisted and retrieved e.g the string String objname worth e.g "/gs/bucketname/819892hjd81dh19gf872g8211", you could split it on "/" (String[] pieces = objname.split("/")) and use the pieces appropriately in the call to GcsFileName.
I'm hoping the answer to this question is quite simple, but I can't get it working after looking at the Azure Java API documentation.
I am trying to create an empty CloudBlockBlob, which will have blocks uploaded to it at a later point. I have successfully uploaded blocks before, when the blob is created upon the first block being uploaded, but I can't seem to get anything other than ("the specified blob does not exist") when I try to create a new blob without any data and then access it. I require this because in my service, a call is first made to create the new blob in Azure, and then later calls are used to upload blocks (at which point a check is made to see if the blob exists). Is it possible to create an empty blob in Azure, and upload data to it later? What have I missed?
I've not worked with Java SDK so I may be wrong but I tried creating an empty blob using C# code (storage client library 2.0) and if I upload an empty input stream an empty blob with zero byte size is created. I did something like the following:
CloudBlockBlob emptyBlob = blobContainer.GetBlockBlobReference("emptyblob.txt");
using (MemoryStream ms = new MemoryStream())
{
emptyBlob.UploadFromStream(ms);//Empty memory stream. Will create an empty blob.
}
I did look at Azure SDK for Java source code on Github here: https://github.com/WindowsAzure/azure-sdk-for-java/blob/master/microsoft-azure-api/src/main/java/com/microsoft/windowsazure/services/blob/client/CloudBlockBlob.java and found this "upload" function where you can specify an input stream. Try it out and see if it works for you.