I'm beginner to aws cdk, I have implemented S3 lifecycle management using AWS CDK to delete bucket (including files) after 10 days of bucket creation.
Cdk implementation as below:
Builder builder = BucketProps.builder().encryption(BucketEncryption.KMS_MANAGED).removalPolicy(RemovalPolicy.DESTROY);
final String bucketName = ApplicationProperties.INSTANCE.getBatchS3Name();
final List<LifecycleRule> lifecycleRules = new ArrayList();
final LifecycleRule rule = new LifecycleRule.Builder().expiration(Duration.days(10)).build();
lifecycleRules.add(rule);
builder.lifecycleRules(lifecycleRules);
this.bucket = new Bucket(stack, "BatchWorkBucket", builder.build());
But s3 object is not getting deleted after 10 days.
It is getting expired as shown in attached image.
We don't have option to setup "Delete expired delete markers or incomplete multipart uploads" or other options in cdk api (https://docs.aws.amazon.com/cdk/api/latest/docs/#aws-cdk_aws-s3.LifecycleRule.html#transitions).
Could you please help me on this subject, how to delete s3 object using aws cdk.
I believe this is working as expected. The objects marked as expired will eventually be deleted, but not exactly after the 10 days, like in your case. As per documentation:
When an object reaches the end of its lifetime based on its lifecycle policy, Amazon S3 queues it for removal and removes it asynchronously. There might be a delay between the expiration date and the date at which Amazon S3 removes an object.
Related
I'm trying to implement multipart upload in Java, following this sample: https://docs.aws.amazon.com/AmazonS3/latest/dev/llJavaUploadFile.html
But my actual task is a bit more complicated: I need to support resuming in case application was shut down during uploading. Also, I can't use TransferManager - I need to use low-level API for particular reason.
The code there is pretty straight-forward, but the problem comes with List<PartETag> partETags part. When finalizing resumed upload, I need to have this collection, previously filled during the upload process. And, obviously, if I'm trying to finalize upload after application restart, I don't have this collection anymore.
So the question is: how do I finalize resumed upload? Is it possible to obtain List<PartETag> partETags from the server using some API? What I have is only a MultipartUpload object.
Get the list of multipart uploads in progress
MultipartUploadListing multipartUploadListing =
s3Client.listMultipartUploads(new ListMultipartUploadsRequest(bucketName));
## for uploadId and keyName
Get the list of parts for each uploadId and key
PartsListing partsListing =
s3Client.listParts(new ListPartsRequest(bucketName, key, uploadId));
Get the List of part summary
List<PartSummary> parts = partsListing.getParts();
From PartSummary getETag() and getPartNumber()
for(PartSummary part: parts)
{
part.getETag();
part.getPartNumber();
}
Amazon S3 SDK Package
AmazonS3 client
I'm trying to manage my ProfitBricks S3 Object Storage Bucket using java, I want to do the basics (add, remove, list) operations but all I have found on internet is to connect to a AWS, Google or IBM Object Storage.
I have tried to use one implementation of those but I don't find how to provide my provider Endpoint.
I have achieved this using the jets3t library and I have set the endpoint property using a Jets3tProperties object.
Jets3tProperties props = new Jets3tProperties();
props.setProperty("s3service.disable-dns-buckets", String.valueOf(true));
props.setProperty("s3service.s3-endpoint", PB_ENDPOINT);
props.setProperty("s3service.s3-endpoint-http-port", PB_ENDPOINT_HTTP_PORT);
props.setProperty("s3service.s3-endpoint-https-port", PB_ENDPOINT_HTTPS_PORT);
props.setProperty("s3service.https-only", String.valueOf(false));
AWSCredentials creds = new AWSCredentials(PB_ACCESS_KEY, PB_SECRET_KEY);
RestS3Service s3Service = new RestS3Service(creds, null, null, props);
And then with the s3Service object I was able to manage my ProfitBricks S3 Object Storage Bucket using java.
Here is a link to the documentation for java 3 sdk version 1. Does version 2.0 has something similar or they removed such option?
Yes! It is possible in AWS SDK v2 to execute S3 operations on regions other than the one configured in the client.
In order to do this, set useArnRegionEnabled to true on the client.
An example of this using Scala is:
val s3Configuration = S3Configuration.builder.useArnRegionEnabled(true).build
val client = S3Client
.builder
.credentialsProvider({$foo})
.region(Region.EU_WEST_1)
.overrideConfiguration({$foo})
.serviceConfiguration(s3Configuration)
.build
Here is the documentation: https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/s3/S3Configuration.Builder.html#useArnRegionEnabled-java.lang.Boolean-
Not supported per here
In version 1.x, services such as Amazon S3, Amazon SNS, and Amazon SQS allowed access to resources across Region boundaries. This is no longer allowed in version 2.x using the same client. If you need to access a resource in a different region, you must create a client in that region and retrieve the resource using the appropriate client.
This works for me when using java AWS SDK 2.16.98 and only requires the name of the bucket rather than the full arn.
private S3Client defaultClient;
private S3Client bucketSpecificClient;
private String bucketName = "my-bucket-in-some-region";
// this client seems to be able to look up the location of buckets from any region
defaultClient = S3Client.builder().endpointOverride(URI.create("https://s3.us-east-1.amazonaws.com")).region(Region.US_EAST_1).build();
public S3Client getClient() {
if (bucketSpecificClient == null) {
String bucketLocation = defaultClient.getBucketLocation(builder -> builder.bucket(this.bucketName)).locationConstraintAsString();
Region region = bucketLocation.trim().equals("") ? Region.US_EAST_1 : Region.of(bucketLocation);
bucketSpecificClient = S3Client.builder().region(region).build();
}
return bucketSpecificClient;
}
Now you can use bucketSpecificClient to perform operations on objects in the bucket my-bucket-in-some-region
Background:
I have two accounts A and B. Account A owns a bucket my-bucket. I have given account B access to this bucket -- account B can read objects from and write objects to this bucket. This is working as expected.
However, account A can only read those objects in my-bucket that it has written on its own. Although it can list even those objects that account B has written, it cannot read those.
Below is what I see when I try to download all objects from my-bucket using AWS CLI with AWS configuration of account A.
download: s3://my-bucket/PN1492646400000.csv to tp/PN1492646400000.csv
download: s3://my-bucket/PN1491264000000.csv to tp/PN1491264000000.csv
download: s3://my-bucket/PN1493942400000.csv to tp/PN1493942400000.csv
download failed: s3://my-bucket/PN1503346865232.csv to tp/PN1503346865232.csv An error occurred (AccessDenied) when calling the GetObject operation: Access Denied
download: s3://my-bucket/PN1495389670000.csv to tp/PN1495389670000.csv
download: s3://my-bucket/PN1496685403000.csv to tp/PN1496685403000.csv
download: s3://my-bucket/PN1497945130000.csv to tp/PN1497945130000.csv
download: s3://my-bucket/PN1500508800000.csv to tp/PN1500508800000.csv
As one can see, I could download all files, but PN1503346865232.csv (this was written by account B using a java method putObject).
What I tried so far:
I have looked into the following two questions:
Amazon S3 file 'Access Denied' exception in Cross-Account: One of the comment asks to do a putObject with acl, but does not specify what acl.
S3: User cannot access object in his own s3 bucket if created by another user: This talks about stopping account B from putting objects into my-bucket without giving ownership access. Does just putting this constraint help me get full access?
This is how I tried to put the ACL while putting the object in the java code.
AccessControlList acl = new AccessControlList();
acl.grantPermission(new CanonicalGrantee(S3_BUCKET_OWNER_ID), Permission.FullControl);
PutObjectRequest putObjectRequest = new PutObjectRequest(S3_BUCKET, remoteCleanedPrismFilename, fileStream, null)
.withAccessControlList(acl);
s3Client.putObject(putObjectRequest);
It throws exception saying: com.amazonaws.services.s3.model.AmazonS3Exception: Invalid id
Questions:
I wonder how am I supposed to get this ID. Is it not the aws account ID i.e. a 10-12 digit number?
Even if I find this ID, will giving this ACL same as bucket-owner-full-
control?
This post got so many views, but not a single answer!
To help others, I am posting some workarounds that I found from my research so far. After these workarounds, account A is able to access the objects created by account B.
In the java code that is running in account B, I explicitly set the ACL on the object that is to be created.
AccessControlList acl = new AccessControlList();
acl.grantPermission(new CanonicalGrantee(S3_BUCKET_OWNER_ID), Permission.FullControl);
PutObjectRequest putObjectRequest = new PutObjectRequest(S3_BUCKET, remoteCleanedPrismFilename, fileStream, null)
.withAccessControlList(acl);
s3Client.putObject(putObjectRequest);
Here S3_BUCKET_OWNER_ID is the canonical ID of account A. Please note that it is not the AWS account ID that we know and I do not know a better way, to find this out, than the following
aws s3api get-object-acl --bucket my-bucket --key PN1491264000000.csv --output text --profile accountA
This is still not an elegant solution according to me and I believe something better exists. I will edit this answer once I find something better.
Edit:
Canonical ID of the bucket owner can be found more elegantly as follows:
String s3BucketOwnerId = s3Client.getBucketAcl(S3_BUCKET).getOwner().getId();
My question is somewhat similar to this SO but I could not find the answer there.
I using the following code snippet to create instance.
Code:
public void test(String accessId, String accessKey){
credentials = new BasicAWSCredentials(accessId, accessKey);
amazonEC2Client = new AmazonEC2Client(credentials);
amazonEC2Client.setEndpoint("ec2.ap-northeast-1.amazonaws.com");
RunInstancesRequest runInstancesRequest = new RunInstancesRequest();
runInstancesRequest.withImageId(imageId)
.withInstanceType("t2.micro")
.withMinCount(2)
.withMaxCount(2);
}
When I ran the above snippet I got the following error.
We currently do not have sufficient t2.micro capacity in zones with support for'gp2'volumes.
Our system will be working on provisioning additional capacity.
(Service: AmazonEC2; Status Code: 500; Error Code:
InsufficientInstanceCapacity; Request ID: c1996284-c208-446a-9f4c-301d8900e503)
After googling I found that AWS does not have that amount of t2.micro instance at that time and it recommended to create instance in different availability zone.
But in the code I have not provided any availability zone but the instances are being created in the ap-northeast-1a availability zone and throws above error and terminates.
When I created in through AWS web console it automatically created an instance in ap-northeast-1c without throwing any error.
Is there any way to create an instance in any available zone of that particular region where ever that instance is available programmatically ?.
My understanding is there is no or less t2.micro instance available in ap-northeast-1 region when I called the API.
Is there any API to check is Instance are available in a particular region ?.
AWS do not provide an API to check what available instances there are in a particular region.
You could try one AZ, if it fails try the next AZ, and so on.
Alternatively you could use an AutoScaling group that is allowed to launch instances in different AZs. It will automatically do the above for you.