This is my first time using amazon s3 and I want to store pdf files that I create using itext in java spring.
The code (hosted on ec2 instance) creates a pdf that I would like to store somewhere. I am exploring if amazon s3 can hold those files. Eventually I would like to retrieve it as well. Can this be done using itext and java spring? Any examples would be great.
To Upload Files to Amazon s3 You need to use putObject method of AmazonS3Client class like this:
AWSCredentials credentials = new BasicAWSCredentials(appId,appSecret);
AmazonS3 s3Client = new AmazonS3Client(credentials);
String bucketPath = "YOUR_BUCKET_NAME/FOLDER_INSIDE_BUCKET";
InputStream is = new FileInputStream("YOUR_PDF_FILE_PATH");
ObjectMetadata meta = new ObjectMetadata();
meta.setContentLength(is.available());
s3Client.putObject(new PutObjectRequest(bucketPath,"YOUR_FILE.pdf", is, meta).withCannedAcl(CannedAccessControlList.Private));
And to get file from S3, You need to generate a pre-signed URL to access private file from S3 or if your files are public then you can directly access your file by hitting link of file in your browser, The link for your file will be available in AWS S3 console.
Also we have specified CannedAccessControlList.Private in the above upload code which means we are making permission of file as private So we need to generate presigned URL to access file like this:
AWSCredentials credentials = new BasicAWSCredentials(appId,appSecret);
AmazonS3 s3Client = new AmazonS3Client(credentials);
GeneratePresignedUrlRequest generatePresignedUrlRequest = new GeneratePresignedUrlRequest("YOUR_BUCKET_NAME", "FOLDER_INSIDE_BUCKET/YOUR_FILE.pdf");
generatePresignedUrlRequest.setMethod(HttpMethod.GET);
Date expiration = new Date();
long milliSeconds = expiration.getTime();
milliSeconds += 1000 * 60 * 60; // Add 1 hour.
expiration.setTime(milliSeconds);
generatePresignedUrlRequest.setExpiration(expiration);
URL url = s3Client.generatePresignedUrl(generatePresignedUrlRequest);
String finalUrl = url.toString();
Related
I have a java application in which I would like to process around 10GB records of file and zip them to a single folder and upload to S3. Since the overall size is around 10GB I cannot add all the files in memory and then upload to S3, and hence I would need to create a zip file in S3 and update the contents of the zip file by partitioning my files. Is there any means by which I can update an existing zip file in S3 without downloading to my local folder?
You can use aws java sdk for it
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk</artifactId>
<version>1.11.398</version>
</dependency>
Create a amazon s3 client using following
BasicAWSCredentials credentials = new BasicAWSCredentials("access_key", "secret_key");
AmazonS3 amazonS3 = AmazonS3ClientBuilder.standard().withCredentials(new AWSStaticCredentialsProvider(credentials)).build();
Create a TransferManager and set the MultipartUploadThresholdoad.
Amazon S3 impose minimum part size of 5mb, So we are using 5mb here. You can increase the size as per your requirement.
TransferManager tm = TransferManagerBuilder.standard()
.withS3Client(amazonS3)
.withMultipartUploadThreshold((long) (5 * 1024 * 1024))
.build();
Set yours S3 bucket name where you want to upload and keyName will used to name the uploaded file. tm.upload will start the uploading process in background.
String bucketName = "my-bucket";
String keyName = "mydata.zip";
File file = new File("path_to_file/mydata.zip");
Upload upload = tm.upload(bucketName, keyName, file);
waitForCompletion is the blocking call and will return result once function completes its execution to upload file to s3.
try {
upload.waitForCompletion();
} catch (AmazonClientException e) {
// ...
}
I am generating a preSignedUrl and then uploading the file through that url.
The issue is that even if I enter the wrong access key or secret key I get the preSignedUrl, though if I try to upload using that url I get 400 error.
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>AuthorizationQueryParametersError</Code>
<Message>Query-string authentication version 4 requires the X-Amz-Algorithm, X-Amz-Credential, X-Amz-Signature, X-Amz-Date, X-Amz-SignedHeaders, and X-Amz-Expires parameters.</Message>
<RequestId>{requestId}</RequestId>
<HostId>{hostId}</HostId>
</Error>
Is there some way I get the error while generating the preSignedUrl so that I don't have to try and upload the file.
AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
.withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials("accessKey", "secretKey")))
.withRegion(clientRegion)
.build();
GeneratePresignedUrlRequest generatePresignedUrlRequest = new GeneratePresignedUrlRequest(bucketName, objectKey)
.withMethod(HttpMethod.PUT)
.withExpiration(expiration);
URL url = s3Client.generatePresignedUrl(generatePresignedUrlRequest);
Generating a pre-signed URL doesn't require an API call; it can be generated by the framework using the specified access key and secret.
The generated URL will be validated by S3 when the request is received, and will obviously only be accepted when valid credentials were used for generating it.
Bottom line: in order to validate your credentials you need to make an API request that actually performs a call to AWS. This can be pretty much any other method on your s3Client.
Let's start with this:
.withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials("accessKey", "secretKey")))
Static credentials go against AWS best practice. Instead, rely on credentials provided via environment variables or an execution role (when running on EC2, ECS, or Lambda).
The only way that you can verify that the credentials are valid is to try them. You could write a small dummy file, however this may cause problems for anything that is supposed to read that file, due to eventual consistency on S3.
There's also the problem that the expiration that you give the URL may not correspond to the lifetime of the credentials.
The best solution to all of these problems is to create a role that has access to PUT the files on S3, and has a duration consistent with your URL expiration (note that the maximum is 12 hours), then explicitly assume that role in order to construct the request:
final String assumedRoleArn = "arn:aws:iam::123456789012:role/Example";
final String sessionName = "example";
final String bucketName = "com-example-mybucket";
final String objectKey = "myfile.txt";
final int expirationSeconds = 12 * 3600;
final Date expiresAt = new Date(System.currentTimeMillis() + expirationSeconds * 1000);
AWSSecurityTokenService stsClient = AWSSecurityTokenServiceClientBuilder.defaultClient();
AWSCredentialsProvider credentialsProvider = new STSAssumeRoleSessionCredentialsProvider.Builder(assumedRoleArn, sessionName)
.withStsClient(stsClient)
.withRoleSessionDurationSeconds(expirationSeconds)
.build();
AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withCredentials(credentialsProvider).build();
URL presignedUrl = s3Client.generatePresignedUrl(bucketName, objectKey, expiresAt, HttpMethod.PUT);
I have a video which is of S3Object type.
I am trying to convert it to a Resource using the following code.
Resource resource = new InputStreamResource(video.getObjectContent());
I get the following error
"InputStream has already been read - do not use InputStreamResource if a stream needs to be read multiple times"
Is it possible to read an S3Object directly as a Resource?
How can I correct it?
How to load S3 bucket file from a private bucket:
//Make sure your credentials are in the aws credentials file or use .withCredentials()
AmazonS3 s3client = AmazonS3ClientBuilder
.standard()
.withRegion("us-east-1")
.build();
if(!s3client.doesBucketExistV2(adfsProperties.keystoreBucket())) {
throw new Exception.... //bucket not found
}
S3Object s3object = s3client.getObject("bucket-name", "file-name");
S3ObjectInputStream inputStream = s3object.getObjectContent();
Resource r = new InputStreamResource(inputStream);
Look like before passing into InputStreamResource you stream used by some method. Or better approach if your s3 url is publicly accessible, then try like that
#Autowired
ResourceLoader resourceLoader;
In method use as follows.
Resource resource = resourceLoader.getResource(<s3 url>)
I create data in the server (gae) and I want to store it in Blobstore. I saw many answers on how to do this giving a BlobStore URL to the client, but there is no client or HTTP request: it's just an asynchronous task.
Then I guess I should use createUploadUrl(), and instead of giving this URL to a client, from my code HTTP Post my data to it via URL Fetch. This looks weird, isn't there another API for this?
Let's say that the files I want in Blobstore are already stored in my GCS default bucket. Can I just tell Blobstore about them using the gcs location "/gs/bucketname/file"? I tried this by
GcsFilename filename = new GcsFilename(bucketName, fileId);
String gcsKey = "/gs/" + bucketName + "/" + filename.getObjectName();
BlobKey blobKey = blobstoreService.createGsBlobKey(gcsKey);
GcsOutputChannel outputChannel = gcsService.createOrReplace(filename, GcsFileOptions.getDefaultInstance());
ObjectOutputStream oout = new ObjectOutputStream(Channels.newOutputStream(outputChannel));
oout.writeObject(myDataObjectToPersist);
oout.close();
// ...at some other point I have checked the file is correctly stored in
// GCS and I can fetch it using /gs/bucket/fileId
// but it doesn't seem to be in Blobstore, so when
InputStream stream = new BlobstoreInputStream(new BlobKey(blobKey.keyString))
// ... this gives a BlobstoreInputStream$BlobstoreIOException: BlobstoreInputStream received an invalid blob key...
Is this something conceptually wrong - like if I use GcsOutputChannel to save it I will not get it from Blobstore even if I create a BlobKey, or is it something that could work but I just did something wrong?
1K thanks
Why would you want to store the file in blobstore as opposed to writing and reading it directly from GCS?
Yes, you can create a BlobKey for a file stored in GCS, and can use the key in some of the blobstore API (such as fetchData and serve) but unfortunately not in all.
Some of the blobstore API (such as BlobstoreInputStream) depends on BlobInfo and that is not created when using the GCS client.
I have some file upload code that works on appspot.com but not on local Java dev server.
I am uploading to Google Cloud Storage, via:
String bucketName = AppIdentityServiceFactory.getAppIdentityService().getDefaultGcsBucketName();
String uploadUrl = blobstoreService.createUploadUrl(
successUrl,
UploadOptions.Builder.withGoogleStorageBucketName(bucketName)
);
in my upload handler (pointed to by "successUrl"), I try and read the newly uploaded blob from the GCS bucket. This is done like this:
BlobstoreService blobstoreService = BlobstoreServiceFactory.getBlobstoreService();
Map<String, List<BlobKey>> uploads = blobstoreService.getUploads(req);
Map<String, List<FileInfo>> files = blobstoreService.getFileInfos(req);
...
// use the FileInfo instance to do fileInfo.getGsObjectName()
...
GcsInputChannel readChannel = gcsService.openPrefetchingReadChannel(gcsFilename, 0, 1024 * 1024);
InputStream is = Channels.newInputStream(readChannel));
up on the actual appengine instance, fileInfo.getGsObjectName() returns a working path like this:
/gs/myapp.appspot.com/L2F..LongIDHere...YmJV
which works and can be read, BUT, on the local dev server, I get something like:
/gs/app_default_bucket/fake-encoded_gs_key:YXB...LongIdHere...6de
which throws FileNotFoundException when tried read using the same code, that works on appEngine :-(