I have an amazon s3 bucket which contains one file and I am looking for a way to download that one file using the file extension. currently, I have the code to download multiple files from s3 bucket using key and then filter based on the extension. Something like this:
s3.listObjects(operaBucketName, keyName)
.getObjectSummaries()
.forEach(s -> keys.add(s.getKey()));
List<String> filteredKeys =
keys.stream().filter(s -> s.contains(extension)).collect(Collectors.toList());
//add to file list
List<File> files = new ArrayList<>();
for (String key : filteredKeys) {
File file = new File(FilenameUtils.getFullPath(localDirectory) + FilenameUtils.getName(key));
downloadFileFromS3(operaBucketName, key, file);
files.add(file);
}
I want to do this but targeting a single file. I tried s3.getObject(operaBucketName, keyName);, but the com.amazonaws.services.s3.model.S3Object doesn't have a way to check the file extension. I think using this How to write an S3 object to a file? I can write S3object contents to file.
Also, just in case, there are multiple files in the given s3 folder(key), will getObject throw an exception?
Sdk version used:
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-s3</artifactId>
<version>1.11.792</version>
</dependency>
Thanks!
Related
I am using JAVA 1.8 and S3TransferManager(Java SDK V2.X) to download all objects from S3 bucket and i am using
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>s3-transfer-manager</artifactId>
<version>2.17.247-PREVIEW</version>
</dependency>
Code appears as below
...
S3TransferManager s3TransferManager = S3TransferManager.create();
DirectoryDownload directoryDownload = s3TransferManager
.downloadDirectory(DownloadDirectoryRequest.builder()
.destinationDirectory(Paths.get("/home/shashank/Documents"))
.bucket(bucketName)
.prefix(bucketPath)
.build());
// Wait for the transfer to complete
CompletedDirectoryDownload completedDirectoryDownload = directoryDownload.completionFuture().join();
// Print out the failed downloads
completedDirectoryDownload.failedTransfers().forEach(System.out::println);
...
This code is downloading all files from S3 but omitting first charater from file name.
Ex : Download files are renamed to
mage.jpg and itle.png
Actual objects in S3 were
image.jpg and title.png
Please guide me if i am doing something wrong.
I have a set of .gz compressed files in s3 bucket. I want to download the csv file inside the .gz file. I tried to extract the .gz file and put it into the s3Object. Now i need to extract the s3 object and download the csv file inside it using java. Please advise.This is the code I used.Now i am able to download gz file. But I need to download csv file inside gz.
S3Object object = s3Client.getObject(“bucket”,“Location/file.gz”);
final String encoding = null;
return ResponseEntity.ok(IOUtils.toString(object.getObjectContent(), encoding));
I need help in unzipping the gz file in s3object and return the decompressed contents in the response.
The below code will convert your gunzip file into plain data, but I'm not sure 100% about your actual issue, whether you want to display the content in browser itself or you want to send it as Save as Option, that's why I did a minimum code change to your code assuming you have only problem in converting gunzip format to CSV data, hope you could modify/enhance it that suits you best.
import java.util.zip.GZIPInputStream;
//your method begins from here
final AmazonS3 s3 = AmazonS3ClientBuilder.defaultClient();
S3Object object = s3.getObject("your-bucket", "file-path");
return ResponseEntity.ok(IOUtils.toString(new GZIPInputStream(object.getObjectContent())));
I am newbie and recently started working on amazon s3 services.
I have create a java maven project and using Java 1.8 and aws-java-sdk version 1.11.6 version in my sample program
Below is source code for the same and it executes successfully.
It returns version id as output of the program.
System.out.println("Started the program to create the bucket....");
BasicAWSCredentials awsCreds = new BasicAWSCredentials(CloudMigrationConstants.AWS_ACCOUNT_KEY, CloudMigrationConstants.AWS_ACCOUNT_SECRET_KEY);
AmazonS3Client s3Client = new AmazonS3Client(awsCreds);
String uploadFileName="G:\\Ebooks\\chap1.doc";
String bucketName="jinesh1522421795620";
String keyName="test/";
System.out.println("Uploading a new object to S3 from a file\n");
File file = new File(uploadFileName);
PutObjectResult putObjectResult=s3Client.putObject(new PutObjectRequest(
bucketName, keyName, file));
System.out.println("Version id :" + putObjectResult.getVersionId());
System.out.println("Finished the program to create the bucket....");
But when I try to see the files using s3browser or amazon console I do not see the files are listed inside the bucket.
Can you please let me know what is wrong with My Java program?
I think I misunderstood the concept. We have to specify the name of the file to store while specifying the key. In above program what I missed was specifying the name of the file along with name of the folder hence I was not able to see the file.
File file = new File(uploadFileName);
PutObjectResult putObjectResult=s3Client.putObject(new PutObjectRequest(
bucketName, keyName+"/chap1.doc", file));
I am new to S3 and I am trying to create multiple directories in Amazon S3 using java by only making one call to S3.
I could only come up with this :-
ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentLength(0);
InputStream emptyContent = new ByteArrayInputStream(new byte[0]);
PutObjectRequest putObjectRequest = new PutObjectRequest(bucket,
"test/tryAgain/", emptyContent, metadata);
s3.putObject(putObjectRequest);
But the problem with this while uploading 10 folders (when the key ends with "/" in the console we can see the object as a folder ) is that I have to make 10 calls to S3.
But I want to do a create all the folders at once like we do a batch delete using DeleteObjectsRequest.
Can anyone please suggest me or help me how to solve my problem ?
Can you be a bit more specific as to what you're trying to do (or avoid doing)?
If you're primarily concerned with the cost per PUT, I don't think there is a way to batch 'upload' a directory with each file being a separate key and avoid that cost. Each PUT (even in a batch process) will cost you the price per PUT.
If you're simply trying to find a way to efficiently and recursively upload a folder, check out the uploadDirectory() method of TransferManager.
http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/transfer/TransferManager.html#uploadDirectory-java.lang.String-java.lang.String-java.io.File-boolean-
public MultipleFileUpload uploadDirectory(String bucketName,
String virtualDirectoryKeyPrefix,
File directory,
boolean includeSubdirectories)
Hey guys I'm currently using jarchivelib which can be found Here I'm stuck on figuring out a way to read the file without having to use the unpack method because it makes a file of the unpacked version. EX:
File archive = new File("/home/jack/archive.zip");
File destination = new File("/home/jack/archive");
Archiver archiver = ArchiverFactory.createArchiver(ArchiveFormat.ZIP);
archiver.extract(archive, destination);
I want to make it so i don't have to unpack it to read the files... If there is no way to do that I'm guessing in my method for Jframe.setDefualtCloseOpperation i'll have to make a custom one so it deletes the files? or is there a better way for handling temp files?
If all you want to do is to extract the file, why not use Java's built in zip to extract the file or if it is password protected you can use Zip4j. These libraries support streams, so that you can extract the contents of the file without writing it a FileStream
As of version 0.4.0, the jarchivelib Archiver API supports streaming an archive rather than extracting it directly onto the filesystem.
ArchiveStream stream = archiver.stream(archive);
ArchiveEntry entry;
while((entry = stream.getNextEntry()) != null) {
// access each archive entry individually using the stream
// or extract it using entry.extract(destination)
// or fetch meta-data using entry.getName(), entry.isDirectory(), ...
}
stream.close();
when the stream is pointing to an entry after calling getNextEntry, you can use the stream.read methods just as you would reading an individual entry.