I am new to S3. I need to fetch a file from S3, update it, and store it back to S3, so I need to fetch the latest time of this file in an existing module; it would be good if the answer is in java.
This gets a list of objects in the bucket. This also prints out each object’s name, the file size, and last modified date.
ObjectListing objects = conn.listObjects(bucket.getName());
do {
for (S3ObjectSummary objectSummary : objects.getObjectSummaries()) {
System.out.println(objectSummary.getKey() + "\t" +
ObjectSummary.getSize() + "\t" +
StringUtils.fromDate(objectSummary.getLastModified()));
}
objects = conn.listNextBatchOfObjects(objects);
} while (objects.isTruncated());
The output will look something like this:
myphoto1.jpg 251262 2011-08-08T21:35:48.000Z
myphoto2.jpg 262518 2011-08-08T21:38:01.000Z
Ref for s3 examples including above one( LISTING A BUCKET’S CONTENT) is at: http://docs.ceph.com/docs/master/radosgw/s3/java/
Related
I am using jdk 11 and virtual-host-style-access (AWS SDK for Java version 2) to create/access objects in AWS s3 bucket following :
https://docs.aws.amazon.com/sdk-for-java/v2/developer-guide/examples-s3-objects.html#list-object
While I was able to create objects in the the designated bucket, I am not able to print the list of contents/objects in the bucket although, as I checked the permission, everyone is granted to view the objects in the bucket. The error message is :
software.amazon.awssdk.services.s3.model.NoSuchKeyException: The specified key does not exist. (Service: S3, Status Code: 404
This is the way the s3client is created :
adapterSmsS3Client = S3Client.builder()
.region(Region.US_WEST_2)
.credentialsProvider(StaticCredentialsProvider.create(AwsBasicCredentials.create(ACCESS_KEY,SECRET_KEY)))
.endpointOverride(URI.create(BASE_URL))
.build();
And this is the way I am trying to print the list:
public static void listBucketObjects( S3Client s3, String bucketName ) {
ListBucketsResponse res1 = s3.listBuckets();
ListObjectsRequest listObjects = ListObjectsRequest
.builder()
.bucket(BUCKET_NAME)
.build();
ListObjectsResponse res = s3.listObjects(listObjects);
List<S3Object> objects = res.contents();
for (ListIterator iterVals = objects.listIterator(); iterVals.hasNext(); ) {
S3Object myValue = (S3Object) iterVals.next();
System.out.print("\n The name of the key is " + myValue.key());
System.out.print("\n The object is " + calKb(myValue.size()) + " KBs");
System.out.print("\n The owner is " + myValue.owner());
}
}
BUCKET_NAME is the name of bucket on s3 ( not any URL)
Although, I would like to mention that if I use Path-style-request (AWS SDK for Java version 1), following :
https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/examples-s3-objects.html
I am able to print contents from the same bucket. However we do not intend to go that way.
Any insight on why am I getting the "key does not exist" error or potential resolution?
If you had any problem with permissions you would have get a 403 forbidden; not a 404 NoSuchKey.
What are the names of your objects in the bucket ? My guess is that you have some special characters or url-encoded characters that causes the problem. See https://aws.amazon.com/premiumsupport/knowledge-center/404-error-nosuchkey-s3/?nc1=h_ls for more details.
And I suggest you to use listObjectsV2 instead of the V1.
I have S3 bucket named 'groceries' and with multiple files inside the folder fruits.
Files name would be like ('APPLE_1','APPLE_11','APPLE_1112','APPLE_3','APPLE_6')
I basically want the file_name and it's version IDs for a given bucket and key.
When I run the below code it also fetches other similar files like (APPLE_11, APPLE_1112).
What changes should I make in the below code to filter only (APPLE_1) ?
ListVersionsRequest request = new ListVersionsRequest();
request.withBucketName("groceries");
request.setPrefix("fruits/APPLE_1");
request.withMaxResults(20);
VersionListing versionListing = s3Client.listVersions(request);
int numVersions = 0, numPages = 0;
while (true) {
numPages++;
for (S3VersionSummary objectSummary:
versionListing.getVersionSummaries()) {
System.out.printf("Retrieved object %s, version %s\n",
objectSummary.getKey(),
objectSummary.getVersionId());
numVersions++;
}
if (versionListing.isTruncated()) {
versionListing = s3Client.listNextBatchOfVersions(versionListing);
} else {
break;
}
current output
Retrieved object fruits/APPLE_1, version LFrP3YxiZu9S0
Retrieved object fruits/APPLE_11, version bHcs6Oh1leiPPvUB6NI07P0GB6
Retrieved object fruits/APPLE_1112, version Q9FD7fmVq_t1L3GPFitf
expected output
Retrieved object fruits/APPLE_1, version LFrP3YxiZu9S0
You can place an if condition inside the for loop. E. g.
for (S3VersionSummary objectSummary:
versionListing.getVersionSummaries()) {
if ( objectSummary.getKey().isEqual("fruits/APPLE_1"){// Or any other condition based on your requirement
System.out.printf("Retrieved object %s, version %s\n",
objectSummary.getKey(),
objectSummary.getVersionId());
}
numVersions++;
}
I am using JNA to read Windows event logs. I can get a fair amount of data out of each record but I can't quite get the field names.
To read logs I am doing
EventLogIterator iter = new EventLogIterator("Security");
while(iter.hasNext()) {
EventLogRecord record = iter.next();
System.out.println("Event ID: " + record.getEventId()
+ ", Event Type: " + record.getType()
+ ", Event Source: " + record.getSource());
String strings[] = record.getStrings();
for(String str : strings) {
System.out.println(str);
}
}
I can get data like the id, type, and source easily. Then I can get the list of strings which may be for SubjectUserSid, SubjectUserName, etc.
I've been trying to get the data that I want with the field names. Is there an easy way to extract the field names/headers for each of the strings from record.getStrings()? I noticed there is a byte[] data variable in the record. I have tried to read this but I haven't been able to get any useful information from it. I know I can get the data length and offset for certain variables which I think I could extract the data that I want that way but I was wondering if that was correct or if there was an easier way.
I am trying to get an id for an existing dir on Google Drive.
com.google.api.services.drive.model.About about = drive.about().get().execute();
com.google.api.services.drive.Drive.Children.List list =
drive.children().list(about.getRootFolderId());
Iterator<Entry<String, Object>> itr = list.entrySet().iterator();
Entry<String, Object> s;
while (itr.hasNext()) {
s = itr.next();
System.out.println(s.getKey() + "::" + s.getValue());
}
Right now this code is giving an output -
folderId::0APcEBFk-CF2pUk9PVA
which is probably not the correct id because I have 2 dirs and 3 files in my google drive.
I must be missing something, what the right way to get the id of an existing dir.
I have seen this question, and it will be helpful if I can get an equivalent java example. I am using the same account's google drive which is owning the app.
Create a List of files and iterate.
Drive service = new Drive.Builder(httpTransport, jsonFactory, credential1).build();
String query = "'" + about.getRootFolderId() +"' " + "in parents";
List<File> files = service.files().list().setQ(query).execute().getItems();
return files;
This should do.
Comparing your code to the example here you didn't call execute() on list, I think you're iterating on the request arguments.
Also it says:
folderId To list all files in the root folder, use the alias root as the value for folderId.
so you can skip getting the About.
I was trying to print all the objects in a bucket but I am getting an error.
Exception in thread "main" com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 301, AWS Service: Amazon S3, AWS Request ID: 758A7CBF1A29FD74, AWS Error Code: PermanentRedirect, AWS Error Message: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint., S3
At the moment I only have the following code :
public class S3Download {
/**
* #param args
*/
public static void main(String[] args) {
AmazonS3 s3 = new AmazonS3Client(new ClasspathPropertiesFileCredentialsProvider());
Region usWest2 = Region.getRegion(Regions.US_WEST_2);
s3.setRegion(usWest2);
String bucketName = "apireleasecandidate1";
ListObjectsRequest listObjectRequest = new ListObjectsRequest().withBucketName(bucketName);
ObjectListing objectListing;
do{
objectListing = s3.listObjects(listObjectRequest);
for(S3ObjectSummary objectSummary : objectListing.getObjectSummaries()){
System.out.println(" - " + objectSummary.getKey() + " " + "(size = " +
objectSummary.getSize() + ")");
}
listObjectRequest.setMarker(objectListing.getNextMarker());
}while(objectListing.isTruncated());
}
}
I found this solution on amazon's website.
Does anyone know what I am missing?
For Scala developers, here it is recursive function to execute a full scan and map of the contents of an AmazonS3 bucket using the official AWS SDK for Java
import com.amazonaws.services.s3.AmazonS3Client
import com.amazonaws.services.s3.model.{S3ObjectSummary, ObjectListing, GetObjectRequest}
import scala.collection.JavaConversions.{collectionAsScalaIterable => asScala}
def map[T](s3: AmazonS3Client, bucket: String, prefix: String)(f: (S3ObjectSummary) => T) = {
def scan(acc:List[T], listing:ObjectListing): List[T] = {
val summaries = asScala[S3ObjectSummary](listing.getObjectSummaries())
val mapped = (for (summary <- summaries) yield f(summary)).toList
if (!listing.isTruncated) mapped.toList
else scan(acc ::: mapped, s3.listNextBatchOfObjects(listing))
}
scan(List(), s3.listObjects(bucket, prefix))
}
To invoke the above curried map() function, simply pass the already constructed (and properly initialized) AmazonS3Client object (refer to the official AWS SDK for Java API Reference), the bucket name and the prefix name in the first parameter list. Also pass the function f() you want to apply to map each object summary in the second parameter list.
For example
map(s3, bucket, prefix)(s => println(s))
will print all the files
val tuple = map(s3, bucket, prefix)(s => (s.getKey, s.getOwner, s.getSize))
will return the full list of (key, owner, size) tuples in that bucket/prefix
val totalSize = map(s3, "bucket", "prefix")(s => s.getSize).sum
will return the total size of its content (note the additional sum() folding function applied at the end of the expression ;-)
You can combine map() with many other functions as you would normally approach by Monads in Functional Programming
It appears that your bucket "apireleasecandidate1" is not in the us-west-1 region. I think it is in the us-classic region. You should modify your code to remove the setRegion() call.