Develop Reference

Is there any direct way to copy one s3 directory to another in java or scala?

I want to archive all the files and sub directories in a s3 directory to some other s3 location using java. Is there any direct way to copy one s3 directory to another in java or scala?

There is no API call to operate on whole directories in Amazon S3.
In fact, directories/folders do not exist in Amazon S3. Rather, each object stores the full path in its filename (Key).
If you wish to copy multiple objects that have the same prefix in their Key, your code will need to loop through the objects, copying one object at a time.

A bit wordy, but does the job: reasonable logging, multithreading via TransferManager, handling continuation token for "folders" with more than 1000 keys:
/**
* Copies all content from s3://sourceBucketName/sourceFolder to s3://destinationBucketName/destinationFolder.
*/
public void copyAll(String sourceBucketName, String sourceFolder, String destinationBucketName, String destinationFolder) {
log.info("Copying data from s3://{}/{} to s3://{}/{}", sourceBucketName, sourceFolder, destinationBucketName, destinationFolder);
TransferManager transferManager = TransferManagerBuilder.standard()
.withS3Client(client)
.build();
try {
ListObjectsV2Request request = new ListObjectsV2Request()
.withBucketName(sourceBucketName)
.withPrefix(sourceFolder);
ListObjectsV2Result objects;
do {
objects = client.listObjectsV2(request);
List<Copy> transfers = new ArrayList<>();
for (S3ObjectSummary object : objects.getObjectSummaries()) {
String sourceKey = object.getKey();
String sourceRelativeKey = sourceKey.substring(sourceFolder.length());
String destinationKey = destinationFolder + sourceRelativeKey;
transfers.add(transferManager.copy(sourceBucketName, sourceKey, destinationBucketName, destinationKey));
}
for (Copy transfer : transfers) {
log.debug(transfer.getDescription());
transfer.waitForCompletion();
}
log.info("Copied batch of {} objects. Last object: {}", transfers.size(), transfers.isEmpty() ? "None" : transfers.get(transfers.size() - 1).getDescription());
request.setContinuationToken(objects.getNextContinuationToken());
} while (objects.isTruncated());
log.info("Copy operation completed successfully from s3://{}/{} to s3://{}/{}", sourceBucketName, sourceFolder, destinationBucketName, destinationFolder);
} catch (InterruptedException e) {
// Resetting interrupt flag and returning control to the caller.
Thread.currentThread().interrupt();
throw new RuntimeException(e);
} finally {
transferManager.shutdownNow(false);
}
}

How to read JSON files from S3 using the S3AsyncClient

I cant figure out how to read a JSON file from S3 into memory as String.
The examples I find calls getObjectContent() however this is not available for the GetObjectResponse I get from the S3AsyncClient.
The code I experiment is the sample code from AWS.
// Creates a default async client with credentials and AWS Region loaded from the
// environment
S3AsyncClient client = S3AsyncClient.create();
// Start the call to Amazon S3, not blocking to wait for the result
CompletableFuture<GetObjectResponse> responseFuture =
client.getObject(GetObjectRequest.builder()
.bucket("my-bucket")
.key("my-object-key")
.build(),
AsyncResponseTransformer.toFile(Paths.get("my-file.out")));
// When future is complete (either successfully or in error), handle the response
CompletableFuture<GetObjectResponse> operationCompleteFuture =
responseFuture.whenComplete((getObjectResponse, exception) -> {
if (getObjectResponse != null) {
// At this point, the file my-file.out has been created with the data
// from S3; let's just print the object version
System.out.println(getObjectResponse.versionId());
} else {
// Handle the error
exception.printStackTrace();
}
});
// We could do other work while waiting for the AWS call to complete in
// the background, but we'll just wait for "whenComplete" to finish instead
operationCompleteFuture.join();
How should this code be modified so that I can get the actual JSON content from the GetObjectResponse?

After response is transformed to bytes it can be transformed to string:
S3AsyncClient client = S3AsyncClient.create();
GetObjectRequest getObjectRequest = GetObjectRequest.builder().bucket("my-bucket").key("my-object-key").build();
client.getObject(getObjectRequest, AsyncResponseTransformer.toBytes())
.thenApply(ResponseBytes::asUtf8String)
.whenComplete((stringContent, exception) -> {
if (stringContent != null)
System.out.println(stringContent);
else
exception.printStackTrace();
});

You can use AsyncResponseTransformer.toBytes in order to save response to a byte array rather than a file. javadoc

Importing csv data from Storage to Cloud SQL not working - status always "pending"

I am new to java (I have experience with C# though)
Sadly, I inherited a terrible project (the code is terrible) and what I need to accomplish is to import some csv files into Cloud SQL
So there's a WS which runs this task, apparently the dev followed this guide to import data. But it is not working. Here's the code (Essential parts, actually it is longer and more ugly)
InstancesImportRequest requestBody = new InstancesImportRequest();
ImportContext ic = new ImportContext();
ic.setKind("sql#importContext");
ic.setFileType("csv");
ic.setUri(bucketPath);
ic.setDatabase(CLOUD_SQL_DATABASE);
CsvImportOptions csv = new CsvImportOptions();
csv.setTable(tablename);
List<String> list = new ArrayList<String>();
// here there is some code that populates the list with the columns
csv.setColumns(list);
ic.setCsvImportOptions(csv);
requestBody.setImportContext(ic);
SQLAdmin sqlAdminService = createSqlAdminService();
SQLAdmin.Instances.SQLAdminImport request = sqlAdminService.instances().sqladminImport(project, instance, requestBody);
Operation response = request.execute();
System.out.println("Executed : Going to sleep.>"+response.getStatus());
int c = 1;
while(!response.getStatus().equalsIgnoreCase("Done")){
Thread.sleep(10000);
System.out.println("sleeped enough >"+response.getStatus());
c++;
if(c==50){
System.out.println("timeout?");
break;
}
}
public static SQLAdmin createSqlAdminService() throws IOException, GeneralSecurityException {
HttpTransport httpTransport = GoogleNetHttpTransport.newTrustedTransport();
JsonFactory jsonFactory = JacksonFactory.getDefaultInstance();
GoogleCredential credential = GoogleCredential.getApplicationDefault();
if (credential.createScopedRequired()) {
credential =
credential.createScoped(Arrays.asList("https://www.googleapis.com/auth/cloud-platform"));
}
return new SQLAdmin.Builder(httpTransport, jsonFactory, credential)
.setApplicationName("Google-SQLAdminSample/0.1")
.build();
}
I am not quite sure how response should be treated, it seems it is an async request. Either way, I always get status Pending; it seems it is not even start to executing.
Of course it ends timing out. What is wrong here, why the requests never starts ? I couldn't find any actual example on the internet about using this java sdk to import files, except the link I gave above

Well, the thing is that the response object is static, so it will always return "Pending" as the initial status since it is a string in the object - it is not actually being updated.
To get the actual status, you have to requested it to google using the sdk. I did something like this (it will be better to use a smaller sleep time, and make it grow as you try more times)
SQLAdmin.Instances.SQLAdminImport request = sqlAdminService.instances().sqladminImport(CLOUD_PROJECT, CLOUD_SQL_INSTANCE, requestBody);
// execution of our import request
Operation response = request.execute();
int tried = 0;
Operation statusOperation;
do {
// sleep one minute
Thread.sleep(60000);
// here we are requesting the status of our operation. Name is actually the unique identifier
Get requestStatus = sqlAdminService.operations().get(CLOUD_PROJECT, response.getName());
statusOperation = requestStatus.execute();
tried++;
System.out.println("status is: " + statusOperation.getStatus());
} while(!statusOperation.getStatus().equalsIgnoreCase("DONE") && tried < 10);
if (!statusOperation.getStatus().equalsIgnoreCase("DONE")) {
throw new Exception("import failed: Timeout");
}

Check if user is logged in aws cognito, JAVA

I am using Java as back end to communicate with AWS Cognito. I am able to login, logout, create users, sign out and other functions. I am also able to verify an access token by following this link:
But I want to verify if a user is logged in or not.
In JAVA, is there a isLoggedin() function that returns a boolean or is there a way to see if the token is revoked? These functions exist for Android and iOS but what about JAVA.
I thought this verifies if the token is active, but it only verifies if the token is in the right format:
// This parses token to JWT.
JWT jwtparser = JWTParser.parse(accessToken);
String JWTissuer = jwtparser.getJWTClaimsSet().getIssuer();
JWSHeader header = (JWSHeader) jwtparser.getHeader();
Object token_use = jwtparser.getJWTClaimsSet().getClaim("token_use");
Object exp = jwtparser.getJWTClaimsSet().getClaim("iat");
Date expirationDate = jwtparser.getJWTClaimsSet().getExpirationTime();
// Read in JSON Key file saved somewhere safe
File file = new File("jwks.json");
String content = FileUtils.readFileToString(file, "utf-8");
JSONObject JsonObjects = new JSONObject(content);
JSONArray keysArray = JsonObjects.getJSONArray("keys");
JSONObject keyString = (JSONObject) keysArray.get(1);
if (header.getKeyID().equals(keyString.get("kid")) && token_use.toString().equals("access") && JWTissuer.equals("https://cognito-idp.us-west-2.amazonaws.com/us-west-2_xxxxxxx")) {
return true;
} else { return false; }
I want to see if a user is logged in. I have not found an appropriate method to do so.
Mahalo

I found a work around:
public boolean isLoggedin(String accessToken) {
GetUserRequest request = new GetUserRequest();
request.withAccessToken(accessToken);
AWSCognitoIdentityProviderClientBuilder builder =
AWSCognitoIdentityProviderClientBuilder.standard();
builder.withRegion("us-west-2");
AWSCognitoIdentityProvider cognitoCreate = builder.build();
GetUserResult result = cognitoCreate.getUser(request);
try {
System.out.println("success");
return true;
} catch (Exception e) {
e.getMessage();
return false;
}
}

Getting file size from S3 bucket

I am trying to get the file size (content-length) using Amazon S3 JAVA sdk.
public Long getObjectSize(AmazonS3Client amazonS3Client, String bucket, String key)
throws IOException {
Long size = null;
S3Object object = null;
try {
object = amazonS3Client.getObject(bucket, key);
size = object.getObjectMetadata().getContentLength();
} finally {
if (object != null) {
//object.close();
1. This results in 50 calls (connection pool size) post that I start getting connection pool errors.
2. If this line is uncommented it takes hell lot of time to make calls.
}
}
return size;
}
I followed this and this. But not sure what I am doing wrong here.
Any help on this?

I'm guessing what your actual question is asking, but I think you can reduce your code and eliminate the need to create an s3Object at all by doing something like:
public Long getObjectSize(AmazonS3Client amazonS3Client, String bucket, String key)
throws IOException {
return amazonS3Client.getObjectMetadata(bucket, key).getContentLength();
}
That should remove the need to call object.close() which you appear to be having issues with.

For v2 of the Amazon S3 Java SDK, try something like this:
HeadObjectRequest headObjectRequest =
HeadObjectRequest.builder()
.bucket(bucket)
.key(key)
.build();
HeadObjectResponse headObjectResponse =
s3Client.headObject(headObjectRequest);
Long contentLength = headObjectResponse.contentLength();

So we have 2 SDKs.
For v1 of the Amazon S3 Java SDK, below
client.getObjectMetadata(bucket, key).getContentLength();
where client is an instance of AmazonS3 coming from import com.amazonaws.services.s3.AmazonS3; and implementation 'com.amazonaws:aws-java-sdk-s3:1.12.353' gradle dependencies.
For v2 of the Amazon S3 Java SDK, below:
return client.headObject(HeadObjectRequest.builder().bucket(bucket).key(key).build()).contentLength();
where client is instance of S3Client coming from import software.amazon.awssdk.services.s3.S3Client; and gradle dependencies implementation 'software.amazon.awssdk:s3:2.18.35'