Develop Reference

Read a file using Java from an S3 bucket and HTTP PUT file to presigned AWS S3 URL of another bucket in a way that simulates an actual file upload

New to Java and HTTP requests.
Why this question is not a duplicate: I'm not using AWS SDK to generate any presigned URL. I get it from an external API.
Here is what I'm trying to accomplish:
Step 1: Read the source S3 bucket for a file (for now .xlsx)
Step 2: Parse this file by converting it to an InputStreamReader (I need help here)
Step 3: Do a HTTP PUT of this file by transferring the contents of the InputStreamReader to an OutputStreamWriter, on a pre-signed S3 URL that I already have obtained from an external team. The file must sit in the destination S3 bucket, in the exact way a file is uploaded manually by dragging and dropping. (Also need help here)
Here is what I've tried:
Step 1: Read the S3 bucket for the file
public class LambdaMain implements RequestHandler<S3Event, String> {
#Override
public String handleRequest(final S3Event event, final Context context) {
System.out.println("Create object was called on the S3 bucket");
S3EventNotification.S3EventNotificationRecord record = event.getRecords().get(0);
String srcBucket = record.getS3().getBucket().getName();
String srcKey = record.getS3().getObject().getUrlDecodedKey();
AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
.withCredentials(DefaultAWSCredentialsProviderChain.getInstance())
.build();
S3Object s3Object = s3Client.getObject(new GetObjectRequest(
srcBucket, srcKey));
String presignedS3Url = //Assume that I have this by making an external API call
InputStreamReader inputStreamReader = parseFileFromS3(s3Object); #Step 2
int responseCode = putContentIntoS3URL(inputStreamReader, presignedS3Url); #Step 3
}
Step 2: Parse the file into an InputStreamReader to copy it to an OutputStreamWriter:
private InputStreamReader parseFileFromS3(S3Object s3Object) {
return new InputStreamReader(s3Object.getObjectContent(), StandardCharsets.UTF_8);
}
Step 3: Make a HTTP PUT call by copying the contents from InputStreamReader to OutputStreamWriter:
private int putContentIntoS3URL(InputStreamReader inputStreamReader, String presignedS3Url) {
URL url = null;
try {
url = new URL(presignedS3Url);
} catch (MalformedURLException e) {
e.printStackTrace();
}
HttpURLConnection httpCon = null;
try {
assert url != null;
httpCon = (HttpURLConnection) url.openConnection();
} catch (IOException e) {
e.printStackTrace();
}
httpCon.setDoOutput(true);
try {
httpCon.setRequestMethod("PUT");
} catch (ProtocolException e) {
e.printStackTrace();
}
OutputStreamWriter outputStreamWriter = null;
try {
outputStreamWriter = new OutputStreamWriter(
httpCon.getOutputStream());
} catch (IOException e) {
e.printStackTrace();
}
try {
IOUtils.copy(inputStreamReader, outputStreamWriter);
} catch (IOException e) {
e.printStackTrace();
}
try {
outputStreamWriter.close();
} catch (IOException e) {
e.printStackTrace();
}
try {
httpCon.getInputStream();
} catch (IOException e) {
e.printStackTrace();
}
int responseCode = 0;
try {
responseCode = httpCon.getResponseCode();
} catch (IOException e) {
e.printStackTrace();
}
return responseCode;
}
The issue with the among approach is that when I read an .xlsx file via an S3 insert trigger and PUT into the URL, when I download the uploaded file - it gets downloaded as some gibberish.
When I try reading in a .png file and PUT into the URL, when I download the uploaded file - it gets downloaded as some text file with some gibberish (I did see the word PNG in it though)
It feels like I'm making mistakes with:
Incorrectly creating an OutputStreamWriter since I don't understand how to send a file via a HTTP request
Assuming that every file type can be handled in a generic way.
Not setting the content-type in the HTTP request
Expecting S3 to magically understand my file type after the PUT operation
I would like to know if my above 4 assumptions are correct or incorrect.
The intention is that, I do the PUT on the file data correctly so it sits in the S3 bucket along with the correct file type/extension. I hope my effort is worthy to garner some help. I've done a lot of searching into HTTP PUT and File/IO, but I'm unable to LINK them together for my use-case, since I perform a File I/O followed by a HTTP PUT.
UPDATE 1:
I've added the setRequestProperty("Content-Type", "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"), but the file doesn't sit in the S3 bucket with the file extension. It simply sits there as an object.
UPDATE 2:
I think this also has something to do with setContentDisposition() header, although I'm not sure how I go about setting these headers for Excel files.
UPDATE 3:
This may simply have to do with how the Presigned S3 URL itself is vended out to us. As mentioned in the question, I said that we get the Presigned S3 URL from some other team. The question itself has multiple parts that need answering.
Does the default Presigned S3 URL ALLOW clients to set the content-type and content-disposition in the HTTP header?: I've set up another separate question here since it's quite unclear: Can a client set file name and extension programmatically when he PUTs file content to a presigned S3 URL that the service vends out?
If the answer to above question is TRUE, then and only then must we go into how to set the file contents and write it to the OutputStream

You are using InputStreamReader and OutputStreamWriter, which are both bridges between a byte stream and a character stream. However, you are using these with byte data, which means you first convert your bytes to characters, and then back to bytes. Since your data is not character data, this conversion might explain why you get gibberish as a result.
I'd start trying to get rid of the reader and writer, instead directly using the InputStream (which you already got from s3Object.getObjectContent()), and the OutputStream (which you got from httpCon.getOutputStream()). IOUtils.copy should also support this.
Also as a side note, when you construct the InputStreamReader you set StandardCharsets.UTF_8 as the charset to use, but when you construct the OutputStreamWriter you don't set the charset. Should the default charset not be UTF-8, this conversion would probably also result in gibberish.

Get size of files using Google drive API

I am trying to list files from google drive along with their sizes for my app & below is the how i tried getting the size
File file = service.files().get(file.getId()).setFields("size").execute();
file.getSize()
I came to know that the size obtained from this call is not right as google drive only populates file size for files apart from google docs, sheet Get size of file created on Google drive using Google drive api in android
Also I tried determining file's size by making http GET to
webContentLink & checking the Content-Length header like below
HttpURLConnection conn = null;
String url = "https://docs.google.com/a/document/d/1Gu7Q2Av2ZokZZyLjqBJHG7idE1dr35VE6rTuSii36_M/edit?usp=drivesdk";
try {
URL urlObj = new URL(url);
conn = (HttpURLConnection) urlObj.openConnection();
conn.setRequestMethod("HEAD");
conn.getInputStream();
System.out.println(conn.getContentLength());
} catch (IOException e) {
e.printStackTrace();
} finally {
conn.disconnect();
}
But in this case , the file size is not correct as it comes out be very large
Is there any way I can determine the file size ?

Found the solution to the problem. For files uploaded to google drive, file size can be determined with file.getSize() call. For Google apps files such as doc, spreadsheet etc file.getSize() will return null as they don't consumer any space in drive. So as a hacky way, I am exporting as a pdf & determining the size. Below is the code for that
try {
Drive service = new Drive.Builder(HTTP_TRANSPORT, JSON_FACTORY, credential)
.setApplicationName("xyz")
.build();
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
service.files().export(fileId, "application/pdf")
.executeMediaAndDownloadTo(outputStream);
int fileSize = outputStream.toByteArray().length;
} catch (Exception ex) {
}

Downloaded file with Google Drive API. Can't find it anywhere

I am trying to download a file with google drive api. The code gives no error but I can't find the file anywhere.
This is the code
String fileId = "...";
OutputStream outputStream = new ByteArrayOutputStream();
try {
service.files().export(fileId, "text/csv")
.executeMediaAndDownloadTo(outputStream);
} catch (IOException e) {
System.out.println("Ceva nu a mers bine");
e.printStackTrace();
}
System.out.println(outputStream == null);
Any ideea?

Based from this example in Google documentation, I don't see any error with your code.
String fileId = "1ZdR3L3qP4Bkq8noWLJHSr_iBau0DNT4Kli4SxNc2YEo";
OutputStream outputStream = new ByteArrayOutputStream();
driveService.files().export(fileId, "application/pdf")
.executeMediaAndDownloadTo(outputStream);
Make sure that you are using the correct fileId of the specific file you want to download.
You can check on these links: How to get Google Drive file ID, How to download a file from google drive using drive api java?

How to download files using java and moving them to a directory?

I've looked around for some script or some tutorial for downloading file using Java, and then moving them to a different directory. I've seen a lot of questions similar to mine answered, but they all vary. Is there a definite way to do this? I currently learning Java and don't have enough experience make functioning script for this. Could anyone help?

I personally think this is the best way to download a file from the web.
When you download the file, it is stored in current running program, not on the hard drive, unless you specify it to.
URL url;
URLConnection con;
DataInputStream dis;
FileOutputStream fos;
byte[] fileData;
try {
url = new URL("http://website.com/file.pdf"); //File Location goes here
con = url.openConnection(); // open the url connection.
dis = new DataInputStream(con.getInputStream());
fileData = new byte[con.getContentLength()];
for (int q = 0; q < fileData.length; q++) {
fileData[q] = dis.readByte();
}
dis.close(); // close the data input stream
fos = new FileOutputStream(new File("/Users/kfang/Documents/Download/file.pdf")); //FILE Save Location goes here
fos.write(fileData); // write out the file we want to save.
fos.close(); // close the output stream writer
}
catch(Exception m) {
System.out.println(m);
}

Reading directly from Google Drive in Java

Please I need to read the content of a file stored in Google Drive programmatically. I'm looking forward to some sort of
InputStream is = <drive_stuff>.read(fileID);
Any help?
I'll also appreciate if I can write back to a file using some sort of
OutputStream dos = new DriveOutputStream(driveFileID);
dos.write(data);
If this sort of convenient approach is too much for what Drive can offer, please I'll like to have suggestions on how I can read/write to Drive directly from java.io.InputStream / OutputStream / Reader / Writer without creating temporary local file copies of the data I want to ship to drive. Thanks!

// Build a new authorized API client service.
Drive service = getDriveService();
// Print the names and IDs for up to 10 files.
FileList result = service.files().list()
.setPageSize(10)
.setFields("nextPageToken, files(id, name)")
.execute();
List<File> files = result.getFiles();
if (files == null || files.size() == 0) {
System.out.println("No files found.");
} else {
System.out.println("Files:");
for (File file : files) {
System.out.printf("%s (%s)\n", file.getName(), file.getId());
String fileId = file.getId();
Export s=service.files().export(fileId, "text/plain");
InputStream in=s.executeMediaAsInputStream();
InputStreamReader isr=new InputStreamReader(in);
BufferedReader br = new BufferedReader(isr);
String line = null;
StringBuilder responseData = new StringBuilder();
while((line = br.readLine()) != null) {
responseData.append(line);
}
System.out.println(responseData);
}
}
}

Please take a look at the DrEdit Java sample that is available on the Google Drive SDK documentation.
This example shows how to authorize and build requests to read metadata, file's data and upload content to Google Drive.
Here is a code snippet showing how to use the ByteArrayContent to upload media to Google Drive stored in a byte array:
/**
* Create a new file given a JSON representation, and return the JSON
* representation of the created file.
*/
#Override
public void doPost(HttpServletRequest req, HttpServletResponse resp)
throws IOException {
Drive service = getDriveService(req, resp);
ClientFile clientFile = new ClientFile(req.getReader());
File file = clientFile.toFile();
if (!clientFile.content.equals("")) {
file = service.files().insert(file,
ByteArrayContent.fromString(clientFile.mimeType, clientFile.content))
.execute();
} else {
file = service.files().insert(file).execute();
}
resp.setContentType(JSON_MIMETYPE);
resp.getWriter().print(new Gson().toJson(file.getId()).toString());
}

Here's a (incomplete) snippet from my app which might help.
URL url = new URL(urlParam);
HttpURLConnection connection = (HttpURLConnection) url
.openConnection();
connection.setDoOutput(true);
connection.setRequestMethod("GET");
connection
.setRequestProperty("Authorization",
"OAuth "+accessToken);
String docText = convertStreamToString(connection.getInputStream());

Using google-api-services-drive-v3-rev24-java-1.22.0:
To read the contents of a file, make sure you set DriveScopes.DRIVE_READONLY when you do GoogleAuthorizationCodeFlow.Builder(...) in your credential authorizing method/code.
You'll need the fileId of the file you want to read. You can do something like this:
FileList result = driveService.files().list().execute();
You can then iterate the result for the file and fileId you want to read.
Once you have done that, reading the contents would be something like this:
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
driveService.files().get(fileId).executeMediaAndDownloadTo(outputStream);
InputStream in = new ByteArrayInputStream(outputStream.toByteArray());