AWS S3 upload file name mismatch - java

I am able to upload multiple files to s3 bucket at once. However there is a mismatch in the file name the one I provided and uploaded file. I am interested in file name as I need to generate cloud front signed url based on that.
File generation code
final String fileName = System.currentTimeMillis() + pictureData.getFileName();
final File file = new File(fileName); //fileName is -> 1594125913522_image1.png
writeByteArrayToFile(img, file);
AWS file upload code
public void uploadMultipleFiles(final List<File> files) {
final TransferManager transferManager = TransferManagerBuilder.standard().withS3Client(amazonS3).build();
try {
final MultipleFileUpload xfer = transferManager.uploadFileList(bucketName, null, new File("."), files);
xfer.waitForCompletion();
} catch (InterruptedException exception) {
if (LOGGER.isInfoEnabled()) {
LOGGER.info("InterruptedException occurred=>" + exception);
}
} catch (AmazonServiceException exception) {
if (LOGGER.isInfoEnabled()) {
LOGGER.info("AmazonServiceException occurred =>" + exception);
}
throw exception;
}
}
Uploaded file name is 94125913522_image1.png. As you can see first two characters disappeared. What am I missing here. I am not able to figure out. Kindly advice.
private static void writeByteArrayToFile(final byte[] byteArray, final File file) {
try (OutputStream outputStream = new BufferedOutputStream(Files.newOutputStream(Paths.get(file.getName())))) {
outputStream.write(byteArray);
} catch (IOException exception) {
throw new FileIllegalStateException("Error while writing image to file", exception);
}
}

The reason of the problem
You lose the first two charecters of the file names because of the third argument of this method:
transferManager.uploadFileList(bucketName, null, new File("."), files);
What happens in this case
So, what is the third argument:
/**
...
* #param directory
* The common parent directory of files to upload. The keys
* of the files in the list of files are constructed relative to
* this directory and the virtualDirectoryKeyPrefix.
...
*/
public MultipleFileUpload uploadFileList(... , File directory, ...){...}
And how will it be used:
...
int startingPosition = directory.getAbsolutePath().length();
if (!(directory.getAbsolutePath().endsWith(File.separator)))
startingPosition++;
...
String key = f.getAbsolutePath().substring(startingPosition)...
Thus, the directory variable is used to define a starting index to trim file paths to get file keys.
When you pass new File(".") as a directory, the parent directory for your files will be {your_path}.
But this is a directory, and you need to work with files inside it. So the common part, retrieved from your directory file, is {your_path}./
That is 2 symbols more than you actually need. And for this reason this method trims the 2 extra characters - an extra shift of two characters when trimming the file path.
The solution
If you only need to work with the current directory, you can pass the current directory as follows:
MultipleFileUpload upload = transferManager.uploadFileList(bucketName, "",
System.getProperty("user.dir"), files);
But if you start working with external sources, it won't work. So you can use this code, which creates one MultipleFileUpload per group of files from one directory.
private final String PATH_SEPARATOR = File.separator;
private String bucketName;
private TransferManager transferManager;
public void uploadMultipleFiles(String prefix, List<File> filesToUpload){
Map<File, List<File>> multipleUploadArguments =
getMultipleUploadArguments(filesToUpload);
for (Map.Entry<File, List<File>> multipleUploadArgument:
multipleUploadArguments.entrySet()){
try{
MultipleFileUpload upload = transferManager.uploadFileList(
bucketName, prefix,
multipleUploadArgument.getKey(),
multipleUploadArgument.getValue()
);
upload.waitForCompletion();
} catch (InterruptedException ex) {
throw new RuntimeException(ex);
}
}
}
private Map<File, List<File>> getMultipleUploadArguments(List<File> filesToUpload){
return filesToUpload.stream()
.collect(Collectors.groupingBy(this::getDirectoryPathForFile));
}
private File getDirectoryPathForFile(File file){
String filePath = file.getAbsolutePath();
String directoryPath = filePath.substring(0, filePath.lastIndexOf(PATH_SEPARATOR));
return new File(directoryPath);
}

Related

java.nio.file.AccessDeniedException error?

Im trying to zip my created folder. Right now im testing localy and it create folder but after that i want to zip it.
This is my code for zip:
public static void pack(final String sourceDirPath, final String zipFilePath) throws IOException {
Path p = Files.createFile(Paths.get(zipFilePath));
try (ZipOutputStream zs = new ZipOutputStream(Files.newOutputStream(p))) {
Path pp = Paths.get(sourceDirPath);
Files.walk(pp).filter(path -> !Files.isDirectory(path)).forEach(path -> {
ZipEntry zipEntry = new ZipEntry(pp.relativize(path).toString());
try {
zs.putNextEntry(zipEntry);
Files.copy(path, zs);
zs.closeEntry();
} catch (IOException e) {
System.err.println(e);
}
});
} }
But im getting an error AccessDeniedException. Is there any option to zip created folder, i dont want to zip file because in that folder i will have subfolders, so i want to zip main folder. Any suggestion how can i achive that?
According to:
Getting "java.nio.file.AccessDeniedException" when trying to write to a folder
I think you should add the filename and the extension to your 'zipFilePath', for example: "C:\Users\XXXXX\Desktop\zippedFile.zip"

Uploading files to S3 in Spring boot and getting stored them in classpath

I have method to upload files from user to S3 bucket. But when I'm testing it, it is both saved on S3 buckend and in my classpath.
This is my function to upload file:
public void uploadFileToNews(byte[] file, String fileName) {
File fileToSave = new File(fileName);
try(FileOutputStream fileOutputStream = new FileOutputStream(fileToSave) ) {
fileOutputStream.write(file);
amazonS3.putObject(new PutObjectRequest("gwnews", fileName, fileToSave));
} catch (IOException e) {
log.error("Error during uploading file to S3", e);
}
}
And this is function in my service:
public News addNewsImage(MultipartFile multipartFile, String newsId) throws IOException {
News news = getById(newsId);
news.setImageUrl(news.getId() + ".png");
fileService.uploadFileToNews(multipartFile.getBytes(), news.getId() + ".png");
return newsRepository.save(news);
}
Am I doing something wrong? How can I avoid saving file to my classpath?
This line is saving the content in your classpath : "fileOutputStream.write(file);"
And the next line is saving it in your S3 bucket for corresponding AWS account.

Relative path to file | Springboot

I am new to Spring-boot/Java and trying to read the contents of a file in a String.
What's the issue:
I'm getting "File not found exception" and unable to read the file. Apparently, I'm not giving the correct file path.
i've attached the directory structure and my code. I'm in FeedProcessor file and want to read feed_template.php (see image)
public static String readFileAsString( ) {
String text = "";
try {
// text = new String(Files.readAllBytes(Paths.get("/src/main/template/feed_template_head.php")));
text = new String(Files.readAllBytes(Paths.get("../../template/feed_template_head.php")));
} catch (IOException e) {
e.printStackTrace();
}
return text;
}
You need to put template folder inside resource folder. And then use following code.
#Configuration
public class ReadFile {
private static final String FILE_NAME =
"classpath:template/feed_template_head.php";
#Bean
public void initSegmentPerformanceReportRequestBean(
#Value(FILE_NAME) Resource resource,
ObjectMapper objectMapper) throws IOException {
new BufferedReader(resource.getInputStream()).lines()
.forEach(eachLine -> System.out.println(eachLine));
}
}
I suggest you to go though once Resource topic in spring.
https://docs.spring.io/spring/docs/3.0.x/spring-framework-reference/html/resources.html

How to update already existing file in google drive api v3 java

I have tried with the following code... Without any luck...
private void updateFile(Drive service, String fileId) {
try {
File file = new File(); /********/
final java.io.File fileToUpdate = new java.io.File("D:/Work Data/Files/pdf.pdf");
FileContent mediaContent = new FileContent("image/pdf", fileToUpdate);
file = service.files().update(fileId, file, mediaContent).execute();
System.out.println(fileId);
} catch (Exception e) {
if (isDebug) {
e.printStackTrace();
}
}
}
also tried with...
private void updateFile(Drive service, String fileId) {
try {
File file = service.files().get(fileId).execute(); /********/
final java.io.File fileToUpdate = new java.io.File("D:/Work Data/Files/pdf.pdf");
FileContent mediaContent = new FileContent("image/pdf", fileToUpdate);
file = service.files().update(fileId, file, mediaContent).execute();
System.out.println(fileId);
} catch (Exception e) {
if (isDebug) {
e.printStackTrace();
}
}
}
With every time i execute the code i get the following stacktrace:
java.lang.IllegalArgumentException
at com.google.api.client.repackaged.com.google.common.base.Preconditions.checkArgument(Preconditions.java:111)
at com.google.api.client.util.Preconditions.checkArgument(Preconditions.java:37)
at com.google.api.client.googleapis.media.MediaHttpUploader.setInitiationRequestMethod(MediaHttpUploader.java:872)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.initializeMediaUpload(AbstractGoogleClientRequest.java:237)
at com.google.api.services.drive.Drive$Files$Update.<init>(Drive.java:3163)
at com.google.api.services.drive.Drive$Files.update(Drive.java:3113)
at com.test.DriveTester.updateFile(DriveTester.java:76)
at com.test.DriveTester.main(DriveTester.java:64)
Can anyone tell what i am doing wrong ? Any sample code for this i.e. updating the content of an already existing file on google drive will be helpful...
For API v3 the solution proposed by Android Enthusiast does not work unfortunately.
The issue is with this bit:
// First retrieve the file from the API.
File file = service.files().get(fileId).execute();
doing this it will create a File object, with it's ID field being set, when executing the update, itt will throw an exception, since the ID meta field is not editable directly.
What you can do is simply create a new file:
File file = new File();
alter the meta you'd like and update file content if required as shown in the example.
then simply update the file as proposed above:
// Send the request to the API.
File updatedFile = service.files().update(fileId, file, mediaContent).execute();
So based a full example would look like this based on Android Enthusiast solution:
private static File updateFile(Drive service, String fileId, String newTitle,
String newDescription, String newMimeType, String newFilename, boolean newRevision) {
try {
// First create a new File.
File file = new File();
// File's new metadata.
file.setTitle(newTitle);
file.setDescription(newDescription);
file.setMimeType(newMimeType);
// File's new content.
java.io.File fileContent = new java.io.File(newFilename);
FileContent mediaContent = new FileContent(newMimeType, fileContent);
// Send the request to the API.
File updatedFile = service.files().update(fileId, file, mediaContent).execute();
return updatedFile;
} catch (IOException e) {
System.out.println("An error occurred: " + e);
return null;
}
}
I can share javascript code for uploading to an already existing file using v3
const url = 'https://www.googleapis.com/upload/drive/v3/files/' + fileId + '?uploadType=media';
if(self.fetch){
var setHeaders = new Headers();
setHeaders.append('Authorization', 'Bearer ' + authToken.access_token);
setHeaders.append('Content-Type', mime);
var setOptions = {
method: 'PATCH',
headers: setHeaders,
body: data
};
fetch(url,setOptions)
.then(response => { if(response.ok){
console.log("save to drive");
}
else{
console.log("Response wast not ok");
}
})
.catch(error => {
console.log("There is an error " + error.message);
});
To update files content, you can use Files:update, this method supports an /upload URI and accepts uploaded media with the following characteristics:
Maximum file size: 5120GB
Accepted Media MIME types: /
This method provides media upload functionality through two separate URIs. For more details, see the document on media upload.
Upload URI, for media upload requests:
PUT https://www.googleapis.com/upload/drive/v2/files/fileId
* Metadata URI, for metadata-only requests:
PUT https://www.googleapis.com/drive/v2/files/fileId
private static File updateFile(Drive service, String fileId, String newTitle,
String newDescription, String newMimeType, String newFilename, boolean newRevision) {
try {
// First retrieve the file from the API.
File file = service.files().get(fileId).execute();
// File's new metadata.
file.setTitle(newTitle);
file.setDescription(newDescription);
file.setMimeType(newMimeType);
// File's new content.
java.io.File fileContent = new java.io.File(newFilename);
FileContent mediaContent = new FileContent(newMimeType, fileContent);
// Send the request to the API.
File updatedFile = service.files().update(fileId, file, mediaContent).execute();
return updatedFile;
} catch (IOException e) {
System.out.println("An error occurred: " + e);
return null;
}
}
You may also check this SO ticket, the ticket discuss the said error.
Source
Unlike the other answers, this solution allows you to upload without having to create a file for the data on the device first.
// Create a File containing any metadata changes.
File metadata = new File().setName(name);
// Convert content to an AbstractInputStreamContent instance.
ByteArrayContent contentStream = ByteArrayContent.fromString("text/plain", content);
// Update the metadata and contents.
mDriveService.files().update(fileId, metadata, contentStream).execute();

How to check if a folder exists?

I am playing a bit with the new Java 7 IO features. Actually I am trying to retrieve all the XML files in a folder. However this throws an exception when the folder does not exist. How can I check if the folder exists using the new IO?
public UpdateHandler(String release) {
log.info("searching for configuration files in folder " + release);
Path releaseFolder = Paths.get(release);
try(DirectoryStream<Path> stream = Files.newDirectoryStream(releaseFolder, "*.xml")){
for (Path entry: stream){
log.info("working on file " + entry.getFileName());
}
}
catch (IOException e){
log.error("error while retrieving update configuration files " + e.getMessage());
}
}
Using java.nio.file.Files:
Path path = ...;
if (Files.exists(path)) {
// ...
}
You can optionally pass this method LinkOption values:
if (Files.exists(path, LinkOption.NOFOLLOW_LINKS)) {
There's also a method notExists:
if (Files.notExists(path)) {
Quite simple:
new File("/Path/To/File/or/Directory").exists();
And if you want to be certain it is a directory:
File f = new File("/Path/To/File/or/Directory");
if (f.exists() && f.isDirectory()) {
...
}
To check if a directory exists with the new IO:
if (Files.isDirectory(Paths.get("directory"))) {
...
}
isDirectory returns true if the file is a directory; false if the file does not exist, is not a directory, or it cannot be determined if the file is a directory or not.
See: documentation.
Generate a file from the string of your folder directory
String path="Folder directory";
File file = new File(path);
and use method exist.
If you want to generate the folder you sould use mkdir()
if (!file.exists()) {
System.out.print("No Folder");
file.mkdir();
System.out.print("Folder created");
}
You need to transform your Path into a File and test for existence:
for(Path entry: stream){
if(entry.toFile().exists()){
log.info("working on file " + entry.getFileName());
}
}
There is no need to separately call the exists() method, as isDirectory() implicitly checks whether the directory exists or not.
import java.io.File;
import java.nio.file.Paths;
public class Test
{
public static void main(String[] args)
{
File file = new File("C:\\Temp");
System.out.println("File Folder Exist" + isFileDirectoryExists(file));
System.out.println("Directory Exists" + isDirectoryExists("C:\\Temp"));
}
public static boolean isFileDirectoryExists(File file)
{
if (file.exists())
{
return true;
}
return false;
}
public static boolean isDirectoryExists(String directoryPath)
{
if (!Paths.get(directoryPath).toFile().isDirectory())
{
return false;
}
return true;
}
}
We can check files and thire Folders.
import java.io.*;
public class fileCheck
{
public static void main(String arg[])
{
File f = new File("C:/AMD");
if (f.exists() && f.isDirectory()) {
System.out.println("Exists");
//if the file is present then it will show the msg
}
else{
System.out.println("NOT Exists");
//if the file is Not present then it will show the msg
}
}
}
File sourceLoc=new File("/a/b/c/folderName");
boolean isFolderExisted=false;
sourceLoc.exists()==true?sourceLoc.isDirectory()==true?isFolderExisted=true:isFolderExisted=false:isFolderExisted=false;
From SonarLint, if you already have the path, use path.toFile().exists() instead of Files.exists for better performance.
The Files.exists method has noticeably poor performance in JDK 8, and can slow an application significantly when used to check files that don't actually exist.
The same goes for Files.notExists, Files.isDirectory and Files.isRegularFile.
Noncompliant Code Example:
Path myPath;
if(java.nio.Files.exists(myPath)) { // Noncompliant
// do something
}
Compliant Solution:
Path myPath;
if(myPath.toFile().exists())) {
// do something
}

Categories

Resources