File size limit at aws s3 file upload - java

I am currently implementing aws s3 file upload. Among the implementation features, we are trying to make uploads less than 300KB. The file size is written in the application.yml file.
However, when comparing the file size after taking it out from the source, if you compare it like file.getsize() >= maxSize, an error occurs because of the data type difference. How should the logic be structured?
My code
#RequiredArgsConstructor
#Service
public class FileUploadService {
private final UploadService s3Service;
public String uploadImage(MultipartFile file) {
String original = createFileName(file.getOriginalFilename());
ObjectMetadata objectMetadata = new ObjectMetadata();
objectMetadata.setContentType(file.getContentType());
objectMetadata.setContentLength(file.getSize());
try (InputStream inputStream = file.getInputStream()) {
s3Service.uploadFile(inputStream, objectMetadata, original);
} catch (IOException e) {
throw new IllegalArgumentException(String.format("file Error!! (%s)", file.getOriginalFilename()));
}
System.out.println("file upload");
System.out.println("fila name : ": + file.getOriginalFilename());
System.out.println("file type : " + getFileType(file));
System.out.println("file size : " + getFileSize(file) + " Bytes");
System.out.println("-----------------------------------------");
return "File Upload Success! , FileName : " + file.getOriginalFilename()
+ " , File Size : " + getFileSize(file) + " Bytes";
}
private String createFileName(String originalFileName) {
return UUID.randomUUID().toString().concat(getFileExtension(originalFileName));
}
private String getFileExtension(String fileName) {
try {
return fileName.substring(fileName.lastIndexOf("."));
} catch (StringIndexOutOfBoundsException e) {
throw new IllegalArgumentException(String.format("잘못된 형식의 파일 (%s) 입니다", fileName));
}
}
private long getFileSize(MultipartFile file) {
return file.getSize();
}
private String getFileType(MultipartFile file) {
return file.getContentType();
}
}
application.yml
spring:
profiles:
include:
- aws
- credentials
servlet:
multipart:
max-file-size: 300KB
max-request-size: 1MB
I don't know right now either, so I'll try to keep trying by posting a question and searching!! I think we still have to keep learning.
Thank you
I wrote and tried my own logic, but I couldn't figure it out, so I wrote a question

Related

File.createTempFile in Java getting Incompatible type error

Till now my code works fine where I am creating file in temporary directory and processing it.
But now I am trying to provide specific directory where I actually want to create xml file. So in method createTmpXmlFile
private static Path createTmpXmlFile(final String prefix) {
try {
log.info("Creating temporary file {}{}", prefix, XML_SUFFIX);
return Files.createTempFile(Paths.get(gleifZipFile), prefix, XML_SUFFIX);
} catch (IOException e) {
throw new IllegalStateException("Could not create tmp file at " + prefix + XML_SUFFIX + ". ", e);
}
}
I changed from
return Files.createTempFile(prefix, XML_SUFFIX);
to
return File.createTempFile(prefix, XML_SUFFIX, "/tmp/in");
and I get following error:
java: incompatible types: java.lang.String cannot be converted to java.io.File.
If I change the logic here then its affecting other method that are calling createTmpXmlFile method.
I really don't understand how to resolve this issue. Below is my code:
#Slf4j
public class InputCS implements Runnable {
public static final String XML_SUFFIX = ".xml";
#Value("${gleifdataimporter.file.dir}")
private String gleifZipFile;
private void processleifZipFile() {
final AtomicBoolean isInsideLeiRecord = new AtomicBoolean();
isInsideLeiRecord.set(false);
final StringBuilder currentLeiRecordXml = new StringBuilder();
try (FileSystem zipFs = FileSystems.newFileSystem(jobRunner.getInputZipPath(), null)) {
Path tmpXMLPath = xmlFileFromLeiZipFile(zipFs);
try (Stream<String> lines = Files.lines(tmpXMLPath)) {
AtomicInteger processedLinesCounter = new AtomicInteger();
AtomicInteger currentLineNumber = new AtomicInteger();
lines.sequential().forEach(handleLineAndIncrementLineNumber(isInsideLeiRecord, currentLeiRecordXml, processedLinesCounter, currentLineNumber));
log.info("{} lines of XML file inside LEIF input ZIP file {} processed.", processedLinesCounter.get(), jobRunner.getInputZipPath());
}catch (IOException e) {
throw new IllegalStateException("Problem reading input file at " + jobRunner.getInputZipPath() + ".", e);
} finally {
Files.delete(tmpXMLPath);
}
} catch (IOException e) {
throw new IllegalStateException("Problem reading input file at " + jobRunner.getInputZipPath() + ".", e);
}
}
private Path xmlFileFromLeiZipFile(FileSystem zipFs) { //extracts the xml file from zip file
log.info("Input file {} exists: {}", jobRunner.getInputZipPath(), Files.exists(jobRunner.getInputZipPath()));
Path tmpXmlPath = createTmpXmlFile("leif__" + System.currentTimeMillis());
for (Path rootDir : zipFs.getRootDirectories()) {
try (Stream<Path> files = treeAt(rootDir)) {
log.info("Trying to extract LEIF XML file from ZIP file into {}.", tmpXmlPath);
final Path xmlFileInsideZip = files
.filter(isNotADir())
.filter(Files::isRegularFile)
.findFirst()
.orElseThrow(() -> new IllegalStateException("No file found in LEI ZIP file."));
log.info("Path to LEIF XML file inside ZIP file: {}.", xmlFileInsideZip);
return copyReplacing(xmlFileInsideZip, tmpXmlPath);
}
}
throw new IllegalStateException("No file found in LEI ZIP file " + jobRunner.getInputZipPath() + ".");
}
private static Path createTmpXmlFile(final String prefix) {
try {
log.info("Creating temporary file {}{}", prefix, XML_SUFFIX);
return Files.createTempFile(Paths.get(gleifZipFile), prefix, XML_SUFFIX);
} catch (IOException e) {
throw new IllegalStateException("Could not create tmp file at " + prefix + XML_SUFFIX + ". ", e);
}
}
#NotNull
private static Path copyReplacing(Path from, Path to) {
requireNonNull(from, "Trying to copy from a path, which is null to path " + to + "."); //trying to copy file where no xml file exist in root directory
requireNonNull(to, "Trying to copy from path " + from + " to a path, which is null.");
try {
return Files.copy(from, to, REPLACE_EXISTING);
} catch (IOException e) {
throw new IllegalStateException("Cannot copy from " + from + " to " + to + ". ", e);
}
}
}
As suggested by Slaw, use Files#createTempFile(Path,String,String,FileAttribute...) to specify the directory to create temp file.
Use Paths#get(String,String...) for java 7 or 8, or Path#of(String,String...) for java 11 or later to convert String to Path. Further reading: Paths.get vs Path.of
private static Path createTmpXmlFile(final String prefix) {
try {
// Java 11 or later
// return Files.createTempFile(Path.of("/tmp/in"), prefix, XML_SUFFIX);
// Java 8
return Files.createTempFile(Paths.get("/tmp/in"), prefix, XML_SUFFIX);
} catch (IOException e) {
throw new IllegalStateException("Could not create tmp file at " + prefix + XML_SUFFIX + ". ", e);
}
}
File.createTempFile is expecting a File object as third parameter. Just wrap your "/tmp/in" into a File
=> return File.createTempFile(prefix, XML_SUFFIX, new File("/tmp/in")); and you should be good to go.
so you can do:
File.createTempFile("prefix", "suffix", new File("/tmp/in"));
Or using NIO (recommended)
Files.createTempFile(Paths.get("/tmp/in"), "prefix", "suffix");

Spring Boot - Upload file to server

The code belows prompts a user to select a file on his local repository, enter some input fields and then upload the file to a server. Currently, it will store it in a /tmp folder, as it is created by createTempFile. The file is successfully created, and an object is created with a reference to that file as needed by the business case. Yay!
However, I want to store all files in a seperate and organizable folder like "/uploadedFiles" on the server repository.
I have tried several things, from creating an empty file on the repository folder and then attempting an overwrite on to it, to just copying the uploaded file to the folder. None of what seemed to be easy fixes worked so far, unless I missed something obvious (which I probably did).
The files created all have a long sequence of numbers after the file extension in their names, like "testfile.xls1612634232432"; is this from the buffer of the inputstream?
The code below is how it currently works, with just writing the uploaded file to a temp file in the /tmp directory. I need to get it to any other directory of my choosing, and then eligibly pass it to the object constructor.
The method begins at newTestUpload.
#MultipartConfig
#RestController
#RequestMapping(value = "/Teacher", produces = "text/html;charset=UTF-8")
public class Teacher {
TestController testcont = TestController.getInstance();
#GetMapping("")
#ResponseBody
public String homePage(#RequestParam(value = "file", required = false) String name, HttpServletRequest request,
HttpServletResponse response) {
StringBuilder sb = new StringBuilder();
sb.append("<p> <a href='/Teacher/NewTest'>New Test upload</a></p>\n"
+ "<p><a href='/SelectTest'>Select Test File</a> <button type='button'>Send Test</button></p>"
+ "\n \n \n" + "<p><a>Current Test for students:</a>\n <a href='/getCurrentTest'></a></p>");
return sb.toString();
}
#PostMapping
#RequestMapping("/NewTest")
#ResponseBody
public String newTestUpload(HttpServletRequest request, HttpServletResponse response) {
StringBuilder sb = new StringBuilder();
try {
if (!request.getParameterNames().hasMoreElements()) {
sb.append("<p><form action='' method='post' enctype='multipart/form-data'>"
+ "<label>Enter file</label><input type='file' name='file'>"
+ "<button type='submit'>Upload</button></p>"
+ "<p><form action='/testName'>Test Name: <input type='text' name='name' value=''></p>"
+ "<p><form action='/addInfo'>Comment: <input type='text' comment='comment' value=''></p>"
+ "<p>Answer 1: <input type='text' Answer='answer1' value=''></p>"
+ "<p>Answer 2: <input type='text' Answer='answer2' value=''></p>"
+ "</form>"
+ "<a href='/Teacher'>Back</a>\n");
return sb.toString();
} else if (request.getParameter("name") != "" && request.getParameter("comment") != ""
&& request.getParameter("answer1") != "" && request.getParameter("answer2") != "") {
try {
// This is where the magic happens
Part filePart = request.getPart("file");
String fileName = Paths.get(filePart.getSubmittedFileName()).getFileName().toString();
InputStream fileContent = filePart.getInputStream();
byte[] buffer = new byte[fileContent.available()];
fileContent.read(buffer);
File testExcel = File.createTempFile(fileName, "", null);
OutputStream outStream = new FileOutputStream(testExcel);
outStream.write(buffer);
// double ans1 =
// Double.parseDouble(request.getParameter("answer1"));
// double ans2 =
// Double.parseDouble(request.getParameter("answer2"));
Test test = new Test(testExcel, request.getParameter("name"), request.getParameter("comment"),
request.getParameter("answer1"), request.getParameter("answer2"));
testcont.addTest(test);
testExcel.deleteOnExit();
outStream.close();
sb.append("New test uploaded!<br/>\n<a href='/Teacher'>Back</a>\n" + testExcel.getPath()
+ "<p>_________</p>" + test.getFile().getPath());
return sb.toString();
} catch (Exception e) {
sb.append("<h1>Couldnt insert test</h1>\n" + e.getMessage() + e.getStackTrace() + e.getCause());
response.setStatus(HttpServletResponse.SC_OK);
e.printStackTrace();
return sb.toString();
}
} else {
sb.append("failed<br/>\n<a href='/Teacher/NewTest'>Back</a>\n");
response.setStatus(HttpServletResponse.SC_BAD_REQUEST);
return sb.toString();
}
} catch (Exception e) {
e.printStackTrace();
}
return "";
}
}
Instead of creating a temp file, create file in the directory that you want to create.
OutputStream outStream = new FileOutputStream(new File("<fileName>"));
If you want to create file first without data. you can use below code, it will create directory as well if not already created:
public static void saveToFile(String folderPath, String fileName) throws IOException {
File directory = new File(folderPath);
if (!directory.exists() && !directory.mkdirs()) {
throw new IOException("Directory does not exist and could not be created");
}
String filePath = folderPath + File.separator + fileName;
File theFile = new File(filePath);
if (!theFile.exists()) {
try {
theFile.createNewFile();
} catch (IOException e) {
throw new IOException("Facing issues in creating file " + filePath, e);
}
}
}
File testExcel = File.createTempFile(fileName, "", null);
replaced with:
File testExcel = new File("/tests/", fileName);
testExcel.getParentFile().mkdirs();
did the trick! Works like a charm now.

How to output a directory meta hash value when the user selects that choice?

Beginner here, I need my program to output hash values for 3 things:
Files
Directories
Directory Meta
I have got the files and directories working but i just can't figure out how to do the directory meta hash values. So far when the user enters the directory and chooses choice '2' the directory meta is shown but with no hash value. So all that i need is to have the program output the hash value for directory meta alongside all the other outputs. My code is shown below: (Thanks in advance to anyone who helps)
Main Class
import java.io.File;
import java.io.IOException;
import java.util.Scanner;
public class Driver {
public static void main(String[] args) throws IOException {
Scanner Scanscan = new Scanner(System.in);
HashFunction hf = new HashFunction();
System.out.println("Input File or Directory Path:");
while(true) {
String filename = Scanscan.nextLine();
File inputFile = new File(filename);
long hash = 0;
if(filename.equals("exit")){
if (inputFile.isFile()) {
hash = hf.produceFileHash(inputFile.getAbsolutePath());
System.out.println(String.format("The File Hash Value is: %016X", hash));
} else if (inputFile.isDirectory()) {
System.out.println("Please choose either '1' - Directory Hash OR '2' - Directory Meta");
String Choice = Scanscan.nextLine();
if (Choice.equals("1")) {
hash = hf.produceDirHash(inputFile.getAbsolutePath());
System.out.println(String.format("The Directory Hash Value is: %016X", hash));
//System.exit(0);
} else if (Choice.equals("2")) {
hash = hf.produceDirMetaHash(inputFile.getAbsolutePath());
}
}
}
}
}
}
HashChecker Class
public interface HashChecker {
long produceFileHash(String filename);
long produceDirHash(String path);
long produceDirMetaHash(String path);
}
HashFunction Class
#Override
public long produceDirMetaHash(String path) {
int FileCount = 0;
File dirmeta = new File(path);
File[] listOfFiles = dirmeta.listFiles();
for (File listOfFile : listOfFiles) {
Path file = listOfFile.toPath();
BasicFileAttributes attr = null;
try {
attr = Files.readAttributes(file, BasicFileAttributes.class);
} catch (IOException e) {
e.printStackTrace();
}
if (listOfFile.isFile()) {
FileCount++;
System.out.println("\nFile Number: " + FileCount);
System.out.println("File Name: " + listOfFile.getName());
System.out.println("Path: " + listOfFile.getPath());
System.out.println("Created: " + attr.creationTime());
System.out.println("Last Accessed: " + attr.lastAccessTime());
System.out.println("Last Modified: " + attr.lastModifiedTime());
System.out.println("Regular File: " + attr.isRegularFile());
System.out.println("Size: " + attr.size());
System.out.println("File Name " + listOfFile.getName());
System.out.println("Path " + listOfFile.getPath());
FileCount++;
System.out.println("File Number " + FileCount);
}
}
return FileCount;
}
}
Example of the hash output i need but instead for directory meta
You can use the following approach:
Create a ByteArrayOutputStream.
Encapsulate the ByteArrayOutputStream in an ObjectOutputStream.
Write all the attributes in the ObjectOutputStream.
Use the method toByteArray() to convert all the attributes into bytes.
Do whatever hash you need to with those bytes... For example you can use Arrays#hashCode(…) for the simplest solution, or use MessageDigest for a better solution.
For the FileTime types use toInstant() method to convert them to Serializable.

How to upload file using java ?

Hello i m trying to upload file using java file.. but i don't get it.. i get file size=0 i'm providing here my java code. tell me why i cant upload on particular folder. i want to store my file in particular folder. i am trying to get file size, file name but i got the null value where am i wrong please tell me.
public void updateTesti(ActionRequest actionRequest,ActionResponse actionResponse) throws IOException, PortletException
{
//image upload logic
String folder_for_upload =(getPortletContext().getRealPath("/"));
//String folder=actionRequest.getParameter("uploadfolder");
realPath=getPortletContext().getRealPath("/");
logger.info("RealPath is" + realPath);
logger.info("Folder is :" + folder_for_upload);
try
{
logger.info("Admin is try to upload");
UploadPortletRequest uploadRequest = PortalUtil.getUploadPortletRequest(actionRequest);
if (uploadRequest.getSize("fileName") == 0) {
SessionErrors.add(actionRequest, "error");
}
String sourceFileName = uploadRequest.getFileName("fileName");
File uploadedFile = uploadRequest.getFile("fileName");
System.out.println("Size of uploaded file: " + uploadRequest.getSize("fileName"));
logger.info("Uploded file name is: " + uploadRequest.getFileName("fileName"));
String destiFolder=("/home/ubuntu/liferay/liferay-portal-6.1.1-ce-ga2/tomcat-7.0.27/webapps/imageUpload-portlet/image");
String newsourcefilename = (uploadRequest.getFileName("fileName"));
File newFile = new File(destiFolder +"/"+ newsourcefilename);
logger.info("New file name: " + newFile.getName());
logger.info("New file path: " + newFile.getPath());
InputStream in = new BufferedInputStream(uploadRequest.getFileAsStream("fileName"));
FileInputStream fis = new FileInputStream(uploadedFile);
FileOutputStream fos = new FileOutputStream(newFile);
byte[] bytes_ = FileUtil.getBytes(in);
int i = fis.read(bytes_);
while (i != -1) {
fos.write(bytes_, 0, i);
i = fis.read(bytes_);
}
fis.close();
fos.close();
Float size = (float) newFile.length();
System.out.println("file size bytes:" + size);
System.out.println("file size Mb:" + size / 1048576);
logger.info("File created: " + newFile.getName());
SessionMessages.add(actionRequest, "success");
}
catch (FileNotFoundException e)
{
System.out.println("File Not Found.");
e.printStackTrace();
SessionMessages.add(actionRequest, "error");
}
catch (NullPointerException e)
{
System.out.println("File Not Found");
e.printStackTrace();
SessionMessages.add(actionRequest, "error");
}
catch (IOException e1)
{
System.out.println("Error Reading The File.");
SessionMessages.add(actionRequest, "error");
e1.printStackTrace();
}
}
You need to do this to upload small files < 1kb
File f2 = uploadRequest.getFile("fileupload", true);
They are stored in memory only. I have it in my catch statement incase I get a null pointer - or incase my original file (f1.length) == 0
I have executed your code.It is working as per expectation.There might be something wrong in your jsp page.I am not sure but might be your name attribute is not same as the one which you are using in processAction(assuming that you are using portlet).Parameter is case sensitive,so check it again.
You will find more on below link.It has good explanation in file upload.
http://www.codeyouneed.com/liferay-portlet-file-upload-tutorial/
I went through a file upload code, and when i implement that in my local system what i got is, portlet is saving the file i upload in tomcat/webbapp/abc_portlet_project location, what i dont understand is from where portlet found
String folder = getInitParameter("uploadFolder");
String realPath = getPortletContext().getRealPath("/");
System.out.println("RealPath" + realPath +"\\" + folder); try {
UploadPortletRequest uploadRequest =
PortalUtil.getUploadPortletRequest(actionRequest);
System.out.println("Size: "+uploadRequest.getSize("fileName"));
if (uploadRequest.getSize("fileName")==0)
{SessionErrors.add(actionRequest, "error");}
String sourceFileName = uploadRequest.getFileName("fileName"); File
file = uploadRequest.getFile("fileName");
System.out.println("Nome file:" +
uploadRequest.getFileName("fileName")); File newFolder = null;
newFolder = new File(realPath +"\" + folder);
if(!newFolder.exists()){ newFolder.mkdir(); }
File newfile = null;
newfile = new File(newFolder.getAbsoluteFile()+"\"+sourceFileName);
System.out.println("New file name: " + newfile.getName());
System.out.println("New file path: " + newfile.getPath());
InputStream in = new
BufferedInputStream(uploadRequest.getFileAsStream("fileName"));
FileInputStream fis = new FileInputStream(file); FileOutputStream fos
= new FileOutputStream(newfile);

How to deal with corrupted files that were created but IOException occured?

Could you please suggest how to deal with these situations ? I understand that in the second example, it is very rare that it would happen on unix, is it ? If access rights are alright. Also the file wouldn't be even created. I don't understand why the IOException is there, either it is created or not, why do we have to bother with IOException ?
But in the first example, there will be a corrupted zombie file. Now if you tell the user to upload it again, the same thing may happen. If you can't do that, and the inputstream has no marker. You loose your data ? I really don't like how this is done in Java, I hope the new IO in Java 7 is better
Is it usual to delete it
public void inputStreamToFile(InputStream in, File file) throws SystemException {
OutputStream out;
try {
out = new FileOutputStream(file);
} catch (FileNotFoundException e) {
throw new SystemException("Temporary file created : " + file.getAbsolutePath() + " but not found to be populated", e);
}
boolean fileCorrupted = false;
int read = 0;
byte[] bytes = new byte[1024];
try {
while ((read = in.read(bytes)) != -1) {
out.write(bytes, 0, read);
}
} catch (IOException e) {
fileCorrupted = true;
logger.fatal("IO went wrong for file : " + file.getAbsolutePath(), e);
} finally {
IOUtils.closeQuietly(in);
IOUtils.closeQuietly(out);
if(fileCorrupted) {
???
}
}
}
public File createTempFile(String fileId, String ext, String root) throws SystemException {
String fileName = fileId + "." + ext;
File dir = new File(root);
if (!dir.exists()) {
if (!dir.mkdirs())
throw new SystemException("Directory " + dir.getAbsolutePath() + " already exists most probably");
}
File file = new File(dir, fileName);
boolean fileCreated = false;
boolean fileCorrupted = false;
try {
fileCreated = file.createNewFile();
} catch (IOException e) {
fileCorrupted = true;
logger.error("Temp file " + file.getAbsolutePath() + " creation fail", e);
} finally {
if (fileCreated)
return file;
else if (!fileCreated && !fileCorrupted)
throw new SystemException("File " + file.getAbsolutePath() + " already exists most probably");
else if (!fileCreated && fileCorrupted) {
}
}
}
I really don't like how this is done in Java, I hope the new IO in Java 7 is better
I'm not sure how Java is different than any other programming language/environment in the way you are using it:
a client sends some data to your over the wire
as you read it, you write it to a local file
Regardless of the language/tools/environment, it's possible for the connection to be interrupted or lost, for the client to go away, for the disk to die, or for any other error to occur. I/O errors can occur in any and all environments.
What you can do in this situation is highly dependent on the situation and the error that occured. For example, is the data structured in some way where you could ask the user to resume uploading from record 1000, for example? However, there is no single solution that fits all here.

Categories

Resources