Different behavior of ZipOutputStream.setLevel(0) in Linux and Windows env - java

I am trying to optimize a method writeZipResults which take list of ByteOutputStream and convert that into a single zip file as ZipOutputStream.
method definition:
public void writeZipResults(Map<String, ByteArrayOutputStream> files, OutputStream outputStream) throws IOException {
Stopwatch writeZipResultsTimer = Stopwatch.createStarted();
if(files != null) {
ZipOutputStream zip = new ZipOutputStream(outputStream);
for(String filename : files.keySet()) {
ByteArrayOutputStream pdfInMemory = files.get(filename);
if(pdfInMemory != null) {
ZipEntry entry = new ZipEntry(filename + fileTypes.getExtension());
zip.putNextEntry(entry);
pdfInMemory.writeTo(zip);
zip.closeEntry();
pdfInMemory.close();
}
}
zip.close();
logger.info("Took {} ms to complete writeZipResults method, writeZipResultsTimer.elapsed(TimeUnit.MILISECONDS));
}
}
To optimize above method I added zip.setLevel(0) i.e. no compression which minimized the method execution time to great extend in my local window system.
But when I am running the same code with zip.setLevel(0) in linux environment I am not getting same performance as it is under windows system.
To put my point here is the application logs(highlighted in yellow) from Linux and my local windows system captured for same scenario with exactly same data set:
To add more information:
Java Version: 7
Use case: for set of attributes create pdf file for each attribute and combine all pdf files into zip file and return in http response. All file creation process is in memory.
Please suggest how to optimize the method under linux environment?

Related

desktop.moveToTrash(srcFile); not for Java 8 [duplicate]

Java is the key here. I need to be able to delete files but users expect to be able to "undelete" from the recycle bin. As far as I can tell this isn't possible. Anyone know otherwise?
Ten years later, with Java 9, finally there is a builtin way to move files to the Trash Bin
java.awt.Desktop.moveToTrash(java.io.File):
public boolean moveToTrash​(File file)
Moves the specified file to the trash.
Parameters:
file - the file
Returns:
returns true if successfully moved the file to the trash.
The availability of this feature for the underlying platform can be tested with Desktop.isSupported​(Desktop.Action.MOVE_TO_TRASH).
For various reasons Windows has no concept of a folder that simply corresponds to the Recycle Bin.
The correct way is to use JNI to invoke the Windows SHFileOperation API, setting the FO_DELETE flag in the SHFILEOPSTRUCT structure.
SHFileOperation documention
Java example for copying a file using SHFileOperation (the Recycle Bin link in the same article doesn't work)
Java 9 has new method but in my case I am restricted to Java 8.
I found Java Native Access Platform that has hasTrash() and moveToTrash() method. I tested it on Win 10 and Mac OS (Worked) for me.
static boolean moveToTrash(String filePath) {
File file = new File(filePath);
FileUtils fileUtils = FileUtils.getInstance();
if (fileUtils.hasTrash()) {
try {
fileUtils.moveToTrash(new File[] { file });
return true;
} catch (IOException e) {
e.printStackTrace();
return false;
}
} else {
System.out.println("No Trash");
return false;
}
}
Maven Repository
https://mvnrepository.com/artifact/net.java.dev.jna/jna-platform/5.1.0
Don't confuse It is Java Native Access Platform not Java Native Access
See the fileutil incubator project (part of the Java Desktop Integration Components project):
This incubator project is created to host those file utility functionalities, most of which are extensions to the java.io.File class in J2SE. There are frequent requests from Java developers for such features like: sending a file to trash bin, checking free disk space, accessing file attributes etc. This project addresses such frequently requested APIs.
Note, this should work not only on Windows, but on other platforms (Linux, Mac OS X) as well.
My 3 cents - use cmd util Recycle.exe with -f to force recycle (no prompt). Works perfectly.
public class Trash {
public void moveToTrash(File ... file) throws IOException {
moveToTrash(false, file);
}
public void promptMoveToTrash(File ... file) throws IOException {
moveToTrash(true, file);
}
private void moveToTrash(boolean withPrompt, File ... file) throws IOException {
String fileList = Stream.of(file).map(File::getAbsolutePath).reduce((f1, f2)->f1+" "+f2).orElse("");
Runtime.getRuntime().exec("Recycle.exe "+(withPrompt ? "" : "-f ")+fileList);
}
}
In JNA platform, the FileUtils doesn't use Win32 API. You should prefer W32FileUtils which supports Undo (restore the file from recycle bin).
Edit: as of the current version of JNA Platform (5.7.0), with FileUtils.getInstance(), this statement has become incorrect, and FileUtils will use the Win32 API.

How to write and read file in spring boot

I have a problem with saving files and then downloading them after generating a .war file.
I need to handle the generation of many files after pressing the button by admin in the application. The files are generated using part of the code that was sent using the POST method and second part is from the database.
The files are hundreds / thousands and it is impossible to do it manually. Admin generates files from time to time. The user should be able to download these files from the application.
When I run the application in IntelliJ, app has access to the folders on the disk, so the following code works:
(part of backend class, responfible for saving files in path)
private void saveTextToFile(String text, String fileName) {
String filePathAndName = "/static/myFiles/" + fileName+ ".txt";
ClassLoader classLoader = getClass().getClassLoader();
File file = new File(classLoader.getResource(".").getFile() + filePathAndName );
FileWriter fileWriter = null;
try {
fileWriter = new FileWriter(file);
PrintWriter printWriter = new PrintWriter(fileWriter);
printWriter.print(text);
printWriter.close();
} catch (IOException e) {
e.printStackTrace();
}
}
The file was saved in folder:
C:\Users...\myProject\target\classes\static.
(and this is link to generated file in thymeleaf)
<html xmlns:th="http://www.thymeleaf.org">
<a th:href="#{|/myFiles/${thisIsMyFileName}|}">Download file</a>
</html>
Unfortunately, when I generate the .war file and run it, the files are not saved in the application's "resources" folder. As a result, the user cannot download this file via the link generated by thymeleaf.
In general, you do not want to upload anything into your application's files - it opens you to many security problems if someone figures out how to overwrite parts of the application, and in most application servers, it is simply not writable.
A much better approach is to have a designated server folder where you can write things. For example, you could have the following in your configuration:
myapp.base-folder = /any/server/folder/you/want
And then, in the code, you would find that folder as follows:
// env is an #AutoWired private Environment
File baseFolder = new File(env.getProperty("myapp.base-folder"));
I find this better than using a database (as #Stultuske suggested in comments), because databases are great for relations, but mostly overkill for actual files. Files can be accessed externally without firing up the database with minimal hassle, and having them separate keeps your database much easier to backup.
To generate links to the file, simply create a link as you would to any other type of request
<a th:href="#{/file/${fileId}|}">Download file</a>
-- and to handle it in the server, but returning the contents of the file:
#GetMapping(value="/file/{id}")
public StreamingResponseBody getFile(#PathVariable long id) throws IOException {
File f = new File(baseFolder, ""+id); // numerical id prevents filesytem traversal
InputStream in;
if (f.exists()) {
in = new BufferedInputStream(new FileInputStream(f));
} else {
// you could also signal error by returning a 404
in = new BufferedInputStream(getClass().getClassLoader()
.getResourceAsStream("static/img/unknown-id.jpg"));
}
return new StreamingResponseBody() {
#Override
public void writeTo(OutputStream os) throws IOException {
FileCopyUtils.copy(in, os);
}
};
}
I prefer numerical IDs to avoid hassles with path traversal - but you can easily use string filenames instead, and deal with security issues by carefully checking that the canonical path of the requested file starts with the canonical path of your baseFolder

Does java.nio.file.Files.copy call sync() on the file system?

i'm developing an application that has to reboot the system after a file has been uploaded and verified. The file system is on an sd card, so it must be synced to be sure the uploaded file has actually been saved on the device.
I was wondering if java.io.file.Files.copy does the sync or not.
My code runs like this:
public int save(MultipartFile multipart) throws IOException {
Files.copy(multipart.getInputStream(), file, standardCopyOption.REPLACE_EXISTING);
if (validate(file)) {
sync(file); <-- is it useless?
reboot();
return 0;
} else {
Files.delete(file);
return -1;
}
}
I tried to find a way to call sync on the fs in the nio package, but the only solution that i've found is:
public void sync(Path file) {
final FileOutputStream fos = new FileOutputStream(file.toFile());
final FileDescriptor fd = fos.getFD();
fd.sync();
}
which relies on old java.io.File .
If you look at the source code for Files.copy(...), you will see that it doesn't perform a sync(). In the end, it will perform a copy of an input stream into an output stream corresponding to the first 2 arguments passed to Files.copy(...).
Furthermore, the FileDescriptor is tied to the stream from which it is obtained. If you don't perform any I/O operation with this stream, other than creating a file with new FileOutputStream(...), there will be nothing to sync() with the fie system, as is the case with the code you shared.
Thus, the only way I see to accomplish your goal is to "revert" to the old-fashioned java.io API and implement a stream-to-stream copy yourself. This will allow you to sync() on the file descriptor obtained from the same FileOutputStream that is used for the copy operation.
I'll say the copy operation is depending on your OS JRE code, so if you want to be sure of the file Copy at OS level, continue to explicitly call the sync() method.
This was because SYNC and DSYNC were annoyingly omitted from StandardCopyOption enum, yet were provided in StandardOpenOption enum for file targets, so you need to use FileChannel and SeekableByteChannel if supported by FileSystemProvider, like :
Set<? extends OpenOption> TARGET_OPEN_OPTIONS = EnumSet.of(StandardOpenOption.CREATE_NEW, StandardOpenOption.WRITE);
FileChannel.out = target.getFileSystem().provider().newFileChannel(target, TARGET_OPEN_OPTIONS);
SeekableByteChannel = Files.newByteChannel(source, StandardOpenOption.READ);
out.transferFrom(source, 0, source.size());
out.force(boolean metadata); // false == DSYNC, true == SYNC
Using java.io.FileOutputStream.getFD().sync() is an obsolete "solution", because you lose all support for NIO2 FileSystems, like the often bundled ZipFileSystem, and it can still fail if not supported by the native class implementations or OS!
Using DSYNC or SYNC when opening an OutputStream via a FileSystemProvider is another option, but may cause premature flushing of a FileSystem cache.

How should a RESTful server save user submitted images to file?

Currently, I am using ImageIO.write() in order to write to file. However, this method opens up a Java App on my computer, which then forcefully aborts the Bootstrap process if closed, thereby killing the 'server'. I'm testing locally, using IntelliJ, and the termination of the Bootstrap process means that we are unable to test the functionality without rebooting the server.
My method is below. It runs on an API call from our front-end.
/**
* Saves image to database, assuming that the input is not null or empty.
* #param filename name of file.
* #param fileext extension of file.
* #param uri uri in string form.
*/
public static void saveImageToDisk(String filename, String fileext, String uri) {
try {
String[] components = uri.split(",");
String img64 = components[1];
byte[] decodedBytes = DatatypeConverter.parseBase64Binary(img64);
BufferedImage bfi = ImageIO.read(new ByteArrayInputStream(decodedBytes));
File outputfile = new File(IMAGESTORAGEFOLDER + filename + "." + fileext);
ImageIO.write(bfi, fileext, outputfile);
bfi.flush();
} catch(Exception e) {
e.printStackTrace();
}
}
My question is as follows: How can I save an image (from Raw Data) to file without the server aborting? If my code can be adapted with minimal rewrite, what other improvements can I make to robustify my existing code? I would like a solution with no external dependencies (relying entirely on standard Java libraries).
I am on MacOSX, running IntelliJ IDEA CE. Our server runs with Spark and uses Maven.
Thank you very much.
ImageIO.write() [...] method opens up a Java App on my computer
The issue here is that when you use the ImageIO class, it will also initialize the AWT because of some dependencies in the Java2D class hierarchy. This causes the Java launcher on OS X to also open up an icon in the dock and some other things, and I believe this is what you experience. There's really no new Java application being launched.
You can easily avoid this by passing a system property to the Java launcher at startup, telling it to run in "headless" mode. This is usually appropriate for a server process. Pass the following on the command line (or in the IntelliJ launch dialog):
-Djava.awt.headless=true
Read more about headless mode from Oracle's pages. Headless mode is the cross-platform way of doing this. There's also an OS X/MacOS specific way to hide the icon from the dock (-Dapple.awt.UIElement=true, but I don't recommend that here.
However, for your use case it's better to avoid the usage of ImageIO altogether. It's easier, more compatible, faster, and uses less memory as a bonus. Simply write the Base64 decoded bytes directly to disk. There's no need to treat a file containing an image differently from any other file in this case.
You can rewrite your method as follows:
public static void saveImageToDisk(String filename, String fileext, String uri) {
try {
String[] components = uri.split(",");
String img64 = components[1];
byte[] decodedBytes = DatatypeConverter.parseBase64Binary(img64);
File outputfile = new File(IMAGESTORAGEFOLDER, filename + "." + fileext);
Paths.write(outputFile.toPath(), decodedBytes);
} catch(Exception e) {
// You really shouldn't swallow this exception, but I'll leave that to you...
e.printStackTrace();
}
}
After running multiple users at the same time, running the process multiple times, etc, it seems to just be an artifact of either Java's ImageIO or IntelliJ. As long as the new process is not closed, Bootstrap continues to run properly, even if multiple browsers try to upload images, etc.

How to read a file from remote system using java?

I have a file copied in one computer and I need to access the file from other computer.
I am not sure, which protocol or which technology to use for this?
Please provide me any hints for this..
Update:
I am using Ubuntu Linux system.
I used the code :
File f = new File("//192.168.1.157/home/renjith/picture.jpg");// 192.168.1.157 is the ip of the computer, where I have the picture file
Image image = ImageIO.read(f);
But it is giving an exception:
javax.imageio.IIOException: Can't read input file!
at javax.imageio.ImageIO.read(ImageIO.java:1275)
I have shared renjith folder also.
There are any number of ways to access files on remote machines, but they virtually all depend on the remote machine having been set up to provide the file in some way first. If you with to access files via java, the easiest method would probably be to set up an HTTP server on the remote machine (this can be done pretty easily using Apache HTTP server on a variety of platforms) and then using Apache Commons HTTPClient on the client side java app. Further discussion of how to install these or configure them is generally beyond the scope of Stack Overflow and would at least require a more specific question
HTTP is an option. However, if these are Windows machines on the same LAN, it would be easier to expose the directory on the remote machine via a file share and access the file through a regular file path. Similarly, if these are Unix-like machines, you could use regular file paths if you're using NFS. FTP's yet another option.
if the remote computer is in the same network and on a shared folder to the computer where your java code is running then try this piece of code for accessing it
File file = new File("\\\\Comp-1\\FileIO\\Stop.txt");
here Comp-1 is the DNS name of the machine containing the file in the network!!!
You might try:
URL url = new URL("file://192.168.1.157/home/renjith/picture.jpg");
Image image = ImageIO.read(url);
You could try to mount that path first, and then load it. Do a :
subst x: \\192.168.1.157
and then:
File f = new File("x:\\home\\renjith\\picture.jpg");
Image image = ImageIO.read(f)
It should work.
Share the directory and access the file thruogh java code
try this one:
File f = new File("//10.22.33.122/images")
File[] files = f.listFiles(new FilenameFilter() {
public boolean accept(File dir, String name) {
// Specify the extentions of files to be included.
return name.endsWith(".bmp") || name.endsWith(".gif");
}
});
// get names of the files
String[] fileNamesArray = null;
for (int indx = 0; indx < files.length(); indx++) {
fileNamesArray[indx] = files[indx].getName();
}
return fileNamesArray;
Map your IP to network drive and try let us say the drive letter is X,
then code changes to File f = new File("x:\\home\\renjith\\picture.jpg");
Infact your file is already loaded in object f , try priting the value of the path f.getAbsolutePath() to console and see.. Actual error is with ImageIO
You can read from remote and write to remote using jcifs-1.3.15.jar jar in java but first you need to share location from remote system then it's possible.
try{
String strLine="";
NtlmPasswordAuthentication auth = new NtlmPasswordAuthentication("WORKGROUP", "username", "passwd"); // Authentication info here, domain can be null
// try (InputStream is = new SmbFile("smb://DESKTOP-0xxxx/usr/local/cache/abc.txt", auth).getInputStream()) {
try (InputStream is = new SmbFile("smb://xx.xx.xx.xxx/dina_share/abc.txt", auth).getInputStream()) {
BufferedReader br = new BufferedReader(new InputStreamReader(is));
while ((strLine = br.readLine()) != null) {
System.out.println(strLine);
}
} catch (IOException e) {
e.printStackTrace();
}
String smbURL="smb://xx.xx.xx.xxx/dina_share/abcOther.txt";
SmbFileOutputStream fos = new SmbFileOutputStream(new SmbFile(smbURL,auth));
byte bytes[]="Wellcome to you".getBytes();
fos.write(bytes);
}catch(Exception e){
e.printStackTrace();
}

Categories

Resources