So I'm writing a Java application that uses Simple to store data as xml file, but it is hellishly slow with big files when it stores on a network drive compared to on a local hard drive. So I'd like to store it locally before copying it over to the desired destination.
Is there some smart way to find a temporary local file storage in Java in a system independent way?
E.g. something that returns something such as c:/temp in windows, /tmp in linux, and likewise for other platforms (such as mac). I could use application path but the problem is that the Java application is run from the network drive as well.
Try:
String path = System.getProperty("java.io.tmpdir");
See: http://java.sun.com/javase/6/docs/api/java/lang/System.html#getProperties%28%29
And to add it here for completeness sake, as wic mentioned in his comment, there's also the methods createTempFile(String prefix, String suffix) and createTempFile(String prefix, String suffix, File directory) methods from Java's File class.
System.getProperty("java.io.tmpdir")
The System and Runtime classes are those whose javadocs you should check first when something related to the system is required.
In the spirit of 'let's solve the problem' instead of 'let's answer the specific question':
What type of input stream are you using when reading into Simple? Be sure to use BufferedInputStream (or BufferedReader) - otherwise you are reading one byte/character at a time from the stream, which will be painfully slow when reading a network resource.
Instead of copying the file to local disk, buffer the inputs and you will be good to go.
try System.getProperty("java.io.tmpdir");
Related
Linux Api has O_TMPFILE flag to be specified with open system call creating unnamed temporary file which cannot be opened by any path. So we can use this to write data to the file "atmoically" and the linkat the given file to the real path. According to the open man page it can be implemented as simple as
char path[1000];
int fd = open("/tmp", O_TMPFILE | O_WRONLY, S_IWUSR);
write(fd, "123456", sizeof("123456"));
sprintf(path, "/proc/self/fd/%d", fd);
linkat(AT_FDCWD, path, AT_FDCWD, "/tmp/1111111", AT_SYMLINK_FOLLOW);
Is there a Java alternative (probably non crossplatform) to do atomic write to a file without writing Linux-specific JNI function? Files.createTempFile does completely different thing.
By atomic write I mean that either it cannot be opened and be read from or it contains all the data required to be writted.
I don't believe Java has an API for this, and it seems to depend on both the OS and filesystem having support, so JNI might be the only way, and even then only on Linux.
I did a quick search for what Cygwin does, seems to be a bit of a hack just to make software work, creating a file with a random name then excluding it only from their own directory listing.
I believe the closest you can get in plain Java is to create a file in some other location (kinda like a /proc/self/fd/... equivalent), and then when you are done writing it, either move it or symbolic link it from the final location. To move the file, you want it on the same filesystem partition so the file contents don't actually need to be copied. Programs watching for the file in say /tmp/ wouldn't see it until the move or sym link creation.
You could possibly play around with user accounts and filesystem permissions to ensure that no other (non SYSTEM/root) program can see the file initially even if they tried to look wherever you hid it.
Is there any way in Java to write out to a temporary file securely?
As far as I can tell, the only way to create a temporary file (createTempFile) does't actually open it at the same time, so there's a race condition between file open & file write. Am I missing something? I couldn't find the C source code behind createFileExclusively(String) in UnixFileSystem.java, but I doubt it can really do anything since the file open occurs in the Java code after the temp file is created (unless it tries to do something with file locks?).
The problem
Between when the temporary file is created & you open it, a malicious attacker could unlink that temporary file & put malicious stuff there. For example, an attacker could create a named pipe to read sensitive data. Or similarly if you eventually copy the file by reading it, then the named pipe could just ignore everything written & supply malicious content to be read.
I remember reading of numerous examples of temporary file attacks in the past 10+ years that exploit the race condition between when the name appears in the namespace and when the file is actually opened.
Hopefully a mitigating factor is that Java set's the umask correctly so a less-privileged user can't read/write to the file and typically the /tmp directory restricts permissions properly so that you can't perform an unlink attack.
Of course if you pass a custom directory for the temporary file that's owned by a less-privileged user who's compromised, the user could do an unlink attack against you. Hell, with inotify, it's probably even easier to exploit the race condition than just a brute force loop that does a directory listing.
http://kurt.seifried.org/2012/03/14/creating-temporary-files-securely/
Java
use java.io.File.createTempFile() – some interesting info at http://www.veracode.com/blog/2009/01/how-boring-flaws-become-interesting/
for directories there is a helpful posting at How to create a temporary directory/folder in Java?
Java 7
for files use java.io.File.createTempFile()
for directories use createTempDirectory()
http://docs.oracle.com/javase/7/docs/api/java/nio/file/Files.html
Since Java 7 we have OpenOption.
An object that configures how to open or create a file.
Objects of this type are used by methods such as newOutputStream, newByteChannel, FileChannel.open, and AsynchronousFileChannel.open when opening or creating a file.
Of particular interest is StandardOpenOptions.CREATE_NEW.
Create a new file, failing if the file already exists. The check for the existence of the file and the creation of the file if it does not exist is atomic with respect to other file system operations.
So, you can do something like this:
FileChannel mkstemp() {
Path path = Files.createTempFile(null, null);
Files.delete(path);
return FileChannel.open(path, WRITE, CREATE_NEW);
}
Implementing the same template behaviour is left as exercise to the reader.
Keep in mind that on many systems, just because a file doesn't have a name doesn't at all mean it's inaccessible. For example, on Linux open file descriptors are available in /proc/<pid>/fd/<fdno>. So you should make sure that your use of temporary files is secure even if someone knows / has a reference to the open file.
You might get a more useful answer if you specify exactly what classes of attacks you are trying to prevent.
Secure against other ordinary userid's? Yes, on any properly functioning multi-user system.
Secure against the your own userid or the superuser? No.
I'm basically just doing some tests to figure out a good way to write large amounts of data to a file. I found this great answer Fastest way to write huge data in text file Java and I'm playing around with the script but it seems to be writing to /tmp instead of the directory I'm actually running the program in.
This normally wouldn't bother me, but I'm using EC2 and I've read that the boot storage is not as fast as the ephemeral but because its writing to the boot drive I cannot test if ephemeral is indeed faster or not.
Any suggestions of how to change the temp directory for File?
Check out the Javadocs for this method of java.io.File:
public static File createTempFile(String prefix,
String suffix,
File directory)
throws IOException
This Java code lists files in a directory on a Windows shared drive. Will it work correctly on a Unix system?
File directory = new File("\\\\server/Shared/stuff/mystuff");
for (File file: directory.listFiles()) {
System.out.println(file);
}
Short answer: No.
Long answer: Do you have samba installed? Even then you need to mount the the share. So it probably won't work.
EDIT
Java delegates the call to the underlying OS eventually. Since Unix doesn't know what the \\SERVERNAME path means, Java doesn't know what it means either. What you have to do, to get this to work is mount the drive explicitly using Samba. Your other option, if you are running Ubuntu, is look under .gvfs in your home directory. Ubuntu creates a mount there for your Samba shares, which you should be able to access using Java. If you don't want to rely on external tools, try JCIFS for a pure-Java solution.
No...
Just let the user select the right path and use an OS dependent file-selection dialog.
On my system (Debian Sid with Gnome 2.30 Desktop) I have to select "smb:///server/Shared/..." to achieve the same behaviour. I think, that GVFS (Gnome Virtual File System) using smbfs drivers handles the real connection in the background...
No, as that is a UNC Path, which is a windowsism.
Are you trying to access a windows share from unix? Then have a look at jcifs.
The counter question I get when seeing this is: "Why would you want to hard-code a path in your application?"
Even if it was just for the example and you intend to load the path from a property file or anything, I still think you are on the wrong track here.
First of all you will want to avoid absolute paths like the plague. Relative paths are sort of ok. You can use slash ('/') characters in paths hardcoded, it will work on both Windows and Linux/Mac. Basically all platforms.
Second of all, why use paths at all? This is the internet age. Use URL's! file: URL's will accomplish the same thing as file paths, but using URL's make your app accept resources from other sources such as web sites and FTP as well.
Third of all, avoid the File class. If you invent a good way to do that, you are out of the woodworks completely. Use URL's together with getResource and getResourceAsStream and your app will work platform independent and across network boundaries over the internet.
I have a Java application that runs on Mac and Windows, directly off a CD/DVD with no installation. Now, I need to store a file containing per-user data (think favourites etc.) somewhere on the local file system, so that it can be written to.
So, where do you think is a good location for this file? I am thinking something like:
<USER_DOCUMENTS_AND_SETTINGS>/application data/myapp/favourites.db for windows
<USER_HOME_DIR>/.myapp/favourites.db for mac/nix
Thoughts?
And does anyone know the best way to determine these paths within Java?
System.getProperty("user.home")
As well as user.home, the following is useful for more temporary file storage:
System.getProperty("java.io.tmpdir")
Use the Java Preferences API, designed specifically to store user preferences values in a platform independent way we all like. Java will take care of saving them and retrieving from the file or other backing store depending on the OS.
These tutorials can get you going
You can consider storing the data at Roaming directory under Windows.