I'm basically just doing some tests to figure out a good way to write large amounts of data to a file. I found this great answer Fastest way to write huge data in text file Java and I'm playing around with the script but it seems to be writing to /tmp instead of the directory I'm actually running the program in.
This normally wouldn't bother me, but I'm using EC2 and I've read that the boot storage is not as fast as the ephemeral but because its writing to the boot drive I cannot test if ephemeral is indeed faster or not.
Any suggestions of how to change the temp directory for File?
Check out the Javadocs for this method of java.io.File:
public static File createTempFile(String prefix,
String suffix,
File directory)
throws IOException
Related
Linux Api has O_TMPFILE flag to be specified with open system call creating unnamed temporary file which cannot be opened by any path. So we can use this to write data to the file "atmoically" and the linkat the given file to the real path. According to the open man page it can be implemented as simple as
char path[1000];
int fd = open("/tmp", O_TMPFILE | O_WRONLY, S_IWUSR);
write(fd, "123456", sizeof("123456"));
sprintf(path, "/proc/self/fd/%d", fd);
linkat(AT_FDCWD, path, AT_FDCWD, "/tmp/1111111", AT_SYMLINK_FOLLOW);
Is there a Java alternative (probably non crossplatform) to do atomic write to a file without writing Linux-specific JNI function? Files.createTempFile does completely different thing.
By atomic write I mean that either it cannot be opened and be read from or it contains all the data required to be writted.
I don't believe Java has an API for this, and it seems to depend on both the OS and filesystem having support, so JNI might be the only way, and even then only on Linux.
I did a quick search for what Cygwin does, seems to be a bit of a hack just to make software work, creating a file with a random name then excluding it only from their own directory listing.
I believe the closest you can get in plain Java is to create a file in some other location (kinda like a /proc/self/fd/... equivalent), and then when you are done writing it, either move it or symbolic link it from the final location. To move the file, you want it on the same filesystem partition so the file contents don't actually need to be copied. Programs watching for the file in say /tmp/ wouldn't see it until the move or sym link creation.
You could possibly play around with user accounts and filesystem permissions to ensure that no other (non SYSTEM/root) program can see the file initially even if they tried to look wherever you hid it.
I'm writing an Application for Mac in Java that needs to store a few preference files. By default, they seem to be storing to the User folder, but I'd like to store the files in the Library/ApplicationSupport folder, but I can't seem to figure out how to do that.
I've tried File file = new File("/Library/ApplicationSupport/AppName"); then file.mkdir() but it keeps returning false. I've tried adding ~ to the front of /Library/, and that didn't work either.
I've also tried just writing the file to the desired directory on a FileOutputStream, but no luck there either. I'm open to other ways of storing my preference files, I just don't want them stored in an obtrusive way to the user.
Thanks!
You can't write to /Library/ApplicationSupport because your user would not have the permission.
The plain Java io classes don't understand "~", so to write to ~/Library/ApplicationSupport, you need:
System.getProperty("user.home") + "/Library/ApplicationSupport"
Is there any way in Java to write out to a temporary file securely?
As far as I can tell, the only way to create a temporary file (createTempFile) does't actually open it at the same time, so there's a race condition between file open & file write. Am I missing something? I couldn't find the C source code behind createFileExclusively(String) in UnixFileSystem.java, but I doubt it can really do anything since the file open occurs in the Java code after the temp file is created (unless it tries to do something with file locks?).
The problem
Between when the temporary file is created & you open it, a malicious attacker could unlink that temporary file & put malicious stuff there. For example, an attacker could create a named pipe to read sensitive data. Or similarly if you eventually copy the file by reading it, then the named pipe could just ignore everything written & supply malicious content to be read.
I remember reading of numerous examples of temporary file attacks in the past 10+ years that exploit the race condition between when the name appears in the namespace and when the file is actually opened.
Hopefully a mitigating factor is that Java set's the umask correctly so a less-privileged user can't read/write to the file and typically the /tmp directory restricts permissions properly so that you can't perform an unlink attack.
Of course if you pass a custom directory for the temporary file that's owned by a less-privileged user who's compromised, the user could do an unlink attack against you. Hell, with inotify, it's probably even easier to exploit the race condition than just a brute force loop that does a directory listing.
http://kurt.seifried.org/2012/03/14/creating-temporary-files-securely/
Java
use java.io.File.createTempFile() – some interesting info at http://www.veracode.com/blog/2009/01/how-boring-flaws-become-interesting/
for directories there is a helpful posting at How to create a temporary directory/folder in Java?
Java 7
for files use java.io.File.createTempFile()
for directories use createTempDirectory()
http://docs.oracle.com/javase/7/docs/api/java/nio/file/Files.html
Since Java 7 we have OpenOption.
An object that configures how to open or create a file.
Objects of this type are used by methods such as newOutputStream, newByteChannel, FileChannel.open, and AsynchronousFileChannel.open when opening or creating a file.
Of particular interest is StandardOpenOptions.CREATE_NEW.
Create a new file, failing if the file already exists. The check for the existence of the file and the creation of the file if it does not exist is atomic with respect to other file system operations.
So, you can do something like this:
FileChannel mkstemp() {
Path path = Files.createTempFile(null, null);
Files.delete(path);
return FileChannel.open(path, WRITE, CREATE_NEW);
}
Implementing the same template behaviour is left as exercise to the reader.
Keep in mind that on many systems, just because a file doesn't have a name doesn't at all mean it's inaccessible. For example, on Linux open file descriptors are available in /proc/<pid>/fd/<fdno>. So you should make sure that your use of temporary files is secure even if someone knows / has a reference to the open file.
You might get a more useful answer if you specify exactly what classes of attacks you are trying to prevent.
Secure against other ordinary userid's? Yes, on any properly functioning multi-user system.
Secure against the your own userid or the superuser? No.
I am currently working on a program to make sitting charts for my teacher's classroom. I have put all of the data files in the jar. These are read in and put in to a table. After running the main function of the program, it updates the files to match what the tables values are. I know I need to explode the jar and then rejar it during excution in order to edit the files, but I can't find any explination on how to rejar during excution. Does anyone have any ideas?
Short answer:
Put data files outside of the binary and ship together with JAR in a separate folder.
Long one:
It seems like you are approaching the problem from the wrong direction. JAR file is something like an executable (.exe) on Windows platform - a read only binary containing code.
You can (although it is a bad practice) put some resources like data files, multimedia, etc. inside JAR (like you can inside .exe). But a better solution would be to place these resources outside of the binary so you can switch them without recompiling/rebuilding.
If you need to modify the resources on-the-fly while the application is running, you basically have no choice. The data files have to be outside the binary. Once again, you'll never see a Windows .exe file modifying itself while running.
Tomasz is right that the following is bad practice, but it is possible.
The contents of the classpath are read into memory during bootstrapping, however the files are modifiable but their changes will not be reflected after initialisation. I would recommend putting the data into another file, separate to your class files, but if you insist on keeping them together, you could look at:
JarInputStream or ZipInputStream to read the contents of the JAR file
Get the JarEntry for the appropriate file
Read and modify the contents as you desire
JarOutputStream or ZipOutputStream to write the contents back out
Make sure you're not reading the resource through the classpath and that it's coming from a file on disk / network.
So I'm writing a Java application that uses Simple to store data as xml file, but it is hellishly slow with big files when it stores on a network drive compared to on a local hard drive. So I'd like to store it locally before copying it over to the desired destination.
Is there some smart way to find a temporary local file storage in Java in a system independent way?
E.g. something that returns something such as c:/temp in windows, /tmp in linux, and likewise for other platforms (such as mac). I could use application path but the problem is that the Java application is run from the network drive as well.
Try:
String path = System.getProperty("java.io.tmpdir");
See: http://java.sun.com/javase/6/docs/api/java/lang/System.html#getProperties%28%29
And to add it here for completeness sake, as wic mentioned in his comment, there's also the methods createTempFile(String prefix, String suffix) and createTempFile(String prefix, String suffix, File directory) methods from Java's File class.
System.getProperty("java.io.tmpdir")
The System and Runtime classes are those whose javadocs you should check first when something related to the system is required.
In the spirit of 'let's solve the problem' instead of 'let's answer the specific question':
What type of input stream are you using when reading into Simple? Be sure to use BufferedInputStream (or BufferedReader) - otherwise you are reading one byte/character at a time from the stream, which will be painfully slow when reading a network resource.
Instead of copying the file to local disk, buffer the inputs and you will be good to go.
try System.getProperty("java.io.tmpdir");