How can we identify whether given File is a named pipe? Any way to use same code for both Linux and Windows.
You may have some success with isOther() in the NIO2 Files API. This tells you that it is not a file, not a directory, and not a link. That leaves a named pipe, a unix socket, or a device..... perhaps that's enough for you?
Related
Linux Api has O_TMPFILE flag to be specified with open system call creating unnamed temporary file which cannot be opened by any path. So we can use this to write data to the file "atmoically" and the linkat the given file to the real path. According to the open man page it can be implemented as simple as
char path[1000];
int fd = open("/tmp", O_TMPFILE | O_WRONLY, S_IWUSR);
write(fd, "123456", sizeof("123456"));
sprintf(path, "/proc/self/fd/%d", fd);
linkat(AT_FDCWD, path, AT_FDCWD, "/tmp/1111111", AT_SYMLINK_FOLLOW);
Is there a Java alternative (probably non crossplatform) to do atomic write to a file without writing Linux-specific JNI function? Files.createTempFile does completely different thing.
By atomic write I mean that either it cannot be opened and be read from or it contains all the data required to be writted.
I don't believe Java has an API for this, and it seems to depend on both the OS and filesystem having support, so JNI might be the only way, and even then only on Linux.
I did a quick search for what Cygwin does, seems to be a bit of a hack just to make software work, creating a file with a random name then excluding it only from their own directory listing.
I believe the closest you can get in plain Java is to create a file in some other location (kinda like a /proc/self/fd/... equivalent), and then when you are done writing it, either move it or symbolic link it from the final location. To move the file, you want it on the same filesystem partition so the file contents don't actually need to be copied. Programs watching for the file in say /tmp/ wouldn't see it until the move or sym link creation.
You could possibly play around with user accounts and filesystem permissions to ensure that no other (non SYSTEM/root) program can see the file initially even if they tried to look wherever you hid it.
The class BasicFileAttributes, for examining the properties of a file in the file system, has the method isRegularFile(). Unfortunately, the Javadoc description is rather lacking:
Tells whether the file is a regular file with opaque content.
What does this mean? What exactly is a regular file with opaque content? I can tell from the other methods in the class that it's not a directory or symbolic link, so I'm inclined to think that it's everything else. However, there apparently is some type of "irregular file" because a method exists called isOther(), which returns true if it's not a directory, symbolic link, or "regular file".
So what exactly is an regular file in Java?
For example in UNIX, a regular file is one that is not special in some way. Special files include symbolic links and directories. A regular file is a sequence of bytes stored permanently in a file system.
Read this answer # UNIX & Linux stackexchange: What is a regular file?
I figure rm -i is an alias, possibly rm -i. The "regular" part doesn't
mean anything in particular, it only means that it's not a pipe,
device, socket or anything other "special".
it means the file is not a symlink, pipe, rand, null, cpu, etc.
Perhaps you have heard the linux philosophy everything is a text. This
isn't literally true, but it suggests a dominant operational context
where string processing tools can be applied to filesystem elements
directly. In this case, it means that in a more literal fashion. To
see the detection step in isolation, try the command file, as in file
/etc/passwd or file /dev/null.
From Files Reference - AIX IBM
A file is a collection of data that can be read from or written to. A file can be a program you create, text you write, data you acquire, or a device you use. Commands, printers, terminals, and application programs are all stored in files. This allows users to access diverse elements of the system in a uniform way and gives the operating system great flexibility. No format is implied when a file is created.
There are three types of files
Regular - Stores data (text, binary, and executable).
Directory - Contains information used to access other files.
Special - Defines a FIFO (first-in, first-out) file or a physical device.
Regular files are the most common. When a word processing program is used to create a document, both the program and the document are contained in regular files.
Regular files contain either text or binary information. Text files are readable by the user. Binary files are readable by the computer. Binary files can be executable files that instruct the system to accomplish a job. Commands, shell scripts, and other programs are stored in executable files.
Directories contain information the system needs to access all types of files, but they do not contain the actual file data. As a result, directories occupy less space than a regular file and give the file-system structure flexibility and depth. Each directory entry represents either a file or subdirectory and contains the name of a file and the file's i-node (index node reference) number. The i-node number represents the unique i-node that describes the location of the data associated with the file. Directories are created and controlled by a separate set of commands. See "Directories" in Operating system and device management for more information.
Special files define devices for the system or temporary files created by processes. There are three basic types of special files: FIFO (first-in, first-out), block, and character. FIFO files are also called pipes. Pipes are created by one process to temporarily allow communication with another process. These files cease to exist when the first process finishes. Block and character files define devices.
All this above is from the first link. I've checked in many other sources regarding Operational Systems differences and it seems this one is the most common definition on all sources i've found.
I am not an expert on this but at the first look BasicFileAttributes is not a class but an interface. So whatever a regular file is depends on the implementation of this interface. I can see that there is e.g. the class WindowsFileAttributs that implements this interface.
If you have a look at the OpenJDK version of this class you will find that it is
!isSymbolicLink() && !isDirectory() && !isOther();
Get all other information from the code ;-)
Is there any way in Java to write out to a temporary file securely?
As far as I can tell, the only way to create a temporary file (createTempFile) does't actually open it at the same time, so there's a race condition between file open & file write. Am I missing something? I couldn't find the C source code behind createFileExclusively(String) in UnixFileSystem.java, but I doubt it can really do anything since the file open occurs in the Java code after the temp file is created (unless it tries to do something with file locks?).
The problem
Between when the temporary file is created & you open it, a malicious attacker could unlink that temporary file & put malicious stuff there. For example, an attacker could create a named pipe to read sensitive data. Or similarly if you eventually copy the file by reading it, then the named pipe could just ignore everything written & supply malicious content to be read.
I remember reading of numerous examples of temporary file attacks in the past 10+ years that exploit the race condition between when the name appears in the namespace and when the file is actually opened.
Hopefully a mitigating factor is that Java set's the umask correctly so a less-privileged user can't read/write to the file and typically the /tmp directory restricts permissions properly so that you can't perform an unlink attack.
Of course if you pass a custom directory for the temporary file that's owned by a less-privileged user who's compromised, the user could do an unlink attack against you. Hell, with inotify, it's probably even easier to exploit the race condition than just a brute force loop that does a directory listing.
http://kurt.seifried.org/2012/03/14/creating-temporary-files-securely/
Java
use java.io.File.createTempFile() – some interesting info at http://www.veracode.com/blog/2009/01/how-boring-flaws-become-interesting/
for directories there is a helpful posting at How to create a temporary directory/folder in Java?
Java 7
for files use java.io.File.createTempFile()
for directories use createTempDirectory()
http://docs.oracle.com/javase/7/docs/api/java/nio/file/Files.html
Since Java 7 we have OpenOption.
An object that configures how to open or create a file.
Objects of this type are used by methods such as newOutputStream, newByteChannel, FileChannel.open, and AsynchronousFileChannel.open when opening or creating a file.
Of particular interest is StandardOpenOptions.CREATE_NEW.
Create a new file, failing if the file already exists. The check for the existence of the file and the creation of the file if it does not exist is atomic with respect to other file system operations.
So, you can do something like this:
FileChannel mkstemp() {
Path path = Files.createTempFile(null, null);
Files.delete(path);
return FileChannel.open(path, WRITE, CREATE_NEW);
}
Implementing the same template behaviour is left as exercise to the reader.
Keep in mind that on many systems, just because a file doesn't have a name doesn't at all mean it's inaccessible. For example, on Linux open file descriptors are available in /proc/<pid>/fd/<fdno>. So you should make sure that your use of temporary files is secure even if someone knows / has a reference to the open file.
You might get a more useful answer if you specify exactly what classes of attacks you are trying to prevent.
Secure against other ordinary userid's? Yes, on any properly functioning multi-user system.
Secure against the your own userid or the superuser? No.
My .htaccess file contains url mappings and my script create these entries once a day.
As .htaccess is a hidden file on Server, when the script tries to overwrite this .htaccess file I get
an error message Access Denied.
Is there a way to overwrite the file.
I suspect that the problem here is that you're on Windows, and Windows doesn't especially like filenames that begin with a dot (it thinks you're creating a file with an empty "name", and a "htaccess" extension).
The fastest solution might just be to change the name of the file that Apache's looking for to e.g. htaccess.txt using the AccessFileName directive.
The fact that the name starts with a . has nothing to do with the access permissions.
Check ls -l /path/to/.htaccess output for the user:group and permissions on the file, and make sure that your script executes with sufficient privileges to write the file. This might mean running your script in the crontab(5) of your webserver, or it might mean running chown(1) to change the owner to whoever should be running your script, or using chown(1) to change the group of the file to the group of the program, and then using chmod(1) to allow group-writes.
It depends upon what you really want to accomplish.
Try deleting the file and let your script create it next time it runs (or force the generation). This way the user which runs the script will be the owner of the file, so it should work from then on.
Also check if your script can create files in that directory.
So I'm writing a Java application that uses Simple to store data as xml file, but it is hellishly slow with big files when it stores on a network drive compared to on a local hard drive. So I'd like to store it locally before copying it over to the desired destination.
Is there some smart way to find a temporary local file storage in Java in a system independent way?
E.g. something that returns something such as c:/temp in windows, /tmp in linux, and likewise for other platforms (such as mac). I could use application path but the problem is that the Java application is run from the network drive as well.
Try:
String path = System.getProperty("java.io.tmpdir");
See: http://java.sun.com/javase/6/docs/api/java/lang/System.html#getProperties%28%29
And to add it here for completeness sake, as wic mentioned in his comment, there's also the methods createTempFile(String prefix, String suffix) and createTempFile(String prefix, String suffix, File directory) methods from Java's File class.
System.getProperty("java.io.tmpdir")
The System and Runtime classes are those whose javadocs you should check first when something related to the system is required.
In the spirit of 'let's solve the problem' instead of 'let's answer the specific question':
What type of input stream are you using when reading into Simple? Be sure to use BufferedInputStream (or BufferedReader) - otherwise you are reading one byte/character at a time from the stream, which will be painfully slow when reading a network resource.
Instead of copying the file to local disk, buffer the inputs and you will be good to go.
try System.getProperty("java.io.tmpdir");