Develop Reference

How to check if a file/directory is a protected OS file?

I'm working on a project which, in part, displays all the files in a directory in a JTable, including sub-directories. Users can double-click the sub-directories to update the table with that new directory's content. However, I've run into a problem.
My lists of files are generated with file.listFiles(), which pulls up everything: hidden files, locked files, OS files, the whole kit and caboodle, and I don't have access to all of them. For example, I don't have permission to read/write in "C:\Users\user\Cookies\" or "C:\ProgramData\ApplicationData\". That's ok though, this isn't a question about getting access to these. Instead, I don't want the program to display a directory it can't open. However, the directories I don't have access to and the directories I do are behaving almost exactly the same, which is making it very difficult to filter them out.
The only difference in behavior I've found is if I call listFiles() on a locked directory, it returns null.
Here's the block of code I'm using as a filter:
for(File file : folder.listFiles())
if(!(file.isDirectory() && file.listFiles() == null))
strings.add(file.getName());
Where 'folder' is the directory I'm looking inside and 'strings' is a list of names of the files in that directory. The idea is a file only gets loaded into the list if it's a file or directory I'm allowed to edit. The filtering aspect works, but there are some directories which contain hundreds of sub-directories, each of which contains hundreds more files, and since listFiles() is O(n), this isn't a feasible solution (list() isn't any better either).
However,
file.isHidden() returns false
canWrite()/canRead()/canExecute() return true
getPath() returns the same as getAbsolutePath() and getCanonicalPath()
createNewFile() returns false for everything, even directories I know are ok. Plus, that's a solution I'd really like to avoid even if that worked.
Is there some method or implementation I just don't know to help me see if this directory is accessible without needing to parse through all of its contents?
(I'm running Windows 7 Professional and I'm using Eclipse Mars 4.5.2, and all instances of File are java.io.File).

The problem you have is that you are dealing with File. By all accounts, in 2016, and, in fact, since 2011 (when Java 7 came out), it has been superseded by JSR 203.
Now, what is JSR 203? It is a totally new API to deal with anything file systems and file system objects; and it extend the definition of a "file system" to include what you find on your local machine (the so called "default filesystem" by the JDK) and other file systems which you may use.
Sample page on how to use it: here
Among the many advantages of this API is that it grants access to metadata which you could not access before; for instance, you specifically mention the case, in a comment, that you want to know which files Windows considers as "system files".
This is how you can do it:
// get the path
final Path path = Paths.get(...);
// get the attributes
final DosAttributes attrs = Files.readAttributes(path, DosFileAttributes.class);
// Is this file a "system file"?
final boolean isSystem = attrs.isSystem();
Now, what is Paths.get()? As mentioned previously, the API gives you access to more than one filesystem at a time; a class called FileSystems gives access to all file systems visible by the JDK (including creating new filesystems), and the default file system, which always exists, is given by FileSystems.getDefault().
A FileSystem instance also gives you access to a Path using FileSystem#getPath.
Combine this and you get that those two are equivalent:
Paths.get(a, b, ...)
FileSystems.getDefault().getPath(a, b, ...)
About exceptions: File handles them very poorly. Just two examples:
File#createNewFile will return false if the file cannot be created;
File#listFiles will return null if the contents of the directory pointed by the File object cannot be read for whatever reason.
JSR 203 has none of these drawbacks, and does even more. Let us take the two equivalent methods:
File#createNewFile becomes Files#createFile;
File#listFiles becomes either of Files#newDirectoryStream (or derivatives; see javadoc) or (since Java 8) Files#list.
These methods, and others, have a fundamental difference in behaviour: in the event of a failure, they will throw an exception.
And what is more, you can differentiate what exception this is:
if it is a FileSystemException or derivative, the error is at the filesystem level (for instance, "access denied" is an AccessDeniedException);
if is is an IOException, then the problem is more fundamental.
This answer cannot contain each and every use case of JSR 203; this API is vast, very complete, although not without flaws, but it is infinitely better than what File has to offer in any case.

I faced the very same problem with paths like C://users/myuser/cookies.
I already used JSR203, so the above answer kind of didn't help me.
In my case the important attribute of those files was the hidden one.
I ended up using the FileSystemview, which excluded those files as I wanted.
File[] files = FileSystemView.getFileSystemView().getFiles(new File(strHomeDirectory), !showHidden);

How to validate a filename in JAVA to resolve CWE ID 73(External Control of File Name or Path) using ESAPI?

I am facing this security flaw in my project at multiple places. I don't have any white-list to do a check at every occurrence of this flaw. I want to use ESAPI call to perform a basic blacklist check on the file name. I have read that we can use SafeFile object of ESAPI but cannot figure out how and where.
Below are a few options I came up with, Please let me know which one will work out?
ESAPI.validator().getValidInput() or ESAPI.validator().getValidFileName()

Blacklists are a no-win scenario. This can only protect you against known threats. Any code scanning tool you use here will continue to report the vulnerability... because a blacklist is a vulnerability. See this note from OWASP:
This strategy, also known as "negative" or "blacklist" validation is a
weak alternative to positive validation. Essentially, if you don't
expect to see characters such as %3f or JavaScript or similar, reject
strings containing them. This is a dangerous strategy, because the set
of possible bad data is potentially infinite. Adopting this strategy
means that you will have to maintain the list of "known bad"
characters and patterns forever, and you will by definition have
incomplete protection.
Also, character encoding and OS makes this a problem too. Let's say we accept an upload of a *.docx file. Here's the different corner-cases to consider, and this would be for every application in your portfolio.
Is the accepting application running on a linux platform or an NT platform? (File separators are \ in Windows and / in linux.)
a. spaces are also treated differently in file/directory paths across systems.
Does the application already account for URL-encoding?
Is the file being sent stored in a database or on the system itself?
Is the file you're receiving executable or not? For example, if I rename netcat.exe to foo.docx does your application actually check to see if the file being uploaded contains the magic numbers for an exe file?
I can go on. But I won't. I could write an encyclopedia.
If this is across multiple applications against your company's portfolio it is your ethical duty to state this clearly, and then your company needs to come up with an app/by/app whitelist.
As far as ESAPI is concerned, you would use Validator.getValidInput() with a regex that was an OR of all the files you wanted to reject, ie. in validation.properties you'd do something like: Validator.blackListsAreABadIdea=regex1|regex2|regex3|regex4
Note that the parsing penalty for blacklists is higher too... every input string will have to be run against EVERY regex in your blacklist, which as OWASP points out, can be infinite.
So again, the correct solution is to have every application team in your portfolio construct a whitelist for their application. If this is really impossible (and I doubt that) then you need to make sure that you've stated the risks cited here clearly to management and you refuse to proceed with the blacklist approach until you have written documentation that the company chooses to accept the risk. This will protect you from legal liability when the blacklist fails and you're taken to court.
[EDIT]
The method you're looking for was called HTTPUtilites.safeFileUpload() listed here as acceptance criteria but this was most likely never implemented due to the difficulties I posted above. Blacklists are extremely custom to the application. The best you'll get is a method HTTPUtilities.getFileUploads() which uses a list defined in ESAPI.properties under the key HttpUtilities.ApprovedUploadExtensions
However, the default version needs to be customized as I doubt you want your users uploading .class files and dll to your system.
Also note: This solution is a whitelist and NOT a blacklist.

The following code snippet works to get past the issue CWE ID 73, if the directory path is static and just the filename is externally controlled :
//'DIRECTORY_PATH' is the directory of the file
//'filename' variable holds the name of the file
//'myFile' variable holds reference to the file object
File dir = new File(DIRECTORY_PATH);
FileFilter fileFilter = new WildcardFileFilter(filename);
File[] files = dir.listFiles(fileFilter);
File myFile = null ;
if(files.length == 1 )
myFile = files[0];

Is there a way to tell if a classpath resource is a file or a directory?

For example, this snippet throws a NullPointerException(!) on the stream.read() line, assuming the com.google package exists in a JAR somewhere (Guava, for example).
ClassLoader classLoader = getClass().getClassLoader();
URL resource = classLoader.getResource("com/google");
InputStream stream = resource.openStream();
System.out.println(stream.toString()); // Fine -- stream is not null
stream.read(); // NPE inside FilterInputStream.read()!
If com/google is swapped with a package that's in the file system rather than a JAR, then the snippet doesn't crash at all. In fact, it seems to read the files in that directory, separated by newlines, though I can't imagine that behaviour is specified anywhere.
Is there a way test if the resource path "com/google" points to a "normal" resource file or to a directory?

This is a bit of a mess due to some unspecified behaviour for the protocol handlers involved in loading these resources. In this particular situation, there are two: sun.net.www.protocol.file.Handler and sun.net.www.protocol.jar.Handler, and they each handle the directory case a bit differently. Based on some experiments, here's what they each do:
sun.net.www.protocol.file.Handler:
What this Handler does is open a FileURLConnection, which does exactly what you discovered it did when confronted with a directory. You can check if it's a directory just with:
if (resource.getProtocol().equals("file")) {
return new File(resource.getPath()).isDirectory();
}
sun.net.www.protocol.jar.Handler:
This Handler, on the other hand, opens a JarURLConnection which eventually makes its way to a ZipCoder. If you take a look at that code, you'll notice something interesting: jzentry will come back null from the native JNI call because the JAR zip file does not, in fact, contain a file called com/google, and so it returns null to the stream that wraps it.
However, there is a solution. Although the ZipCoder won't find com/google, it will find com/google/ (this is how most ZIP interfaces work, for some reason). In that case, the jzentry will be found, and it'll just return a null byte.
So, cutting through all these random implementation-specific behaviours, you can probably figure out if it's a directory by first trying to access the resource with a trailing / (which is what URLClassLoaders expect for directories anyway). If ClassLoader.getResource() returns non-null, then it's a directory. If it doesn't, try without the trailing slash. If it returns non-null, it's a file. If it still returns null, then it's not even an existing resource.
Kinda hacky, but I don't think there's anything better. I hope this helps!

There is no safe and generic way to detect this. When you use ClassLoader.getResource(), the ClassLoader can return practically anything in the URL, in principle even something you have never seen before if the ClassLoader implements its own URL scheme (and protocol).
Your only option is to analyze the URL returned by getResource(), the protocol should hint at what it is (e.g. "file://"). But beware, depending on environment it may return things you did not plan for.
But to just access a resource, you don't care where it comes from (you may care if you're debugging a configuration issue, but your code should not care).
In general you should not make assumptions about the returned InputStream's capabilities, i.e. do not rely on it supporting mark/reset etc. The only safe operation would be simply reading the Stream. If an IOException occurs during read it indicates a problem with access to the resource (network connection lost etc.).
EDIT: getResource() should IMO only return resources (e.g. files or zip file entries), but never directories (since they are not resources). However I wouldn't count on every possible ClassLoader to do so, and I'm not sure what the correct behavior is (if its even specified somewhere).

I think that there are 2 solutions.
Naive solution based on analysis of the path itself. If it ends with .jar or .zip or .war or .ear it is a file. Otherwise it is a directory. I think that this approach will work in 99.99% of cases unless somebody tries to make you you to fail on purpose. For example by defining soft link that looks like a directory but is a file or vise versa.
Try to mimic the JVM logic that interprets paths of classpath relatively to the current working directory. So, retrieve current working directory by using new File("."), then take classpath, split it and for each its element use new File(".", classPathElement) unless it is defined using absolute path.
Good luck with this.

Using the JNotify library, how can you tell if a deleted file was a file or a directory?

For those unfamiliar with JNotify, this is a library which provides an easy way to monitor events in a directory.
For instance, when a file gets deleted in the selected folder, the method "fileDeleted" gets called, along with a few parameters. Here's an example of the fileDeleted method:
public void fileDeleted(int wd, String rootPath, String name) {
print("deleted " + rootPath + " : " + name);
}
Now, I would like to know if the deleted file was a file or directory. My usual approach is to create a new File object with the given path, and use the methods isFile() and isDirectory()
However, since this file is already deleted, these methods always return false.
So here's my concrete question: I have the path to a deleted file or directory, how can I tell wether it was a file or a directory? Is there a workaround to this? What's the best practice to do here?
Thank you in advance.

I suggest using a better API for this, like Commons IO. It has this distinction in its interface org.apache.commons.io.monitor.FileAlterationListener and its methods onFile...(), onDirectory...(). Alternatively, and this is probably the best approach, use the new standard feature for this that comes with Java 7, WatchService, as discussed here.

How big is the directory structure you're looking at?
My first instinct is to build an internal representation of the directory structure, using some simple graph traversal algorithm, and then do a lookup every time something is removed to figure out what it was.
<edit>
If you know your directory structure is a strict tree you can use a simple recursion to traverse the file system, and create a map of Files or Strings to boolean, so you can do an easy lookup. Then, once you've got the map built it should be easy to maintain using the JNotify events.
<edit/>
even for medium-sized directories I would think this could be made pretty quick. What is this for? Might there be another way of going about achieving the same goal?

I am facing the same problem. Yet as far as I understand it, Java's WatchService does not allow monitoring of subdirectories, so I cannot use it (task is to monitor changes to a structure containing ~40K folders). I will try and go ahead using the simple (and fallible) heuristic
If it contains a dot ('.'), it's a file.
I will post updates if I come across something more sophisticated...

Rename file onto another in java

I have a file I need to rename to that of an existing file. This is a copy, modify, replace original operation on an existing JAR file. I've got the first two steps done, I just need help with the replace original bit. What's the best way to rename the new version of the JAR to that of the old. The old JAR doesn't need preserving and I don't want to have a copy of the new with its initial name sticking around.
I have commons lang and io already, so if there's a method I've missed, that would be great.

Java.io.File.renameTo(java.io.File)
You might need to call File.delete() first on the original file first - some systems won't rename a file onto an existing file.

You're going to need to create two java.io.File objects: one for the new file, one for the old file.
Lets call these oldFile and newFile.
oldFile.delete()
newFile.renameTo(oldFile);
Edit: mmyers beat me to it.

This should get you reasonably close:
public boolean replaceOldJar(String originalJarPath, java.io.File newJar) {
java.io.File originalJar = new java.io.File(originalJarPath);
if (!originalJar.isFile()) {
return false;
}
boolean deleteOldJarSucceeded = originalJar.delete();
if (!deleteOldJarSucceeded) {
return false;
}
newJar.renameTo(originalJar);
return originalJar.exists();
}

Is there a problem with deleting the old one and renaming the new one?

I'm not completely sure if you are asking simply how to rename the file in the filesystem or if you also want to reload this new version of your jar?
The rename part sounds easy... just use File.renameTo... unfortunately, there are many platform specific problems related to doing this. Some platforms will not allow you to overwrite an existing file, others will not allow you to rename a file so it changes location onto another partition. If you want to make the process completely safe, you need to do the process yourself by removing the old file first and then renaming the new (or copying if a partition change is possible). This is naturally prone to problems if your application/machine crashes while doing this, since it is no longer an atomic operation. You will thus need to add a check to your applications startup process that looks for rename operations that were in the middle when the crash occured. If you are just updating a single file in this way, it should be pretty easy.
However, if you actually want to reload the jar, there are a few more issues to thing about, but you would need to give a bit more detailed view of the situation to get proper advice on how to do it.