Develop Reference

File.exists() returns false for file (directory) that actually exists

TLDR: File.exists() is buggy and i would like to understand why!
I am facing a weird issue (as so often happens) in my Android App. I will try to be as brief as i can.
First, i will show you the code and then provide some additional info. This is not the full code. Just the core of the issue.
Example code:
String myPath = "/storage/emulated/0/Documents";
File directory= new File(myPath);
if (!directory.exists() && !directory.mkdirs()) {
throw new IllegalArgumentException("Could not create the specified directory: " + directory.getAbsolutePath() + ".");
}
Most of the time this works fine. A few times however the exception is thrown which means that the directory did not exist and could not be created. Out of every 100 runs, it works fine on 95-96 times and fails 4-5 times.
I have declared the permissions for storage/read external storage/write external storage in my manifest and asked for the permissions on runtime. The problem does not lie there. (If anything i have too many permissions at this point :D ). After all, if it was a permission issue it would fail every time but in my case it fails at a rate of 4% or 5%.
With the above code i am attempting to create a file that points to the 'Documents' folder. In my app i am actually using String myPath = Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_DOCUMENTS).getPath();
In the specific device where the error occurs this path happens to be "/storage/emulated/0/Documents" and this is why i hardcoded it in the example code i gave you.
If i use a file explorer app on the device (i.e. 'Astro file manager' i can see that the folder does exist and has some contents and also confirm that the path really is "/storage/emulated/0/Documents".
This has never happened to me locally. Only the users of the app experience the issue and i know the issue exists thanks to Firebase/Crashlytics. The users have the exact same tablet as the one i am using for development, namely a Lenovo TB-8504X. (I work for a company and we provide both the software and the hardware).
So, do you have any thoughts on why this issue occurs?
Has anyone ever experienced something similar?
Could the path to the 'Documents' folder sometimes be "/storage/emulated/0/Documents" and sometimes become something else on the same physical device?
I am an experienced Android developer but i am quite novice in Android architecture and the Android filesystem. Could it be that on start-up (when device is powered on or after a reboot) the filesystem has not yet 'mounted' the 'disk' at the point when my code checks if the directory exists? Here i am using the terms 'mount' and 'disk' as loosely as possible. Also my app is actually a launcher/parental control app so it is the first thing that gets fired when device starts. I am almost conviced that this does not make sense at all but at this point i am trying to see the greater picture and explore solutions that transcend typical Android development.
I would really appreciate your help as this issue is starting to get on my nerves.
Looking forward to any helpful responses.
Thanks in advance.
EDIT (27/08/2019) :
I came across this Java Bug Report although it is pretty outdated. According to this, when operating on NFS-mounted volumes, java.io.File.exists ends up performing a stat(2). If the stat fails (which it may do for several reasons), then File.exists (mistakenly) assumes that the file being stat'ed does not exist. Could this be the source of my troubles?
EDIT (28/08/2019) :
Today i am able to add a bounty to this question in an attempt to draw some more attention. I would encourage you to read the question carefully, look through the comments disregarding the one that claims that this has to do with costumer support from Realm. Realm code is indeed the one using the unreliable method but what i want to know is why the method is unreliable. Whether or not Realm can work around this and use some other code instead, is beyond the scope of the question. I simply want to know if one can safely use File.exists() and if not, why?
Once again, thank you all in advance. It would be really important to me to get an answer even if it is overly technical and involves a deeper understanding of NFS file systems, Java, Android, Linux, or whatever!
EDIT (30/08/2019) :
Because some users suggest replacing File.exists() with some other method, i'd like to state that what i am interested in at this point is understating why the method fails and not what one could use instead as a workaround.
Even if i wanted to replace File.exists() with something else, i am not able to do that because this piece of code resides in RealmConfiguration.java file (Read-only) which is part of the Realm Library that i use in my app.
To make things even more clear i will provide two pieces of code. The code i use in my activity and the method that get's called in RealmConfiguration.java as a consequence:
Code i use in my activity :
File myfile = new File("/storage/emulated/0/Documents");
if(myFile.exists()){ //<---- Notice that myFile exists at this point.
Realm.init(this);
config = new RealmConfiguration.Builder()
.name(".TheDatabaseName")
.directory(myFile) //<---- Notice this line of code.
.schemaVersion(7)
.migration(new MyMigration())
.build();
Realm.setDefaultConfiguration(config);
realm = Realm.getDefaultInstance();
}
At this point myFile exists and the code that resides in RealmConfiguration.java get's called.
The RealmConfiguration.java method that crashes :
/**
* Specifies the directory where the Realm file will be saved. The default value is {#code context.getFilesDir()}.
* If the directory does not exist, it will be created.
*
* #param directory the directory to save the Realm file in. Directory must be writable.
* #throws IllegalArgumentException if {#code directory} is null, not writable or a file.
*/
public Builder directory(File directory) {
//noinspection ConstantConditions
if (directory == null) {
throw new IllegalArgumentException("Non-null 'dir' required.");
}
if (directory.isFile()) {
throw new IllegalArgumentException("'dir' is a file, not a directory: " + directory.getAbsolutePath() + ".");
}
------> if (!directory.exists() && !directory.mkdirs()) { //<---- Here is the problem
throw new IllegalArgumentException("Could not create the specified directory: " + directory.getAbsolutePath() + ".");
}
if (!directory.canWrite()) {
throw new IllegalArgumentException("Realm directory is not writable: " + directory.getAbsolutePath() + ".");
}
this.directory = directory;
return this;
}
So, myFile exists in my activity, the Realm code get's called and suddenly myFile no longer exists.. Again i wish to point out that this is not consistent. I am noticing crashes at a rate of 4-5% meaning that most of the time myFile exists both in the activity and when the realm code makes it's check.
I hope this will be helpful.
Again thanks in advance!

First of all, if you are using Android, bug reports in the Java Bugs database are not relevant. Android does not use the Sun / Oracle codebase. Android started out as a clean-room re-implementation of the Java class libraries.
So if there are bugs in File.exists() on Android the bugs would be in the Android codebase, and any reports would be in the Android issue tracker.
But when you say this:
According to this, when operating on NFS-mounted volumes, java.io.File.exists ends up performing a stat(2). If the stat fails (which it may do for several reasons), then File.exists (mistakenly) assumes that the file being stat'ed does not exist.
Unless you are using NFS, that bug report is not directly relevant.
It is not a mistake / bug. It is a limitation.
At the file system level, it is a fact of life that Linux supports many different kinds of file system, and that many of them behave in unexpected ways ... compared to an "ordinary" file system. It is not possible for the JVM to hide all of the weird filesystem-specific edge cases at the Java API level.
On the API level, File.exists cannot report any errors. The signature doesn't allow it to throw an IOException, and throwing an unchecked exception would be a breaking change. All it can say is true or false.
If you want to distinguish the various reasons for a false, you should use the newer Files.exists(Path, LinkOptions...) method instead.
Could this be the source of my troubles?
Yes it could, and not just in the NFS case! See below. (With Files.exist, an NFS stat failure would most likely be an EIO, and that would raise an IOException rather than returning false.)
The File.java code in the Android codebase (version android-4.2.2_r1) is:
public boolean exists() {
return doAccess(F_OK);
}
private boolean doAccess(int mode) {
try {
return Libcore.os.access(path, mode);
} catch (ErrnoException errnoException) {
return false;
}
}
Note how it turns any ErrnoException into a false.
A bit more digging reveals that the os.access call is performing a native call which makes an access syscall, and throws ErrnoException if the syscall fails.
So now we need look at the documented behavior of the access syscall. Here's what man 2 access says:
F_OK tests for the existence of the
file.
On error (at least one bit in mode
asked for a permission that is denied, or mode is F_OK and the file
does not exist, or some other error occurred), -1 is returned, and
errno is set appropriately.
access() shall fail if:
EACCES The requested access would be denied to the file, or search per‐
mission is denied for one of the directories in the path prefix
of pathname. (See also path_resolution(7).)
ELOOP Too many symbolic links were encountered in resolving pathname.
ENAMETOOLONG
pathname is too long.
ENOENT A component of pathname does not exist or is a dangling symbolic
link.
ENOTDIR
A component used as a directory in pathname is not, in fact, a
directory.
EROFS Write permission was requested for a file on a read-only
filesystem.
access() may fail if:
EFAULT pathname points outside your accessible address space.
EINVAL mode was incorrectly specified.
EIO An I/O error occurred.
ENOMEM Insufficient kernel memory was available.
ETXTBSY
Write access was requested to an executable which is being executed.
I have struck out the errors that I think are technically impossible or implausible, but the still leaves quite few to consider.
Another possibility is something (e.g. some other part of your application) is deleting or renaming the file or a (hypothetical) symlink, or changing file permissions ... behind your back.
But I don't think that File.exist() is broken1, or that the host OS is broken. It is theoretically possible, but you would need some clear evidence to support the theory.
1 - It is not broken in the sense that it is not behaving differently to the known behavior of the method. You could argue until the cows come home about whether the behavior is "correct", but it has been like that since Java 1.0 and it can't be changed in OpenJDK or in Android without breaking thousands of existing applications written over the last 20+ years. It won't happen.
What to do next?
Well my recommendation would be to use strace to track the syscalls that your app is making and see if you can get some clues as to why some access syscalls are giving you unexpected results; e.g. what the paths are and what the errno is. See https://source.android.com/devices/tech/debug/strace .

I have had a similar issue, but with a higher trouble rate, where the Anti Virus was locking FileSystem, and thus failing any requests (almost instantly)
the workaround was using java.nio.Files.exists() instead.

Trying to suppress errors while attaching browser

EDIT: I'm using the LeanFT Java SDK 14.50
EDIT2: for text clarification
I'm writing test scripts for a web application that sometimes opens popup browsers for specific actions. So natually when that happens, I will attach the new browser using BrowserFactory.attach(...). The problem is that leanFT does not seem to have a way to validate that the browser exists before attaching it, and if I try to attach it too early, it will fail. And I don't like to use an arbitrairy wait/sleep time as I can never really know how much time it's going to take for the browser to get be ready. So my solution to this is below
private Browser attachPopUpBrowser(BrowserType bt, RegExpProperty url){
Browser browser = null;
int iteration = 0;
//TimeoutLimit.SHORT = 15000
while (browser == null && iteration < TimeoutLimit.SHORT.getLimit()) {
try {
Reporter.setReportLevel(ReportLevel.Off);
browser = BrowserFactory.attach(
new BrowserDescription.Builder()
.type(bt)
.url(url)
.build()
);
Reporter.setReportLevel(ReportLevel.All);
} catch (GeneralLeanFtException e) {
try {
Thread.sleep(1000);
iteration += 1000;
} catch (InterruptedException e1) {
}
}
}
return browser;
}
Now, this works wonderfully with one exception. It generates errors in the leanft test result. Errors that I want to ignore because I know that it will fail a few times before it will succeed. As you can see, I've tried changing the ReportLevel while doing this in order to suppress the error logging, but it doesn't work. I've tried using
Browser[] browsers = BrowserFactory.getallOpenBrowsers(BrowserDescription);
thinking that it will return an empty Array if it finds nothing, but I still get errors while the browser is not ready. Does anyone have suggestions as to how I could work around this?
TL;DR
I'm looking for a way to either suppress the errors generated within my While..Loop or to validate that the browser is ready before attaching it. All of that, so that I can have a nice and clean Run Result at the end of my test (because these errors will present false negatives in all nearly all of my tests)
Addendum
Also, when the attach fails for the first time, I get a an exception
com.hp.lft.sdk.ReplayObjectNotFoundException: attachApplication
as expected, but all subsequent failures are throwing
com.hp.lft.sdk.GeneralLeanFtException: Cannot read property 'match' of null
I've compared both stack traces and they are identical except for the last 2 lines which happen within the ReplayExceptionFactory.CreateDefault() so I think that there is something that gets corrupted during the exception generation, but that is within the leanft.sdk.internal package so there might not be a lot we can do about it right now.I'm guessing that if I did not get that second "cannot read property" exception, I would correctly get the ReplayObjectNotFoundException until the browser is correctly attached.

I'd rather not force an attach endlessly until it works. Even if we'd solve the false negatives, we'd still have a not so good approach to the problem.
The cleanest solution would be to see if there is anything to attach to in the first place.
And you can do just that by getting all the browser instances that meets your description.
Browser[] browsers = BrowserFactory.getAllOpenBrowsers(new BrowserDescription.Builder().build());
Any element in this collection is an already "attached" browser - you can start using it.
If the list doesn't contain your browser instance, rerun the query.

How to get the intermediate symlink using Java [duplicate]

In a DirectoryWalker class I want to find out if a File instance is actually a symbolic link to a directory (assuming, the walker walks on UNIX systems). Given, I already know the instance is a directory, would the following be a reliable condition to determine the symbolic link?
File file;
// ...
if (file.getAbsolutePath().equals(file.getCanonicalPath())) {
// real directory ---> do normal stuff
}
else {
// possible symbolic link ---> do link stuff
}

The technique used in Apache Commons uses the canonical path to the parent directory, not the file itself. I don't think that you can guarantee that a mismatch is due to a symbolic link, but it's a good indication that the file needs special treatment.
This is Apache code (subject to their license), modified for compactness.
public static boolean isSymlink(File file) throws IOException {
if (file == null)
throw new NullPointerException("File must not be null");
File canon;
if (file.getParent() == null) {
canon = file;
} else {
File canonDir = file.getParentFile().getCanonicalFile();
canon = new File(canonDir, file.getName());
}
return !canon.getCanonicalFile().equals(canon.getAbsoluteFile());
}

Java 1.6 does not provide such low level access to the file system. Looks like NIO 2, which should be included in Java 1.7, will have support for symbolic links. A draft of the new API is available. Symbolic links are mentioned there, creating and following them is possible. I'm not exactly sure that which method should be used to find out whether a file is a symbolic link. There's a mailing list for discussing NIO 2 - maybe they will know.

Also, watch out for file.isFile() and file.isDirectory() both returning results based on the resolved file and therefore both returning false when file refers to a symlink where the target doesn't exist.
(I know this isn't a useful answer in itself but it tripped me up a couple of times so thought I should share)

It looks like getCanonicalPath() can do other things that might make it different from the absolute path.
This method first converts this pathname to absolute form if necessary, as if by invoking the getAbsolutePath() method, and then maps it to its unique form in a system-dependent way. This typically involves removing redundant names such as "." and ".." from the pathname, resolving symbolic links (on UNIX platforms), and converting drive letters to a standard case (on Microsoft Windows platforms).
But it might work for the vast majority of your use cases; your mileage may vary.

If you are already coding something specifically for *nix, then you could do a shell command from Java like this:
Process p = Runtime.getRuntime().exec(new String[]{"test", "-h", yourFileName});
p.waitFor();
if (p.exitValue() == 0)
System.out.println("This file is a symbolic link");
else
System.out.println("This file is not a symbolic link");
That's very specific to *nix, but it does at least work.

Sorry to reply to such an old post, but I was looking for a solution for Windows systems some time back, and some of the previous answers didn't work out for me. If you're not concerned with cross platform compatibility and only need a solution for Windows, the following technique worked well for my purposes.
File f = new File("whatever file or folder");
if (f instanceof ShellFolder) {
ShellFolder sf = (ShellFolder)f;
if (sf.isLink()) {
// Your code when it's a link
}
}

I thought I would share some good fortune I had in dealing with this issue. I am using JDK 1.6.0_23 and so I cannot benefit from NIO2. I am building and running on Windows 7 /x64 ONLY so mileage may vary in other environments. Unfortunately, other solutions here did not work for me in avoiding NullPointerExceptions caused when attempting to traverse a junction (probably because junction != symlink....). While I am not constrained by JDK version, I decided to keep at the problem for a bit longer.
I had this code which would cause a NullPointerException if used on a symbolic link or when encountering the 'System Volume Information' directory. (Note, traverseItem.f() returns an object of type java.io.File)
if (traverseItem.f().isDirectory) {
for (File item : traverseItem.f().listFiles()) {
So, it is supposedly a directory but calling listFiles() on it causes an NPE. What to do? I spied the list() method and wondered if it would exhibit the same behavior. What I discovered was the following:
Calling list() on a File describing an empty folder returns a String[] array of length zero. However, calling list() on a File describing a junction which would otherwise crash from listFiles() returns null
I was able to avoid the NullPointerExceptions by adding the following test before calling listFiles()
String[] contents = traverseItem.f().list();
if (contents != null) { //Non-traversible if null, possibly junction or ???
It remains to exhaustively test all cases of junction, symbolic link, hard link, and dare I mention it, shortcut, but this may help some.

How can I control the usage of a custom jar library?

I need a way to essentially secure my jar library to allow registered apps to use it in their projects and deny usage to apps that weren't approved by me.
It is fine if I hard code things in the lib for each distribution. I currently have this jar obfuscated.
What are good approaches to restrict the usage of a jar?
One idea was to lock the lib to a specific package so if the developer tries to use it in another project they can't. But I'm not sure if they can easily provide a custom fake Context to make it work...

To me the best approach if you would like your library to stay standalone (without involving the network for checking or downloading pieces of the library, I mean) would be to make mandatory the use of an initializer class that would receive a token from the client application.
This would be crackable as the token validity test would be performed by your lib: one may modify the lib in a way is would just skip that test, but this would be made harder by the obfuscation. But this is probably sufficient, unless using your lib without having registered it is a really critical issue.
So you would have something like:
boolean Initializer.initLib(String passcode)
That would prevent the lib to work unless passcode is correct.
You can make the obfuscation more efficient by avoiding checking that way:
public void initLib(String passcode) {
if (passcode == A_GIVEN_PUBLIC_STATIC_THAT_STORESTHE_CODE) {
// do the proper initializations
}
else {
throw new RuntimeException("Bad passcode, sorry!");
}
}
But doing that way instead:
public void initLib(String passcode) {
final char[] PASS_ENCRYPTED = "f5uhjgf56ik8kv214d5".toCharArray();
final char[] PASS_MINUSMASK = "bc".toCharArray();
final int PASS_SHIFT = 11;
final int PASS_MASK_MINUS = 2;
for (int ctr = 0; ctr < PASS_MINUSMASK.length; ++ctr) {
final char next = PASS_ENCRYPTED[PASS_SHIFT + ctr - PASS_MASK_MINUS];
if (passcode.charAt(ctr) != next - (PASS_MINUSMASK[ctr] - 'a')) {
// make the lib unusable by some inits. But it should look as a proper initialization
return;
}
}
// make the lib usable by some inits.
}
This looks stupid, but if you have a look at the obfuscated code, you will see a big difference. This code is just an example (it accepts "hi" as a valid passcode), any algorithm would be fine as long as its obfuscated version is not too straightforward to reverse.
Now the question is: what passcode?
As the library's protection concerns the developpers of the client apps that will use it, and not the final users of these apps, you cannot rely on any piece of data specific to the devices on which the applications will run. So no IMEI or anything like that.
If these developpers are trustworthy that's fine. A fixed passcode is sufficient.
But if they are subject to give this passcode to other people to allow them using your library, this is more difficult. In this case I don't think you can solve it without a real "industrial" process such as registering the client apps and their code checksums, for example. Such a process needs a specific design and cannot be solved "just by the code", but as it also has a cost (time, resources, involvment of the client...) you can only consider this if the use of library is very critical.

Can't you make your jar call your server with a specific code and the application name, to check if they are registered ?

When you build an Android app with a jar, that jar is compiled into the app and becomes a part of it. You can't just copy the jar out of the package and use it elsewhere. Unless I'm not understanding the question, this shouldn't be an issue you need to worry about.

Java 1.6 - determine symbolic links

In a DirectoryWalker class I want to find out if a File instance is actually a symbolic link to a directory (assuming, the walker walks on UNIX systems). Given, I already know the instance is a directory, would the following be a reliable condition to determine the symbolic link?
File file;
// ...
if (file.getAbsolutePath().equals(file.getCanonicalPath())) {
// real directory ---> do normal stuff
}
else {
// possible symbolic link ---> do link stuff
}

The technique used in Apache Commons uses the canonical path to the parent directory, not the file itself. I don't think that you can guarantee that a mismatch is due to a symbolic link, but it's a good indication that the file needs special treatment.
This is Apache code (subject to their license), modified for compactness.
public static boolean isSymlink(File file) throws IOException {
if (file == null)
throw new NullPointerException("File must not be null");
File canon;
if (file.getParent() == null) {
canon = file;
} else {
File canonDir = file.getParentFile().getCanonicalFile();
canon = new File(canonDir, file.getName());
}
return !canon.getCanonicalFile().equals(canon.getAbsoluteFile());
}

Java 1.6 does not provide such low level access to the file system. Looks like NIO 2, which should be included in Java 1.7, will have support for symbolic links. A draft of the new API is available. Symbolic links are mentioned there, creating and following them is possible. I'm not exactly sure that which method should be used to find out whether a file is a symbolic link. There's a mailing list for discussing NIO 2 - maybe they will know.

Also, watch out for file.isFile() and file.isDirectory() both returning results based on the resolved file and therefore both returning false when file refers to a symlink where the target doesn't exist.
(I know this isn't a useful answer in itself but it tripped me up a couple of times so thought I should share)

It looks like getCanonicalPath() can do other things that might make it different from the absolute path.
This method first converts this pathname to absolute form if necessary, as if by invoking the getAbsolutePath() method, and then maps it to its unique form in a system-dependent way. This typically involves removing redundant names such as "." and ".." from the pathname, resolving symbolic links (on UNIX platforms), and converting drive letters to a standard case (on Microsoft Windows platforms).
But it might work for the vast majority of your use cases; your mileage may vary.

If you are already coding something specifically for *nix, then you could do a shell command from Java like this:
Process p = Runtime.getRuntime().exec(new String[]{"test", "-h", yourFileName});
p.waitFor();
if (p.exitValue() == 0)
System.out.println("This file is a symbolic link");
else
System.out.println("This file is not a symbolic link");
That's very specific to *nix, but it does at least work.

Sorry to reply to such an old post, but I was looking for a solution for Windows systems some time back, and some of the previous answers didn't work out for me. If you're not concerned with cross platform compatibility and only need a solution for Windows, the following technique worked well for my purposes.
File f = new File("whatever file or folder");
if (f instanceof ShellFolder) {
ShellFolder sf = (ShellFolder)f;
if (sf.isLink()) {
// Your code when it's a link
}
}

I thought I would share some good fortune I had in dealing with this issue. I am using JDK 1.6.0_23 and so I cannot benefit from NIO2. I am building and running on Windows 7 /x64 ONLY so mileage may vary in other environments. Unfortunately, other solutions here did not work for me in avoiding NullPointerExceptions caused when attempting to traverse a junction (probably because junction != symlink....). While I am not constrained by JDK version, I decided to keep at the problem for a bit longer.
I had this code which would cause a NullPointerException if used on a symbolic link or when encountering the 'System Volume Information' directory. (Note, traverseItem.f() returns an object of type java.io.File)
if (traverseItem.f().isDirectory) {
for (File item : traverseItem.f().listFiles()) {
So, it is supposedly a directory but calling listFiles() on it causes an NPE. What to do? I spied the list() method and wondered if it would exhibit the same behavior. What I discovered was the following:
Calling list() on a File describing an empty folder returns a String[] array of length zero. However, calling list() on a File describing a junction which would otherwise crash from listFiles() returns null
I was able to avoid the NullPointerExceptions by adding the following test before calling listFiles()
String[] contents = traverseItem.f().list();
if (contents != null) { //Non-traversible if null, possibly junction or ???
It remains to exhaustively test all cases of junction, symbolic link, hard link, and dare I mention it, shortcut, but this may help some.