Develop Reference

File.exists() returns false for file (directory) that actually exists

TLDR: File.exists() is buggy and i would like to understand why!
I am facing a weird issue (as so often happens) in my Android App. I will try to be as brief as i can.
First, i will show you the code and then provide some additional info. This is not the full code. Just the core of the issue.
Example code:
String myPath = "/storage/emulated/0/Documents";
File directory= new File(myPath);
if (!directory.exists() && !directory.mkdirs()) {
throw new IllegalArgumentException("Could not create the specified directory: " + directory.getAbsolutePath() + ".");
}
Most of the time this works fine. A few times however the exception is thrown which means that the directory did not exist and could not be created. Out of every 100 runs, it works fine on 95-96 times and fails 4-5 times.
I have declared the permissions for storage/read external storage/write external storage in my manifest and asked for the permissions on runtime. The problem does not lie there. (If anything i have too many permissions at this point :D ). After all, if it was a permission issue it would fail every time but in my case it fails at a rate of 4% or 5%.
With the above code i am attempting to create a file that points to the 'Documents' folder. In my app i am actually using String myPath = Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_DOCUMENTS).getPath();
In the specific device where the error occurs this path happens to be "/storage/emulated/0/Documents" and this is why i hardcoded it in the example code i gave you.
If i use a file explorer app on the device (i.e. 'Astro file manager' i can see that the folder does exist and has some contents and also confirm that the path really is "/storage/emulated/0/Documents".
This has never happened to me locally. Only the users of the app experience the issue and i know the issue exists thanks to Firebase/Crashlytics. The users have the exact same tablet as the one i am using for development, namely a Lenovo TB-8504X. (I work for a company and we provide both the software and the hardware).
So, do you have any thoughts on why this issue occurs?
Has anyone ever experienced something similar?
Could the path to the 'Documents' folder sometimes be "/storage/emulated/0/Documents" and sometimes become something else on the same physical device?
I am an experienced Android developer but i am quite novice in Android architecture and the Android filesystem. Could it be that on start-up (when device is powered on or after a reboot) the filesystem has not yet 'mounted' the 'disk' at the point when my code checks if the directory exists? Here i am using the terms 'mount' and 'disk' as loosely as possible. Also my app is actually a launcher/parental control app so it is the first thing that gets fired when device starts. I am almost conviced that this does not make sense at all but at this point i am trying to see the greater picture and explore solutions that transcend typical Android development.
I would really appreciate your help as this issue is starting to get on my nerves.
Looking forward to any helpful responses.
Thanks in advance.
EDIT (27/08/2019) :
I came across this Java Bug Report although it is pretty outdated. According to this, when operating on NFS-mounted volumes, java.io.File.exists ends up performing a stat(2). If the stat fails (which it may do for several reasons), then File.exists (mistakenly) assumes that the file being stat'ed does not exist. Could this be the source of my troubles?
EDIT (28/08/2019) :
Today i am able to add a bounty to this question in an attempt to draw some more attention. I would encourage you to read the question carefully, look through the comments disregarding the one that claims that this has to do with costumer support from Realm. Realm code is indeed the one using the unreliable method but what i want to know is why the method is unreliable. Whether or not Realm can work around this and use some other code instead, is beyond the scope of the question. I simply want to know if one can safely use File.exists() and if not, why?
Once again, thank you all in advance. It would be really important to me to get an answer even if it is overly technical and involves a deeper understanding of NFS file systems, Java, Android, Linux, or whatever!
EDIT (30/08/2019) :
Because some users suggest replacing File.exists() with some other method, i'd like to state that what i am interested in at this point is understating why the method fails and not what one could use instead as a workaround.
Even if i wanted to replace File.exists() with something else, i am not able to do that because this piece of code resides in RealmConfiguration.java file (Read-only) which is part of the Realm Library that i use in my app.
To make things even more clear i will provide two pieces of code. The code i use in my activity and the method that get's called in RealmConfiguration.java as a consequence:
Code i use in my activity :
File myfile = new File("/storage/emulated/0/Documents");
if(myFile.exists()){ //<---- Notice that myFile exists at this point.
Realm.init(this);
config = new RealmConfiguration.Builder()
.name(".TheDatabaseName")
.directory(myFile) //<---- Notice this line of code.
.schemaVersion(7)
.migration(new MyMigration())
.build();
Realm.setDefaultConfiguration(config);
realm = Realm.getDefaultInstance();
}
At this point myFile exists and the code that resides in RealmConfiguration.java get's called.
The RealmConfiguration.java method that crashes :
/**
* Specifies the directory where the Realm file will be saved. The default value is {#code context.getFilesDir()}.
* If the directory does not exist, it will be created.
*
* #param directory the directory to save the Realm file in. Directory must be writable.
* #throws IllegalArgumentException if {#code directory} is null, not writable or a file.
*/
public Builder directory(File directory) {
//noinspection ConstantConditions
if (directory == null) {
throw new IllegalArgumentException("Non-null 'dir' required.");
}
if (directory.isFile()) {
throw new IllegalArgumentException("'dir' is a file, not a directory: " + directory.getAbsolutePath() + ".");
}
------> if (!directory.exists() && !directory.mkdirs()) { //<---- Here is the problem
throw new IllegalArgumentException("Could not create the specified directory: " + directory.getAbsolutePath() + ".");
}
if (!directory.canWrite()) {
throw new IllegalArgumentException("Realm directory is not writable: " + directory.getAbsolutePath() + ".");
}
this.directory = directory;
return this;
}
So, myFile exists in my activity, the Realm code get's called and suddenly myFile no longer exists.. Again i wish to point out that this is not consistent. I am noticing crashes at a rate of 4-5% meaning that most of the time myFile exists both in the activity and when the realm code makes it's check.
I hope this will be helpful.
Again thanks in advance!

First of all, if you are using Android, bug reports in the Java Bugs database are not relevant. Android does not use the Sun / Oracle codebase. Android started out as a clean-room re-implementation of the Java class libraries.
So if there are bugs in File.exists() on Android the bugs would be in the Android codebase, and any reports would be in the Android issue tracker.
But when you say this:
According to this, when operating on NFS-mounted volumes, java.io.File.exists ends up performing a stat(2). If the stat fails (which it may do for several reasons), then File.exists (mistakenly) assumes that the file being stat'ed does not exist.
Unless you are using NFS, that bug report is not directly relevant.
It is not a mistake / bug. It is a limitation.
At the file system level, it is a fact of life that Linux supports many different kinds of file system, and that many of them behave in unexpected ways ... compared to an "ordinary" file system. It is not possible for the JVM to hide all of the weird filesystem-specific edge cases at the Java API level.
On the API level, File.exists cannot report any errors. The signature doesn't allow it to throw an IOException, and throwing an unchecked exception would be a breaking change. All it can say is true or false.
If you want to distinguish the various reasons for a false, you should use the newer Files.exists(Path, LinkOptions...) method instead.
Could this be the source of my troubles?
Yes it could, and not just in the NFS case! See below. (With Files.exist, an NFS stat failure would most likely be an EIO, and that would raise an IOException rather than returning false.)
The File.java code in the Android codebase (version android-4.2.2_r1) is:
public boolean exists() {
return doAccess(F_OK);
}
private boolean doAccess(int mode) {
try {
return Libcore.os.access(path, mode);
} catch (ErrnoException errnoException) {
return false;
}
}
Note how it turns any ErrnoException into a false.
A bit more digging reveals that the os.access call is performing a native call which makes an access syscall, and throws ErrnoException if the syscall fails.
So now we need look at the documented behavior of the access syscall. Here's what man 2 access says:
F_OK tests for the existence of the
file.
On error (at least one bit in mode
asked for a permission that is denied, or mode is F_OK and the file
does not exist, or some other error occurred), -1 is returned, and
errno is set appropriately.
access() shall fail if:
EACCES The requested access would be denied to the file, or search per‐
mission is denied for one of the directories in the path prefix
of pathname. (See also path_resolution(7).)
ELOOP Too many symbolic links were encountered in resolving pathname.
ENAMETOOLONG
pathname is too long.
ENOENT A component of pathname does not exist or is a dangling symbolic
link.
ENOTDIR
A component used as a directory in pathname is not, in fact, a
directory.
EROFS Write permission was requested for a file on a read-only
filesystem.
access() may fail if:
EFAULT pathname points outside your accessible address space.
EINVAL mode was incorrectly specified.
EIO An I/O error occurred.
ENOMEM Insufficient kernel memory was available.
ETXTBSY
Write access was requested to an executable which is being executed.
I have struck out the errors that I think are technically impossible or implausible, but the still leaves quite few to consider.
Another possibility is something (e.g. some other part of your application) is deleting or renaming the file or a (hypothetical) symlink, or changing file permissions ... behind your back.
But I don't think that File.exist() is broken1, or that the host OS is broken. It is theoretically possible, but you would need some clear evidence to support the theory.
1 - It is not broken in the sense that it is not behaving differently to the known behavior of the method. You could argue until the cows come home about whether the behavior is "correct", but it has been like that since Java 1.0 and it can't be changed in OpenJDK or in Android without breaking thousands of existing applications written over the last 20+ years. It won't happen.
What to do next?
Well my recommendation would be to use strace to track the syscalls that your app is making and see if you can get some clues as to why some access syscalls are giving you unexpected results; e.g. what the paths are and what the errno is. See https://source.android.com/devices/tech/debug/strace .

I have had a similar issue, but with a higher trouble rate, where the Anti Virus was locking FileSystem, and thus failing any requests (almost instantly)
the workaround was using java.nio.Files.exists() instead.

Java method that writes to file does nothing when invoked from a JSP

Hey, all! I have a class method who's primary function is to get a Map object, which works fine; however, it's an expensive operation that doesn't need to be done every time, so I'd like to have the results stored in an XML file using JAXB, to be read from for the majority of calls and updated infrequently.
When I run a class that calls it out of NetBeans the file is created no problem with exactly what I want -- but when I have my JSP call the method nothing happens whatsoever, even though the rest of the information is passed normally. I have the feeling it's somehow lacking write privileges, but the file is just in the root directory so I'm not sure what I'm missing. Thanks for the help!
The code looks roughly like this:
public class DataHandler() {
...
public void config() {
MapHolder bucket = new MapHolder();
MapExporter exp = new MapExporter();
Map map = makeMap();
bucket.setMap(map);
exp.exportMap(bucket);
}
}
And then the JSP has a javabean of Datahandler, and this line:
databean.config();
It's probably a tad more fragmented than it needs to be; the whole bucket rigamarole was because I was stumbling trying to learn how to write a map to an xml file. Mapholder is just a class that I wrap around the map, and MapExporter just uses a JAXB marshaller, and it all does work properly when run from NetBeans.

OK turns out I'm just dumb; everything was working fine, the file was just being stored in a folder at the localhost location. Whoops! That'd be my inexperience with web development at work.

Issue with running JAR from Desktop, but not from command line or Eclipse

I am running into a peculiar issue (peculiar for me anyways) that seems to happen in a SwingWorker that I use for saving the result of another 'SwingWorker' thread as a tab-delimited file (just a spreadsheet of data).
Here is the worker, that initializes and declares an object which organizes the data and writes each table row to a file (using BufferedWriter):
// Some instance variables outside of the SwingWorker:
// model: holds a matrix of numerical data (double[][])
// view: the GUI class
class SaveWorker extends SwingWorker<Void, Void> {
/* The finished reordered matrix axes */
private String[] reorderedRows;
private String[] reorderedCols;
private String filePath; // the path of the file that will be generated
public SaveWorker(String[] reorderedRows, String[] reorderedCols) {
// variables have been checked for null outside of the worker
this.reorderedRows = reorderedRows;
this.reorderedCols = reorderedCols;
}
#Override
protected Void doInBackground() throws Exception {
if (!isCancelled()) {
LogBuffer.println("Initializing writer.");
final CDTGenerator cdtGen = new CDTGenerator(
model, view, reorderedRows, reorderedCols);
LogBuffer.println("Generating CDT.");
cdtGen.generateCDT();
LogBuffer.println("Setting file path.");
filePath = cdtGen.getFilePath(); // stops inside here, jumps to done()
LogBuffer.println("Path: " + filePath);
}
return null;
}
#Override
protected void done() {
if (!isCancelled()) {
view.setLoadText("Done!");
LogBuffer.println("Done saving. Opening file now.");
// need filePath here to load and then display generated file
visualizeData(filePath);
} else {
view.setReorderOngoing(false);
LogBuffer.println("Reordering has been cancelled.");
}
}
}
When I run the program from Eclipse, this all works perfectly fine. No issues whatsoever. Now I know there have been tons of question on here that are about Eclipse running fine while the runnable JAR fails. It's often due to not including dependencies or referring to them in the wrong way. But what's weird is that the JAR also works completely fine when it's being started from command line (Windows 8.1):
java -jar reorder.jar
Et voilà, everything as expected. The CDTGenerator will finish, write all the matrix rows to a file, and return the filePath. With the filePath I can subsequently open the new file and display the matrix.
In the case of double-clicking the JAR on my desktop, where I placed it when creating it from Eclipse, this is where the program will let me know that stuff happens. I get the error message I created for the case of filePath == null and using some logging I closed in on where the CDTGenerator object stops executing its method generateCDT() (Eclipse debugger also won't reproduce the error and do everything as planned).
What the log shows made me think it's an issue with concurrency, but I am actually leaning against that because Eclipse and command line both run the code fine. The log just tells me that the code suddenly stops executing during a loop which transforms double values from a matrix row (double[]) to Strings to be stored in a String[] for later writing with BufferedWriter.
If I use more logging in that loop, the loop will stop at a different iterator (???).
Furthermore, the code does work for small matrices (130x130) but not for larger ones (1500x3500) but I haven't tested where the limit is. This makes it seem almost time dependent, or memory.
I also used jVisualVM to look at potential memory issues, but even for the larger matrices I am on ~250MB which is nowhere near problematic regarding potential OutOfMemoryExceptions.
And finally, the last potential factor I can think of: Generating the JAR 'fails' due to some classpath issues (clean & rebuild have no effect...) but this has never been an issue before as I have run the code many many times using the 'broken' JAR and execute from Desktop.
I am a real newbie to programming, so please point in some direction if possible. I have tried to find logged exceptions, logged the values of variables, I am checking for null and IndexOutOfBound issues at the array where it stops executing... I am at a complete loss especially because this runs fine from command line.

It looks like the problem had to see with the java versions installed in OP's computer. They checked the file extensions and the programs associated to each one in order to see if it was the same java version as executed from Eclipse and the command line.
Once they cleaned older java versions the jar started to work by double-clicking it :)

Cause I do not have enough points (need 50 to directly answer your question), I need to ask this way:
If you double click a JAR you won't see a console which is often the problem because you can't see stack traces. They get just written to "nowhere". Maybe you get an NPE ore something else.
Try to attach an Exceptionhandler like this Thread.setDefaultUncaughtExceptionHandler(UncaughtExceptionHandler) and let this handler write down a message to a file or such...
Just an idea.

Play Framework await() makes the application act wierd

I am having some strange trouble with the method await(Future future) of the Controller.
Whenever I add an await line anywhere in my code, some GenericModels which have nothing to do with where I placed await, start loading incorrectly and I can not access to any of their attributes.
The wierdest thing is that if I change something in another completely different java file anywhere in the project, play will try to recompile I guess and in that moment it starts working perfectly, until I clean tmp again.

When you use await in a controller it does bytecode enhancement to break a single method into two threads. This is pretty cool, but definitely one of the 'black magic' tricks of Play1. But, this is one place where Play often acts weird and requires a restart (or as you found, some code changing) - the other place it can act strange is when you change a Model class.
http://www.playframework.com/documentation/1.2.5/asynchronous#SuspendingHTTPrequests
To make it easier to deal with asynchronous code we have introduced
continuations. Continuations allow your code to be suspended and
resumed transparently. So you write your code in a very imperative
way, as:
public static void computeSomething() {
Promise delayedResult = veryLongComputation(…);
String result = await(delayedResult);
render(result); }
In fact here, your code will be executed in 2 steps, in 2 different hreads. But as you see it, it’s very
transparent for your application code.
Using await(…) and continuations, you could write a loop:
public static void loopWithoutBlocking() {
for(int i=0; i<=10; i++) {
Logger.info(i);
await("1s");
}
renderText("Loop finished"); }
And using only 1 thread (which is the default in development mode) to process requests, Play is able to
run concurrently these loops for several requests at the same time.
To respond to your comment:
public static void generatePDF(Long reportId) {
Promise<InputStream> pdf = new ReportAsPDFJob(report).now();
InputStream pdfStream = await(pdf);
renderBinary(pdfStream);
and ReportAsPDFJob is simply a play Job class with doJobWithResult overridden - so it returns the object. See http://www.playframework.com/documentation/1.2.5/jobs for more on jobs.
Calling job.now() returns a future/promise, which you can use like this: await(job.now())

Notification when file download completes in Play framework

I'm successfully using Play 1.2.4 to serve large binary file downloads to users using the renderBinary() method.
I'd like to have a hint of when the user actually completes the download. Generally speaking, I know this is somewhat possible as I've done it before. In an old version of my website, I wrote a simple servlet that served up binary file downloads. Once that servlet finished writing out the contents of the file, a notification was sent. Certainly not perfect, but useful nonetheless. In my testing, it did provide an indication of how long the user took to download a file.
Reviewing the Play source, I see that the play.mvc.results.RenderBinary class has a handy apply() method that I could use. I wrote my own version of RenderBinary so I could send the notification after the apply() method finished writing out the file contents.
The problem I found is that calls to response.out.write() obviously cache the outgoing bytes (via Netty?), so even though I am writing out several megabytes of data, the calls to play.mvc.Http.Response.out.write() complete in seconds, even though it takes the downloader a couple minutes to download the file.
I don't mind writing custom classes, although I'd prefer to use a stock Play 1.2.4 distribution.
Any ideas on how to get a notification of when the end of a file download is pushed out towards the user's browser?

It seems this may help you, as it tackles a somehow similar problem:
Detect when browser receives file download
I'm not sure you'll eb able to do it via renderBinary nor an #After annotation in the controller. Some browser-side detection of the download and then a notification to the server (pinging the download's end) would work.
There may be an alternative: using WebSockets (streaming the file via the socket and then having teh client to answer) but it may be overkill for this :)

you can use ArchivedEventStream.
first create a serializable ArcivedEventStream class..
public class Stream<String> extends ArchivedEventStream<String> implements Serializable{
public Stream(int arg0) {
super(arg0);
}
}
then on your controller...
public static void downloadPage(){
Stream<String> userStream = Cache.get(session.getId(),Stream.class);
if( userStream == null){
userStream = new Stream<String>(5);
Cache.add(session.getId(), userStream);
}
render();
}
public static void download(){
await(10000);// to provide some latency. actually no needed
renderBinary(Play.getFile("yourfile!"));
}
public static void isDownloadFinished(){
Stream<String> userStream = Cache.get(session.getId(),Stream.class);
List<IndexedEvent<String>> list = await(userStream.nextEvents(0));
renderJSON(list.get(0).data);
}
#After(only="download")
static void after(){
Stream<String> userStream = Cache.get(session.getId(),Stream.class);
userStream.publish("ok");
}
on your html...
#{extends 'main.html' /}
#{set title:'downloadPage' /}
download
<script>
$(document).ready(function(){
$.ajax('/application/isDownloadFinished',{success:function(data){
if(data){
console.log("downloadFinished");
}
}});
});
</script>
when your download finished, the original page will retrieve the notification.
This code is just a sample. You could read the api of ArchivedEventStream and make your own implementation..