Develop Reference

Possible side effects when several CommandLine instance "work" on the same instance of an annotated class?

picoCLI's #-file mechanism is almost what I need, but not exactly. The reason is that I want to control the exact location of additional files parsed -- depending on previous option values.
Example: When called with the options
srcfolder=/a/b optionfile=of.txt, my program should see the additional options read from /a/b/of.txt, but when called with srcfolder=../c optionfile=of.txt, it should see those from ../c/of.txt.
The #-file mechanism can't do that, because it expands ALL the option files (always relative to the current folder, if they're relative) prior to processing ANY option values.
So I'd like to have picoCLI...
process options "from left to right",
recursively parse an option file when it's mentioned in an optionfile option,
and after that continue with the following options.
I might be able to solve this by recursively starting to parse from within the annotated setter method:
...
Config cfg = new Config();
CommandLine cmd = new CommandLine(cfg);
cmd.parseArgs(a);
...
public class Config {
#Option(names="srcfolder")
public void setSrcfolder(String path) {
this.srcfolder=path;
}
#Option(names="optionfile")
public void parseOptionFile(String pathAndName) {
// validate path, do some other housekeeping...
CommandLine cmd = new CommandLine(this /* same Config instance! */ );
cmd.parseArgs(new String[] { "#"+this.srcfolder + pathAndName });
}
...
This way several CommandLine instances would call setter methods on the same Config instance, recursively "interrupting" each other. Now comes the actual question: Is that a problem?
Of course my Config class has state. But do CommandLine instances also have state that might get messed up if other CommandLine instances also modify cfg "in between options"?
Thanks for any insights!
Edited to add: I tried, and I'm getting an UnmatchedArgumentException on the #-file option:
Exception in thread "main" picocli.CommandLine$UnmatchedArgumentException: Unmatched argument at index 0: '#/path/to/configfile'
at picocli.CommandLine$Interpreter.validateConstraints(CommandLine.java:13490)
...
So first I have to get around this: Obviously picoCLI doesn't expand the #-file option unless it's coming directly from the command line.

I did get it to work: several CommandLine instance can indeed work on the same instance of an annotated class, without interfering with each other.
There are some catches and I had to work around a strange picoCLI quirk, but that's not exactly part of an answer to this question, so I explain them in this other question.

File.exists() returns false for file (directory) that actually exists

TLDR: File.exists() is buggy and i would like to understand why!
I am facing a weird issue (as so often happens) in my Android App. I will try to be as brief as i can.
First, i will show you the code and then provide some additional info. This is not the full code. Just the core of the issue.
Example code:
String myPath = "/storage/emulated/0/Documents";
File directory= new File(myPath);
if (!directory.exists() && !directory.mkdirs()) {
throw new IllegalArgumentException("Could not create the specified directory: " + directory.getAbsolutePath() + ".");
}
Most of the time this works fine. A few times however the exception is thrown which means that the directory did not exist and could not be created. Out of every 100 runs, it works fine on 95-96 times and fails 4-5 times.
I have declared the permissions for storage/read external storage/write external storage in my manifest and asked for the permissions on runtime. The problem does not lie there. (If anything i have too many permissions at this point :D ). After all, if it was a permission issue it would fail every time but in my case it fails at a rate of 4% or 5%.
With the above code i am attempting to create a file that points to the 'Documents' folder. In my app i am actually using String myPath = Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_DOCUMENTS).getPath();
In the specific device where the error occurs this path happens to be "/storage/emulated/0/Documents" and this is why i hardcoded it in the example code i gave you.
If i use a file explorer app on the device (i.e. 'Astro file manager' i can see that the folder does exist and has some contents and also confirm that the path really is "/storage/emulated/0/Documents".
This has never happened to me locally. Only the users of the app experience the issue and i know the issue exists thanks to Firebase/Crashlytics. The users have the exact same tablet as the one i am using for development, namely a Lenovo TB-8504X. (I work for a company and we provide both the software and the hardware).
So, do you have any thoughts on why this issue occurs?
Has anyone ever experienced something similar?
Could the path to the 'Documents' folder sometimes be "/storage/emulated/0/Documents" and sometimes become something else on the same physical device?
I am an experienced Android developer but i am quite novice in Android architecture and the Android filesystem. Could it be that on start-up (when device is powered on or after a reboot) the filesystem has not yet 'mounted' the 'disk' at the point when my code checks if the directory exists? Here i am using the terms 'mount' and 'disk' as loosely as possible. Also my app is actually a launcher/parental control app so it is the first thing that gets fired when device starts. I am almost conviced that this does not make sense at all but at this point i am trying to see the greater picture and explore solutions that transcend typical Android development.
I would really appreciate your help as this issue is starting to get on my nerves.
Looking forward to any helpful responses.
Thanks in advance.
EDIT (27/08/2019) :
I came across this Java Bug Report although it is pretty outdated. According to this, when operating on NFS-mounted volumes, java.io.File.exists ends up performing a stat(2). If the stat fails (which it may do for several reasons), then File.exists (mistakenly) assumes that the file being stat'ed does not exist. Could this be the source of my troubles?
EDIT (28/08/2019) :
Today i am able to add a bounty to this question in an attempt to draw some more attention. I would encourage you to read the question carefully, look through the comments disregarding the one that claims that this has to do with costumer support from Realm. Realm code is indeed the one using the unreliable method but what i want to know is why the method is unreliable. Whether or not Realm can work around this and use some other code instead, is beyond the scope of the question. I simply want to know if one can safely use File.exists() and if not, why?
Once again, thank you all in advance. It would be really important to me to get an answer even if it is overly technical and involves a deeper understanding of NFS file systems, Java, Android, Linux, or whatever!
EDIT (30/08/2019) :
Because some users suggest replacing File.exists() with some other method, i'd like to state that what i am interested in at this point is understating why the method fails and not what one could use instead as a workaround.
Even if i wanted to replace File.exists() with something else, i am not able to do that because this piece of code resides in RealmConfiguration.java file (Read-only) which is part of the Realm Library that i use in my app.
To make things even more clear i will provide two pieces of code. The code i use in my activity and the method that get's called in RealmConfiguration.java as a consequence:
Code i use in my activity :
File myfile = new File("/storage/emulated/0/Documents");
if(myFile.exists()){ //<---- Notice that myFile exists at this point.
Realm.init(this);
config = new RealmConfiguration.Builder()
.name(".TheDatabaseName")
.directory(myFile) //<---- Notice this line of code.
.schemaVersion(7)
.migration(new MyMigration())
.build();
Realm.setDefaultConfiguration(config);
realm = Realm.getDefaultInstance();
}
At this point myFile exists and the code that resides in RealmConfiguration.java get's called.
The RealmConfiguration.java method that crashes :
/**
* Specifies the directory where the Realm file will be saved. The default value is {#code context.getFilesDir()}.
* If the directory does not exist, it will be created.
*
* #param directory the directory to save the Realm file in. Directory must be writable.
* #throws IllegalArgumentException if {#code directory} is null, not writable or a file.
*/
public Builder directory(File directory) {
//noinspection ConstantConditions
if (directory == null) {
throw new IllegalArgumentException("Non-null 'dir' required.");
}
if (directory.isFile()) {
throw new IllegalArgumentException("'dir' is a file, not a directory: " + directory.getAbsolutePath() + ".");
}
------> if (!directory.exists() && !directory.mkdirs()) { //<---- Here is the problem
throw new IllegalArgumentException("Could not create the specified directory: " + directory.getAbsolutePath() + ".");
}
if (!directory.canWrite()) {
throw new IllegalArgumentException("Realm directory is not writable: " + directory.getAbsolutePath() + ".");
}
this.directory = directory;
return this;
}
So, myFile exists in my activity, the Realm code get's called and suddenly myFile no longer exists.. Again i wish to point out that this is not consistent. I am noticing crashes at a rate of 4-5% meaning that most of the time myFile exists both in the activity and when the realm code makes it's check.
I hope this will be helpful.
Again thanks in advance!

First of all, if you are using Android, bug reports in the Java Bugs database are not relevant. Android does not use the Sun / Oracle codebase. Android started out as a clean-room re-implementation of the Java class libraries.
So if there are bugs in File.exists() on Android the bugs would be in the Android codebase, and any reports would be in the Android issue tracker.
But when you say this:
According to this, when operating on NFS-mounted volumes, java.io.File.exists ends up performing a stat(2). If the stat fails (which it may do for several reasons), then File.exists (mistakenly) assumes that the file being stat'ed does not exist.
Unless you are using NFS, that bug report is not directly relevant.
It is not a mistake / bug. It is a limitation.
At the file system level, it is a fact of life that Linux supports many different kinds of file system, and that many of them behave in unexpected ways ... compared to an "ordinary" file system. It is not possible for the JVM to hide all of the weird filesystem-specific edge cases at the Java API level.
On the API level, File.exists cannot report any errors. The signature doesn't allow it to throw an IOException, and throwing an unchecked exception would be a breaking change. All it can say is true or false.
If you want to distinguish the various reasons for a false, you should use the newer Files.exists(Path, LinkOptions...) method instead.
Could this be the source of my troubles?
Yes it could, and not just in the NFS case! See below. (With Files.exist, an NFS stat failure would most likely be an EIO, and that would raise an IOException rather than returning false.)
The File.java code in the Android codebase (version android-4.2.2_r1) is:
public boolean exists() {
return doAccess(F_OK);
}
private boolean doAccess(int mode) {
try {
return Libcore.os.access(path, mode);
} catch (ErrnoException errnoException) {
return false;
}
}
Note how it turns any ErrnoException into a false.
A bit more digging reveals that the os.access call is performing a native call which makes an access syscall, and throws ErrnoException if the syscall fails.
So now we need look at the documented behavior of the access syscall. Here's what man 2 access says:
F_OK tests for the existence of the
file.
On error (at least one bit in mode
asked for a permission that is denied, or mode is F_OK and the file
does not exist, or some other error occurred), -1 is returned, and
errno is set appropriately.
access() shall fail if:
EACCES The requested access would be denied to the file, or search per‐
mission is denied for one of the directories in the path prefix
of pathname. (See also path_resolution(7).)
ELOOP Too many symbolic links were encountered in resolving pathname.
ENAMETOOLONG
pathname is too long.
ENOENT A component of pathname does not exist or is a dangling symbolic
link.
ENOTDIR
A component used as a directory in pathname is not, in fact, a
directory.
EROFS Write permission was requested for a file on a read-only
filesystem.
access() may fail if:
EFAULT pathname points outside your accessible address space.
EINVAL mode was incorrectly specified.
EIO An I/O error occurred.
ENOMEM Insufficient kernel memory was available.
ETXTBSY
Write access was requested to an executable which is being executed.
I have struck out the errors that I think are technically impossible or implausible, but the still leaves quite few to consider.
Another possibility is something (e.g. some other part of your application) is deleting or renaming the file or a (hypothetical) symlink, or changing file permissions ... behind your back.
But I don't think that File.exist() is broken1, or that the host OS is broken. It is theoretically possible, but you would need some clear evidence to support the theory.
1 - It is not broken in the sense that it is not behaving differently to the known behavior of the method. You could argue until the cows come home about whether the behavior is "correct", but it has been like that since Java 1.0 and it can't be changed in OpenJDK or in Android without breaking thousands of existing applications written over the last 20+ years. It won't happen.
What to do next?
Well my recommendation would be to use strace to track the syscalls that your app is making and see if you can get some clues as to why some access syscalls are giving you unexpected results; e.g. what the paths are and what the errno is. See https://source.android.com/devices/tech/debug/strace .

I have had a similar issue, but with a higher trouble rate, where the Anti Virus was locking FileSystem, and thus failing any requests (almost instantly)
the workaround was using java.nio.Files.exists() instead.

Iterate over certain subpaths in Path in Java

I am working with code which makes use of java.nio.file.Path. I have a path like
/tmp/something/first/second/third/last
which I see only as
{parent.dir}/first/second/third/{path.end}
In this example /tmp/something or {parent.dir} is a path which can be different during runtime and which is irrelevant for me. The same applies for the last element in the path {path.end}
What I would need is to iterate over elements between {parent.dir} and {path.end}. Basically to test each element in the path if it exists and if it is a file or
a directory or something else (it doesn't matter).
I am therefore looking for some elegant/simple and proper way to iterate over instance of Pathwhere I can access paths like these:
/tmp
/tmp/something/
/tmp/something/first
...
/tmp/something/first/second/third/last
ideally in this case without the first 2 and the last iteration.
I am looking for a good solution using Path and java.nio and not the old way. I know I can achieve this using old io API but I am interested in the proper way of using nio.

Here I print all directories child of your parent dir:
Files.walk(Paths.get(${parent.dir})).filter(path -> Files.isDirectory(path, LinkOption.NOFOLLOW_LINKS)).forEach(System.out::println);
You can pass another lambda to the forEach method for your own purpose.
Also replace ${parent.dir} with your correct value as a String.
(The code above may throw IOException).

Assuming that the base and end parts are parameters, while the in-between part is fixed, a solution can look like this:
static void iterate(Path base, Path end) {
if(!base.isAbsolute() || end.isAbsolute()) throw new
IllegalArgumentException("base must be absolute, end must be relative");
// test the fixed in-between paths
Path fixed=Paths.get("first", "second", "third");
for(Path part: fixed) {
base=base.resolve(part);
System.out.print(base);
if(Files.isDirectory(base)) {
System.out.println(" is a directory");
}
else {
System.out.println(Files.exists(part)?" is not a directory":" does not exist");
return;
}
}
// test the end path
end=base.resolve(end);
System.out.print(end+(
Files.isDirectory(end)? " is a directory":
Files.exists(end)? " is not a directory": " does not exist"));
}
It stops the iteration once it encounters a non-directory path component. You will have to adapt this behavior if you want to enforce a specific policy regarding following symbolic links…

Handling non-fatal errors in Java

I've written a program to aid the user in configuring 'mechs for a game. I'm dealing with loading the user's saved data. This data can (and some times does) become partially corrupt (either due to bugs on my side or due to changes in the game data/rules from upstream).
I need to be able to handle this corruption and load as much as possible. To be more specific, the contents of the save file are syntactically correct but semantically corrupt. I can safely parse the file and drop whatever entries that are not semantically OK.
Currently my data parser will just show a modal dialog with an appropriate warning message. However displaying the warning is not the job of the parser and I'm looking for a way of passing this information to the caller.
Some code to show approximately what is going on (in reality there is a bit more going on than this, but this highlights the problem):
class Parser{
public void parse(XMLNode aNode){
...
if(corrupted) {
JOptionPane.showMessageDialog(null, "Corrupted data found",
"error!", JOptionPane.WARNING_MESSAGE);
// Keep calm and carry on
}
}
}
class UserData{
static UserData loadFromFile(File aFile){
UserData data = new UserData();
Parser parser = new Parser();
XMLDoc doc = fromXml(aFile);
for(XMLNode entry : doc.allEntries()){
data.append(parser.parse(entry));
}
return data;
}
}
The thing here is that bar an IOException or a syntax error in the XML, loadFromFile will always succeed in loading something and this is the wanted behavior. Somehow I just need to pass the information of what (if anything) went wrong to the caller. I could return a Pair<UserData,String> but this doesn't look very pretty. Throwing an exception will not work in this case obviously.
Does any one have any ideas on how to solve this?

Depending on what you are trying to represent, you can use a class, like SQLWarning from the java.sql package. When you have a java.sql.Statement and call executeQuery you get a java.sql.ResultSet and you can then call getWarnings on the result set directly, or even on the statement itself.
You can use an enum, like RefUpdate.Result, from the JGit project. When you have a org.eclipse.jgit.api.Git you can create a FetchCommand, which will provide you with a FetchResult, which will provide you with a collection of TrackingRefUpdates, which will each contain a RefUpdate.Result enum, which can be one of:
FAST_FORWARD
FORCED
IO_FAILURE
LOCK_FAILURE
NEW
NO_CHANGE
NOT_ATTEMPTED
REJECTED
REJECTED_CURRENT_BRANCH
RENAMED
In your case, you could even use a boolean flag:
class UserData {
public boolean isCorrupt();
}
But since you mentioned there is a bit more than that going on in reality, it really depends on your model of "corrupt". However, you will probably have more options if you have a UserDataReader that you can instantiate, instead of a static utility method.

What does eclipse do when you inspect variables (while debugging)

I have an instance of org.hibernate.envers.entities.mapper.relation.lazy.proxy.ListProxy that is causing some grief: whenever I programmatically try to access it I get a null pointer exception (ie calling list.size()) but when I first inspect the object using Eclipse's variable inspector I see Hibernate generate a SQL statement and the list changes dynamically. Then everything works. How can I do the same thing programmatically? I've tried list.toString() but that doesn't seem to help.
Update 1
Don't know if this helps but when I first click on the list instance I see in the display:
com.sun.jdi.InvocationException occurred invoking method.
Then database query runs and when I click again I get the correct .toString() result.
Update 2
Here is the original exception I get (when I don't inspect the element in debug mode).
java.lang.NullPointerException
at org.hibernate.envers.query.impl.EntitiesAtRevisionQuery.list(EntitiesAtRevisionQuery.java:72)
at org.hibernate.envers.query.impl.AbstractAuditQuery.getSingleResult(AbstractAuditQuery.java:104)
at org.hibernate.envers.entities.mapper.relation.OneToOneNotOwningMapper.mapToEntityFromMap(OneToOneNotOwningMapper.java:74)
at org.hibernate.envers.entities.mapper.MultiPropertyMapper.mapToEntityFromMap(MultiPropertyMapper.java:118)
at org.hibernate.envers.entities.EntityInstantiator.createInstanceFromVersionsEntity(EntityInstantiator.java:93)
at org.hibernate.envers.entities.mapper.relation.component.MiddleRelatedComponentMapper.mapToObjectFromFullMap(MiddleRelatedComponentMapper.java:44)
at org.hibernate.envers.entities.mapper.relation.lazy.initializor.ListCollectionInitializor.addToCollection(ListCollectionInitializor.java:67)
at org.hibernate.envers.entities.mapper.relation.lazy.initializor.ListCollectionInitializor.addToCollection(ListCollectionInitializor.java:39)
at org.hibernate.envers.entities.mapper.relation.lazy.initializor.AbstractCollectionInitializor.initialize(AbstractCollectionInitializor.java:67)
at org.hibernate.envers.entities.mapper.relation.lazy.proxy.CollectionProxy.checkInit(CollectionProxy.java:50)
at org.hibernate.envers.entities.mapper.relation.lazy.proxy.CollectionProxy.size(CollectionProxy.java:55)
at <MY CODE HERE, which checks list.size()>
Final Solution (Actually more of a temporary hack)
boolean worked = false;
while (!worked) {
try {
if(list.size() == 1) {
// do stuff
}
worked = true;
} catch (Exception e) {
// TODO: exception must be accessed or the loop will be infinite
e.getStackTrace();
}
}

Well what happends there is you're seing Hibernate's lazy loading in deep action :)
Basically hibernate loads proxy classes for you lazily associated relations, such that instead of a List of classes C you get a List (actually a PersistenceBag implementation) of Hibernate autogenerated proxie for your C class. THis is hibernate's way of deferring load of that association's values until they are actually accessed. So that's why when you access it in the eclipse debugger (which basically accesses an instance's fields/methids via introspection) you see the sql hibernate triggers to fetch the needed data.
The trick here is that depending on WHEN you access a lazy collection you might get different results. If you access it using the eclipse debugger you're more likely still in the Hibernate session that started loading that thing, so everything works as expected, an sql is (lazily) triggered when the thing is accessed and the data is loaded). Problem is that if you wanna access that same data in your code, but at a point where the session is already closed, you'll either get a LazyInitializationException or null (the latter if you're using some library for cleaning put hibenrate proxises such as Gilead)