Im answering this question below; I opened it because it is more general than what I originally asked in an earlier question, so it wouldn't really fit there. It took me quite some tinkering, so I thought I'd share the solution here.
My situation:
I use picoCLI to parse multiple configuration files that in turn can "include" other config files, to arbitrary depth. Unfortunately, for some of my options the order in which they are parsed does also matter.
In my application, there are "section" options like section=A:teacher that request section A and cause it to be processed (I'll leave out what that exactly means) for teachers, students or other groups. Among a number of other options, there is also one called configfile= that "includes" another option file. That situation can be described by a "tree" of configuration details:
# options given on actual command line ("root of tree")
section=A:teacher
configfile=cf-1.txt # include options from cf-1.txt
section=A:student # this indentation: options read from cf-1.txt
section=B:principal
configfile=cf-2.txt # read options from cf-2.txt
section=A:parent # this indentation: options read from cf-2.txt
section=C:parent
section=C:teacher # back in cf-1.txt
section=D:admin # back to actual command line
I want this tree to be traversed depth-first, with "later" options overwriting "earlier" ones if they refer to the same section name: In the end, section A should get parent and C should get teacher.
For parsing configfile= options, I can't use picoCLI's #-syntax because these files are not necessarily in the "current" folder, so I want to control where the application looks for them. That's problem #1. It is solved by the parseConfigfile method listed below.
Unfortunately, picoCLI has a peculiar quirk when an option occurs multiple times in the same file (as section does with A, B and C): It does call the annotated setter method each time but with accumulating option values in the list parameter of that method. The first call only gets (A:student), the second (A:student,B:prof), the third (A:student,B:prof,C:teacher) etc.
I learned here that this behaviour is intended but for me it is problem #2 because the repeated evaluation of section=A:student messes up my later-options-overwrite-earlier-ones semantics: In the end, A is incorrectly configured for teacher. For many options (those with "one-dimensional" values), that's not a problem, but it is for section= and, somewhat ironically, also for configfile=.
Here's my solution; maybe it's useful to someone. The basic approach is to use a new CommandLine instance for each nested config file.
All code snippets below are from the annotated class; project-specific housekeeping, error checking, path construction etc. were removed.
Problem #1 is solved by the parseConfigFile(...) method:
#Option(names = { "configfile", "cf" } )
public void parseConfigfile(final List<String> accumulatedCfgFiles) {
// prune the list, keeping only "new" entries that haven't been seen yet:
List<String> newCfgFiles = this.cfgfileHelper.retainNewOptionValuesOnly(
accumulatedCfgFiles, "configfile");
if(newCfgFiles.isEmpty()) {
// another picoCLI quirk: this happens even if there always are values
return;
} else if(newCfgFiles.size() > 1) {
// loop over the files if you want to allow this, or report an error
}
// some path tinkering left out
File cfgFile = new File(newCfgFiles.get(0));
if(this.stackOfConfigFiles.contains(cfgFile)) {
// report error because of cyclic reference
} else {
this.stackOfConfigFiles.push(cfgFile);
// task a new CommandLine instance with processing that file:
CommandLine cmd = new CommandLine(this);
String[] optionsFromFile = FileUtils.readLines(cfgFile); // Apache Commons
this.cfgfileHelper.wrapParseArgsCall(cmd, optionsFromFile);
this.stackOfConfigFiles.pop();
}
}
The method uses an instance of NestedCfgfileHelper (see source below) which does all the nested-config-specific housekeeping to solve problem #2. Your annotated class needs one helper instance as a public attribute. Call the constructor with the names of all "problem" options (those that the helper class should take care of):
...
public final NestedCfgfileHelper cfgfileHelper =
new NestedCfgfileHelper(new String[] { "configfile", "section" });
...
The following steps make all this work:
Identify those options that are sensitive to "spurious setter method calls" (most are not);
If there are any, paste NestedCfgfileHelper's source to your annotated class as an inner class;
Create an instance of NestedCfgfileHelper as a public member of your annotated class, telling the constructor the names of all those "problematic" options;
Never call yourInstanceOfCommandLine.parseArgs(...) directly. Instead, instantiate and initialize it, but pass it to the helper using instanceOfYourAnnotatedClass.cfgfileHelper.wrapParseArgs(...)
Let the setter method(s) for those "difficult" options...
... first get rid of "old" values from previous invocations by calling retainNewOptionValuesOnly, passing the name of the option;
... then process the remaining option value(s) normally.
Finally, here's the source of NestedCfgfileHelper:
/** NestedCfgfileHelper ensures that the values of certain options are
* processed just once, despite the picoCLI quirk. */
public final class NestedCfgfileHelper {
/** Maps an (option name|CommandLine instance) pair to the number of
* option values that instance has so far passed for that option.
* Because Java doesn't have Maps with two keys, it's implemented as
* a Map of Maps: */
private Map<String, Map<CommandLine, Integer>> mapOptionAndCLToCount =
new HashMap<>();
/** Constructs a helper instance and prepares it to handle the options
* given as parameters.
*
* #param optionNames any number of Strings, with each String denoting
* one option whose values should be protected against being processed
* multiple times */
public NestedCfgfileHelper(String... optionNames) {
// make one mapping for each option name given:
for(String optionName: optionNames) {
mapOptionAndCLToCount.put(optionName, new HashMap<CommandLine, Integer>());
}
}
/** This stack keeps track of CommandLine instances that are currently
* working on this annotated class instance. A stack is needed because
* config files can be nested. */
private Stack<CommandLine> stackOfCmdLineInstances = new Stack<>();
/** Wraps the call to {#link CommandLine#parseArgs(String...)} with some
* housekeeping so that when an annotated setter method is being called
* during option parsing, the helper method can look up from which
* CommandLine instance the call is coming.
* Because parseArg invocations will be nested recursively for nested config
* files, the respective CommandLine instances are kept on a stack.
* #param cl CommandLine instance that's been about to start parsing
* #param args options that are to be parsed */
public void wrapParseArgsCall(final CommandLine cl, final String[] args) {
// the brand new CommandLine instance hasn't passed any values yet,
// so put 0 in all maps:
mapOptionAndCLToCount.forEach(
(String s, Map<CommandLine, Integer> m) -> m.put(cl, 0));
this.stackOfCmdLineInstances.push(cl);
cl.parseArgs(args);
this.stackOfCmdLineInstances.pop();
}
/** This method filters its list parameter, discarding the first n
* entries (assuming they've already been processed), where n is retrieved
* from a Map instance kept for each option name.
* As a side effect, the updated number of option values is stored.
* This method should be called exactly once per invocation of an annotated
* setter method, and only by setter methods whose options values shouldn't
* be set multiple times.
*
* #param accumulated List containing all values (old and new ones
* accumulated) of the option named in the other parameter.
* #param optionName describes the option that's being parsed.
* #return pruned list containing only the "new" values that haven't
* been seen before. */
private List<String> retainNewOptionValuesOnly(
final List<String> accumulated,
final String optionName) {
// get the CommandLine instance currently working on this TFConfig instance:
CommandLine currentCL = this.stackOfCmdLineInstances.peek();
// get the CommandLine->int map for the option name passed:
Map<CommandLine, Integer> map = mapOptionAndCLToCount.get(optionName);
if(map == null) {
throw new IllegalArgumentException("unknown option: " + optionName);
}
/* Find out how many option values it has already passed to the setter. */
int n = map.get(currentCL);
/* discard the first n entries (they have already been processed) of
* accumulated, keeping only the "new" ones: */
List<String> optionValuesNewThisTime =
accumulated.subList(n, accumulated.size());
// associate the new number of patterns with the current CommandLine:
int newNumber = n + optionValuesNewThisTime.size();
map.put(currentCL, newNumber);
return optionValuesNewThisTime;
}
}
Related
picoCLI's #-file mechanism is almost what I need, but not exactly. The reason is that I want to control the exact location of additional files parsed -- depending on previous option values.
Example: When called with the options
srcfolder=/a/b optionfile=of.txt, my program should see the additional options read from /a/b/of.txt, but when called with srcfolder=../c optionfile=of.txt, it should see those from ../c/of.txt.
The #-file mechanism can't do that, because it expands ALL the option files (always relative to the current folder, if they're relative) prior to processing ANY option values.
So I'd like to have picoCLI...
process options "from left to right",
recursively parse an option file when it's mentioned in an optionfile option,
and after that continue with the following options.
I might be able to solve this by recursively starting to parse from within the annotated setter method:
...
Config cfg = new Config();
CommandLine cmd = new CommandLine(cfg);
cmd.parseArgs(a);
...
public class Config {
#Option(names="srcfolder")
public void setSrcfolder(String path) {
this.srcfolder=path;
}
#Option(names="optionfile")
public void parseOptionFile(String pathAndName) {
// validate path, do some other housekeeping...
CommandLine cmd = new CommandLine(this /* same Config instance! */ );
cmd.parseArgs(new String[] { "#"+this.srcfolder + pathAndName });
}
...
This way several CommandLine instances would call setter methods on the same Config instance, recursively "interrupting" each other. Now comes the actual question: Is that a problem?
Of course my Config class has state. But do CommandLine instances also have state that might get messed up if other CommandLine instances also modify cfg "in between options"?
Thanks for any insights!
Edited to add: I tried, and I'm getting an UnmatchedArgumentException on the #-file option:
Exception in thread "main" picocli.CommandLine$UnmatchedArgumentException: Unmatched argument at index 0: '#/path/to/configfile'
at picocli.CommandLine$Interpreter.validateConstraints(CommandLine.java:13490)
...
So first I have to get around this: Obviously picoCLI doesn't expand the #-file option unless it's coming directly from the command line.
I did get it to work: several CommandLine instance can indeed work on the same instance of an annotated class, without interfering with each other.
There are some catches and I had to work around a strange picoCLI quirk, but that's not exactly part of an answer to this question, so I explain them in this other question.
Our application involves an external device that mounts as a USB mass storage device. I need to be able to check the integrity of that USB mass storage device from a Java application, running on Windows.
Currently I execute "chkdsk", which works fine unless the user's computer isn't configured for English. (Because I need to examine the chkdsk output to determine the state of the drive.) And, surprise, surprise, not every computer in the world is configured for English!
There is a Win32 class called "Win32_Volume" with a method called "Chkdsk" that is just what I'm looking for.
I need to call it from Java, and JNA seems the way to go.
How can I use COM to call a method from a Win32 WMI class?
You can send an FSCTL_IS_VOLUME_DIRTY message using DeviceIoControl.
Here is and example of calling DeviceIoControl.
Or you can use WMI, both the the Win32_Volume and the Win32_LogicalDisk class expose a DirtyBitSet property. However WMI is really inteded for scripting languages and is not preferred because of it's incredible performance overhead.
The Win32_Volume documentation with the Chkdsk method you cite indicates it is a legacy API, but has not yet been deprecated so you could use it.
The current Storage Management API equivalent is MSFT_Volume. It has a Diagnose method that looks to perform similar functionality.
To execute a method on a WMI object via COM you need to obtain a pointer to a WbemServices object. It has an ExecMethod method you can invoke that does what you want for WMI classes. The code to retrieve this object is in JNA's WbemcliUtil class as a return type from the connectServer() method.
You will also need the full path to the Class Object; this is available by querying WMI for the __PATH, similarly to how you'd query any other field. That same WbemcliUtil class has a WmiQuery class you can instantiate to execute the query, collecting that path from the WmiResult. See the JNA test classes for examples on using these.
Finally, you can execute the WMI method. The code below will accomplish it for String properties, e.g., the MSFT_StorageDiagnoseResult object you'd get from MSFT_Volume's Diagnose method. You'd have to treat other return types differently.
/**
* Convenience method for executing WMI methods without any input parameters
*
* #param svc
* The WbemServices object
* #param clsObj
* The full path to the class object to execute (result of WMI
* "__PATH" query)
* #param method
* The name of the method to execute
* #param properties
* One or more properties returned as a result of the query
* #return An array of the properties returned from the method
*/
private static String[] execMethod(WbemServices svc, String clsObj, String method, String... properties) {
List<String> result = new ArrayList<>();
PointerByReference ppOutParams = new PointerByReference();
HRESULT hres = svc.ExecMethod(new BSTR(clsObj), new BSTR(method), new NativeLong(0L), null, null, ppOutParams,
null);
if (COMUtils.FAILED(hres)) {
return new String[0];
}
WbemClassObject obj = new WbemClassObject(ppOutParams.getValue());
VARIANT.ByReference vtProp = new VARIANT.ByReference();
for (String prop : properties) {
hres = obj.Get(new BSTR(prop), new NativeLong(0L), vtProp, null, null);
if (!COMUtils.FAILED(hres)) {
result.add(vtProp.getValue() == null ? "" : vtProp.stringValue());
}
}
obj.Release();
return result.toArray(new String[result.size()]);
}
A more complete example of querying a COM object, where I extracted the above code, is in this utility class which queries the GetOwner() and GetOwnerSid() methods on the Win32_Process object. It's legacy code from before I contributed the WbemCli class to JNA, so you'll have to adapt some of it, but it should be a good starting point.
I'm using additional jar file in my spring boot project. The problem is that one of the methods I want to test is using method from that jar that returns a value, but also sets a value to variable that is posted to it.
String valueThatWillBeReturned;
int returnMessage = method(valueThatWillBeReturned);
I don't know why the method is write by that. I'm not the writer of that jar and I cant debug it, but it works like that. It will return int that will be stored in int returnMessage, and also will set valueThatWillBeReturned that is posted to it.
It's a problem for me to test it. I'm setting the value of int returnMessage by:
when(external.method(valueThatWillBeReturned).thenReturn(1);
But how should I set value of String valueThatWillBeReturned?
The main problem is that my code depends on String valueThatWillBeReturned that should be returned.
After Edit
I made the upper example more simpler, but will give you additional details. I'm using an external library ImageSDK.jar. It uses .dll or .so file depending on the operation system.
So by documentation I should have int[] pnMatchTemplate2Index = new int[1]; to post to the method below.
int result = libSDK.UFM_Identify(hMatcherContainer[0],
templateForIdentification,
sizeOfTemplateForIdentification,
templatesFromDBForIdentification,
sizeOfEachTemplateFromDB,
numberOfTemplatesForCompare,
5000,
pnMatchTemplate2Index);
What method returns is int result that stores the return status, but the method also sets pnMatchTemplate2Index where index of matched template is stored. After that I'm using templates.get(pnMatchTemplate2Index[0]) to get information I need.
So in the end my method depends on that parameter to return value, and I dont know how to set it by Junit to test my method return.
I found a solution to my problem. I changed the location of the pnMatchTemplate2Index variable from method to class variable.
Service.class
// Receives the index of matched template in the template array;
private int[] pnMatchTemplate2Index = new int[1];
Then in the test method I mocked it like oliver_t suggested.
ServiceTest.class
int [] mockResults = {0};
ReflectionTestUtils.setField(underTest,"pnMatchTemplate2Index", mockResults);
Where:
underTest is the class we are testing
"pnMatcherTemplate2Index" is the private variable we want to mock in the class we are testing
mockResults is the value we want to set for the private variable
That way I managed to use pnMatchTemplate2Index to get the index I want from my other method.
I am trying to generate a class and methods in it, using Byte Buddy, based on some configuration that is available at runtime. The class is trying to create a Hazelcast Jet pipeline to join multiple IMaps.
Based on the provided configuration, the no. of IMaps to join can vary. In the sample below, I am trying to join three IMaps.
private Pipeline getPipeline(IMap<String, Object1> object1Map, IMap<String, Object2> object2Map,
IMap<String, Object3> object3Map) {
Pipeline p = Pipeline.create();
BatchStage<Entry<String, Object1>> obj1 = p.drawFrom(Sources.map(object1Map));
BatchStage<Entry<String, Object2>> obj2 = p.drawFrom(Sources.map(object2Map));
BatchStage<Entry<String, Object3>> obj3 = p.drawFrom(Sources.map(object3Map));
DistributedFunction<Tuple2<Object1, Object2>, String> obj1Obj2JoinFunc = entry -> entry.f1().getField31();
DistributedBiFunction<Tuple2<Object1, Object2>, Object3, Tuple2<Tuple2<Object1, Object2>, Object3>> output = (
in1, in2) -> (Tuple2.tuple2(in1, in2));
BatchStage<Tuple2<Object1, Object2>> obj1_obj2 = obj1.map(entry -> entry.getValue())
.hashJoin(obj2.map(entry -> entry.getValue()),
JoinClause.onKeys(Object1::getField11, Object2::getField21), Tuple2::tuple2).filter(entry -> entry.getValue() != null);
BatchStage<Tuple2<Tuple2<Object1, Object2>, Object3>> obj1_obj2_obj3 = obj1_obj2.hashJoin(
obj3.map(entry -> entry.getValue()),
JoinClause.onKeys(obj1Obj2JoinFunc, Object3::getField31), output)
.filter(entry -> entry.getValue() != null);
// the transformResult method will get the required fields from above operation and create object of AllObjectJoinClass
BatchStage<Entry<String, AllObjectJoinClass>> result = transformResult(obj1_obj2_obj3);
result.drainTo(Sinks.map("obj1_obj2_obj3"));
return p;
}
The problem here is that the no. of arguments to my method depend on the runtime configuration and that determines the method body as well.
I am able to generate the method signature using TypeDescription.Generic.Builder.parameterizedType.
But, I am having trouble generating the method body. I tried using MethodDelegation.to so that the method resides in a separate class. The trouble with this approach is that the method in the separate class needs to be very generic so that it can take arbitrary no. of arguments of different types and also needs to know about the fields of each of the objects in the IMap.
I wonder if there's an an alternate approach to achieving this with maybe templates of some type so that a separate class can be generated for each pipeline with this body. I didn't find any documentation for generating a method with a defined body (maybe I missed something).
-- Anoop
It very much depends on what you are trying to do:
With Advice, you can write a template as byte code that is inlined into your method.
With StackManipulations you can compose individual byte code instructions.
It seems to me that option (2) is what you are aiming for. For individually composed code, this is often the easiest option.
Writing individual byte code is of course not the most convenient option but if you can easily compose handling of each input, you might be able to compose multiple Advice classes to avoid using byte code instructions directly.
When I was studying android source code, I noticed that the general class loader in app is an instance of PathClassLoader, and there is two constructors in this class. One is like:
public PathClassLoader(String dexPath, ClassLoader parent) {
super(dexPath, null, null, parent);
}
and the other is like:
public PathClassLoader(String dexPath, String libraryPath,
ClassLoader parent) {
super(dexPath, null, libraryPath, parent);
}
But I cannot find the call of the second constructor in the source code during the procedure of app launching. So where does the value of libraryPath param come from? As it is known that libraryPath refers to the list of directories containing native libraries, and is used for initializing the value of nativeLibraryDirectories, which is a field of DexPathList object. So if there is no call of the second constructor with three params, how can the value of nativeLibraryDirectories be initialized? Therefore how can an app find its native libraries?
Actually, I am wondering who determines the value of nativeLibraryDirectories?
Hope some one can guide me in this. Thanks a lot.
So where does the value of libraryPath param come from?
You can use Android Studio search to find it out. Perform "Find in Path", specifying the "Scope" parameter to directory of Android sources. As a text to find paste following regex expression:
new PathClassLoader\(\w+, \w+, \w+\)\;
This matches a call of the constructor with three params. Also, do not forget to check "Regular expression" checkbox:
Then in the preview tab you'll be able to see the results:
With the same technique you can find out who is calling PathClassLoaderFactory#createClassLoader() function:
In ZygoneInit.java you'll be able to locate following piece of code:
/**
* Creates a PathClassLoader for the system server. It also creates
* a shared namespace associated with the classloader to let it access
* platform-private native libraries.
*/
private static PathClassLoader createSystemServerClassLoader(String systemServerClasspath,
int targetSdkVersion) {
String librarySearchPath = System.getProperty("java.library.path");
return PathClassLoaderFactory.createClassLoader(systemServerClasspath,
librarySearchPath,
null /* libraryPermittedPath */,
ClassLoader.getSystemClassLoader(),
targetSdkVersion,
true /* isNamespaceShared */);
}
Now, back to your questions.
So if there is no call of the second constructor with three params...
There is, ZygoteInit#handleSystemServerProcess() calls createSystemServerClassLoader(), which would eventually call 3 args constructor of PathClassLoader.
Actually, I am wondering who determines the value of nativeLibraryDirectories?
As you can see from the code above, it defaults to system property "java.library.path".