I have an application that consists of a grid of hazelcast nodes which uses extensive runtime bytecode generation (asm). Specifically I am dynamically building predicate<> Java functions from user entered filter expressions. I would like to store the predicates in a map so that they are available across the cluster without having to be recompiled. Predicates are not the only instance of this. I also have ORM style mapping classes that are generated at runtime which need to be shared across the cluster. These classes are loaded by a custom classloader called DynamiClassLoader.
At first I was unable to store my custom generated classes in Hazelcast IMaps, getting a ClassNotFoundexception. However, I found if tell Hazelcast to use my custom DynamicClassLoader using Config.setClassLoader() then it is able to find my dynamic classes and use them in the local member so I have no issues serializing and deserializing instances of my custom classes in IMaps in the same member.
However, I am still unable to deserialize instances of my predicates which were inserted into the map by a different member. I have enabled UserCodeDeployment and stepped through the code in my debugger to confirm that if it cannot find the class locally then it is hitting UserCodeDeploymentClassLoader.java and on the classNotFoundException it is proceeding to check other members for the classes but it is unable to find them. I haven't been able to discover how exactly this works. it seems to look in an internal map for which member any given class can be found on and it does not find my classes in there. I believe it dispatches an operation to other members then to look for the class, but in this case it looks like my custom classloader isn't used so it's not able to find my custom classes.
How can I make dynamically generated classes work with UserCodeDeployment on Hazelcast? Is there a way I can 'register' my dynamic classes with the member's code service or something like that?
Thankyou,
Troy.
I've finally figured this out after extensive debugging. It turns out that the operation in Hazelcast on the target member to find a class calls loadBytecodeFromParent() in ClassDataProvider.java. This looks for a .class file using getResourceAsStream on the classloader:
String resource = className.replace('.', '/').concat(".class");
...
is = parent.getResourceAsStream(resource);
Which basically looks for the class file on the filesystem. Since my dynamic classes are entirely in memory there is no .class file resource for it to find.
I resolved this, by putting a hashmap in my DynamicClassLoader to keep the generated bytecode in and overriding getResourceAsStream to return that bytecode when it's available before looking further. Now it works!
Related
As we know java class loading works like in picture below.
When we have a notion of plugin in our application (like app servers) we sometimes need some classes to be loaded in the instances class loader rather than in parent, so that each new instance (plugin, webapp or whatever) loads that particular class not delegating the parent...
For example Log4j classes, we need them to be loaded for each instance.
So in order to do this the best approach that came into my mind is to write custom classloader that will take a list of class names which shall be prevented from being delegated to parent classloader (ideally we want instances to be in complete isolation).
So the the application that will load other "instances" will use that custom classloader while loading those "instances"...
Is there an implementation of such classloader that solves this issue? (given that we dont want to know what OSGi is and we work with pure java no frameworks etc...)
My searches end up pointing some frameworks or some web container specific solutions, yet this is quite a simple task that is solved with one class, I'd like to find out if i'm missing something before i start implementing it.
Update (digging deeper) : suppose there is a class loaded by bootstrap which has static state that can be shared between instances (and we really badly want to make sure instances are completely isolated), now that class is obviously not included in our classpath, but it is loaded, and if we copy it instead of reffering it we will achieve the required isolation.
So do we have the notion of copying or cloneing a class from one classloader to other?
If a java class loaded at runtime as a bytecode (e.g. via ASM library or other mechanism), is it on a classpath of java? I don't see it.
How to scan all annotations of classes loaded this way, if it's not on the java classpath?
I use google reflections and a custom classloader of a 3-rd party library.
Scanning for classes works as follows:
Reflections reflections = new Reflections(ClasspathHelper.forPackage("com.mypackage",
MyCustomClassLoader),
new SubTypesScanner(), new TypeAnnotationsScanner());
Set<Class<?>> myClasses = reflections.getTypesAnnotatedWith(MyAnnotation.class);
MyAnnotation - is marked as #Retention(RetentionPolicy.RUNTIME).
The above (in a class) is loaded dynamically at runtime by JVM.
As can be seen, behind the scenes, Reflections tries to get all URLs to be scanned using 2 classloaders by default static and context.
Reflection.scan()
Update: I have found an answer Can you find all classes in a package using reflection? saying that "If there are classes that get generated, or delivered remotely, you will not be able to discover those classes." But there is no proof, however.
Can please anybody give more details on this and confirm?
Dynamically instantiating classes at run time does not change the classpath that your JVM is using. What happens is that some ClassLoader class fetches bytecode from somewhere; and makes it "available" to you. But that does in no way change the "search order" for loading classes (and that is basically what the classpath is about: it only tells the JVM where and in which order to look for classes to load).
Meaning: any "loading" of a class results in some object of class java.lang.Class.
If you want to query the structure of any loaded class; you "just" need to get to the corresponding Class object. Class provides methods like getAnnotation(). It also provides methods to retrieve other objects that represent the methods and fields of the specific Class; and those objects can be queried for annotations in similar ways.
Update, regarding the updates in the question: there are situations where you don't have access to the file system where classes are coming from. You can load classes, when you know their name, but you have no way looking into the "place" where these classes live. And that basically breaks your ability to use reflection as intended.
There is no need to "prove" that, it is simply a consequence of the fact that Java allows you to load classes when you know there name, but "hides" the exact content of "where" those classes are coming from.
I'm currently working on a client/server application that supports dynamicly loaded classes.
These classes are plugins which should not be able to access certain methods.
My first idea was a SecurityManager that checked class names in the stacktrace but that does not seem to be the ideal way to approache this.
How would one employ such restrictions to specific classes?
Is there a (realisable) way to read every referenced class from class file content?
It seems like everybody has had an unpleasant brush with the Java Service Provider, that thing you can do with a file named like META-INF/services/com.example.Interface, but that nobody uses except for trying to load the right XML parser. I'm trying to work with a library that uses the Service Provider API, and trick it so that I can provide some runtime-extended classes (using cglib) that don't actually implement the interface but can be made to do so easily.
Basically, I think the steps I need to perform are:
Create a custom class loader that will respond to getResources(...) and return an "extra" URL
Also have that class loader hook getResourceAsStream(...) to return a list of the classes I am going to manipulate with cglib, when asked for the "extra" resource
Finally, have that class loader load those classes when requested
But here's where I get lost. For example, when the library tries to determine what implementers are out there, it calls getResources(...) which returns a bunch of URLs. But getResourceAsStream(...) doesn't take URLs, it takes "names". Names which seem to be classpath-relative, and therefore the same everywhere. So META-INF/services/com.example.Interface in has the same "name" as META-INF/services/com.example.Interface in their JAR, right? Except somehow this works with those blasted XML parsers...
Of course, all of this assumes they were smart/kind enough to call ClassLoader.getSystemClassLoader() rather than using ClassLoader.getSystemResources(...), ClassLoader.getSystemResourceAsStream(...), etc., as in the latter case there's no way to hook the ClassLoader and provide the faked file.
I guess in that case I could use BCEL to manipulate the class files when my code is being packaged by Maven, rather than waiting until runtime to do it with cglib?
The idea I described was along the right track. The mistake I made was in thinking that using ClassLoader.getResourceAsStream(..) to access the contents of the URLs. Instead, you should just URL.openStream().
Had I found it before posting, java.util.ServiceLoader (#since 1.6) provides some insight into how to do things correctly.
My application consists of several JAR files. I would like to iterate over all of the JAR files and find all classes that subclass a particular class, so that I can cause their static initializers to run.
I looked though the Javadocs for java.lang.ClassLoader, but can't find any method that does this.
I am trying to implement the "Product Trader" pattern (http://www.ubilab.com/publications/print_versions/pdf/plop-96-producttrader.pdf), with the classes "self-registering" in the abstract superclass. The superclass will have a HashMap that maps the service a subclass provides and the java.lang.Class file that handles that service.
The problem is that a class does not run it's static initializers until it is loaded. If I can iterate all of the subclasses, I can force each to load and run it's initializers.
Thanks, Ralph
This is not generally solvable. A ClassLoader is not required to be able to iterate over the Classes it could load. The easiest example of a ClassLoader that can't do that is when you load classes from a http:// base URL: HTTP provides no standardized way to enumerate the content of any given URL.
An better way to implement this is to use the ServiceLoader and let all implementing classes register themselves via a simple entry in their jar file.
This is obviously not solvable very simply.
However, there are solutions, like the one mentionned in this Javaworld article.
Generally speaking, it consists into exploration all elements from the CLASSPATH. if they're jar files, explore them looking for classes, load those classes and see if they extend you reference base class.
However, i would strongly suggest you use a more "mature" mechanism like IoC or a plugin architecture.
I'm obviously thinking about JSPF, as an example.
If you really need to do this the only real way is to iterate through the classpath, scanning the jars and directories for class files, loading that class and looking at it's parent class.
Note, some classes will have static initialising code that can have bad side affects, e.g. loading X11 classes from the runtime can hang for a long long time. If possible try to restrict what classes you load to specific packages, e.g. com.company (you can of course identify the package by the path of the class file relative to the root to the classpath item).
[Note the ServiceLoader suggested by Joachim Sauer, or a framework that provides a similar mechanism is a much better solution]