I have a question about java ClassLoaders. I am trying to use different ClassLoaders to be able to run different versions of a JAR from within the same program.
I have heard somewhere that if you load one class using one ClassLoader all classes called (being loaded) from within that class would use the same ClassLoader. Is this correct?
If not, is there a neat way to set the context of a ClassLoader (let's say, everything being called from a specific class/library should use the same ClassLoader).
This is not a simple subject and i would advise doing more research online as no answer here will be nearly in depth enough. but, as a quick synopsis:
classes loaded via normal class references (i.e. a line of code in Class A which uses a variable of static type B) will be loaded using the same classloader as the initial class.
however, due to classloader delegation, a class may not actually be loaded by the ClassLoader from which the search originally started. example, i have Class A loaded by classloader LA with parent classloader LP. Class B is referenced by A, so the search for Class B will start with LA. however, the class bytes for B are actually found in LP, so LP loads the class and hands it to LA which returns it. ultimately, however, B is owned by LP, not LA.
with utilities which load classes via reflection (e.g. serialization, JAXB, Hibernate, etc.) or frameworks which are typically used with nested classloaders (e.g. Java EE appservers), all bets are off. typically utilities/frameworks like this load classes using the context classloader, but that is not always the case. each utility may have different priorities and fallbacks regarding which classloader is used. additionally, many have ways of explicitly providing a classloader at runtime.
as a rule of thumb, while executing code which you know is from a nested classloader (probably because you set it up), you should set the current context classloader appropriately.
Related
I am desperate for help but was unable to find anything on the web about this particular subject (many related ones that leave my particular problem unanswered).
Specifically, I need to be able to download code (jars) from a central and external code repository. This is done by the bootstrap code that needs to add this to the classpath of a class loader to be used thereafter. This is when we enter the subject that has been discussed so many times. I don't like hacks, so I tried the following:
Attempt #1: Create an instance of URLClassLoader configured for this purpose, then invoke the "rest" of the code through it.
Failure: There are 1.5 problems here (one may be the cause of another). One is that URLClassLoader, normally, prefers to load stuff from its parent. Some code has to exist in both, possibly different versions. If the code from the parent is used, it continues using the "outer" class loader for the rest of loading, which is not what we want, even when the initial loading is OK. Secondly, some third party libraries seem to access the system class loader directly, either by design or accidentally (may get it from one of the classes loaded by it).
Attempt #2: Create my subclass of the URLClassLoader that prefers self over the parent. Overrode loadClass, getResource, getResources, getPackage, getPackages... later other methods too to make sure of this.
Failure: Didn't help (enough). That third party code still couldn't load some resources.
Attempt #3 Create another custom subclass of the URLClassLoader and set it as the system class loader using -Djava.system.class.loader=...
Failure: This worked better - went further, but still failed trying to get resources. This time it was different resources, though. I added logging to all the overridden methods to log their calls and resource names. Regular resources were MOSTLY found. Some still weren't, even though they are there (confirmed). But something I don't know about even though I tried hard to learn is about many calls with resource names that end with a slash. Some also have slashes where a dollar sign would normally appear (nested/inner class resources). Some examples that were requested but NOT found:
com/acme/foo/bar/ClassName/
com/acme/foo/bar/ClassName/InnerClassName/
When I run the downloaded code with all content on the initial/boot classpath (and do not use my classloader), everything works fine - thus my class loader breaks things, but I need it to work.
My closest guesses are:
Third party code gets hold of the true system class loader somehow, perhaps via some class that was loaded by it, then uses that. I don't see requests to it and they are bound to fail because it does not have the entire class path.
This business with resource names ending in slashes is the cause by being supported by the true system class loader but not by the URLClassLoader I am subclassing. I can only guess that the expected return URL somehow locates the collection of resources with that name as prefix. That would be tough to match, although possible. Furthermore, it appears that some slashes are in positions where a dollar sign separating the inner class name should be, i.e. in the above example (spaces added for clarity):
com/acme/foo/bar/ClassName / InnerClassName/
com/acme/foo/bar/ClassName $ InnerClassName/
Please note that I cannot rely on hacking the actual system classloader by assuming that it is a subclass of the URLClassLoader and using reflection to call its addURL(URL) method.
Is there a way to make this work? Please help!
UPDATE
I just made an additional attempt. I created a dummy wrapper classloader (extending ClassLoader, not URLClassLoader) that only logs requests, then passes them on to the parent (public methods) or superclass (protected methods). I set this to be the system class loader and manually added the entire "inner" class path to the actual outer one, then tried to run the code. That works correctly, just as it does without the custom system class loader. What was logged also identified that even the system class loader return null for these resources ending in slashes for MOST of them, but not all. I did not check whether these also work in the my real code but guessing they may - as they were not the stumbling block. Somehow the custom system classloader is still being bypassed. How?
UPDATE 2
In my custom system class loaders I have let some classes come from the outer/true system class loader, e.g. those in java.lang. I am now suspecting that I should not have and that the inner "world" must be completely isolated. That would make it problematic, though, to communicate with it and all I would have left is reflection... but not sure whether that would even work - i.e. can there be more than one java.lang.Class and/or java.lang.Object?
Given all constraints this does not appear entirely possible in a rock solid fashion as I wanted it:
a) Third party libraries may always "misbehave" and get hold of lassloaders they are not supposed to use one way or another. I looked at OneJar as suggested by fge but they have the same issue - they only detect a possibility of it.
b) There is no way to completely replace/hide the system class loader.
c) Casting the system class loader to a URLClassLoader may stop working at any moment.
It seems, you didn’t understand the class loader structure. Within an ordinary JVM of the current version, there are at least three class loaders:
The bootstrap loader which is responsible for loading the core classes. Since this involves classes like Class and ClassLoader itself, it can’t be represented by a ClassLoader instance. All classes whose getClassLoader() returns null were loaded by the bootstrap loader
The extension loader. It is responsible for loading classes within the ext/ directory of the JRE. Afaik, it may vanish in future versions. Its parent loader is the bootstrap loader
The application loader. This is the one which will be returned by ClassLoader.getSystemClassLoader() and which will be used if no other parent was specified. In the current configurations, it’s parent is the extension loader, but maybe it will have the bootstrap loader as its direct parent in future versions
The conclusion is, if you want to reload your application’s classes without the delegation to the parent loader destroying your effort, you don’t need to manipulate the class loader’s implementation. You just have to specify the right parent. It’s as simple as
URLClassLoader cl=new URLClassLoader(urls, ClassLoader.getSystemClassLoader().getParent());
That way, the new class loader’s parent will be the original application class loader’s parent, thus the already loaded application classes are not in the scope of the new loader while everything else works as usual.
Regarding the resources ending with a slash, they are rather uncommon. They may get resolved when they actually refer to a directory but that depends on the protocol of the URL and the actual handler for that protocol. E.g. it might work for file: URLs but usually doesn’t for jar: URLs unless the jar file contains pseudo-entries for directories. I’ve also seen it working for ftp: URLs.
Another thing to understand is that if one class directly refers to another class, its original defining class loader will be queried, not necessarily the application class loader. E.g. when the class java.lang.String contains a reference to java.lang.Object (it does), this reference will be directly resolved using the bootstrap loader as this is the defining loader of java.lang.String.
This implies that if you manipulate the parent lookup of a loader to not follow the standard parent delegation you are risking to resolve names to different runtime classes as the resolving of the same names when being referenced by classes loaded by the parent loader. You avoid such problems by following the standard procedure as in the solution above. The JRE classes will never contain references to your application classes and the new loader not having the original application loader as its parent will never interfere with the classes loaded by the original application loader.
What is the difference between a thread's context class loader and a normal class loader?
That is, if Thread.currentThread().getContextClassLoader() and getClass().getClassLoader() return different class loader objects, which one will be used?
This does not answer the original question, but as the question is highly ranked and linked for any ContextClassLoader query, I think it is important to answer the related question of when the context class loader should be used. Short answer: never use the context class loader! But set it to getClass().getClassLoader() when you have to call a method that is missing a ClassLoader parameter.
When code from one class asks to load another class, the correct class loader to use is the same class loader as the caller class (i.e., getClass().getClassLoader()). This is the way things work 99.9% of the time because this is what the JVM does itself the first time you construct an instance of a new class, invoke a static method, or access a static field.
When you want to create a class using reflection (such as when deserializing or loading a configurable named class), the library that does the reflection should always ask the application which class loader to use, by receiving the ClassLoader as a parameter from the application. The application (which knows all the classes that need constructing) should pass it getClass().getClassLoader().
Any other way to obtain a class loader is incorrect. If a library uses hacks such as Thread.getContextClassLoader(), sun.misc.VM.latestUserDefinedLoader(), or sun.reflect.Reflection.getCallerClass() it is a bug caused by a deficiency in the API. Basically, Thread.getContextClassLoader() exists only because whoever designed the ObjectInputStream API forgot to accept the ClassLoader as a parameter, and this mistake has haunted the Java community to this day.
That said, many many JDK classes use one of a few hacks to guess some class loader to use. Some use the ContextClassLoader (which fails when you run different apps on a shared thread pool, or when you leave the ContextClassLoader null), some walk the stack (which fails when the direct caller of the class is itself a library), some use the system class loader (which is fine, as long as it is documented to only use classes in the CLASSPATH) or bootstrap class loader, and some use an unpredictable combination of the above techniques (which only makes things more confusing). This has resulted in much weeping and gnashing of teeth.
When using such an API, first, try to find an overload of the method that accepts the class loader as a parameter. If there is no sensible method, then try setting the ContextClassLoader before the API call (and resetting it afterwards):
ClassLoader originalClassLoader = Thread.currentThread().getContextClassLoader();
try {
Thread.currentThread().setContextClassLoader(getClass().getClassLoader());
// call some API that uses reflection without taking ClassLoader param
} finally {
Thread.currentThread().setContextClassLoader(originalClassLoader);
}
Each class will use its own classloader to load other classes. So if ClassA.class references ClassB.class then ClassB needs to be on the classpath of the classloader of ClassA, or its parents.
The thread context classloader is the current classloader for the current thread. An object can be created from a class in ClassLoaderC and then passed to a thread owned by ClassLoaderD. In this case the object needs to use Thread.currentThread().getContextClassLoader() directly if it wants to load resources that are not available on its own classloader.
There is an article on infoworld.com that explains the difference
=> Which ClassLoader should you use
(1)
Thread context classloaders provide a
back door around the classloading
delegation scheme.
Take JNDI for instance: its guts are
implemented by bootstrap classes in
rt.jar (starting with J2SE 1.3), but
these core JNDI classes may load JNDI
providers implemented by independent
vendors and potentially deployed in
the application's -classpath. This
scenario calls for a parent
classloader (the primordial one in
this case) to load a class visible to
one of its child classloaders (the
system one, for example). Normal J2SE
delegation does not work, and the
workaround is to make the core JNDI
classes use thread context loaders,
thus effectively "tunneling" through
the classloader hierarchy in the
direction opposite to the proper
delegation.
(2) from the same source:
This confusion will probably stay with
Java for some time. Take any J2SE API
with dynamic resource loading of any
kind and try to guess which loading
strategy it uses. Here is a sampling:
JNDI uses context classloaders
Class.getResource() and Class.forName() use the current classloader
JAXP uses context classloaders (as of J2SE 1.4)
java.util.ResourceBundle uses the caller's current classloader
URL protocol handlers specified via java.protocol.handler.pkgs system property are looked up in the bootstrap and system classloaders only
Java Serialization API uses the caller's current classloader by default
Adding to #David Roussel answer, classes may be loaded by multiple class loaders.
Lets understand how class loader works.
From javin paul blog in javarevisited :
ClassLoader follows three principles.
Delegation principle
A class is loaded in Java, when its needed. Suppose you have an application specific class called Abc.class, first request of loading this class will come to Application ClassLoader which will delegate to its parent Extension ClassLoader which further delegates to Primordial or Bootstrap class loader
Bootstrap ClassLoader is responsible for loading standard JDK class files from rt.jar and it is parent of all class loaders in Java. Bootstrap class loader don't have any parents.
Extension ClassLoader delegates class loading request to its parent, Bootstrap and if unsuccessful, loads class form jre/lib/ext directory or any other directory pointed by java.ext.dirs system property
System or Application class loader and it is responsible for loading application specific classes from CLASSPATH environment variable, -classpath or -cp command line option, Class-Path attribute of Manifest file inside JAR.
Application class loader is a child of Extension ClassLoader and its implemented by sun.misc.Launcher$AppClassLoader class.
NOTE: Except Bootstrap class loader, which is implemented in native language mostly in C, all Java class loaders are implemented using java.lang.ClassLoader.
Visibility Principle
According to visibility principle, Child ClassLoader can see class loaded by Parent ClassLoader but vice-versa is not true.
Uniqueness Principle
According to this principle a class loaded by Parent should not be loaded by Child ClassLoader again
I have two different webapps, and each load the same class A with different classloader. When I put one instance in the session and then get it from the other webapp, a ClassCastException is thrown.
For example, in webapp A, I store a in the session, then in webapp B, I get the a from the session and cast it to A, the ClassCastException is thrown.
Is there a way to resolve this?
Is there a way to resolve this?
Basically no.
As far as the JLS is concerned, the types are different types, and there is no way that the JVM will allow you to pretend otherwise. For instance, the classes could have different code and different object layouts. If you could trick the JVM into treating the types as the same, you would be able to blow away JVM runtime safety. That way lies insanity.
The solution is to make sure that you don't have two different class loaders loading the same class. In the context of Tomcat, this means that if two or more webapps need to share instances of a class, then that class must be defined in a classloader that is common to both; e.g. put the JAR file in the $CATALINA_HOME/lib or $CATALINA_HOME/common directory.
If there is a technical reason why the classes have to be loaded by different classloaders (maybe because the classes really are different), then you could work around the problem by defining an interface that both versions of the class implement, and then programming to the interface rather than the implementation class. Of course, the interface needs to be loaded by a shared classloader ... or else you run into the same problem again.
You should avoid this situation, basically - either put both bits of functionality in the same webapp, or move the library containing class A into an appropriate location such that only one classloader will be used. Two classes loaded by different classloaders are entirely distinct in the JVM - you simply won't be able to cast between them.
See the Tomcat classloader documentation for more details about the various classloaders used. It looks like you'd want to put this common class into the common classloader area. As the documentation notes, this is pretty unusual, but if you really want to share an object between two webapps (which is also unusual) it's probably the easiest way forward.
You can't. Two classes loaded by different classloaders are different.
Perhaps you can serialize the shared objects?
I'm not necessarily advocating this approach, but then again that's what serialization essentially does --- you serialize from some JVM or ClassLoader X and load it in (deserialize) it into another JVM/ClassLoader Y...
You can't cast two object from different classes even if the classes have the same package name and signature, however you can copy the data from one to another simply by using apache bean utils library, BeanUtils.copyProperties(o1, o2);
This is possible with a workaround.
While it is true that you can't cast an object from one class loaded by classloader A to the same class loaded by classloader B (classes with the same name are not compatible if loaded under different classloaders as explained here), in a webapp container such as Jetty or Tomcat, you can load that class once in a parent classloader, which will be used by all webapps in the JVM. Each webapp classloader will defer to the shared (parent) classloader for the class definition and casting it back and forth works just fine.
For example, with Jetty, use WebAppContext.addSystemClass() as described here.
I have a custom class loader so that a desktop application can dynamically start loading classes from an AppServer I need to talk to. We did this since the amount of jars that are required to do this are ridiculous (if we wanted to ship them). We also have version problems if we don't load the classes dynamically at run time from the AppServer library.
Now, I just hit a problem where I need to talk to two different AppServers and found that depending on whose classes I load first I might break badly... Is there any way to force the unloading of the class without actually killing the JVM?
Hope this makes sense
The only way that a Class can be unloaded is if the Classloader used is garbage collected. This means, references to every single class and to the classloader itself need to go the way of the dodo.
One possible solution to your problem is to have a Classloader for every jar file, and a Classloader for each of the AppServers that delegates the actual loading of classes to specific Jar classloaders. That way, you can point to different versions of the jar file for every App server.
This is not trivial, though. The OSGi platform strives to do just this, as each bundle has a different classloader and dependencies are resolved by the platform. Maybe a good solution would be to take a look at it.
If you don't want to use OSGI, one possible implementation could be to use one instance of JarClassloader class for every JAR file.
And create a new, MultiClassloader class that extends Classloader. This class internally would have an array (or List) of JarClassloaders, and in the defineClass() method would iterate through all the internal classloaders until a definition can be found, or a NoClassDefFoundException is thrown. A couple of accessor methods can be provided to add new JarClassloaders to the class. There is several possible implementations on the net for a MultiClassLoader, so you might not even need to write your own.
If you instanciate a MultiClassloader for every connection to the server, in principle it is possible that every server uses a different version of the same class.
I've used the MultiClassloader idea in a project, where classes that contained user-defined scripts had to be loaded and unloaded from memory and it worked quite well.
Yes there are ways to load classes and to "unload" them later on. The trick is to implement your own classloader which resides between high level class loader (the System class loader) and the class loaders of the app server(s), and to hope that the app server's class loaders do delegate the classloading to the upper loaders.
A class is defined by its package, its name, and the class loader it originally loaded. Program a "proxy" classloader which is the first that is loaded when starting the JVM. Workflow:
The program starts and the real "main"-class is loaded by this proxy classloader.
Every class that then is normally loaded (i.e. not through another classloader implementation which could break the hierarchy) will be delegated to this class loader.
The proxy classloader delegates java.x and sun.x to the system classloader (these must not be loaded through any other classloader than the system classloader).
For every class that is replaceable, instantiate a classloader (which really loads the class and does not delegate it to the parent classloader) and load it through this.
Store the package/name of the classes as keys and the classloader as values in a data structure (i.e. Hashmap).
Every time the proxy classloader gets a request for a class that was loaded before, it returns the class from the class loader stored before.
It should be enough to locate the byte array of a class by your class loader (or to "delete" the key/value pair from your data structure) and reload the class in case you want to change it.
Done right there should not come a ClassCastException or LinkageError etc.
For more informations about class loader hierarchies (yes, that's exactly what you are implementing here ;- ) look at "Server-Based Java Programming" by Ted Neward - that book helped me implementing something very similar to what you want.
I wrote a custom classloader, from which it is possible to unload individual classes without GCing the classloader. Jar Class Loader
Classloaders can be a tricky problem. You can especially run into problems if you're using multiple classloaders and don't have their interactions clearly and rigorously defined. I think in order to actually be able to unload a class youlre going go have to remove all references to any classes(and their instances) you're trying to unload.
Most people needing to do this type of thing end up using OSGi. OSGi is really powerful and surprisingly lightweight and easy to use,
You can unload a ClassLoader but you cannot unload specific classes. More specifically you cannot unload classes created in a ClassLoader that's not under your control.
If possible, I suggest using your own ClassLoader so you can unload.
Classes have an implicit strong reference to their ClassLoader instance, and vice versa. They are garbage collected as with Java objects. Without hitting the tools interface or similar, you can't remove individual classes.
As ever you can get memory leaks. Any strong reference to one of your classes or class loader will leak the whole thing. This occurs with the Sun implementations of ThreadLocal, java.sql.DriverManager and java.beans, for instance.
If you're live watching if unloading class worked in JConsole or something, try also adding java.lang.System.gc() at the end of your class unloading logic. It explicitly triggers Garbage Collector.
I have a custom class loader so that a desktop application can dynamically start loading classes from an AppServer I need to talk to. We did this since the amount of jars that are required to do this are ridiculous (if we wanted to ship them). We also have version problems if we don't load the classes dynamically at run time from the AppServer library.
Now, I just hit a problem where I need to talk to two different AppServers and found that depending on whose classes I load first I might break badly... Is there any way to force the unloading of the class without actually killing the JVM?
Hope this makes sense
The only way that a Class can be unloaded is if the Classloader used is garbage collected. This means, references to every single class and to the classloader itself need to go the way of the dodo.
One possible solution to your problem is to have a Classloader for every jar file, and a Classloader for each of the AppServers that delegates the actual loading of classes to specific Jar classloaders. That way, you can point to different versions of the jar file for every App server.
This is not trivial, though. The OSGi platform strives to do just this, as each bundle has a different classloader and dependencies are resolved by the platform. Maybe a good solution would be to take a look at it.
If you don't want to use OSGI, one possible implementation could be to use one instance of JarClassloader class for every JAR file.
And create a new, MultiClassloader class that extends Classloader. This class internally would have an array (or List) of JarClassloaders, and in the defineClass() method would iterate through all the internal classloaders until a definition can be found, or a NoClassDefFoundException is thrown. A couple of accessor methods can be provided to add new JarClassloaders to the class. There is several possible implementations on the net for a MultiClassLoader, so you might not even need to write your own.
If you instanciate a MultiClassloader for every connection to the server, in principle it is possible that every server uses a different version of the same class.
I've used the MultiClassloader idea in a project, where classes that contained user-defined scripts had to be loaded and unloaded from memory and it worked quite well.
Yes there are ways to load classes and to "unload" them later on. The trick is to implement your own classloader which resides between high level class loader (the System class loader) and the class loaders of the app server(s), and to hope that the app server's class loaders do delegate the classloading to the upper loaders.
A class is defined by its package, its name, and the class loader it originally loaded. Program a "proxy" classloader which is the first that is loaded when starting the JVM. Workflow:
The program starts and the real "main"-class is loaded by this proxy classloader.
Every class that then is normally loaded (i.e. not through another classloader implementation which could break the hierarchy) will be delegated to this class loader.
The proxy classloader delegates java.x and sun.x to the system classloader (these must not be loaded through any other classloader than the system classloader).
For every class that is replaceable, instantiate a classloader (which really loads the class and does not delegate it to the parent classloader) and load it through this.
Store the package/name of the classes as keys and the classloader as values in a data structure (i.e. Hashmap).
Every time the proxy classloader gets a request for a class that was loaded before, it returns the class from the class loader stored before.
It should be enough to locate the byte array of a class by your class loader (or to "delete" the key/value pair from your data structure) and reload the class in case you want to change it.
Done right there should not come a ClassCastException or LinkageError etc.
For more informations about class loader hierarchies (yes, that's exactly what you are implementing here ;- ) look at "Server-Based Java Programming" by Ted Neward - that book helped me implementing something very similar to what you want.
I wrote a custom classloader, from which it is possible to unload individual classes without GCing the classloader. Jar Class Loader
Classloaders can be a tricky problem. You can especially run into problems if you're using multiple classloaders and don't have their interactions clearly and rigorously defined. I think in order to actually be able to unload a class youlre going go have to remove all references to any classes(and their instances) you're trying to unload.
Most people needing to do this type of thing end up using OSGi. OSGi is really powerful and surprisingly lightweight and easy to use,
You can unload a ClassLoader but you cannot unload specific classes. More specifically you cannot unload classes created in a ClassLoader that's not under your control.
If possible, I suggest using your own ClassLoader so you can unload.
Classes have an implicit strong reference to their ClassLoader instance, and vice versa. They are garbage collected as with Java objects. Without hitting the tools interface or similar, you can't remove individual classes.
As ever you can get memory leaks. Any strong reference to one of your classes or class loader will leak the whole thing. This occurs with the Sun implementations of ThreadLocal, java.sql.DriverManager and java.beans, for instance.
If you're live watching if unloading class worked in JConsole or something, try also adding java.lang.System.gc() at the end of your class unloading logic. It explicitly triggers Garbage Collector.