Recently I came accross the java custom class loader api. I found one use over here, kamranzafar's blog
I am a bit new to the class loader concept. Can any one explain in detail, what are the different scenarios where we may need it or we should use it?
Custom class loaders are useful in larger architectures consisting of several module/applications. Here are the advantages of the custom class loader:
Provides Modular architecture Allows to define multiple class loader allowing modular architecture.
Avoiding conflicts Clearly defines the scope of the class to within the class loader.
Support Versioning Supports different versions of class within same VM for different modules.
Better Memory Management Unused modules can be removed which unloads the classes used by that module, which cleans up memory.
Load classes from anywhere Classes can be loaded from anywhere, for ex, Database, Networks, or even define it on the fly.
Add resources or classes dynamically All the above features allows you add classes or resources dynamically.
Runtime Reloading Modified Classes Allows you to reload a class or classes runtime by creating a child class loader to the actual class loader, which contains the modified classes.
The primary use is in Application servers so that they can run two applications and not have the classes conflict. i.e. if application 1 has a class with the same name as application 2, with a custom class loader application 1 will load its class and application 2 will load its class.
Also if a class is loaded by a custom class loader it is possible to unload that class from the JVM. Again useful in application servers.
Another use would be for instrumentation - One way of doing aspect oriented programming or when using some persistence API's. With a custom classloader you can add behaviour to the loaded classes before they are passed over to the running application.
Java class loaders do pretty much what the name suggests: load classes into memory so that they can be used.
Classes are also linked with the ClassLoader that loaded them.
Custom class loaders therefore open up a variety of interesting possibilities:
Loading multiple versions of the same class with different classloaders (e.g. to resolve possible versioning conficts for example)
Loading and unloading classes dynamically at runtime
Generating new classes (e.g. JVM languages like Clojure use various classloading tricks to generate new compiled classes to represent Clojure functions at runtime)
Loading classes from non-standard sources
Normal Java applications don't usually need to worry about classloaders. But if you are writing a framework or platform that needs to host other code then they become much more important / relevant.
Related
Since Tomcat can load more than one webapp at once, and those webapps can work separately, and do not disturb each other, and they work in same JVM.
So I am very confused about how tomcat handle The Object scope in the same JVM.
For example I have a singleton object in both of the two different Webapps, and tomcat will generate two different singleton Object for each. I always thought that the singleton object have only one object in the same JVM, but in tomcat JVM there maybe has two or more.
I have read some info about the ClassLoader, Tomcat has its own WebAppClassLoader to load webapps. So does it mean the Object Scope here is the ClassLoader or am I wrong. Does anyone know about this or can give me some info about tomcat work memory layout?
All the secrets is behind those ClassLoader instances.
The state of the class (like all static variables, byte code and so on) is scoped by the class loader which loads that class (a class is identified in JVM by its fully qualiflied name and the class loader loading the class. This is not exactly a scope, but thinking as scope usually helps understanding this better).
So if a class is loaded by two different class loaders, this class exists twice within the VM, it has two sets of static fields, can have different byte code (like different method implementations) and all such. Note that these two objects cannot be cast to each other even their names are identical.
"Normal" Java applications have all classes loaded by a class loader hierarchy and every class is only loaded once.
For more complex scenarios, you will need different behaviours. Sometimes you want to isolate a library from messing with your code (like plugins in eclipse or web applications in an application server).
The basic idea to isolate your program from other classes is to load those with an extra class loader and use a lot of reflection. If you want to read up on this have a look at Oracle's documentation on ClassLoaders or OSGI.
Tomcat (and a lot of other web containers / application servers) load the application with separate ClassLoader hierarchies. This isolates all classes against other (web) applications and thus also makes sure, that singletons, different class versions and all this stuff does not collide.
Remember that a class in Java is identified by its fully qualified name and the classloader that loaded it. Tomcat uses separate classloaders for each context (web application) that you deploy, thus keeping them separate. In addition, the system classloader loads the tomcat specific libraries and JVM bootstrap loader load the Java core libraries.
In normal Java applications when a classloader is asked to load a class it forst delegates the request to it's parent class loader and then loads it if parent class loaders cannot find the requested class.
For web application servers this slightly differs. There are generally different class loader for each web app deployed in a web application server like tomcat. For Tomcat it looks like below -
So for web apps class loading resource happens in following order -
Bootstrap classes of your JVM (Core java classes)
/WEB-INF/classes of your web application
/WEB-INF/lib/*.jar of your web application
System class loader classes (Tomcat / Classpath specific classes)
Common class loader classes (classes common to all web apps)
But note if web application class loader is configured with delegate="true" then order is changed -
Bootstrap classes of your JVM (Core java classes)
System class loader classes (Tomcat / Classpath specific classes)
Common class loader classes (classes common to all web apps)
/WEB-INF/classes of your web application
/WEB-INF/lib/*.jar of your web application
For more details you can check Apache Tomcat's Class Loader HOW-TO page.
One thing that always gets left out when talking about singletons, is that a singleton can have only one instance per classloader. A ClassLoader limits class visibility, so the same class can exist under several different classloaders in the same VM. This allows you, among other things, to have different versions of jars loaded at the same time.
This question: Java Class Loaders seems to have some nice links and resources for further studying.
The "ID" of a class in the JVM consists of the fully qualified class name and the class loader that was used to load it. This means, if you load two classes with the same name by different class loaders they are considered different classes.
So a singleton would be singleton for a classloader - in a container/JVM; as a container/JVM might have multiple classloaders.
Different App in tomcat using different classloader to seperate. For example app1 using ClassLoaderA, app2 using classloaderB.
Each class will use its own classloader to load other classes. So if ClassA.class references ClassB.class then ClassB needs to be on the classpath of the classloader of ClassA, or its parents. For example In app1, com.exmaple.test1 loaded from ClassLoaderA. And com.exmaple.test1 want to new com.exmaple.test2(). By default It use its own classloader ClassLoaderA to load com.exmaple.test2. So In the view of com.exmaple.test1 it can only see its own classpath's class(app1/webapp/classes or app1/webapp/lib). And In app2, It will see a different view.
In summary learn classloader you must understand the delegation model. And the visibility is the child can see the parent. but the parent can not see the child and the sibling can not see the sibling. So we can isolate different app.
I'm trying to understand the security model used when the JVM is asked to load classes.
From the JVM specification on Sandboxing, I'm given to believe that a standard JVM implementation should maintain at least one other ClassLoader, independent of the primordial ClassLoader. This is used to load the application class files (from a provided classpath for example).
If the class is requested from the ClassLoader that is not in it's namespace, java/lang/String for example, then it forwards the request to the primordial ClassLoader, which attempts to load the class from the Java API, if its not there then it throws a NoClassDefFoundError.
Am I right in thinking that the primordial ClassLoader only loads classes from the Java API namespace, and all other classes are loaded via a separate ClassLoader implementation?
And that this makes the loading of classes more secure because it means that a malicious class cannot masquerade as a member of the Java API (lets say java/lang/Virus) because this is a protected namespace, and cannot be used in the current ClassLoader?
But is there anything to prevent the Classes of the Java API being replaced by malicious classes, or would that be detected during class verification?
For historical reasons the names used for class loaders are a little peculiar. The boot class loader loads the systems classes. The system class loader, by default, loads classes from the class path not the system classes. The system classes are in jre/lib (mostly in rt.jar), endorsed directories and anywhere added through -Xbootclasspath.
On the Sun/Oracle JRE, rt.jar contains classes with packages starting with java., javax., sun., com.sun., org.omg, org.w3c and org.xml.
Untrusted code (including configuration) should not be able to add to the system classes. Some packages name prefixed may be restricted through a security property. The java. prefix is specially protected against for non-technical reasons.
Generally a class loader will delegate to its parent before defining a new class, preventing any classes from an ancestor loader from being replaced. Java EE recommends (even though Java SE bans) having some class loaders prefer their own classes, typically to use a more up to date API or a different implementation. This allows shadowing of classes, but only as seen through that loader (and its children). All other classes continue to link to the original.
I am looking for a simple scenario when can be requirement to create a custom class loader?
Apart from that, just wanted to confirm that Does even bootstrap and application class loader extends java.lang.ClassLoader internally?
In short, it can be used to add new code or to change existing code on the fly. Here's a more detailed explanation from this URL.
Why write a custom class loader?
The three main reasons for wanting to create a custom class loader are:
To allow class loading from alternative repositories.
This is the most common case, in which an application developer might want to load classes from other locations, for example, over a network connection.
To partition user code.
This case is less frequently used by application developers, but widely used in servlet engines.
To allow the unloading of classes.
This case is useful if the application creates large numbers of classes that are used for only a finite period. Because a class loader maintains a cache of the classes that it has loaded, these classes cannot be unloaded until the class loader itself has been dereferenced. For this reason, system and extension classes are never unloaded, but application classes can be unloaded when their classloader is.
I have a custom class loader so that a desktop application can dynamically start loading classes from an AppServer I need to talk to. We did this since the amount of jars that are required to do this are ridiculous (if we wanted to ship them). We also have version problems if we don't load the classes dynamically at run time from the AppServer library.
Now, I just hit a problem where I need to talk to two different AppServers and found that depending on whose classes I load first I might break badly... Is there any way to force the unloading of the class without actually killing the JVM?
Hope this makes sense
The only way that a Class can be unloaded is if the Classloader used is garbage collected. This means, references to every single class and to the classloader itself need to go the way of the dodo.
One possible solution to your problem is to have a Classloader for every jar file, and a Classloader for each of the AppServers that delegates the actual loading of classes to specific Jar classloaders. That way, you can point to different versions of the jar file for every App server.
This is not trivial, though. The OSGi platform strives to do just this, as each bundle has a different classloader and dependencies are resolved by the platform. Maybe a good solution would be to take a look at it.
If you don't want to use OSGI, one possible implementation could be to use one instance of JarClassloader class for every JAR file.
And create a new, MultiClassloader class that extends Classloader. This class internally would have an array (or List) of JarClassloaders, and in the defineClass() method would iterate through all the internal classloaders until a definition can be found, or a NoClassDefFoundException is thrown. A couple of accessor methods can be provided to add new JarClassloaders to the class. There is several possible implementations on the net for a MultiClassLoader, so you might not even need to write your own.
If you instanciate a MultiClassloader for every connection to the server, in principle it is possible that every server uses a different version of the same class.
I've used the MultiClassloader idea in a project, where classes that contained user-defined scripts had to be loaded and unloaded from memory and it worked quite well.
Yes there are ways to load classes and to "unload" them later on. The trick is to implement your own classloader which resides between high level class loader (the System class loader) and the class loaders of the app server(s), and to hope that the app server's class loaders do delegate the classloading to the upper loaders.
A class is defined by its package, its name, and the class loader it originally loaded. Program a "proxy" classloader which is the first that is loaded when starting the JVM. Workflow:
The program starts and the real "main"-class is loaded by this proxy classloader.
Every class that then is normally loaded (i.e. not through another classloader implementation which could break the hierarchy) will be delegated to this class loader.
The proxy classloader delegates java.x and sun.x to the system classloader (these must not be loaded through any other classloader than the system classloader).
For every class that is replaceable, instantiate a classloader (which really loads the class and does not delegate it to the parent classloader) and load it through this.
Store the package/name of the classes as keys and the classloader as values in a data structure (i.e. Hashmap).
Every time the proxy classloader gets a request for a class that was loaded before, it returns the class from the class loader stored before.
It should be enough to locate the byte array of a class by your class loader (or to "delete" the key/value pair from your data structure) and reload the class in case you want to change it.
Done right there should not come a ClassCastException or LinkageError etc.
For more informations about class loader hierarchies (yes, that's exactly what you are implementing here ;- ) look at "Server-Based Java Programming" by Ted Neward - that book helped me implementing something very similar to what you want.
I wrote a custom classloader, from which it is possible to unload individual classes without GCing the classloader. Jar Class Loader
Classloaders can be a tricky problem. You can especially run into problems if you're using multiple classloaders and don't have their interactions clearly and rigorously defined. I think in order to actually be able to unload a class youlre going go have to remove all references to any classes(and their instances) you're trying to unload.
Most people needing to do this type of thing end up using OSGi. OSGi is really powerful and surprisingly lightweight and easy to use,
You can unload a ClassLoader but you cannot unload specific classes. More specifically you cannot unload classes created in a ClassLoader that's not under your control.
If possible, I suggest using your own ClassLoader so you can unload.
Classes have an implicit strong reference to their ClassLoader instance, and vice versa. They are garbage collected as with Java objects. Without hitting the tools interface or similar, you can't remove individual classes.
As ever you can get memory leaks. Any strong reference to one of your classes or class loader will leak the whole thing. This occurs with the Sun implementations of ThreadLocal, java.sql.DriverManager and java.beans, for instance.
If you're live watching if unloading class worked in JConsole or something, try also adding java.lang.System.gc() at the end of your class unloading logic. It explicitly triggers Garbage Collector.
I have a custom class loader so that a desktop application can dynamically start loading classes from an AppServer I need to talk to. We did this since the amount of jars that are required to do this are ridiculous (if we wanted to ship them). We also have version problems if we don't load the classes dynamically at run time from the AppServer library.
Now, I just hit a problem where I need to talk to two different AppServers and found that depending on whose classes I load first I might break badly... Is there any way to force the unloading of the class without actually killing the JVM?
Hope this makes sense
The only way that a Class can be unloaded is if the Classloader used is garbage collected. This means, references to every single class and to the classloader itself need to go the way of the dodo.
One possible solution to your problem is to have a Classloader for every jar file, and a Classloader for each of the AppServers that delegates the actual loading of classes to specific Jar classloaders. That way, you can point to different versions of the jar file for every App server.
This is not trivial, though. The OSGi platform strives to do just this, as each bundle has a different classloader and dependencies are resolved by the platform. Maybe a good solution would be to take a look at it.
If you don't want to use OSGI, one possible implementation could be to use one instance of JarClassloader class for every JAR file.
And create a new, MultiClassloader class that extends Classloader. This class internally would have an array (or List) of JarClassloaders, and in the defineClass() method would iterate through all the internal classloaders until a definition can be found, or a NoClassDefFoundException is thrown. A couple of accessor methods can be provided to add new JarClassloaders to the class. There is several possible implementations on the net for a MultiClassLoader, so you might not even need to write your own.
If you instanciate a MultiClassloader for every connection to the server, in principle it is possible that every server uses a different version of the same class.
I've used the MultiClassloader idea in a project, where classes that contained user-defined scripts had to be loaded and unloaded from memory and it worked quite well.
Yes there are ways to load classes and to "unload" them later on. The trick is to implement your own classloader which resides between high level class loader (the System class loader) and the class loaders of the app server(s), and to hope that the app server's class loaders do delegate the classloading to the upper loaders.
A class is defined by its package, its name, and the class loader it originally loaded. Program a "proxy" classloader which is the first that is loaded when starting the JVM. Workflow:
The program starts and the real "main"-class is loaded by this proxy classloader.
Every class that then is normally loaded (i.e. not through another classloader implementation which could break the hierarchy) will be delegated to this class loader.
The proxy classloader delegates java.x and sun.x to the system classloader (these must not be loaded through any other classloader than the system classloader).
For every class that is replaceable, instantiate a classloader (which really loads the class and does not delegate it to the parent classloader) and load it through this.
Store the package/name of the classes as keys and the classloader as values in a data structure (i.e. Hashmap).
Every time the proxy classloader gets a request for a class that was loaded before, it returns the class from the class loader stored before.
It should be enough to locate the byte array of a class by your class loader (or to "delete" the key/value pair from your data structure) and reload the class in case you want to change it.
Done right there should not come a ClassCastException or LinkageError etc.
For more informations about class loader hierarchies (yes, that's exactly what you are implementing here ;- ) look at "Server-Based Java Programming" by Ted Neward - that book helped me implementing something very similar to what you want.
I wrote a custom classloader, from which it is possible to unload individual classes without GCing the classloader. Jar Class Loader
Classloaders can be a tricky problem. You can especially run into problems if you're using multiple classloaders and don't have their interactions clearly and rigorously defined. I think in order to actually be able to unload a class youlre going go have to remove all references to any classes(and their instances) you're trying to unload.
Most people needing to do this type of thing end up using OSGi. OSGi is really powerful and surprisingly lightweight and easy to use,
You can unload a ClassLoader but you cannot unload specific classes. More specifically you cannot unload classes created in a ClassLoader that's not under your control.
If possible, I suggest using your own ClassLoader so you can unload.
Classes have an implicit strong reference to their ClassLoader instance, and vice versa. They are garbage collected as with Java objects. Without hitting the tools interface or similar, you can't remove individual classes.
As ever you can get memory leaks. Any strong reference to one of your classes or class loader will leak the whole thing. This occurs with the Sun implementations of ThreadLocal, java.sql.DriverManager and java.beans, for instance.
If you're live watching if unloading class worked in JConsole or something, try also adding java.lang.System.gc() at the end of your class unloading logic. It explicitly triggers Garbage Collector.