This question already has answers here:
Unloading classes in java?
(7 answers)
Closed 6 years ago.
How to unload a class from the class loader, so that I can use the recently changed class on the fly without restarting my application (hot deployment)? Is it possible to do that?
Java rebel reloads changed classes on the fly. See the jrebel website for details. I don't know that I would recommend this for a production environment though, because of performance issues. JRebel on occasion bogs things down momentarily when you've recompiled for a server.
You can't unload a class that is actually in use. But you might want to have a look at platforms like OSGi, if you want to reload or redeploy your application during runtime.
You can't explicitly unload a class.
In principle, you can expicitly reload a class via Classloader.loadClass(). From that moment onwards, all new instances of that class will use the new definition.
In any case, I would proceed with extreme caution...
You would have to create a custom ClassLoader: which returns all Objects wrapped inside a Proxy. The ClassLoader must have a list of all referenced Proxies (keep this list WeakReferenced). If you decide to "unload" and thus "reload" any class, you wait untill the class is loaded, find all Proxy's in your list, and replace the actual object.
There are several problems: you need Reflection, to get all private variables and reset the internal state (see setAccesible on Field). Also multi-threading problems might occour, thus the Proxy must be synchronized: which makes it performing poor.
You can better look for Google's Guice dependency solution, which enables unplugging and reloading modules at runtime. This might be a solution, but way too bloated for a small app. Still: I'm not sure wheter the GC also unloads unused classes.
Related
I have a Java class that gets instantiated by a third-party application as an extension. That is, as per the 3rd party software design, customers like us register our Java class to their application and their application will install it to execute custom logic at the right place and time.
Our custom Java class needs to marshal and unmarshal XML, for which it uses JAXB. It therefore needs a JAXB context.
I naively called JAXBContext.newInstance(MyClass.class) on every call and not-so-quickly discovered that that's a well known recipe for a memory leak. The common prescription is to make one (or at most only a few) JAXB contexts to share among your whole application.
Fine, except the third party application that invokes my class makes each invocation on a new ClassLoader instance that is private to that invocation.
So, even if I put the JAXBContext in a static field or static HashMap<>, it would still be private to the invocation!
QUESTION
How can I, in a class that is instantiated from a private ClassLoader instance, create a singleton to be shared across the JVM?
I am thinking along two possible lines, but I'd like advice on how to make either of them work or any completely different approach anyone has.
The three ideas I had were:
Find somewhere in a JVM class where I could write an object. E.g., if System.setProperty could write instance of Object instead of just String, the idea would be to create the JAXB context and put it in a property, since it is sure that System will already have been instantiated and that the custom classloader instance would inherit it. But System.setProperty does not take an Object value, so I don't know a practical way to do this.
Somehow force a class to load on the parent or root classloader where I could store by JAXB context. I don't know how to do this.
Use a ThreadLocal to store the JAXBContexts. I don't think each invocation is a brand new thread (they're probably reused from a thread pool), so this could maybe be the way to limit my contexts. But how to create the ThreadLocal variable so its shared across the instances? It seems like this leaves me with the same problem.
It sounds like your code is sandboxed with in the 3rd party application. So using static or ThreadLocal won't help since it will only exist within the same classloader, and if that classloader is changed... then the context is lost.
The closest solution is to inject your context into the application code. This is not the best idea, since it can be affected by outsiders and have unexpected consequences. Do note that this means a value you created in your classloader will remain in the application and thus the creating classloader will never be garbage collected. There's also the problem of where JAXB jar comes from. I assume from your code and not the 3rd party. So that jar is loaded each time in a different classloader, so to share an object from there might be a problem and require some proxy.
Honestly there can be so many unforeseen results.
The best idea is to ask the 3rd party to provide you with some API to enable that.
Before continuing, let's see other options:
Is there a replacement for using JAXB? Something that won't present the same memory leak problem.
Is the memory leak that serious? What if there is a temporary memory leak until an API is provided by the 3rd party application.
If you really want to try to inject the object, I'd be happy to help. But, from experience, these kinds of things are messy.
I have built in my application the ability to reload classes. However, I need to clarification on the behaviour of the class loader.
Let me explain what I know and then ask the questions...
What I do is provide a special jar that is loaded by a custom classloader. Then during the bootstrap of the jar a bunch of spring beans are created and the ones which implement certain interfaces get registered for use by the central app.
Now I kick off a process in the application which uses these new classes. I can successfully unload the "jar", change classes in the jar and reload and I get the changes. Unloading in class loader parlance means making the class loader that loaded the classes unreachable - this causes any classes loaded by that class loader to be unreachable and thus effectively unloaded.
However, from what I know so far, once the vm has loaded the class it stores it in some shared space so does not have to load it again.
The issue that I have is that sometimes the unload and reload of a new class does not work. The old class stays around. It stands to reason that if I unload a class loader (i.e. make the class loader unreachable) and one of the classes in that class loader is currently in use (an object of that type exists) then the class cannot be unloaded.
Is this true? It seems to be true in practice.
If that is so, how do I successfully unload classes that are in use at the time the class loader goes unreachable. Could I for example, put a weak reference on each class so I can detect when the class goes unreachable and act when the class goes unreachable? (not sure what action I could take though).
Update in response to #Kayaman
My use case is that I have the core application and then based on the customers requirements I can load different classes that implement known interfaces in the core application (so they can be accessed). Then the core application kicks off various processes which use these classes. The big strength of this is that I can update these plugin classes without doing a big redeploy and every customer does not need every one of these. The problem comes in when I want to load a new version of one of these and the current version is in use. Kinda stands to reason that this is not possible.
Conclusion
#Kayamam many thanks for your consult. It has been very helpful. This has codified my thinking somewhat. The conclusion is that no matter what technique I use you cannot ever, with the VM the way it is, reload a class for which there is currently a strongly reachable object. For some of my reloads I have control over these objects as I can make them unreachable before I unload and reload but for other classes I cannot do this... that is where my problem lies. What I need to do is to ring fence the objects of the classes that I wish to reload so that I can the process that is using them to pause while I reload the classes for those objects.
As you said, in order to unload a class, you need to get rid of the classloader. For example URLClassLoader can be used to load classes, and then null out the reference to make it eligible for GC and therefore unload the classes it loaded.
However, all classes know which classloader loaded them. That means that if you've got instances of your classes in use, they have a reference to the Class which has a reference to the ClassLoader and will prevent it from being collected and classes being unloaded. This is understandable, since having an object without a class would be quite an interesting situation.
So for a full reload you need to get rid of old instances and get rid of the classloader. That also avoids situations where you get mysterious exceptions about MyClass != MyClass.
WeakReference (or PhantomReference would probably be better here) would allow you to notice when your existing objects get collected, you just need to make sure you're tracking them all.
Spring adds to the complexity here, so I strongly recommend spending some time imagining that this approach is impossible, and to see if Spring has something that could be used to fulfil your business requirement. After all it does a lot of classloading itself, so you might be reinventing the wheel in a less clear way.
With quick Googling I found this http://docs.spring.io/spring-boot/docs/current/reference/html/howto-hotswapping.html which mentions among other things Spring Loaded which can apparently do class reloading and then some.
I don't want to use the URL Classloader to load classes.
I want to implement this myself.
I don't want to use a solution like JRebel (although it's great).
I've got prior experience of JavaAssist, bytecode generation, implementing javaagent class transformers etc.
I would like to write a javaagent which hooks into the classloader or defines it's own system classloader.
I'll store the class files in an in memory cache, and for particular files, periodically reload them from disk.
I'd prefer to do this in a way which doesn't involve continuously polling the file system and manually invalidating specific classes. I'd much rather intercept class loading events.
I last messed around with this stuff 4 years ago, and I'm sure, although my memory may deceive me that it was possible to do, but 8 hours of searching google doesn't present an obvious solution beyond building a patched JVM.
Is this actually possible?
I've created a stub implementation at https://github.com/packetops/poc_agent if anyone's interested in a simple example of javaagent use.
update
Just found this post - I may have been using the wrong approach, I'll investigate further.
It depends on what you want to do. If you want to reload your classes and define new ones, then you are fine with implementing your own classloader, as you already found.
If you want to replace existing classes, things become more "envolved". You can do this by implementing your own tiny Java agent. See the Java documentation, how to do this: http://docs.oracle.com/javase/7/docs/api/java/lang/instrument/package-summary.html
With the instrumentation mechanism you can not freely redefine classes, quote from Instrumentation.redefineClass:
The redefinition may change method bodies, the constant pool and attributes. The redefinition must not add, remove or rename fields or methods, change the signatures of methods, or change inheritance. These restrictions maybe be lifted in future versions. The class file bytes are not checked, verified and installed until after the transformations have been applied, if the resultant bytes are in error this method will throw an exception.
If you want to do more, you need to load it again. This can be done under the same name, by using a different classloader. The previous class definition will be unloaded, if no one else is using it any more. So, you need to reload any class that uses your previous class also. Utlimatly, you end up reinventing something like OSGi. Take a look at: Unloading classes in java?
The situation at hand is not as simple as the title seems to indicate.
Java 1.6_17 running via JWS.
I have a class, lets say MyClass and one of its instance member variables is a Type from an errant 3rd party library where during class initialization it dynamically tries loading some of its own classes with Class.forName(String). In one of these cases it happens to dynamically call: Class.forName("foo/Bar").This class name doesn't follow the JLS for binary names and ultimately leads to a java.lang.NoClassDefFoundError: foo/Bar.
We have a custom ClassLoader which I've added a sanitize method to ClassLoader.findClass(String) and ClassLoader.loadClass(String) which fixes this problem.
I can call stuff like:
myCustomClassLoader.findClass("foo/Bar")
Which then loads the class without any problems. But even if I load the class ahead of time, I still get the exception later. This is because during initialization of MyClass which refers to Bar - their code ends up calling Class.forName("foo/Bar") in a static block somewhere. This actually would be OK if the ClassLoader it was trying to use was my custom class loader. But it isn't. It is the com.sun.jnlp.JNLPClassLoader which doesn't do such sanitation, thus my problem.
I've made sure that Thread.currentThread().getContextClassLoader() is set to my custom class loader. But this (as you know) has no effect. I even set it as the first thing i do in main() due to some stuff I read and still, MyClass.class.getClassLoader() - is the JNLPClassLoader. If I could force it to NOT be the JNLPClassLoader and to use mine instead, problem solved.
How can I control which ClassLoader is used to load the class via their static Class.forName("foo/Bar") call made during class initialization? I believe if I can force MyClass.class.getClassLoader() to return my custom class loader, my problem will be resolved.
I'm open to other options if anyone has ideas.
TL;DR: Help me force all Class.forName(String) calls in a third party library which are referenced by MyClass - to use the classloader of my choosing.
This reminds me of an article I read 10 years ago about the classloading arrangements in Java. It's still there on JavaWorld.
The article won't answer your question directly, but it may help understand your problem. You need to cause MyClass to be loaded through your custom class loader and trump the default class loading behavior, which is to first delegate class loading to the parent classloader and only attempt to load a class if that fails.
Allowing MyClass to get loaded by a classloader other than yours will store a relationship from the instantiated class to that classloader (via getClassLoader) and cause Java to use that other classloader to try to discover any referenced classes found at compile time, effectively bypassing your custom class loader by virtue of the class loader hierarchy and the delegation model. If MyClass is instead defined by your class loader, you get a second chance.
It sounds like a job for something like URLClassLoader, overriding loadClass and trumping the delegation model for classes residing in your JARs. You'll probably want to use a bootstrap approach (as suggested by Thomas in a comment above) to force a single entrypoint class to be loaded through your custom class loader, dragging all the others with it.
Also informative is this other JavaWorld article by the same guy, which warns you about the caveats of Class.forName. That too may trip your classloading arrangements.
I hope this helps and proves informative. In any case, it sounds like a difficult solution that is easy to break as your code evolves.
I think everyone gave good solid attempts at answering the problem. However, it turns out that I misdiagnosed the problem.
I had a coworker take over the problem and asked him to get a JDK with debug flags on so we could debug the JNLPClassLoader to see what was going on as I had tried all of the suggestions here + some.
We ended up getting OpenJDK because recompiling the JDK from scratch is a total nightmare (we tried). After getting OpenJDK working with our product and debugging through the JNLPClassLoader - it turns out that it was still using a REALLY old .jnlp from months earlier that had the resource path wrong and thus why it couldn't find the class.
We were confused why it was still using the ancient .jnlp even though we had redeployed the server correctly many times with the correct .jnlp and lots of code changes between which were reflected in our client application when run.
Well, it turns out that on client machines, Java caches the .jnlp file. Even if your application changes and it redownloads your application, it still won't re-download the new .jnlp for whatever reason. So it will use all of the new code, but look up resources/class paths using the cached .jnlp.
If you run:
javaws -uninstall
On the client machine then that will clear the .jnlp cache and next time it will use the correct .jnlp file.
Really sad that this was the problem. Hopefully, this saves someone else endless hours of frustration like it caused us.
If you run out of ideas with patching the ClassLoaders themselves, you might consider rewriting the library bytecode itself -- just replace the "foo/bar" constant with the correct value, and then you don't need to customize further class loading at all!
You could do this either at runtime or beforehand.
I'm currently wondering what the actual overhead, in the JVM, is for loading extra classes which are never used.
We have code which iterates all the classes in the class path to find classes which implement a certain interface, we then load them.
This allows custom classes to be simply dropped in a directory and they get loaded and registered.
The side affect is that we hit every class in the class path, causing the classes to load. What would be the affect on the JVMs memory?
Does simply loading classes affect the memory much at all?
As usual, I would advise measuring this for your particular scenario.
Having said that, I'm not sure I'd advise scanning the whole classpath. If you don't control the classpath (it's your customer's, or similar), potentially they could add anything to it, and your process is going to scan anything they drop into their classpath (possibly unrelated to your app).
I'd suggest that you nominate only certain directories/repositories that classes can be uploaded to, and that way you'll restrict the classpath scanning and reduce the chances of inadvertently picking up stuff you don't intend to.
If you use a separate ClassLoader to load those classes and are very careful not to create any references to those classes or instances of them, then when the ClassLoader becomes eligible for garbage collection, so do the classes.
Thus, you could avoid unnecessarily clogging your PermGen space by doing 2 passes with separate ClassLoaders: one to load all the classes and identify those you want to keep, and another to actually use them.
Won't using ClassLoaders in this way have unintended side-effects? Like running static initialisers and so on.
You could use the ServiceLoader mechanism, but if that doesn't suit, you can inspect classes without using ClassLoaders - byte manipulation libraries like BCEL and ASM can be used to just inspect classes.
Yes, this forces the VM to load the class file and examine it (which can be a performance problem). Moreover, if you're using a Sun VM, then these classes will stay in memory forever. the Sun VM puts classes in the so called "PermGen" space which is never garbage collected unless you specify a special option.
So this is generally a bad idea but there are two simple workarounds:
Check the name of the class (the filename). Repeat the name of the interface in the name, so you can easily notice what you have to load and what not.
Use two directories. One contains normal classes, the other the one all those which you want to always load.
A "far out there" suggestion:
could you do the same thing but without actually using the VM? the class file spec is documented, couldn't you write your own app to just read the class files and figure out if they implement your interface/whatever without actually loading them?
That would get you the ability to scan any directories but without any worry of loading classes or static intializers or anything like that.