I am investigating ways to implement some hot-reloading of Java classes. The technique I am thinking about is something like this:
keep all classes from foreign/core libraries in memory
if one of my files changes, remove all my classes from memory and reload all my classes. I won't have to reload any libraries in memory, because they won't change and don't depend on my files/classes.
public class Server implements Runnable {
private Thread current;
public void run(){
// create a new classloader and load all my classes from disk?
}
public Server start(){
if(this.current != null){
this.current.destroy(); // not sure if this works
}
this.current = new Thread(r);
this.current.start();
return this;
}
public static void main(String[] args){
var s = new Server().start();
onFileChanges(filePath -> {
// we don't really care what file changed
// as long as it's one of our project's files, we reload all classes
s.start();
});
}
}
I think the key idea is that I can just reload all the classes from my project, instead of trying to calculate some dependency tree.
My main questions are -
(a) how do I stop/kill a thread? Thread#destroy is deprecated.
(b) how can I remove all classes from the classloader in memory?
(c) how can I keep all the classes for libraries in memory, but remove all references to classes/instances of my code from memory?
Does anyone think this technique will work? Is an implementation possible?
The key concept is not to reload the classes in the Classloader but to create a separate ClassLoader instance that will be "deleted" in case you are performing a code reload. Shared libraries that do not change can be placed in your main project so that they are available via SystemClassLoader.
Create your custom "MyClassloader" and load all your JAR files and their dependencies into it.
Use the MyClassLoader to load and execute the main class of the loaded code.
If you want to reload the code stop all threads started by the re-loadable code. This usually has to be supported by the thread itself e.g. by regularly checking a MyClassLoader global static field AtomicBoolean shutdown plus interrupting the thread in case it is waiting for something.
Once all threads have ended the only reference to the code is your MyClassLoader instance. If you throw away this instance all the code will also be removed and you are free to start over new.
Related
I am building a dynamic Web Project (in Eclipse with Tomcat as server) using servlets and JSPs. The general purpose of the project is to let the user solve small code problems. In order to do that I take a method written by the user and use it to build a java file that I can run via Reflection. The problem I can't seem to figure out is that Tomcat (or Eclipse?) does not update the file at runtime. So when I create the file using the code of the current user and try to compile it, my program always executes the file as it was when I started the server using the code of the previous user. How can I tell it to update the file before running it?
Edit:
That's how I create the file:
public boolean writeFile() {
try {
PrintWriter writer = new PrintWriter(relativePath + "src\\testfiles\\TestFile.java");
writer.print(content);
writer.close();
return true; }...
Here I call the writer and try running the file:
FileWriter writer = new FileWriter(content);
if(writer.writeFile()){
Class<?> TestFile;
Method m;
try {
TestFile = cl.loadClass("testfiles.TestFile");
m = TestFile.getDeclaredMethod("testSolution");
m.invoke(null);
Thanks in advance!
Ok, it's now clear what the issue is. Your issue is not with Tomcat not reloading the file, but with the classloader not reloading the class.
Normal classloaders will only load a class once, and keep it cached forever. The only way for a class to get unloaded is by its classloader being garbage collected. To reload a class you either have to use a different classloader each time (with the previous one getting garbage collected or you'll run out of memory), or to have a custom loader thar doesn't cache.
See this article for an implementation of a custom classloader that does what you want.
You could theoretically just have a new class each time (by changing its name on each save) but you'd eventually run out of memory.
Maybe the easiest way is to instantiate a new classloader in the method itself, load a class, run the method on it, and don't keep any other references to the loader or the class. That way, you'll get a freshly loaded class each time, and the old instances of both classes and loaders will get garbage collected.
UPDATE: I was operating under the assumption that you already know how to compile a class at runtime but, based on the comments, that's not the case.
A classloader can, of course, only load a compiled class, so a source directly is not very useful.
Java internally provides a a compiler interface under javax.tools.JavaCompiler, but it's not easy to use and requires a different handling of Java versions before and after Java 9. So it is much easier to use a library like jOOR that hides the gory parts:
Class clazz = Reflect.compile("com.example.Test",
"package com.example;" +
"public class Test {\n" +
" public String hello() {\n" +
" return \"hello\";\n" +
" }\n" +
" }")
.type();
Instead of type() to simply get the class, you can actually keep using jOOR's fluent reflection API to trigger the methods on the generated class or whatever it is you'd normally do via regular reflection.
For direct JavaCompiler usage, see e.g. this or, even better, jOOR's source code.
I have a case where I need to create a lot of class loaders in my application to temporarily make some code visible while user supplied scripts are running. I'm using an URLClassLoader for this and it works pretty well.
When the script terminates, I want to "unload" or "close" the class loader to free the resources.
Is it enough to set the reference to the class loader to null? I'm especially wondering if I'll eventually run out of file handles because the extra classes are in JAR files.
PS: Must work with Java 5 and up. Yeah, I know...
A little late, but hopefully this'll be helpful for those who come to this Question later (like me).
With Java 7, a close() method has been added to URLClassLoader, which is exactly what OP was asking for.
EDIT (thanks to #Hot Licks): OK, so it isn't exactly what the OP has asked for. It doesn't free up all the resources, or make the resources and the loader collectible. It simply prevents the loading of more resources using the class loader. It does, however, close the jar file that was loaded with the URLClassLoader.
If you can't use Java7 and it's close() method, use reflection to close all open JAR archives of a classloader, like so:
public void close() {
try {
Class clazz = java.net.URLClassLoader.class;
java.lang.reflect.Field ucp = clazz.getDeclaredField("ucp");
ucp.setAccessible(true);
Object sun_misc_URLClassPath = ucp.get(this);
java.lang.reflect.Field loaders =
sun_misc_URLClassPath.getClass().getDeclaredField("loaders");
loaders.setAccessible(true);
Object java_util_Collection = loaders.get(sun_misc_URLClassPath);
for (Object sun_misc_URLClassPath_JarLoader :
((java.util.Collection) java_util_Collection).toArray()) {
try {
java.lang.reflect.Field loader =
sun_misc_URLClassPath_JarLoader.getClass().getDeclaredField("jar");
loader.setAccessible(true);
Object java_util_jar_JarFile =
loader.get(sun_misc_URLClassPath_JarLoader);
((java.util.jar.JarFile) java_util_jar_JarFile).close();
} catch (Throwable t) {
// if we got this far, this is probably not a JAR loader so skip it
}
}
} catch (Throwable t) {
// probably not a SUN VM
}
return;
}
When all classes that the class loader loaded no longer have any references, and all references to the class loader itself have been erased, the class loader and the classes it loaded will be garbage collected as a group.
Note that this is dependent on having the JVM attribute set that causes unreferenced classes to be unloaded. It's set by default in most environments, but may not be in some embedded cases.
[Note that it's a non-trivial matter to remove references to a class. Any other class that references it by name will of course prevent removal. So the class must be loaded using ClassLoader.findClass or something similar.]
If you do not longer have classes (and object) loaded from that classloader, and if you do not keep any reference to that classloader, it will automatically handled by the garbage collector.
There are no close() methods in URL class loader or any of its parent classes, so you're out of luck.
Shouldn't GC handle this?
I extended URLClassLoader and made a close method based on Java 7s. I wanted to develop my IRC bot on my iPad 2, so I did what was needed. Now my plugin system is stable on Java 6 and 7, hurray.
I am working on an existing application which has quite a lot of external JAR dependencies. I need to enhance it to integrate with a third-party application which has an API. Sadly, the API is not well contained and also comes with a large number of its own dependencies some of which clash with mine.
I believe I should solve this using Classloaders, but I'm struggling to see how to structure them correctly.
To keep it simple, assume we have myapp.jar with a hibernate3.jar dependency, and vendor-api.jar with a hibernate2.jar dependency (and assume these are incompatible).
My new piece of code will reside in the myapp.jar library (although it could be in a separate jar if this would help). Due to the way the vendor API works, my new code needs to extend a class from the vendor-api.jar library.
How can I structure the Classloaders in such a way that anything within the vendor-api.jar accesses only its own dependencies, and anything on my side accesses only the myapp.jar and dependencies?
Thanks, Jon
I've not tried this myself, but from memory each clashing class needs to be in a sibling classloader and any communication between the two needs to go through a common ancestor. However, the ancestor cannot (AFAIK) "directly" reference classes from its children and must access them through the reflection API.
Something along these lines ought to work (dry-coded) YMMV. Comments and error-spotting welcome.
class Orchestrator {
URL[] otherAppClasspath = new URL[] { new URL("file:///vendor-api.jar"),
new URL("file:///hibernate2.jar"),
new URL("file:///code-extending-vendor-api.jar" };
URLClassLoader otherAppLoader = new URLClassLoader(otherAppClasspath);
URL[] yourAppClasspath = new URL[] { new URL("file:///myapp.jar"),
new URL("file:///hibernate3.jar") };
URLClassLoader yourAppLoader = new URLClassLoader(yourAppClasspath);
public void start() {
Method yourAppEntryPoint = yourAppLoader.findClass("com/company/Main").getMethod("start", new Class[] { Orchestrator.class } );
yourAppEntryPoint.invoke(null, new Object[] { this });
}
public static void main(String[] args) {
new Orchestrator().start();
}
// define some abstracted API here that can be called from your app
// and calls down into classes in the other app
public String getSomeResultFromOtherApp(int someArgument) {
Method otherAppAPI = otherAppLoader.findClass("com/company/ExtendingAPIClass").getMethod("getSomeResult", new Class[] { Integer.class });
return (String)otherAppAPI.invoke(null, new Object[] { someArgument });
}
}
#fd's answer gives a technical mechanism that ought to work - give or take some typos, exception handling, etc.
However, I think you would be better off not trying to do this ... unless your dependencies on the 3rd party APIs are restricted to a very small number of methods on a very small number of classes. Each class that you depend on has to be dynamically loaded and each method has to be looked up and invoked reflectively. Too much of that and your codebase will suffer.
If I were you, I'd try to resolve the dependency issue some other way:
Try to get the 3rd party vendor to use hibernate3.jar
Change your application to use hibernate2.jar
Refactor so that your application code and the 3rd party library are in separate JVMs or separate webapps.
From what you say, this might be hard.
Using OSGi may help you in the long term. Here is an implementation I am trying now- http://felix.apache.org
I am writing a static analysis tool for an assignment, it analyses Java bytecode using the ASM library. One of the parts of ASM that we use requires (or at least, appears to require) that the class be loaded from the ClassLoader.
We were hoping the tool would be able to analyse .class files without requiring them on the classpath. We already load the .classes from a specified directory at run time and read them in using an InputStream. This is acceptable for ASM in most cases. There are some classes, such as SimpleVerifier, which attempt to load the classes though.
Is it possible, under this scenario, to register the .class files to be loaded so that calls to Class.forName() will load them? Or is there an easy way to extend the ClassLoader to allow this?
Edit: the information on URLClassLoader was useful. Unfortunately, using Thread.currentThread().setContextClassLoader() to an instance of that didn't work in this scenario. The library code I'm calling into uses a loader it retrieves on instance initialisation using getClass().getClassLoader().
By the time I set the URLClassLoader the class hasn't been initialised so I guess the contextClassLoader does not load that class.
Have I understand the responses correctly? Would using the URLClassLoader to load the 3rd party class be a possibility?
Almost.
If you have classes compiled somewhere, you can load them with a URLClassLoader. You can then set this ClassLoader to be the ClassLoader for the current Thread: Thread.setContextClassLoader(ClassLoader)
Users can that get the current threads context class loader and use that to access the class definition.
First of all, ASM can be used in a such way that it won't use ClassLoader to obtain information about classes.
There are several places in ASM framework where it loads classes by default but all those places can be overridden in your own subclasses. Out of the top of my head:
ClassWriter.getCommonSuperClass() method is called only when ClassWriter.COMPUTE_FRAMES flag is used and can be overwriten to not use ClassLoader to get inforamtion about classes. You can find an example of that in ClassWriterComputeFramesTest that introduces a ClassInfo abstraction
Similarly SimpleVerifier.getClass() method is used by SimpleVerifier.isAssignableFrom() and you can overwrite the latter and use the ClassInfo abstraction to find the common super type. If I am not mistaken, AspectWerkz project had implemented similar thing in its type pattern matching code. Also note that there is SimpleVerifier.setClassLoader() method, which you can use if you still want to load your own classes.
On a side note, on a Sun's JVMs, loaded classes gets to PermGen area and can't be unloaded, so it is not a good idea to load classes only for static code analysis purposes if you can avoid that, especially if tool would be integrated into a long-live process, such as IDE.
You can't, as far as I know, extend the System class loader at runtime, but you can dynamically load classes from an arbitrary location (jar or directory) using URLClassLoader.
You could try to setup a "launcher" in the startup of your application that creates an URLClassLoader passing it the locations on the classpath and your own .class locations and start the application from that classloader.
When the SimpleVerifier is loaded by the URLClassLoader it will also be able to load the classes from the extra locations.
Yes, you can use URLClassLoader
I have a test where I do load the class at runtime. This class is not in the classpath (nor even exist when the test is run for that matter ), later is it loaded and works great.
Here's the code.
void testHello() throws MalformedURLException, ClassNotFoundException {
URL[] url = {
new URL("file:/home/oreyes/testwork/")
};
try {
new URLClassLoader(url).loadClass("Hello");
throw new AssertionError("Should've thrown ClassNotFoundException");
} catch ( ClassNotFoundException cnfe ){}
c.process();// create the .class file
new URLClassLoader(url).loadClass("Hello");
// it works!!
}
Taken from this question.
I created my own ClassLoader its quite simple.
/**
* Used to hold the bytecode for the class to be loaded.
*/
private final static ThreadLocal<byte[]> BYTE_CODE = new ThreadLocal<byte[]>();
#Override
protected Class<?> findClass(final String name) throws ClassNotFoundException {
final byte[] bytes = BYTE_CODE.get();
if (null == bytes) {
throw new ClassNotFoundException(name);
}
return this.defineClass(null, bytes, 0, bytes.length);
}
From the perspective of a cross application/applet java accessibility service, how would you link to a package but only optionally execute an action based on existence/availability of a package (being already loaded) at runtime?
I think what I'm interested in here is a way to resolve the class identity crisis but rather than the issue being between 2 apps sharing objects, being a service loaded at a higher level of the class loaders.
It seems like reflection is the way to go, but I am not sure how or if I can implement a derived class this way. I need to add a specific listener derived from the specific optional classes, I can load the listener using the applet class loader but the internals still fail. Say you wanted to add an JInternalFrameListener, but Swing wasn't guaranteed to be available, using reflection you can find the method to add the listener, but how can you create and have the frame listener work if it cannot find any of the related classes because they can't be found in the base classloader! Do I need to create a thread and use setContextClassLoader to the classloader that knows about swing so that I can get the class to be loaded reliably? simply trying to set the class loader on my existing thread didn't seem to work.
Earlier description of issues
Sorry, I'm not quite sure what to ask or how to make this clear, so it rambles on a bit.
Say a class uses some feature of another, but the other class may not always be available - say finding the website from JNLP if this is a JNLP app.
At one stage I thought that simply compiling against JNLP would mean that my class would not load unless JNLP was available, and so to identify this optional section I simply wrapped a try{} catch( NoClassDefFoundError ) around it.
Later something changed (perhaps changing jdk or ?? I don't recall) and it seemed that I should also use a try{} catch( ClassNotFoundException ).
Now I wanted to extend this idea to other optional features, but it doesn't seem to work consistently.
Say I wanted to add some feature to do something more advanced in a JRE1.6 runtime using the same jar and classes as I run in a JRE1.3, or say I want to handle some controls in a specific gui toolkit which may not always be used like SWT or oracle.forms.
Is there some way of doing this more reliably? It just seems wrong to cause an exception and catch it to ignore it all the time.
The current issue comes down to being able to compile against oracle.forms but then the accessibility component installed in ext is unable to access the oracle.forms classes even though objects from the oracle.forms package have been created. If I throw the frmall.jar into the ext directory to test then the accessibility component works up to the point that the whole lot gets flakey because of the different versions of the same package.
I seem to be caught up on an issue with the class loader not being the right one or something (??). How do I find the right one?
Edit:
The answers so far are kindof interesting but not quite getting me where I want to be.
In the case of the gui components I currently compile in the form of a factory something like...
import oracle.forms.ui.*;
import java.awt.*;
static public IComponentNode newNode( INode parent, Component component ) {
System.out.println( component.getClass().toString() );
try{
if( component instanceof FormDesktopContainer )
... does stuff here like return new FormDesktopNode( parent, (FormDesktopContainer) component )
} catch ( NoClassDefFoundError a ) {
System.out.println( a.getMessage() );
}
where it prints out class oracle.forms.ui.FormDesktopContainer and then throws and exception on the instanceof call with NoClassDefFound thus printing out oracle/forms/ui/FormDesktopContainer
So how can it have an instance of a class yet not be able to find it?
How about this? messy, but it ought to work:
public boolean exists(String className){
try {
Class.forName(className);
return true;
}
catch (ClassNotFoundException){
return false;
}
}
You can check the availability of a class by calling
ClassLoader.getSystemClassLoader().loadClass("my.package.MyClass")
if it throws a ClassNotFoundException, it's not available. If you get the Class object, it is. You can then choose behaviour based on whether or not the class is available.
I suggest compiling the majority of your code against your minimum target. Have code that uses particular optional libraries clearly separated, but dependent upon the bulk of your code. Dynamically load the code that uses optional libraries once. The main class should do something that checks for the presence of the required library/version in its static initialiser.
In the case of JNLP, your JNLP main class load the JNLP dependent code statically.
(Note that attempting to catch class loading related exceptions from normally linked code is unreliable.)
getSystemClass loader was not useful for this purpose as there where multiple possible class loaders to interact with based on which applet the given window was in. The accessibility components being loaded at a more base class loader cannot see the applet specific classes.
To interact with the objects reflection does the job, though it does add so much more to maintain.
// statically linking would be
return component.getText();
// dynamically is
try {
return (String)component.getClass().getMethod("getText", new Class [] {}).invoke(component, new Object [] {});
} catch (Throwable e) {
e.printStackTrace();
}
The trickier bit is in writing a class derived from an interface that is not directly accessible, using the Proxy service allows this to be accomplished, providing the proxy service the applet specific class loader and the dynamically loaded class for the interface.
public void addListener(Container parent) {
if (parent == null) { return; }
if ("oracle.forms".equals(parent.getClass().getName())) {
// Using the class loader of the provided object in the applet
// get the "class" of the interface you want to implement
Class desktopListenerClass = Class.forName( "oracle.DesktopListener"
, true, parent.getClass().getClassLoader());
// Ask the proxy to create an instance of the class,
// providing your implementation through the InvocationHandler::invoke
Object desktopListener = Proxy.newProxyInstance(
parent.getClass().getClassLoader()
, new Class[] { desktopListenerClass }, new InvocationHandler() {
public Object invoke(Object proxy, Method method, Object[] args)
throws Throwable {
if ("functionName".equals(method.getName())) {
// do stuff
}
return null;
}
});
// do something with your new object
Method addDesktopListener = parent.getClass().getMethod("");
addDesktopListener.invoke(parent, desktopListener);
}
}
examples cut down to show general method