I want to select different implementations of classes dynamically, based on a runtime condition. Let's say I have a class with fully qualified class name C. My running system may have many definitions of class C, each of which is in its own jar. I have a runtime condition (held in a ThreadLocal) that tells which definition should be chosen.
I was asked in a comment to clarify the original requirement, so I will clarify the requirement as best I can. There are multiple teams writing software to contribute to this system - something like 4000 classes in many independent modules. What's more they can change over time. They are currently running in separate JVMs so there is no issue with classes overlapping. Now we are considering running them in the same JVM with multiple releases running simultaneously on the same JVM; which specific set of implementations used being differentiated by the ThreadLocal. So the original problem was how to allow a thread to at one time run one set of implementations and at another time run another.
I have a tomcat application that is currently using OpenJDK 8.
I believe I can write a custom ClassLoader that manipulates the classpath to choose the definition of C differently based on the ThreadLocal. But I'm afraid the results will be cached somewhere such as JVM Code Cache. Unless I can override that behavior too, the next time the class is needed, the runtime condition may have changed and the version in the cache would be wrong.
Is there any way to do what I need to do?
Well, the simple solution would be to NOT have multiple definitions of class C, but instead have 'class C1', 'class C2', etc (i.e. they don't overlap and can be simultaneously loaded) and then your runtime property just picks the right one as appropriate. That is way-way the easiest solution, so strongly consider it first. But it may not meet your needs.
If you truly need to have multiple separate implementations of a single 'class C', then what you are effectively talking about is a 'hot swap' scenario. Fortunately, Tomcat and other tools got good (with limitations) at hotswapping long-time-back. You likely already know this, but 'hot swapping' meets the need of a developer who codes 'class C', deploys it to a container, tries it out, realizes it has a glitch, makes a quick code edit and wants to run the modified code without relaunching the container. Hotswapping does this by basically overlaying the new implementation in the JVM. It only works up to a point because each 'redeploy' pollutes the 'class space' of the JVM and you eventually run of of 'class space memory' and/or the JVM starts going unstable. Depending on your needs and tolerances though, hotswapping might work.
The clean way to do what you want is to define C as an interface, then you can load and use any class that implements interface C
Related
I don't want to use the URL Classloader to load classes.
I want to implement this myself.
I don't want to use a solution like JRebel (although it's great).
I've got prior experience of JavaAssist, bytecode generation, implementing javaagent class transformers etc.
I would like to write a javaagent which hooks into the classloader or defines it's own system classloader.
I'll store the class files in an in memory cache, and for particular files, periodically reload them from disk.
I'd prefer to do this in a way which doesn't involve continuously polling the file system and manually invalidating specific classes. I'd much rather intercept class loading events.
I last messed around with this stuff 4 years ago, and I'm sure, although my memory may deceive me that it was possible to do, but 8 hours of searching google doesn't present an obvious solution beyond building a patched JVM.
Is this actually possible?
I've created a stub implementation at https://github.com/packetops/poc_agent if anyone's interested in a simple example of javaagent use.
update
Just found this post - I may have been using the wrong approach, I'll investigate further.
It depends on what you want to do. If you want to reload your classes and define new ones, then you are fine with implementing your own classloader, as you already found.
If you want to replace existing classes, things become more "envolved". You can do this by implementing your own tiny Java agent. See the Java documentation, how to do this: http://docs.oracle.com/javase/7/docs/api/java/lang/instrument/package-summary.html
With the instrumentation mechanism you can not freely redefine classes, quote from Instrumentation.redefineClass:
The redefinition may change method bodies, the constant pool and attributes. The redefinition must not add, remove or rename fields or methods, change the signatures of methods, or change inheritance. These restrictions maybe be lifted in future versions. The class file bytes are not checked, verified and installed until after the transformations have been applied, if the resultant bytes are in error this method will throw an exception.
If you want to do more, you need to load it again. This can be done under the same name, by using a different classloader. The previous class definition will be unloaded, if no one else is using it any more. So, you need to reload any class that uses your previous class also. Utlimatly, you end up reinventing something like OSGi. Take a look at: Unloading classes in java?
I want to redefine the bytecode of the StackOverflowError constructor so I have a "hook" for when a stack overflow occurs. All I want to do is insert a single method call to a static method of my choosing at the start of the constructor. Is it possible to do this?
You should be able to do it using one of two ways (unless something changed in the last 1-2 years, in which case I'd love some links to changelogs/docs):
Mentioned in a comment, not very feasible I guess, modify the classes you are interested in, put them in a jar and then use the -bootclasspath option to load them instead of the default ones. As was mentioned before this can have some legal issues (and is a pain to do in general).
You should be able to (or at least you used to be able to) instrument almost all core classes (iirc Class was the only exception I've seen). One of many problems you might have is the fact that many of core classes are being initialized before the agents you provide (or well their premain methods to be exact) are consulted. To overcome this you will have to add Can-Retransform-Classes property to your agent jar and then re-transform the classes you are interested in. Be aware that re-transformation is a bit less powerful and doesn't give you all the options you'd have normally with instrumentation, you can read more about it in the doc.
I am assuming you know how to do instrumentation?
There are several things to consider.
It is possible to redefine java.lang.StackOverflowError. I tried it successfully on 1.7.0_40. isModifiableClass(java.lang.StackOverflowError.class) return true and I successfully redefined it inserting a method invocation into all of its constructors
You should be aware that when you insert a method call into a class via Instrumentation you still have to obey the visibility imposed by the ClassLoader relationships. Since StackOverflowError is loaded by the bootstrap loader it can only invoke methods of classes loaded by the bootstrap loader. You would have to add the target method’s class(es) to the bootstrap loader
This works if the application’s code throws a StackOverflowError manually. However, when a real stackoverflow occurs, the last thing the JVM will do is to invoke additional methods (keep in mind what the error says, the stack is full). Consequently it creates an instance of StackOverflowError without calling its constructor (a JVM can do that). So your instrumentation is pointless in this situation.
As already pointed out by others, a “Pure Java Application” must not rely on modified JRE classes. It is only valid to use Instrumentation as add-on, i.e. development or JVM management tool. You should keep in mind that the fact that Oracle’s JVM 1.7.0_40 supports the redefinition of StackOverflowError does not imply that other versions or other JVMs do as well.
I'm working on a sandbox feature for my java antivirus, and I've come into a question: Does the specified package on a class matter for compilation?
Example:
I'm running a program that wants to use Runtime.getRuntime().exec(), when the classloader attempts to load that to run a method, does it check the package qualified in the file, if they exist? I would prefer not to try and change files in the JVM, but to simply load ones from a different package. I can accomplish the loading and such, but my only dilemma, will it crash and burn? Inside the java, it would be registered as say, java.lang.Runtime, but the compiled code will say for example pkg.pkg.Runtime and will it need to extend the old runtime? My guess is that extending the old runtime would just break it. Does anyone know anything about this? I'm working on making a testable example, but I'm still a bit away and wanted to get some answers, as well as this might benefit some people.
Does the specified package on a class matter for compilation?
Yes it does matter. A class called pkg.pkg.Runtime() cannot be loaded as if it was java.lang.Runtime.
Furthermore, if my memory is correct, the JVM has some additional security measures in it to prevent normal applications from injecting classes into core packages such as java.lang.
If you need to change the behaviour of the java.lang.Runtime class (for experimental purposes!) then I think you will need to put your modified version on the boot classpath, ahead of the "rt.jar" file.
However:
This level of tinkering can easily result in JVM instability; i.e. hard JVM crashes that are difficult to diagnose.
If your aim is to produce a "production quality" tool, then you will find that things that involve tinkering with the JVM are not considered acceptable. People are going to be very suspicious of installation instructions that say things like "add this to your installed JVM's bootclasspath".
Distributing a "tinkered with" JVM may fall foul of Oracle's Java licensing agreement.
My advice would be to look for a less intrusive way of doing what you are trying to do. For instance, if you are trying to do virus checking, either do it outside of the JVM, or in a custom application classloader.
You commented:
I have a custom classloader, my question is: If I compile a class that is labelled as say, pkg.pkg.Runtime, can I register in my classloader as java.lang.Runtime?
As I said above, no you can't. A bytecode file has the classname embedded in it. If you attempt to "pull a swifty" by loading a class with a different name, the JVM will throw an Error.
And:
If not, then how can I replace the class? If the compiled package name has to equal the request referenced naming, then can I modify the .class file to to match, or perhaps compile it as if it were in the java.lang package?
That's what you would have to do. You need to name the class java.lang.Runtime in the source code and compile it as such.
But what I meant by my advice above is that you should use do the virus checking in the class loader. Forget about trying to replace / modify the behaviour of Runtime. It is a bad idea for the reasons I listed above.
I'm currently wondering what the actual overhead, in the JVM, is for loading extra classes which are never used.
We have code which iterates all the classes in the class path to find classes which implement a certain interface, we then load them.
This allows custom classes to be simply dropped in a directory and they get loaded and registered.
The side affect is that we hit every class in the class path, causing the classes to load. What would be the affect on the JVMs memory?
Does simply loading classes affect the memory much at all?
As usual, I would advise measuring this for your particular scenario.
Having said that, I'm not sure I'd advise scanning the whole classpath. If you don't control the classpath (it's your customer's, or similar), potentially they could add anything to it, and your process is going to scan anything they drop into their classpath (possibly unrelated to your app).
I'd suggest that you nominate only certain directories/repositories that classes can be uploaded to, and that way you'll restrict the classpath scanning and reduce the chances of inadvertently picking up stuff you don't intend to.
If you use a separate ClassLoader to load those classes and are very careful not to create any references to those classes or instances of them, then when the ClassLoader becomes eligible for garbage collection, so do the classes.
Thus, you could avoid unnecessarily clogging your PermGen space by doing 2 passes with separate ClassLoaders: one to load all the classes and identify those you want to keep, and another to actually use them.
Won't using ClassLoaders in this way have unintended side-effects? Like running static initialisers and so on.
You could use the ServiceLoader mechanism, but if that doesn't suit, you can inspect classes without using ClassLoaders - byte manipulation libraries like BCEL and ASM can be used to just inspect classes.
Yes, this forces the VM to load the class file and examine it (which can be a performance problem). Moreover, if you're using a Sun VM, then these classes will stay in memory forever. the Sun VM puts classes in the so called "PermGen" space which is never garbage collected unless you specify a special option.
So this is generally a bad idea but there are two simple workarounds:
Check the name of the class (the filename). Repeat the name of the interface in the name, so you can easily notice what you have to load and what not.
Use two directories. One contains normal classes, the other the one all those which you want to always load.
A "far out there" suggestion:
could you do the same thing but without actually using the VM? the class file spec is documented, couldn't you write your own app to just read the class files and figure out if they implement your interface/whatever without actually loading them?
That would get you the ability to scan any directories but without any worry of loading classes or static intializers or anything like that.
I have a web application and two class files,
First class is MyClass.class which is inside the abc.jar file (WEB-INF/lib/abc.jar) and
Second class is YourClass.class which is inside classes folder (WEB-INF/classes/ YourClass.class).
My question is which class would load first when the Application starts? And Why ?
In my experience you can't predict the order in which the classes are loaded by the JVM.
Once I made a test runner (kinda Maven's Surefire) and with the same JVM and OS it loaded classes in different order when run in different machines. The lesson learnt:
You shouldn't build your applications
to depend on class loading order
Classes are loaded as needed, for some definition of "needed". Exactly when a class is loaded is dependent upon the JRE implementation, javac implementation, exactly what thread are up to, server code and, of course, application code. It's a bad idea to make assumptions in this area. If you want to see what happens for a particular run, you can use -verbose:class
Sun's class loader docs always say WEB-INF/classes OR WEB-INF/lib, but doesn't say which one will be checked first.
From IBM docs:
"The rules for loading classes are spelled out in detail in the JVM specification. The basic principle is that classes are only loaded when needed (or at least appear to be loaded this way -- the JVM has some flexibility in the actual loading, but must maintain a fixed sequence of class initialization). Each class that gets loaded may have other classes that it depends on, so the loading process is recursive."
So I think the answer is: It depends on which classes is needed in your application first.
As duffymo points out, this can vary. One way you might ascertain the sequence for this specific app is to insert Response.Write text in the class constructors and web web app page loading methods. "Instantiated object in Class A", "Opened web page MyPage", and so on.
Once you've figured out the sequence, comment out the code for those so you can reuse them later to verify that you haven't made a change (such as instantiating an object earlier or later) that affected the sequence.