If you want to load multiple versions of a class, you can do so if they implement a shared interface and are in separate JARs, using a separate class loader for each version.
If you have a JAR that calls native code, you can store the shared library (DLL) for the native code in its JAR by extracting the shared library to a temporary file and then using System.load to load the library from the temporary file.
But if you do both, will it work? What happens if both versions of the JAR call native code, and both contain a different version of the shared library?
Let us assume that both JARs use a different temporary file to store the copy of the shared library. But the two versions of the shared library have native code that call native (C) functions that have identical declarations (but the implementations of those functions are different). Will the JVM/class loader/System.load delegate from the Java code to the correct native code? Or will the JVM complain about name conflicts?
If that scheme does fail, how do I use multiple versions of a class that uses native code?
Examining the Open JDK 7 implementation, it seems that, yes, loading multiple versions of Java classes that use native code will work:
Library Loading
Crucial information is, how does System.load behave? The implementation of that method will be system dependent, but the semantics of the various implementations should be the same.
System.load delegates to the package-private method Runtime.load0.
Runtime.load0 delegates to the package-private static method ClassLoader.loadLibrary.
ClassLoader.loadLibrary delegates to the private static method ClassLoader.loadLibrary0.
ClassLoader.loadLibrary0 creates an object of the package-private inner class ClassLoader.NativeLibrary and delegates to its load method.
ClassLoader.NativeLibrary.load is a native method, which delegates to the function JVM_LoadLibrary.
JVM_LoadLibrary delegates to os::dll_load.
os::dll_load is system dependent.
The Linux variant of os::dll_load delegates to the dlopen system call, giving the RTLD_LAZY option.
The Linux variant of the POSIX dlopen system call has RTLD_LOCAL behaviour by default, so the shared library is loaded with RTLD_LOCAL semantics.
RTLD_LOCAL semantics are that the symbols in the loaded library are not made available for (automatic) symbol resolution of subsequently loaded libraries. That is, the symbols do not enter the global namespace, and different libraries may define the same symbols without generating conflicts. The shared libraries could even have identical content without problems.
Hence it does not matter if different shared libraries, loaded by different class loaders, define the same symbols (have the same names of extern functions for native methods): the JRE and JVM together avoid name clashes.
Native Function Lookup
That ensures that the multiple versions of the shared libraries do not generate name conflicts. But how does OpenJDK ensure that the correct JNI code is used for the native method calls?
The procedure followed by the JVM to call a native method is rather lengthy, but it is all contained within one function, SharedRuntime::generate_native_wrapper. Ultimately, however, that needs to know the address of the JNI function to be called.
That wrapper function makes use of a methodHandle C++ object, getting the address of the JNI function from either the methodHandle::critical_native_function() or methodHandle::native_function(), as appropriate.
The address of the JNI function is recorded in the methodHandle by a call to methodHandle::set_native_function from NativeLookup::lookup.
NativeLookup::lookup delegates, indirectly, to NativeLookup::lookup_style
NativeLookup::lookup_style delegates to the Java package-private static method ClassLoader.findNative.
ClassLoader.findNative iterates through the list (ClassLoader.nativeLibraries) of ClassLoader.NativeLibrary objects set up by ClassLoader.loadLibrary0, in the order that the libraries were loaded. For each library, it delegates to NativeLibrary.find to try to find the native method of interest. Although this list of objects is not public, the JNI specification requires that the JVM "maintains a list of loaded native libraries for each class loader", so all implementations must have something similar to this list.
NativeLibrary.find is a native method. It simply delegates to JVM_FindLibraryEntry.
JVM_FindLibraryEntry delegates to the system dependent method os::dll_lookup.
The Linux implementation of os::dll_lookup delegates to the dlsym system call to lookup the address of the function in the shared library.
Because each class-loader maintains its own list of loaded libraries, it is guaranteed that the JNI code called for a native method will be the correct version, even if a different class loader loads a different version of the shared library.
If you try to load the same library in different class loaders you will get an UnsatisfiedLinkError with the message "Native Library: ... already loaded in another class loader". This might have something to do with the VM calling the library's unload method when the class loader is garbage collected (https://docs.oracle.com/javase/8/docs/technotes/guides/jni/spec/design.html#compiling_loading_and_linking_native_methods).
But if you - as you say - "use a different temporary file to store the copy of the shared library" the two are effectively different libraries regardless of the files' contents (might be binary identical, doesn't matter). So there isn't a problem.
Related
I'm interested in the subject of language interop.
For the Kotlin/JVM target we're able to call Java code directly without having to define any binding interfaces or using tools such as the SWIG library, JNI and others. How was that achieved ?
Short answer: the Java runtime handles it all; no bindings are needed.
All classes and interfaces that the JVM runs (except for native calls using JNI or similar) are provided to it as Java bytecode (typically in the form of .class files); it's the same whether that bytecode was compiled from Java, or Kotlin, or Groovy, or Scala, or any other JVM language.
The bytecode contains details of all the constructors, fields, and methods of a class or interface. When that refers to other classes, it does so via their fully-qualified names (e.g. java.lang.String) — and the JVM knows how to find (e.g. by searching the classpath) and load any class given its fully-qualified name. (Specifically, it uses a ClassLoader — usually the system one, though custom ones can be used where appropriate.) Having loaded the class, instances can be constructed and their methods called directly.
So the JVM doesn't need any secondary means of identifying or accessing classes/interfaces or their methods; it's all specified in the bytecode and accessible directly.
(If you want more details, the Java Virtual Machine Specification is probably the ultimate reference.)
I can access Java class and methods when executing xquery with Saxon when they are declared correctly (through namespace pointing to package and class) but I wonder if there is a way to create a kind of "dynamic" class path at each run to load external jar file and search classes in it instead of in the current project/program classpath (as I cannot add all possible class in it).
So for instance I have something like :
declare namespace dpr="java:com.*****.atm.dpr.common.util.DPRConfigurationLoader";
declare variable $rules as node()* := doc(dpr:getApplicationProperty('Common','RulesFileLocation'))//category;
I can replace the path the real class with an emulated version but it means I must create each possible class on my side (not really a good way as it means a "patch" for each new java call...).
So if I provide a jar containing the classes I need is there a way to load it so that the namespace point to it ?
I know I can load .class file if they are on classpath, but 3 jar files entirely ?
Thanks.
Technically, Saxon doesn't require external classes to be on the classpath - it requires them to be accessible using the appropriate ClassLoader. If you understand ClassLoaders and are prepared to write your own or configure third-party offerings, then you can load classes from anywhere. All the hooks are there in Saxon if you want to do such things; but there's nothing packaged with the product.
Some of the thing you could try include:
With Configuration.setDynamicLoader() you can change the way Saxon does dynamic loading of external classes, including the classes used for Java extension functions.
With Configuration.getDynamicLoader().setClassLoader() you could provide a different ClassLoader for loading classes, for example a URLClassLoader.
With ProfessionalConfiguration.setExtensionBinder("java", XXX) you could register a customized JavaExtensionLibrary, typically as a subclass of the standard one, allowing you to change the way URIs are mapped to Java classes and the way methods are selected (for example)
This is all very low-level system programming and is not for the faint-hearted.
I am trying to open a Java class file, instrument the bytecode and save the class file before the class is loaded into the JVM. My problem is that a class is being loaded "too soon" into the JVM. The bytecode is instrumented after the class is loaded into the JVM.
-verbose:class prints when each class is loaded but it doesn't tell me what triggered the JVM to load the class. How do I get a call stack which shows the class being loaded?
Putting a breakpoint in the following code, shows the call stack when the class is initialized and not loaded.
static
{
System.out.println("Initialized!");
}
Note: I know I could use a Java agent to do this and guarantee the bytecode is instrumented. But, I choose this route for various reasons.
I opened java.lang.ClassLoader and set a conditional breakpoint in loadClass(String name, boolean resolve). The condition is arg0.endsWith("MyClass") where arg0 is the name parameter. When the breakpoint is triggered, the IDE displays the call stack. Several frames down on the call stack shows me why the class is being loaded.
Note: This condition works in Eclipse IDE and may need a little tweaking in other IDEs.
On the Hotspot JVM, which is what is used with the the Oracle and OpenJDK java distributions, classes are loaded when they are first referenced. Here are the relevant snippets from https://docs.oracle.com/javase/specs/jvms/se8/html/jvms-5.html
The Java Virtual Machine starts up by creating an initial class, which is specified in an implementation-dependent manner, using the bootstrap class loader (§5.3.1). The Java Virtual Machine then links the initial class, initializes it, and invokes the public class method void main(String[]). The invocation of this method drives all further execution. Execution of the Java Virtual Machine instructions constituting the main method may cause linking (and consequently creation) of additional classes and interfaces, as well as invocation of additional methods.
Creation of a class or interface C denoted by the name N consists of the construction in the method area of the Java Virtual Machine (§2.5.4) of an implementation-specific internal representation of C. Class or interface creation is triggered by another class or interface D, which references C through its run-time constant pool. Class or interface creation may also be triggered by D invoking methods in certain Java SE platform class libraries (§2.12) such as reflection.
If you wish to actually perform the bytecode manipulation yourself, and at runtime, then there are at least two models that you may choose to follow
Use something akin to AspectJ's load time weaving, which involves using a separate classloader and an agent to respond to class loading
How JRebel does it, via monitoring known .class files for updated timestamps http://zeroturnaround.com/software/jrebel/learn/faq/
Unless you have a very strong need to do this yourself however, consider using one of the above mentioned tools. Better yet, determine if Java annotations and reflection can solve what you are attempting to do instead.
I have a C++ application that uses plugins (it dynamically loads certain shared lib files). I'm having some trouble understanding what I need to do to call into dynamically loaded native code from Java.
Do I have to use System.loadLibrary/System.load to 'preload' native shared objects ("plugins") to be able to call into that native code? Or would I be okay only calling RegisterNatives from within the plugin after loading it in my native code? I've already called System.loadLibrary for my main native application lib -- its the subsequent plugins the native lib loads that this question is for.
And if I can get away with only calling RegisterNatives what would happen if I suddenly decide to unload a plugin and the JVM tries calling a native method that was inside it?
At least Android VM requires that all native methods must be "registered" when the class is instantiated. That's why System.load() is usually called in static constructor, and not later.
The two ways to resolve native methods are through RegisterNatives() in JNI_OnLoad() or though name matching (C exported functions names as reported by javah).
You can look for function pointers for RegisterNatives() in all loaded modules, or load more modules and get pointers from them. RegisterNatives() may be called at any time, and if you really want to unload some plugins you can make use of UnregisterNatives().
The latter was introduced to support the following flow (pseudo-code follows):
SwitchPlugin() {
UnregisterNatives();
unloadPlugin(oldHandle);
newHandle = loadPlugin(newPluginName);
RegisterNatives();
}
Your app will probably crash miserably if it tries to use a native method that is implemented in an unloaded plugin or after it is unregistered with UnregisterNatives().
I have a JNI library that interacts with a set of third party libraries, and there may be more than one version of the third party library on the system. For each version of the third party library, I have to re-compile the JNI code for comparability reasons. Right now I deal with this by loading a DLL with a specific name, and if the version changes I change the names of the JNI interface DLLs so that the correct one for the version has the right name to get loaded.
I'd like to be able to dynamically load the dll though, based on which version the user wants to use. What happens if I call System.loadLibrary twice on DLLs with different names but the same method signatures?
System.loadLibrary("JNIv1");
// Same code compiled against a different third party version
System.loadLibrary("JNIv2");
I only need to use one of the versions at a time, so it's fine if the old version is no longer accessable.
Is it possible to load two different versions of a DLL with the same method signatures without re-starting the program?
It is possible and in fact it is totally supported and works brilliantly.
I have had to do this in a production environment and on a sun JVM it is rock solid.
Basically, if you load the library from a different classloader, then it will load a different copy of the library. Its as simple as that.
I wouldn't recommend doing this unless you really have to... but it does work.
As an alternative, depending on your specific requirements, you could just make it out of process, and have a simple protocol (using say jetty/xstream/httpclient, or netty) between the client and the different servers, each of which has a different dll version loaded.
Essentially this involves you writing a classloader
public class MyClassLoader extends URLClassLoader {
protected String findLibrary(String libName) {
if ( libName.equals("mylib.dll")) {
return "full/path/to/library";
}
else {
super.findLibrary(libName);
}
}
}
Then you arrange to load the implementation of your class using the relevant classloader...
public interface Implementation {
}
public class ImplementationLookerUpper {
Classloader v1 = new MyClassloader(version1);
Classloader v2 = new MyClassloader(version2);
public Implementation implementationUsingVersion(Version someversion) {
Classloader classloader = pickCorrectClassLoaderForVersion(someVersion);
return (Implementation) classloader.loadClass(RealImplementation.class.getName()).newInstance();
}
}
That kind of thing.....
I don't know the details of how DLL loading in windows works, but my gut feeling tells me it definitely would not be safe to have overlapping libraries loaded at the same time. But maybe if you first unloaded the first one, it might be.
As far as I see, there's no way to explicitly unload a library. If you look at java.lang.Classloader.NativeLibrary you'll see that a library is unloaded when the classloader that caused it to be loaded gets garbage collected (in its finalize() method). So if you load it in a separate classloader, and then wait for it to be garbage collected (with some evil calls to System.gc() to make that happen faster) before loading the new one, that might help.
This is not safe and is quite messy to implement. It is possible to have different classloaders for different versions of the dll. You will have to ensure the classloader is garbage collected to be sure that the dll has been unloaded. (See Java JNI - DLL Unloading)
In the past, we have solved this problem by writing a lightweight java mediator process which handles a user's request and creates a new child java process with the version of the dll requested. When a user requests a different version, the mediator shuts down the existing child and spawns a new one with a different library path. This approach works well. The only downside is the time taken to startup new JVMs, but this will only be noticeable if you have many requests for version switches coming in very frequently.