I am trying to read the list of modules available in a given Java 9+ installation, given its Java Home, using the method described in How to extract the file jre-9/lib/modules?.
The solution works, but it appears that the resources allocated to read the content of the Java Runtime Image are never freed, causing a memory leak, observable with VisualVM for instance:
How can I fix the memory leak in the following reproduction?
package leak;
import java.net.URI;
import java.net.URL;
import java.net.URLClassLoader;
import java.nio.file.FileSystem;
import java.nio.file.FileSystems;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Collections;
import java.util.Map;
import java.util.stream.Stream;
public class JrtfsLeak {
public static void main(String[] args) throws Exception {
Path javaHome = Paths.get(args[0]);
for (int i = 0; i < 100000; ++i) {
modules(javaHome).close();
}
}
private static Stream<Path> modules(Path javaHome) throws Exception {
Map<String, String> env = Collections.singletonMap("java.home", javaHome.toString());
Path jrtfsJar = javaHome.resolve("lib").resolve("jrt-fs.jar");
try (URLClassLoader classloader = new URLClassLoader(new URL[] { jrtfsJar.toUri().toURL() })) {
try (FileSystem fs = FileSystems.newFileSystem(URI.create("jrt:/"), env, classloader)) {
Path modulesRoot = fs.getPath("modules");
return Files.list(modulesRoot);
}
}
}
}
This is a JDK bug JDK-8260621 that has been fixed in JDK 17.
It was caused by a careless use of thread locals in ImageBufferCache.
Note that when you are running under Java 9 or newer, the underlying implementation will create a new class loader for the jrt-fs.jar when you specify the java.home option. So, these classes are not loaded by your URLClassLoader but a different class loader. When you don’t need support for versions prior to 9, you can omit the creation of a class loader.
In either case, they’re loaded by a custom class loader and could be unloaded when the garbage collector supports it. But the class jdk.internal.jimage.ImageBufferCache contains:
private static final ThreadLocal<BufferReference[]> CACHE =
new ThreadLocal<BufferReference[]>() {
#Override
protected BufferReference[] initialValue() {
// 1 extra slot to simplify logic of releaseBuffer()
return new BufferReference[MAX_CACHED_BUFFERS + 1];
}
};
As explained in How does this ThreadLocal prevent the Classloader from getting GCed, a backreference from the value to the thread local can prevent its garbage collection and when the thread local is stored in a static variable, a reference to one of the classes loaded by the same class loader is enough.
And here, the value is an array of BufferReference, which means even when all entries of that array have been cleared, the array type itself has an implicit reference to the class loader of that filesystem.
But since its a thread local variable, we can work-around it by letting the key thread die. When I change your code to
public static void main(String[] args) throws InterruptedException {
Path javaHome = Paths.get(args[0]);
Runnable r = () -> test(javaHome);
for(int i = 0; i < 1000; ++i) {
Thread thread = new Thread(r);
thread.start();
thread.join();
}
}
static void test(Path javaHome) {
for (int i = 0; i < 1000; ++i) {
try(var s = modules(javaHome)) {}
catch(IOException ex) {
throw new UncheckedIOException(ex);
}
catch(Exception ex) {
throw new IllegalStateException(ex);
}
}
}
the classes get unloaded.
Refer to javadoc for method list (in class java.nio.file.Files). Here is the relevant part.
API Note:
This method must be used within a try-with-resources statement or similar control structure to ensure that the stream's open directory is closed promptly after the stream's operations have completed.
In other words you need to close the Stream returned by your modules method.
Related
I am trying to delete a DLL which has been loaded into JNA and later disposed. I have tried all the solutions described in the answer to this question, but they are not working: How to dispose library loaded with JNA
Here is code I've tried without a time delay:
import java.io.File;
import com.sun.jna.Library;
import com.sun.jna.Native;
import com.sun.jna.NativeLibrary;
class Filter {
private static ExtDLLTool DLLUtil;
final private static String dllPath = "./ExternalDownloader_64.dll";
static {
DLLUtil = (ExtDLLTool) Native.loadLibrary(dllPath, ExtDLLTool.class);
}
public static void main(String[] args) {
if (DLLUtil != null) {
DLLUtil = null;
NativeLibrary lib = NativeLibrary.getInstance(dllPath);
lib.dispose();
}
File dllFile = new File(dllPath);
if(dllFile.exists()){
boolean isDeleted = dllFile.delete();
if(!isDeleted){
System.out.println("Unable to delete dll file, since it hold by jvm");
}
}
}
private interface ExtDLLTool extends Library {
String validateNomination(String dloadProps);
}
}
I added a time delay to give the native code time to release the handle:
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;
import com.sun.jna.Library;
import com.sun.jna.Native;
import com.sun.jna.NativeLibrary;
class Filter {
private static ExtDLLTool DLLUtil;
final private static String dllPath = "./ExternalDownloader_64.dll";
static {
DLLUtil = (ExtDLLTool) Native.loadLibrary(dllPath, ExtDLLTool.class);
}
public static void main(String[] args) throws Exception{
if (DLLUtil != null) {
DLLUtil = null;
NativeLibrary lib = NativeLibrary.getInstance(dllPath);
lib.dispose();
Thread.sleep(3000);
}
File dllFile = new File(dllPath);
if(dllFile.exists()){
Files.delete(Paths.get(dllPath));
// boolean isDeleted = dllFile.delete();
if(dllFile.exists()){
System.out.println("Unable to delete dll file, since it hold by jvm");
}
}
}
private interface ExtDLLTool extends Library {
String validateNomination(String dloadProps);
}
}
This code results in an exception implying the JVM has not released the file.
Exception in thread "main" java.nio.file.AccessDeniedException: .\ExternalDownloader_64.dll at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102) at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
In the end the problem is, that Native#open is called twice and Native#close only once. The assumption behind the presented code is, that:
NativeLibrary lib = NativeLibrary.getInstance(dllPath);
yields the same NativeLibrary instance, that is used by:
DLLUtil = (ExtDLLTool) Native.loadLibrary(dllPath, ExtDLLTool.class);
This assumption does not hold. Indeed NativeLibrary#load does use caching and if invoked with the same parameters it will yield only a single instance.
The codepath behind Native.loadLibrary passes two options to Native#loadLibrary: calling-convention and classloader. The calling-convention is equal to the default calling convention, so can be ignored. It is/would be automatically added in NativeLibrary#getInstance. The classloader though is not set to a default value and there is the difference. The options are part of the caching key and thus a second instance of the NativeLibrary is created and not the first returned.
To make it work, the call to NativeLibrary#getInstance must pass the correct classloader. If you modify the sample like this:
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;
import com.sun.jna.Library;
import com.sun.jna.Native;
import com.sun.jna.NativeLibrary;
class Filter {
private static ExtDLLTool DLLUtil;
final private static String dllPath = "./ExternalDownloader_64.dll";
static {
DLLUtil = (ExtDLLTool) Native.loadLibrary(dllPath, ExtDLLTool.class);
}
public static void main(String[] args) throws Exception{
if (DLLUtil != null) {
DLLUtil = null;
NativeLibrary lib = NativeLibrary.getInstance(dllPath, ExtDLLTool.class.getClassLoader());
lib.dispose();
Thread.sleep(3000);
}
File dllFile = new File(dllPath);
if(dllFile.exists()){
Files.delete(Paths.get(dllPath));
// boolean isDeleted = dllFile.delete();
if(dllFile.exists()){
System.out.println("Unable to delete dll file, since it hold by jvm");
}
}
}
private interface ExtDLLTool extends Library {
String validateNomination(String dloadProps);
}
}
it works as expected.
After discussion there is another requirement: The cache path is only hit in a limited number of cases:
the library name is the filename of the library (without a prefix)
the library name is the absolute path to the library
the library name is the "base" name without any prefixes or suffixes the default library search mechanism adds (on windows ".dll" should be stripped, on linux "lib" prefix and ".so" suffix should be stripped) (UNTESTED!)
The TL;DR version: find the absolute path name and use that for interface loading and NativeLibrary loading.
I was able to reproduce the problem with your code, but only on Windows. When reproducible, I was able to successfully delete the file by adding a garbage collection suggestion before the time delay:
if (DLLUtil != null) {
DLLUtil = null;
NativeLibrary lib = NativeLibrary.getInstance(dllPath);
lib.close();
System.gc();
System.gc();
Thread.sleep(3000);
}
When JNA loads a Windows DLL via Native.loadLibrary(), it internally executes the WinAPI LoadLibraryExW function.
Internally the Java instance is stored in a map to be re-used when possible -- however for this to happen, it requires two things to look up the same Java object:
the DLL Path must be an absolute path
the options must match. In this case, you would need to pass the classloader as an argument as Matthias Bläsing indicated in his answer:
// if loaded like this:
DLLUtil = (ExtDLLTool) Native.loadLibrary(dllPath, ExtDLLTool.class);
// fetch from cache like this:
NativeLibrary lib = NativeLibrary.getInstance(dllPath, ExtDLLTool.class.getClassLoader());
lib.dispose();
This should allow you to delete the file.
However, in your case, with the relative path, the library is getting unloaded but the old java object isn't getting closed until GC occurs.
The dispose() (or close() as of 5.12) call in JNA eventually calls the Native.close() method which uses the Windows API FreeLibrary function. This unloads the DLL from the Process memory, so the advice on the linked question on how to dispose is still accurate in the case that you want to re-load the library. If you're not reloading the library, using dispose() (5.11-) or close() (5.12+) is optional.
If you must use a relative path, consider this approach using a PhantomReference inspired by this answer to track the deletion:
if (DLLUtil != null) {
// Unload the DLL from process memory
// Optional here, as it will be called by a cleaner on GC below
NativeLibrary lib = NativeLibrary.getInstance(dllPath);
lib.close();
System.out.println("Closed.");
// Remove any internal JVM references to the file
final ReferenceQueue rq = new ReferenceQueue();
final PhantomReference phantom = new PhantomReference(DLLUtil, rq);
DLLUtil = null;
// Poll until GC removes the reference
int count = 0;
while (rq.poll() == null) {
System.out.println("Waiting...");
Thread.sleep(1000);
if (++count > 4) {
// After 5 seconds prompt for GC!
System.out.println("Suggesting GC...");
System.gc();
}
}
System.out.println("Collected.");
}
The DLL was successfully deleted following this sequence. It did take a second GC call to take effect:
Closed.
Waiting...
Waiting...
Waiting...
Waiting...
Waiting...
Suggesting GC...
Waiting...
Suggesting GC...
Collected.
Deleted!
My objective is to look at some lines of codes of an external file and count the number of functions of a class are called then.
For example, if I have the following code:
import java.io.BufferedReader;
import whatever.MyClass;
import java.util.ArrayList;
...
...
public void example(){
InputStreamReader isr = new InputStreamReader (whatever);
MyClass object = new MyClass();
someArrayList.add(whatever2)
someArrayList.add(whatever3)
}
In this case, BufferedReader and MyClass functions were called once, and ArrayList functions were called twice.
My solution for that is get a list of all methods inside the used classes and try to match with some string of my code.
For classes created in my project, I can do the following:
jar -tf jarPath
which returns me the list of classes inside a JAR . And doing:
javap -cp jarPath className
I can get a list of all methods inside a JAR whit a specific class name. However, what can I do to get a external methods names, like add(...) of an "external" class java.util.ArrayList?
I can't access the .jar file of java.util.ArrayList correct? Anyone have another suggestion to reach the objective?
The compiler doesn't put the imports into the object file. It throws them away. Import is just a shorthand to the compiler.(Imports are a compile-time feature ).
first step :
use Qdox https://github.com/paul-hammant/qdox to get all the imports in a class :
String fileFullPath = "Your\\java\\ file \\full\\path";
JavaDocBuilder builder = new JavaDocBuilder();
builder.addSource(new FileReader( fileFullPath ));
JavaSource src = builder.getSources()[0];
String[] imports = src.getImports();
for ( String imp : imports )
{
System.out.println(imp);
}
second step :
inspire from that code , loop through your imports (String array) and apply the same code and you will get the methods .
import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
public class Tes {
public static void main(String[] args) {
Class c;
try {
c = Class.forName("java.util.ArrayList");
Arrays.stream(getAccessibleMethods(c)).
forEach(System.out::println);
} catch (ClassNotFoundException e) {
e.printStackTrace();
}
}
public static Method[] getAccessibleMethods(Class clazz) {
List<Method> result = new ArrayList<Method>();
while (clazz != null) {
for (Method method : clazz.getDeclaredMethods()) {
result.add(method);
}
clazz = clazz.getSuperclass();
}
return result.toArray(new Method[result.size()]);
}
}
Output :
public void java.util.ArrayList.add(int,java.lang.Object)
public boolean java.util.ArrayList.add(java.lang.Object)
public boolean java.util.ArrayList.remove(java.lang.Object)
public java.lang.Object java.util.ArrayList.remove(int)
public java.lang.Object java.util.ArrayList.get(int)
public java.lang.Object java.util.ArrayList.clone()
public int java.util.ArrayList.indexOf(java.lang.Object)
public void java.util.ArrayList.clear()
.
.
.
All the code - one class :
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import com.thoughtworks.qdox.JavaDocBuilder;
import com.thoughtworks.qdox.model.JavaSource;
public class Tester {
public static void main(String[] args) {
// put your .java file path
// CyclicB is a class within another project in my pc
String fileFullPath =
"C:\\Users\\OUSSEMA\\Desktop\\dev\\OCP_Preparation\\src\\w\\CyclicB.java";
JavaDocBuilder builder = new JavaDocBuilder();
try {
builder.addSource(new FileReader( fileFullPath ));
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
JavaSource src = builder.getSources()[0];
String[] imports = src.getImports();
for ( String imp : imports )
{
Class c;
try {
c = Class.forName(imp);
Arrays.stream(getAccessibleMethods(c)).
forEach(System.out::println);
} catch (ClassNotFoundException e) {
e.printStackTrace();
}
}
}
public static Method[] getAccessibleMethods(Class clazz) {
List<Method> result = new ArrayList<Method>();
while (clazz != null) {
for (Method method : clazz.getDeclaredMethods()) {
result.add(method);
}
clazz = clazz.getSuperclass();
}
return result.toArray(new Method[result.size()]);
}
}
Output all the methods within the classes imported in the file CyclicB.java :
private void java.lang.Throwable.printStackTrace(java.lang.Throwable$PrintStreamOrWriter)
public void java.lang.Throwable.printStackTrace(java.io.PrintStream)
public void java.lang.Throwable.printStackTrace()
public void java.lang.Throwable.printStackTrace(java.io.PrintWriter)
public synchronized java.lang.Throwable java.lang.Throwable.fillInStackTrace()
.
.
.
You may look into OpenJDK project that has a Java compiler. Learn to build the modified versions. Investigate the syntax analysis layer of this compiler and find where the method calls are handled. Put the logging into these locations and now you only need to build your java file with the modified compiler to get the information about the calls.
The build is complex, but you will likely only need a careful editing in a few files. It is not exactly very low hanging fruit but I think it should be possible to discover these files and make changes in them, and still may be a simpler/cleaner approach than to implement the own Java syntax parser (also doable with JavaCC).
If you also need to track calls from the external libraries, build them with the modified compiler as well and you will have the needed records.
GNU Classpath is another open source project where you can do the similar thing, and it may be easier to build. However, unlike OpenJDK, GNU Classpath java system library is not complete.
This approach may not discover some methods called during reflection. But it would discover that reflection framework methods have been called. If it is a security - related project, the simplest would be to agree that reflection is not allowed. It is uncommon to use reflection in a normal Java application that is not a framework.
Pretty much, I'm trying to write a simple program that lets the user choose a file. Unfortunately, JFileChooser through Swing is a little outdated, so I am trying to use JavaFX FileChooser for this. The goal is to run FileGetter as a thread, transfer the file data to the Main Class, and continue from there.
Main Class:
package application;
import java.io.File;
import javafx.application.Application;
import javafx.stage.Stage;
import javafx.scene.Scene;
import javafx.scene.layout.BorderPane;
public class Main {
public static void main(String[] args) {
Thread t1 = new Thread(new FileGetter());
FileGetter fg = new FileGetter();
t1.start();
boolean isReady = false;
while(isReady == false){
isReady = FileGetter.getIsReady();
}
File file = FileGetter.getFile();
System.out.println(file.getAbsolutePath());
...
}
}
FileGetter Class:
package application;
import java.io.File;
import javafx.application.Application;
import javafx.application.Platform;
import javafx.stage.FileChooser;
import javafx.stage.Stage;
import javafx.scene.Scene;
import javafx.scene.layout.BorderPane;
public class FileGetter extends Application implements Runnable {
static File file;
static boolean isReady = false;
#Override
public void start(Stage primaryStage) {
try {
FileChooser fc = new FileChooser();
while(file == null){
file = fc.showOpenDialog(primaryStage);
}
isReady = true;
Platform.exit();
} catch(Exception e) {
e.printStackTrace();
}
}
#Override
public void run() {
launch();
}
public static boolean getIsReady(){
return isReady;
}
public static File getFile(){
return file;
}
}
Problem is that the value of isReady in the while loop doesn't update to true when the user picked a file (the reason I have it is to prevent the code in Main from continuing with a File set to null).
Any help, alternative suggestions, or explanations as of why this happens is very much appreciated!
The java memory model does not require variable values to be the same in different threads except under specific conditions.
What is happening here is that the FileGetter thread is updating the value in the own memory that is only accessed from this thread, but your main thread doesn't see the updated value, since it only sees the version of the variable stored in it's own memory that is different from the one of the FileGetter thread. Each of the threads has it's own copy of the field in memory, which is perfectly fine according to the java specification.
To fix this, you can simply add the volatile modifier to isReady:
static volatile boolean isReady = false;
which makes sure the updated value will be visible from your main thread.
Furthermore I recommend reducing the number of FileGetter instances you create. In your code 3 instances are created, but only 1 is used.
Thread t1 = new Thread(() -> Application.launch(FileGetter.class));
t1.start();
...
The easiest way to implement this
Instead of trying to drive the horse with the cart, why not just follow the standard JavaFX lifecycle? In other words, make your Main class a subclass of Application, get the file in the start() method, and then proceed (in a background thread) with the rest of the application?
public class Main extends Application {
#Override
public void init() {
// make sure we don't exit when file chooser is closed...
Platform.setImplicitExit(false);
}
#Override
public void start(Stage primaryStage) {
File file = null ;
FileChooser fc = new FileChooser();
while(file == null){
file = fc.showOpenDialog(primaryStage);
}
final File theFile = file ;
new Thread(() -> runApplication(theFile)).start();
}
private void runApplication(File file) {
// run your application here...
}
}
What is wrong with your code
If you really want the Main class to be separate from the JavaFX Application class (which doesn't really make sense: once you have decided to use a JavaFX FileChooser, you have decided you are writing a JavaFX application, so the startup class should be a subclass of Application), then it gets a bit tricky. There are several issues with your code as it stands, some of which are addressed in other answers. The main issue, as shown in Fabian's answer, is that you are referencing FileGetter.isReady from multiple threads without ensuring liveness. This is exactly the issue addressed in Josh Bloch's Effective Java (Item 66 in the 2nd edition).
Another issue with your code is that you won't be able to use the FileGetter more than once (you can't call launch() more than once), which might not be an issue in your code now, but almost certainly will be at some point with this application as development progresses. The problem is that you have mixed two issues: starting the FX toolkit, and retrieving a File from a FileChooser. The first thing must only be done once; the second should be written to be reusable.
And finally your loop
while(isReady == false){
isReady = FileGetter.getIsReady();
}
is very bad practice: it checks the isReady flag as fast as it possibly can. Under some (fairly unusual) circumstances, it could even prevent the FX Application thread from having any resources to run. This should just block until the file is ready.
How to fix without making Main a JavaFX Application
So, again only if you have a really pressing need to do so, I would first create a class that just has the responsibility of starting the FX toolkit. Something like:
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.atomic.AtomicBoolean;
import javafx.application.Application;
import javafx.application.Platform;
import javafx.stage.Stage;
public class FXStarter extends Application {
private static final AtomicBoolean startRequested = new AtomicBoolean(false);
private static final CountDownLatch latch = new CountDownLatch(1);
#Override
public void init() {
Platform.setImplicitExit(false);
}
#Override
public void start(Stage primaryStage) {
latch.countDown();
}
/** Starts the FX toolkit, if not already started via this method,
** and blocks execution until it is running.
**/
public static void startFXIfNeeded() throws InterruptedException {
if (! startRequested.getAndSet(true)) {
new Thread(Application::launch).start();
}
latch.await();
}
}
Now create a class that gets a file for you. This should ensure the FX toolkit is running, using the previous class. This implementation allows you to call getFile() from any thread:
import java.io.File;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.FutureTask;
import javafx.application.Platform;
import javafx.stage.FileChooser;
public class FileGetter {
/**
** Retrieves a file from a JavaFX File chooser. This method can
** be called from any thread, and will block until the user chooses
** a file.
**/
public File getFile() throws InterruptedException {
FXStarter.startFXIfNeeded() ;
if (Platform.isFxApplicationThread()) {
return doGetFile();
} else {
FutureTask<File> task = new FutureTask<File>(this::doGetFile);
Platform.runLater(task);
try {
return task.get();
} catch (ExecutionException exc) {
throw new RuntimeException(exc);
}
}
}
private File doGetFile() {
File file = null ;
FileChooser chooser = new FileChooser() ;
while (file == null) {
file = chooser.showOpenDialog(null) ;
}
return file ;
}
}
and finally your Main is just
import java.io.File;
public class Main {
public static void main(String[] args) throws InterruptedException {
File file = new FileGetter().getFile();
// proceed...
}
}
Again, this is pretty complex; I see no reason not to simply use the standard FX Application lifecycle for this, as in the very first code block in the answer.
In this code
while(isReady == false){
isReady = FileGetter.getIsReady();
}
there is nothing that is going to change the state of FileGetter's isReady to true
The code is as follows
what it does is it loads all the classes inside a jar file which I placed inside my home directory .
import java.io.File;
import java.util.jar.JarFile;
import java.util.jar.JarEntry;
import java.net.URLClassLoader;
import java.net.URL;
import java.util.Enumeration;
import java.lang.ClassLoader;
public class Plugin extends ClassLoader {
public static void main(String[] args) throws Exception {
File file = new File(System.getProperty("user.home") + "/HelloWorld.jar");
URLClassLoader clazzLoader = URLClassLoader.newInstance(new URL[]{file.toURI().toURL()});
JarFile jarFile = new JarFile(file);
Enumeration<JarEntry> entries = jarFile.entries();
while (entries.hasMoreElements()) {
JarEntry element = entries.nextElement();
if (element.getName().endsWith(".class")) {
try {
Class c = clazzLoader.loadClass(element.getName().replaceAll(".class", "").replaceAll("/", "."));
c.newInstance(); // this proves that class is loaded
} catch (Exception e) {
e.printStackTrace();
}
}
}
Class cls = Class.forName("HelloWorld");
cls.newInstance();
Plugin p = new Plugin();
p.checkIfLoaded();
}
public void checkIfLoaded() {
System.out.println("coming in");
if (findLoadedClass("HelloWorld") != null){
System.out.println("Yepee, HelloWorld class is loaded !");
}
}
}
My HelloWorld is as in https://github.com/HarishAtGitHub/doc/blob/master/makeExecutableJar/HelloWorld.java
and the jar is got using the instructions in my github account mentioned above .
c.newInstance() works .
How did I confirm ?
the static block got executed ...
but Class.forName("HelloWorld") throws ClassNotFoundException
also findLoadedClass("HelloWorld") is null ..
I cannot understand why this strange behaviour ?
Please guide ...
This is a classloader issue.
As per the Javadocs to Class.forName, you are looking up the class using the classloader of the current class. As your main class, this will be the JVM's bootstrap classloader (and will more or less just include the standard library plus anything you provided as a -cp command line argument). It is not going to delegate to the classloader that you instantiated as a local variable, and so will not return classes that that classloader could find.
If you were to specify the classloader explicitly, and call
Class.forName("HelloWorld", true, clazzloader)
then the classloader you just created will be searched instead and your class should be found.
Because Class.forName(String) uses currentClassLoader and you have load the class in different ClassLoader.
According with javadoc, invoking Class.forName(String) is equivalent to:
Class.forName(className, true, currentLoader)
I'm trying to load JRuby dynamically at runtime (so I can execute Ruby code using arbitrary JRuby installations and versions). My plan is roughly to create a ClassLoader that has access to jruby.jar, then use that to load the necessary JRuby runtime etc. All was well until I needed to do this multiple times. If I destroy the first JRuby runtime, the third or fourth will cause an OutOfMemory: PermGen space.
I've reduced this to a minimal example. The example uses both the "direct" API as well as the JRuby Embed API. The "direct" API section is commented out, but both exhibit the same behavior: after a few iterations, PermGen is out of memory. (tested with JRuby 1.6.7 and JRuby 1.6.5.1)
import java.lang.reflect.Method;
import java.net.URL;
import java.net.URLClassLoader;
import org.junit.Test;
public class JRubyInstantiationTeardownTest {
#Test
public void test() throws Exception {
for (int i = 0; i < 100; ++i) {
URL[] urls = new URL[] {
new URL("file://path/to/jruby-1.6.7.jar")
};
ClassLoader cl = new URLClassLoader(urls, this.getClass().getClassLoader());
// "Direct" API
/*
Class<?> klass = cl.loadClass("org.jruby.Ruby");
Method newInstance = klass.getMethod("newInstance");
Method evalScriptlet = klass.getMethod("evalScriptlet", String.class);
Method tearDown = klass.getMethod("tearDown");
Object runtime = newInstance.invoke(null);
System.out.println("have " + runtime);
evalScriptlet.invoke(runtime, "puts 'hello, world'");
tearDown.invoke(runtime);
*/
// JRuby Embed API
Class<?> scriptingContainerClass = cl.loadClass("org.jruby.embed.ScriptingContainer");
Method terminate = scriptingContainerClass.getMethod("terminate");
Method runScriptlet = scriptingContainerClass.getMethod("runScriptlet", String.class);
Object container = scriptingContainerClass.newInstance();
System.out.println("have " + container);
runScriptlet.invoke(container, "puts 'hello, world'");
terminate.invoke(container);
}
}
}
Questions: is this a reasonable thing to try to do with a ClassLoader? If so, is this a bug in JRuby, or am I doing something wrong with my class loading?
Bonus: if this were a bug in JRuby, how might something like Eclipse Memory Analysis tool help find the source? I can open a heap dump and see several Ruby objects (where I'd expect no more than one at any given time), but I'm not sure how to find why these aren't being garbage collected...
Try to look at stackoverflow: loading classes with different classloaders to unload them from the JVM when not needed and references from there. Sources of a mature web-container (like Tomcat) should have answers for your problem somewhere in load/unload stack.
PermGen stores bytecode for loaded classes (and generated dynamic proxies). It should be properly compacted by GC, when all references to the classes and their class loader cleared. But your code proves that something keeps your JRuby classes locked and accessible from the main class loader. It could be callback map of somekind the JRuby registers itself on load.
Edit: reported this as a bug: JRUBY-6522, now fixed.
After digging around in the Eclipse Memory Analyzer, I clicked "path to GC" on one of the URLClassLoader instances. It was referenced by org.jruby.RubyEncoding$2 which was referenced by java.lang.ThreadLocal$ThreadLocalMap$Entry.
Looking inside that source file, I see a static ThreadLocal variable being created: RubyEncoding.java:266. ThreadLocals are presumably hanging around forever, referencing my ClassLoader and leaking memory.
This code example succeeds:
import java.lang.reflect.Method;
import java.net.URL;
import java.net.URLClassLoader;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import org.junit.Test;
public class JRubyInstantiationTeardownTest {
public static int i;
#Test
public void test() throws Exception {
for (i = 0; i < 100; ++i) {
URL[] urls = new URL[] {
new URL("file:///home/pat/jruby-1.6.7/lib/jruby.jar")
};
final ClassLoader cl = new URLClassLoader(urls, this.getClass().getClassLoader());
final Class<?> rubyClass = cl.loadClass("org.jruby.Ruby");
final Method newInstance = rubyClass.getMethod("newInstance");
final Method evalScriptlet = rubyClass.getMethod("evalScriptlet", String.class);
final Method tearDown = rubyClass.getMethod("tearDown");
// "Direct" API
Callable<Void> direct = new Callable<Void>() {
public Void call() throws Exception {
// created inside thread because initialization happens immediately
final Object ruby = newInstance.invoke(null);
System.out.println("" + i + ": " + ruby);
evalScriptlet.invoke(ruby, "puts 'hello, world'");
tearDown.invoke(ruby);
return null;
}
};
// JRuby Embed API
final Class<?> scriptingContainerClass = cl.loadClass("org.jruby.embed.ScriptingContainer");
final Method terminate = scriptingContainerClass.getMethod("terminate");
final Method runScriptlet = scriptingContainerClass.getMethod("runScriptlet", String.class);
// created outside thread because ruby instance not created immediately
final Object container = scriptingContainerClass.newInstance();
Callable<Void> embed = new Callable<Void>() {
public Void call() throws Exception {
System.out.println(i + ": " + container);
runScriptlet.invoke(container, "puts 'hello, world'");
terminate.invoke(container);
return null;
}
};
// separate thread for each loop iteration so its ThreadLocal vars are discarded
final ExecutorService executor = Executors.newSingleThreadExecutor();
executor.submit(direct).get();
executor.submit(embed).get();
executor.shutdown();
}
}
}
Now I'm wondering if this is acceptable behavior of JRuby, or what JRuby-Rack does in the context of a servlet container where the servlet container is managing its own thread pool to process requests. It seems like one would need to maintain a completely separate thread pool, only execute Ruby code in those threads, and then ensure they get destroyed when the servlet is undeployed...
This is very relevant: Tomcat Memory Leak Protection
See also JVM bug report: Provide reclaimable thread local values without Thread termination