Develop Reference

Cannot delete library loaded with JNA after disposing

I am trying to delete a DLL which has been loaded into JNA and later disposed. I have tried all the solutions described in the answer to this question, but they are not working: How to dispose library loaded with JNA
Here is code I've tried without a time delay:
import java.io.File;
import com.sun.jna.Library;
import com.sun.jna.Native;
import com.sun.jna.NativeLibrary;
class Filter {
private static ExtDLLTool DLLUtil;
final private static String dllPath = "./ExternalDownloader_64.dll";
static {
DLLUtil = (ExtDLLTool) Native.loadLibrary(dllPath, ExtDLLTool.class);
}
public static void main(String[] args) {
if (DLLUtil != null) {
DLLUtil = null;
NativeLibrary lib = NativeLibrary.getInstance(dllPath);
lib.dispose();
}
File dllFile = new File(dllPath);
if(dllFile.exists()){
boolean isDeleted = dllFile.delete();
if(!isDeleted){
System.out.println("Unable to delete dll file, since it hold by jvm");
}
}
}
private interface ExtDLLTool extends Library {
String validateNomination(String dloadProps);
}
}
I added a time delay to give the native code time to release the handle:
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;
import com.sun.jna.Library;
import com.sun.jna.Native;
import com.sun.jna.NativeLibrary;
class Filter {
private static ExtDLLTool DLLUtil;
final private static String dllPath = "./ExternalDownloader_64.dll";
static {
DLLUtil = (ExtDLLTool) Native.loadLibrary(dllPath, ExtDLLTool.class);
}
public static void main(String[] args) throws Exception{
if (DLLUtil != null) {
DLLUtil = null;
NativeLibrary lib = NativeLibrary.getInstance(dllPath);
lib.dispose();
Thread.sleep(3000);
}
File dllFile = new File(dllPath);
if(dllFile.exists()){
Files.delete(Paths.get(dllPath));
// boolean isDeleted = dllFile.delete();
if(dllFile.exists()){
System.out.println("Unable to delete dll file, since it hold by jvm");
}
}
}
private interface ExtDLLTool extends Library {
String validateNomination(String dloadProps);
}
}
This code results in an exception implying the JVM has not released the file.
Exception in thread "main" java.nio.file.AccessDeniedException: .\ExternalDownloader_64.dll at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102) at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)

In the end the problem is, that Native#open is called twice and Native#close only once. The assumption behind the presented code is, that:
NativeLibrary lib = NativeLibrary.getInstance(dllPath);
yields the same NativeLibrary instance, that is used by:
DLLUtil = (ExtDLLTool) Native.loadLibrary(dllPath, ExtDLLTool.class);
This assumption does not hold. Indeed NativeLibrary#load does use caching and if invoked with the same parameters it will yield only a single instance.
The codepath behind Native.loadLibrary passes two options to Native#loadLibrary: calling-convention and classloader. The calling-convention is equal to the default calling convention, so can be ignored. It is/would be automatically added in NativeLibrary#getInstance. The classloader though is not set to a default value and there is the difference. The options are part of the caching key and thus a second instance of the NativeLibrary is created and not the first returned.
To make it work, the call to NativeLibrary#getInstance must pass the correct classloader. If you modify the sample like this:
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;
import com.sun.jna.Library;
import com.sun.jna.Native;
import com.sun.jna.NativeLibrary;
class Filter {
private static ExtDLLTool DLLUtil;
final private static String dllPath = "./ExternalDownloader_64.dll";
static {
DLLUtil = (ExtDLLTool) Native.loadLibrary(dllPath, ExtDLLTool.class);
}
public static void main(String[] args) throws Exception{
if (DLLUtil != null) {
DLLUtil = null;
NativeLibrary lib = NativeLibrary.getInstance(dllPath, ExtDLLTool.class.getClassLoader());
lib.dispose();
Thread.sleep(3000);
}
File dllFile = new File(dllPath);
if(dllFile.exists()){
Files.delete(Paths.get(dllPath));
// boolean isDeleted = dllFile.delete();
if(dllFile.exists()){
System.out.println("Unable to delete dll file, since it hold by jvm");
}
}
}
private interface ExtDLLTool extends Library {
String validateNomination(String dloadProps);
}
}
it works as expected.
After discussion there is another requirement: The cache path is only hit in a limited number of cases:
the library name is the filename of the library (without a prefix)
the library name is the absolute path to the library
the library name is the "base" name without any prefixes or suffixes the default library search mechanism adds (on windows ".dll" should be stripped, on linux "lib" prefix and ".so" suffix should be stripped) (UNTESTED!)
The TL;DR version: find the absolute path name and use that for interface loading and NativeLibrary loading.

I was able to reproduce the problem with your code, but only on Windows. When reproducible, I was able to successfully delete the file by adding a garbage collection suggestion before the time delay:
if (DLLUtil != null) {
DLLUtil = null;
NativeLibrary lib = NativeLibrary.getInstance(dllPath);
lib.close();
System.gc();
System.gc();
Thread.sleep(3000);
}
When JNA loads a Windows DLL via Native.loadLibrary(), it internally executes the WinAPI LoadLibraryExW function.
Internally the Java instance is stored in a map to be re-used when possible -- however for this to happen, it requires two things to look up the same Java object:
the DLL Path must be an absolute path
the options must match. In this case, you would need to pass the classloader as an argument as Matthias Bläsing indicated in his answer:
// if loaded like this:
DLLUtil = (ExtDLLTool) Native.loadLibrary(dllPath, ExtDLLTool.class);
// fetch from cache like this:
NativeLibrary lib = NativeLibrary.getInstance(dllPath, ExtDLLTool.class.getClassLoader());
lib.dispose();
This should allow you to delete the file.
However, in your case, with the relative path, the library is getting unloaded but the old java object isn't getting closed until GC occurs.
The dispose() (or close() as of 5.12) call in JNA eventually calls the Native.close() method which uses the Windows API FreeLibrary function. This unloads the DLL from the Process memory, so the advice on the linked question on how to dispose is still accurate in the case that you want to re-load the library. If you're not reloading the library, using dispose() (5.11-) or close() (5.12+) is optional.
If you must use a relative path, consider this approach using a PhantomReference inspired by this answer to track the deletion:
if (DLLUtil != null) {
// Unload the DLL from process memory
// Optional here, as it will be called by a cleaner on GC below
NativeLibrary lib = NativeLibrary.getInstance(dllPath);
lib.close();
System.out.println("Closed.");
// Remove any internal JVM references to the file
final ReferenceQueue rq = new ReferenceQueue();
final PhantomReference phantom = new PhantomReference(DLLUtil, rq);
DLLUtil = null;
// Poll until GC removes the reference
int count = 0;
while (rq.poll() == null) {
System.out.println("Waiting...");
Thread.sleep(1000);
if (++count > 4) {
// After 5 seconds prompt for GC!
System.out.println("Suggesting GC...");
System.gc();
}
}
System.out.println("Collected.");
}
The DLL was successfully deleted following this sequence. It did take a second GC call to take effect:
Closed.
Waiting...
Waiting...
Waiting...
Waiting...
Waiting...
Suggesting GC...
Waiting...
Suggesting GC...
Collected.
Deleted!

How to free all resources after reading a JRT?

I am trying to read the list of modules available in a given Java 9+ installation, given its Java Home, using the method described in How to extract the file jre-9/lib/modules?.
The solution works, but it appears that the resources allocated to read the content of the Java Runtime Image are never freed, causing a memory leak, observable with VisualVM for instance:
How can I fix the memory leak in the following reproduction?
package leak;
import java.net.URI;
import java.net.URL;
import java.net.URLClassLoader;
import java.nio.file.FileSystem;
import java.nio.file.FileSystems;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Collections;
import java.util.Map;
import java.util.stream.Stream;
public class JrtfsLeak {
public static void main(String[] args) throws Exception {
Path javaHome = Paths.get(args[0]);
for (int i = 0; i < 100000; ++i) {
modules(javaHome).close();
}
}
private static Stream<Path> modules(Path javaHome) throws Exception {
Map<String, String> env = Collections.singletonMap("java.home", javaHome.toString());
Path jrtfsJar = javaHome.resolve("lib").resolve("jrt-fs.jar");
try (URLClassLoader classloader = new URLClassLoader(new URL[] { jrtfsJar.toUri().toURL() })) {
try (FileSystem fs = FileSystems.newFileSystem(URI.create("jrt:/"), env, classloader)) {
Path modulesRoot = fs.getPath("modules");
return Files.list(modulesRoot);
}
}
}
}

This is a JDK bug JDK-8260621 that has been fixed in JDK 17.
It was caused by a careless use of thread locals in ImageBufferCache.

Note that when you are running under Java 9 or newer, the underlying implementation will create a new class loader for the jrt-fs.jar when you specify the java.home option. So, these classes are not loaded by your URLClassLoader but a different class loader. When you don’t need support for versions prior to 9, you can omit the creation of a class loader.
In either case, they’re loaded by a custom class loader and could be unloaded when the garbage collector supports it. But the class jdk.internal.jimage.ImageBufferCache contains:
private static final ThreadLocal<BufferReference[]> CACHE =
new ThreadLocal<BufferReference[]>() {
#Override
protected BufferReference[] initialValue() {
// 1 extra slot to simplify logic of releaseBuffer()
return new BufferReference[MAX_CACHED_BUFFERS + 1];
}
};
As explained in How does this ThreadLocal prevent the Classloader from getting GCed, a backreference from the value to the thread local can prevent its garbage collection and when the thread local is stored in a static variable, a reference to one of the classes loaded by the same class loader is enough.
And here, the value is an array of BufferReference, which means even when all entries of that array have been cleared, the array type itself has an implicit reference to the class loader of that filesystem.
But since its a thread local variable, we can work-around it by letting the key thread die. When I change your code to
public static void main(String[] args) throws InterruptedException {
Path javaHome = Paths.get(args[0]);
Runnable r = () -> test(javaHome);
for(int i = 0; i < 1000; ++i) {
Thread thread = new Thread(r);
thread.start();
thread.join();
}
}
static void test(Path javaHome) {
for (int i = 0; i < 1000; ++i) {
try(var s = modules(javaHome)) {}
catch(IOException ex) {
throw new UncheckedIOException(ex);
}
catch(Exception ex) {
throw new IllegalStateException(ex);
}
}
}
the classes get unloaded.

Refer to javadoc for method list (in class java.nio.file.Files). Here is the relevant part.
API Note:
This method must be used within a try-with-resources statement or similar control structure to ensure that the stream's open directory is closed promptly after the stream's operations have completed.
In other words you need to close the Stream returned by your modules method.

Java Security Manager: Restrictions on code from external jar loaded via ServiceLoader

What am I trying to achieve?
I am working on a java application that can be extended by additional jars that get integrated via ServiceLoader. These loaded extensions should run with some restrictions by the SecurityManager, of course simply to improve the security. As an example each Extension shall get one specific directory where it can store whatever, but access to any other file/folder should be restricted. The main application is trusted code and can therefore run without any restrictions. Furthermore the main application provides some api implementations for each extension that shall also run without restrictions. That means an extension mustn't access a file outside of its directory but when the extension is calling an api method that tries to access any other file, the access should be granted.
Question
How can I achieve the mentioned behaviour that only 'direct' calls from extension classes get restricted but not any code from the main application?
Running extensions in different threads/threadGroups might be a good solution anyway but since calls to the api might run under the same thread(group) it might not help to identify whether access should be restricted or not based only on the thread.
Example
I created a simplified test environment. On one hand there are these two interfaces:
public interface Extension {
void doSomethingRestricted();
void doSameViaApi(ExtensionApi api);
}
public interface ExtensionApi {
void doSomethingWithHigherPermissions();
}
For testing I created a jar containing this extension:
public class SomeExtension implements Extension {
public void doSomethingRestricted() {
System.out.println(System.getProperty("user.home"));
}
public void doSameViaApi(final ExtensionApi api) {
api.doSomethingWithHigherPermissions();
}
}
In the main application I would like do something like this:
final ExtensionApi api = () -> System.out.println(System.getProperty("user.home"));
try {
final URLClassLoader urlClassLoader = new URLClassLoader(new URL[] { jarFile.toURI().toURL() });
for(final Extension extension : ServiceLoader.load(Extension.class, urlClassLoader)) {
extension.doSomethingRestricted();
extension.doSameViaApi(api);
}
}
So when I call extension.doSomethingRestricted(); it should result in a SecurityException but calling extension.doSameViaApi(api); should work just fine.
So both methods try to do the same but one does try to do it via the api call. The only approach I could think of is iterating through the call history and checking the classloaders to analyze whether the access request is based on trusted code or extension code. But I feel like this might be a nasty error-prone solution so maybe I missed some better approaches?

First ensure your "main" JAR's classes get to enjoy full privileges. Programmatically this may be accomplished as follows:
package q46991566;
import java.nio.file.Files;
import java.nio.file.Path;
import java.security.Policy;
import java.util.Collections;
public class Main {
public static void main(String... args) throws Exception {
// policy configuration contents: this JAR gets all permissions, others get nothing
StringBuilder sb = new StringBuilder("grant {};\n\ngrant codebase \"")
.append(Main.class.getProtectionDomain().getCodeSource().getLocation())
.append("\" {\n\tpermission java.security.AllPermission;\n};\n");
// temp-save the policy configuration
Path policyPath = Files.createTempFile(null, null);
Files.write(policyPath, Collections.singleton(sb.toString()));
// convey to the default file-backed policy provider where to obtain its configuration from;
// leading equals ensures only the specified config file gets processed
System.setProperty("java.security.policy", "=".concat(policyPath.toUri().toURL().toString()));
// establish a policy; "javaPolicy" is the default provider's standard JCA name
Policy.setPolicy(Policy.getInstance("javaPolicy", null));
// policy loaded; backing config no longer needed
Files.delete(policyPath);
// establish a security manager for enforcing the policy (the default implementation is more than
// sufficient)
System.setSecurityManager(new SecurityManager());
// ...
}
}
Alternatively, you will either have to a) modify the JRE distribution's java.policy (or specify a different configuration via the policy.url.n properties in java.security), or b) replace the implementation of the System ClassLoader with one that statically grants AllPermission to the ProtectionDomain associated with classes loaded from the "main" JAR.
Secondly, when loading Extensions from some JAR, employ a URLClassLoader subclass that a) manages extension-specific directories and b) includes a java.io.FilePermission in the permission collection being statically accorded to the protection domain mapped to its defined classes. Crude sample implementation (note that there is no persistent relationship between an extension JAR and a directory; also note that two Extensions originating from the same JAR (but loaded by different class loaders, of course) will get different directories):
package q46991566;
import java.io.FilePermission;
import java.io.IOException;
import java.net.URL;
import java.net.URLClassLoader;
import java.nio.file.Files;
import java.nio.file.Path;
import java.security.CodeSource;
import java.security.Permission;
import java.security.PermissionCollection;
import java.security.Permissions;
import java.security.cert.Certificate;
import java.util.Enumeration;
import java.util.Objects;
public final class ExtensionLoader extends URLClassLoader {
private static void copyPermissions(PermissionCollection src, PermissionCollection dst) {
for (Enumeration<Permission> e = src.elements(); e.hasMoreElements();) {
dst.add(e.nextElement());
}
}
private final CodeSource origin;
private final PermissionCollection perms = new Permissions();
private final Path baseDir;
public ExtensionLoader(URL extensionOrigin) {
super(new URL[] { extensionOrigin });
origin = new CodeSource(Objects.requireNonNull(extensionOrigin), (Certificate[]) null);
try {
baseDir = Files.createTempDirectory(null);
perms.add(new FilePermission(baseDir.toString().concat("/-"), "read,write,delete"));
copyPermissions(super.getPermissions(origin), perms);
perms.setReadOnly();
}
catch (IOException ioe) {
throw new RuntimeException(ioe);
}
}
#Override
protected PermissionCollection getPermissions(CodeSource cs) {
return (origin.implies(cs)) ? perms : super.getPermissions(cs);
}
// ExtensionApiImpl (or ExtensionImpl directly -- but then ExtensionLoader would have to be relocated
// into a separate, also fully privileged JAR, accessible to the extension) can call this to relay to
// extensions where they can persist their data
public Path getExtensionBaseDir() {
return baseDir;
}
// optionally override close() to delete baseDir early
}
Lastly, for unprivileged Extensions to be able to execute privileged operations via ExtensionApi, the latter's implementation must wrap privileged method (methods issuing SecurityManager::checkXXX requests) invocations within Privileged(Exception)Actions and pass them to AccessController::doPrivileged; e.g.:
ExtensionApi api = () -> {
AccessController.doPrivileged((PrivilegedAction<Void>) () -> {
try {
Files.write(Paths.get("/root/Documents/highly-sensitive.doc"), Collections.singleton("trusted content"),
StandardOpenOption.CREATE, StandardOpenOption.WRITE, StandardOpenOption.APPEND);
return null;
}
catch (IOException ioe) {
throw new RuntimeException(ioe);
}
});
};
For details on the (proper) use of "privileged blocks", refer to the AccessController documentation and the "Secure Coding Guidelines for Java SE" document.

GWT generator to get compile time

I am trying to understand GWT generators but facing few issues. I am trying to display the compile time in an app using generators and running into this error -
Rebind result 'com.example.client.Function' must be a class
Here is what i have -
This is how i am calling my generated method -
Function b = GWT.create(Function.class);
label.setText(b.getBuildTime());
gwt.xml-
<generate-with class="example.frontend.client.gin.FunctionGenerator">
<when-type-assignable class="com.example.frontend.client.gin.Function" />
</generate-with>
Function.java
package com.example.frontend.client.gin;
public interface Function{
public String getBuildTime();
}
Generator class -
package com.example.frontend.egenerator;
import java.io.PrintWriter;
import java.util.Date;
import com.google.gwt.core.ext.Generator;
import com.google.gwt.core.ext.GeneratorContext;
import com.google.gwt.core.ext.TreeLogger;
import com.google.gwt.core.ext.UnableToCompleteException;
import com.google.gwt.core.ext.typeinfo.JClassType;
import com.google.gwt.core.ext.typeinfo.TypeOracle;
import com.google.gwt.user.rebind.ClassSourceFileComposerFactory;
import com.google.gwt.user.rebind.SourceWriter;
import com.example.frontend.client.gin.Function;
public class FunctionGenerator extends Generator {
private static final String IMPL_TYPE_NAME = Function.class.getSimpleName() + "Impl";
private static final String IMPL_PACKAGE_NAME = Function.class.getPackage().getName();
#Override
public String generate(final TreeLogger logger, final GeneratorContext context, final String requestedClass) throws UnableToCompleteException {
TypeOracle typeOracle = context.getTypeOracle();
JClassType functionType = typeOracle.findType(requestedClass);
assert Function.class.equals(functionType.getClass());
ClassSourceFileComposerFactory composerFactory = new ClassSourceFileComposerFactory(IMPL_PACKAGE_NAME, IMPL_TYPE_NAME);
composerFactory.addImport(Function.class.getCanonicalName());
composerFactory.addImplementedInterface(Function.class.getName());
PrintWriter printWriter = context.tryCreate(logger, IMPL_PACKAGE_NAME, IMPL_TYPE_NAME);
SourceWriter sourceWriter = composerFactory.createSourceWriter(context, printWriter);
if(sourceWriter != null) {
sourceWriter.print("public String getBuildTime() {");
sourceWriter.print(" return \"" + new Date() + "\" ;");
sourceWriter.print("}");
sourceWriter.commit(logger);
}
return IMPL_PACKAGE_NAME + "." + IMPL_TYPE_NAME;
}
}
Any ideas, what I am missing?

I believe you also need to null check the PrintWriter created by tryCreate, as it may return null. On the other hand, createSourceWriter will not return null, so no need to null check that.
Your generate-with is also incorrect, at least for the sample that you have here. It should have a different package (according to your FunctionGenerator source at least), com.example.frontend.egenerator, not com.example.frontend.client.gin:
<generate-with class="com.example.frontend.egenerator.FunctionGenerator">
<when-type-assignable class="com.example.frontend.client.gin.Function" />
</generate-with>
In general, your generators should not be in the client package, if for no other reason than preventing spurious errors which slow down the compiler (and really slow down super dev mode).
Beyond that, the full log could help a lot to track down the issue, though without mapping the generator correctly there wouldn't be much of an error. Also be sure to compile with strict turned on when working on generators to ensure that the compiler fails as soon as possible and you can stop at the very first error.
With all of that said, tend to avoid new Generators at this point - they will slow down Super Dev Mode slightly (since they must be re-run every time you refresh), and they will not be supported in future versions of GWT. Annotation Processors (aka APT) are the preferred way to do this, but in your case you might also just be able to generate the class in ant or maven with a plugin.

java.rmi.NoSuchObjectException

I have had a look here, here and here however I have absolutely no idea why I am coming across this error.
I am of the opinion that I have strong references (code at end of post), I am not explicitly calling GC (not that that particular method does much) and yet I just cannot boot this damn RMI server.
It should be noted I am extremely new to RMI. I have read the Oracle trail, that didn't work. Tried another tutorial, same problem. In fact, no matter how I approach this, I cannot get past this ObjectNotFoundException.
Now I know that this means the object no longer exists, but why doesn't it exist... I have a static reference to it in the main class, the VM is seriously misbehaving if it is doing this.
Running Arch Linux, oracle's VM and Eclipse kepler.
package engine;
import java.rmi.RMISecurityManager;
import java.rmi.registry.LocateRegistry;
import java.rmi.registry.Registry;
import java.rmi.server.UnicastRemoteObject;
import service.Service;
public class Main {
private static Service stub;
private static Service engine;
public static void main(String[] args) {
System.setProperty("java.security.policy", "file:///home/jameshey/git/MicroSense/release/conf/rmisecurity.policy");
if (System.getSecurityManager() == null) {
System.setSecurityManager(new RMISecurityManager());
}
try {
String name = "HelloWorldService";
engine = new ServiceEngine();
Registry registry = LocateRegistry.getRegistry(1099);
stub = (Service) UnicastRemoteObject.exportObject(engine, 1099);
registry.bind(name, stub);
System.out.println("ServiceEngine bound");
} catch (Exception e) {
System.err.println("ServiceEngine exception:");
e.printStackTrace();
}
}
}
I have tried pretty much every forum solution I can find. I have tried no statics and combinations of which variables are static. Completely stumped.
The Stack Trace of the Exception is:
ServiceEngine exception:
java.rmi.NoSuchObjectException: no such object in table
at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:275)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:252)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:378)
at sun.rmi.registry.RegistryImpl_Stub.bind(Unknown Source)
at engine.Main.main(Main.java:25)

LocateRegistry.getRegistry() doesn't create a Registry. It just creates a stub that may or may not actually work, depending on whether the Registry is running. Change it to createRegistry(). Make the Registry variable static in the server.