Develop Reference

Getting a specific version of an image with Jib (Maven, Docker, testcontainers)

I'm trying to understand a comment that a colleague made. We're using testcontainers to create a fixture:
import org.testcontainers.containers.GenericContainer;
import org.testcontainers.utility.DockerImageName;
public class SalesforceFixture extends GenericContainer<SalesforceFixture> {
private static final String APPLICATION_NAME = "salesforce-emulator";
public SalesforceFixture() {
// super(ImageResolver.resolve(APPLICATION_NAME));
super(DockerImageName.parse("gcr.io/ad-selfserve/salesforce-emulator:latest"));
...
}
...
The commented code is what it used to be. The next line is my colleague's suggestion. And on that line he commented:
This is the part I don't know. The [ImageResolver] gets the specific version of the emulator, rather than the latest. You need a docker-info file for that though, which jib doesn't automatically generate (but I think it can).
This is what I know or have figured so far:
SalesforceFixture is a class that will be used by other projects to write tests. It spins up a container in Docker, running a service that emulates the real service's API. It's like a local version of the service that behaves enough like the real thing that if one writes code and tests using the fixture, it should work the same in production. (This is where my knowledge ends.)
I looked into ImageResolver—it seems to be a class we wrote that searches a filesystem for something:
public static String resolve(String applicationName, File... roots) {
Stream<File> searchPaths = Arrays.stream(roots).flatMap((value) -> {
return Stream.of(new File(value, "../" + applicationName), new File(value, applicationName));
});
Optional<File> buildFile = searchPaths.flatMap((searchFile) -> {
if (searchFile.exists()) {
File imageFile = new File(searchFile + File.separator + "/target/docker/image-name");
if (imageFile.exists()) {
return Stream.of(imageFile);
}
}
return Stream.empty();
}).findAny();
InputStream build = (InputStream)buildFile.map(ImageResolver::fileStream).orElseGet(() -> {
return searchClasspath(applicationName);
});
if (build != null) {
try {
return IOUtils.toString(build, Charset.defaultCharset()).trim();
} catch (IOException var6) {
throw new RuntimeException("An exception has occurred while reading build file", var6);
}
} else {
throw new RuntimeException("Could not resolve target image for application: " + applicationName);
}
}
But I'm confused. What filesystem? Like, what is the present working directory? My local computer, wherever I ran the Java program from? Or is this from within some container? (I don't think so.) Or maybe the directory structure inside a .jar file? Or somewhere in gcr.io?
What does he mean about a "specific version number" vs. "latest"? I mean, when I build this project, whatever it built is all I have. Isn't that equivalent to "latest"? In what case would an older version of an image be present? (That's what made me think of gcr.io.)
Or, does he mean, that in the project using this project's image, one will not be able to specify a version via Maven/pom.xml—it will always spin up the latest.
Sorry this is long, just trying to "show my work." Any hints welcome. I'll keep looking.

I can't comment on specifics of your own internal implementations, but ImageResolver seems to work on your local filesystem, e.g. it looks into your target/ directory and also touches the classpath. I can imagine this code was just written for resolving an actual image name (not an image), since it also returns a String.
Regarding latest, using a latest tag for a Docker image is generally considered an anti-pattern, so likely your colleague is commenting about this. Here is a random article from the web explaining some of the issues with latest tag:
https://vsupalov.com/docker-latest-tag/
Besides, I don't understand why you ask these questions which are very specific to your project here on SO rather than asking your colleague.

AWS Lambda and Java Reflections (Guava)

I am trying to run Guava reflections in my AWS Lambda function but it seems to not work in production..
The Code i am trying to run is supposed to create a Map<String, Class> with class name and class.
Code:
val converterClassMap by lazy {
val cl = ClassLoader.getSystemClassLoader()
ClassPath.from(cl).getTopLevelClasses("converters").asSequence().mapNotNull { it.load().kotlin }
.filter { it.simpleName?.endsWith("Converter") == true }
.associateBy( { it.simpleName }, { it } )
}
Running this code locally works perfectly, but running it in production on a lambda return an error where the map is empty.
Key PaginationConverter is missing in the map.: java.util.NoSuchElementException
Has anyone else run into this problem?

One more case. You have the
val cl = ClassLoader.getSystemClassLoader()
the line in the code. It means it takes the system classloader to scan for classes.
Try using
class SomeClassFromYouCodeNotALibrary
val cl = SomeClassFromYouCodeNotALibrary::class.java.classLoader
That one will work stable, independent from the number of classloaders, that are used in the application. AWS Lambda runtime may have specific classloaders, for example.
If it does not work, try logging the classloader type and classpath, e.g. println(cl) and println((cl as? URLClassLoader).getURLs().joinToString(", "))

Java Urlclass loader cast exception [duplicate]

We learned from the release notes of Java 9 that
The application class loader is no longer an instance of java.net.URLClassLoader (an implementation detail that was never specified in previous releases). Code that assumes that ClassLoader::getSytemClassLoader returns a URLClassLoader object will need to be updated.
This breaks old code, which scans the classpath as follows:
Java <= 8
URL[] ressources = ((URLClassLoader) classLoader).getURLs();
which runs into a
java.lang.ClassCastException:
java.base/jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to
java.base/java.net.URLClassLoader
So for Java 9+ the following workaround was proposed as a PR at the Apache Ignite Project, which works as intended given adjustments in the JVM runtime options: --add-opens java.base/jdk.internal.loader=ALL-UNNAMED. However, as mentioned in the comments below, this PR was never merged into their Master branch.
/*
* Java 9 + Bridge to obtain URLs from classpath...
*/
private static URL[] getURLs(ClassLoader classLoader) {
URL[] urls = new URL[0];
try {
//see https://github.com/apache/ignite/pull/2970
Class builtinClazzLoader = Class.forName("jdk.internal.loader.BuiltinClassLoader");
if (builtinClazzLoader != null) {
Field ucpField = builtinClazzLoader.getDeclaredField("ucp");
ucpField.setAccessible(true);
Object ucpObject = ucpField.get(classLoader);
Class clazz = Class.forName("jdk.internal.loader.URLClassPath");
if (clazz != null && ucpObject != null) {
Method getURLs = clazz.getMethod("getURLs");
if (getURLs != null) {
urls = (URL[]) getURLs.invoke(ucpObject);
}
}
}
} catch (NoSuchMethodException | InvocationTargetException | NoSuchFieldException | IllegalAccessException | ClassNotFoundException e) {
logger.error("Could not obtain classpath URLs in Java 9+ - Exception was:");
logger.error(e.getLocalizedMessage(), e);
}
return urls;
}
However, this causes some severe headache due to the use of Reflection here. This is kind of an anti-pattern and is strictly criticized by the forbidden-apis maven plugin:
Forbidden method invocation: java.lang.reflect.AccessibleObject#setAccessible(boolean) [Reflection usage to work around access flags fails with SecurityManagers and likely will not work anymore on runtime classes in Java 9]
Question
Is there a safe way to access the list of all resource URLs in the class- / module path, which can be accessed by the given classloader, in OpenJDK 9/10 without using sun.misc.* imports (e.g. by using Unsafe)?
UPDATE (related to the comments)
I know, that I can do
String[] pathElements = System.getProperty("java.class.path").split(System.getProperty("path.separator"));
to obtain the elements in the classpath and then parse them to URLs. However - as far as I know - this property only returns the classpath given at the time of the application launch. However, in a container environment this will be the one of the application server and might not be sufficient, e.g. then using EAR bundles.
UPDATE 2
Thank your for all your comments. I will test, if System.getProperty("java.class.path") will work for our purposes and update the question, if this fullfills our needs.
However, it seems that other projects (maybe for other reasons, e.g Apache TomEE 8) suffer the same pain related to the URLClassLoader- for this reason, I think it is a valueable question.
UPDATE 3
Finally, we did switch to classgraph and migrated our code to this library to resolve our use-case to load ML resources bundled as JARs from the classpath.

I think this is an XY problem. Accessing the URLs of all resources on the classpath is not a supported operation in Java and is not a good thing to try to do. As you have already seen in this question, you will be fighting against the framework all the way if you try to do this. There will be a million edge cases that will break your solution (custom classloaders, EE containers, etc. etc.).
Please could you expand on why you want to do this?
If you have some kind of plugin system and are looking for modules that interface with your code which may have been provided at runtime, then you should use the ServiceLoader API, i.e.:
A service provider that is packaged as a JAR file for the class path is identified by placing a provider-configuration file in the resource directory META-INF/services. The name of the provider-configuration file is the fully qualified binary name of the service. The provider-configuration file contains a list of fully qualified binary names of service providers, one per line.
For example, suppose the service provider com.example.impl.StandardCodecs is packaged in a JAR file for the class path. The JAR file will contain a provider-configuration file named:
META-INF/services/com.example.CodecFactory
that contains the line:
com.example.impl.StandardCodecs # Standard codecs

AFAIK you can parse the java.class.path system property to get the urls:
String classpath = System.getProperty("java.class.path");
String[] entries = classpath.split(File.pathSeparator);
URL[] result = new URL[entries.length];
for(int i = 0; i < entries.length; i++) {
result[i] = Paths.get(entries[i]).toAbsolutePath().toUri().toURL();
}
System.out.println(Arrays.toString(result)); // e.g. [file:/J:/WS/Oxygen-Stable/jdk10/bin/]

How to safely access the URLs of all resource files in the classpath in Java 9+?

We learned from the release notes of Java 9 that
The application class loader is no longer an instance of java.net.URLClassLoader (an implementation detail that was never specified in previous releases). Code that assumes that ClassLoader::getSytemClassLoader returns a URLClassLoader object will need to be updated.
This breaks old code, which scans the classpath as follows:
Java <= 8
URL[] ressources = ((URLClassLoader) classLoader).getURLs();
which runs into a
java.lang.ClassCastException:
java.base/jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to
java.base/java.net.URLClassLoader
So for Java 9+ the following workaround was proposed as a PR at the Apache Ignite Project, which works as intended given adjustments in the JVM runtime options: --add-opens java.base/jdk.internal.loader=ALL-UNNAMED. However, as mentioned in the comments below, this PR was never merged into their Master branch.
/*
* Java 9 + Bridge to obtain URLs from classpath...
*/
private static URL[] getURLs(ClassLoader classLoader) {
URL[] urls = new URL[0];
try {
//see https://github.com/apache/ignite/pull/2970
Class builtinClazzLoader = Class.forName("jdk.internal.loader.BuiltinClassLoader");
if (builtinClazzLoader != null) {
Field ucpField = builtinClazzLoader.getDeclaredField("ucp");
ucpField.setAccessible(true);
Object ucpObject = ucpField.get(classLoader);
Class clazz = Class.forName("jdk.internal.loader.URLClassPath");
if (clazz != null && ucpObject != null) {
Method getURLs = clazz.getMethod("getURLs");
if (getURLs != null) {
urls = (URL[]) getURLs.invoke(ucpObject);
}
}
}
} catch (NoSuchMethodException | InvocationTargetException | NoSuchFieldException | IllegalAccessException | ClassNotFoundException e) {
logger.error("Could not obtain classpath URLs in Java 9+ - Exception was:");
logger.error(e.getLocalizedMessage(), e);
}
return urls;
}
However, this causes some severe headache due to the use of Reflection here. This is kind of an anti-pattern and is strictly criticized by the forbidden-apis maven plugin:
Forbidden method invocation: java.lang.reflect.AccessibleObject#setAccessible(boolean) [Reflection usage to work around access flags fails with SecurityManagers and likely will not work anymore on runtime classes in Java 9]
Question
Is there a safe way to access the list of all resource URLs in the class- / module path, which can be accessed by the given classloader, in OpenJDK 9/10 without using sun.misc.* imports (e.g. by using Unsafe)?
UPDATE (related to the comments)
I know, that I can do
String[] pathElements = System.getProperty("java.class.path").split(System.getProperty("path.separator"));
to obtain the elements in the classpath and then parse them to URLs. However - as far as I know - this property only returns the classpath given at the time of the application launch. However, in a container environment this will be the one of the application server and might not be sufficient, e.g. then using EAR bundles.
UPDATE 2
Thank your for all your comments. I will test, if System.getProperty("java.class.path") will work for our purposes and update the question, if this fullfills our needs.
However, it seems that other projects (maybe for other reasons, e.g Apache TomEE 8) suffer the same pain related to the URLClassLoader- for this reason, I think it is a valueable question.
UPDATE 3
Finally, we did switch to classgraph and migrated our code to this library to resolve our use-case to load ML resources bundled as JARs from the classpath.

I think this is an XY problem. Accessing the URLs of all resources on the classpath is not a supported operation in Java and is not a good thing to try to do. As you have already seen in this question, you will be fighting against the framework all the way if you try to do this. There will be a million edge cases that will break your solution (custom classloaders, EE containers, etc. etc.).
Please could you expand on why you want to do this?
If you have some kind of plugin system and are looking for modules that interface with your code which may have been provided at runtime, then you should use the ServiceLoader API, i.e.:
A service provider that is packaged as a JAR file for the class path is identified by placing a provider-configuration file in the resource directory META-INF/services. The name of the provider-configuration file is the fully qualified binary name of the service. The provider-configuration file contains a list of fully qualified binary names of service providers, one per line.
For example, suppose the service provider com.example.impl.StandardCodecs is packaged in a JAR file for the class path. The JAR file will contain a provider-configuration file named:
META-INF/services/com.example.CodecFactory
that contains the line:
com.example.impl.StandardCodecs # Standard codecs

AFAIK you can parse the java.class.path system property to get the urls:
String classpath = System.getProperty("java.class.path");
String[] entries = classpath.split(File.pathSeparator);
URL[] result = new URL[entries.length];
for(int i = 0; i < entries.length; i++) {
result[i] = Paths.get(entries[i]).toAbsolutePath().toUri().toURL();
}
System.out.println(Arrays.toString(result)); // e.g. [file:/J:/WS/Oxygen-Stable/jdk10/bin/]

Problem reloading a jar using URLClassLoader

I need to add plugin functionality to an existing application for certain parts of the application. I want to be able to add a jar at runtime and the application should be able to load a class from the jar without restarting the app. So far so good. I found some samples online using URLClassLoader and it works fine.
I also wanted the ability to reload the same class when an updated version of the jar is available. I again found some samples and the key to achieving this as I understand is that I need to use a new classloader instance for each new load.
I wrote some sample code but hit a NullPointerException. First let me show you guys the code:
package test.misc;
import java.io.File;
import java.net.URL;
import java.net.URLClassLoader;
import plugin.misc.IPlugin;
public class TestJarLoading {
public static void main(String[] args) {
IPlugin plugin = null;
while(true) {
try {
File file = new File("C:\\plugins\\test.jar");
String classToLoad = "jartest.TestPlugin";
URL jarUrl = new URL("jar", "","file:" + file.getAbsolutePath()+"!/");
URLClassLoader cl = new URLClassLoader(new URL[] {jarUrl}, TestJarLoading.class.getClassLoader());
Class loadedClass = cl.loadClass(classToLoad);
plugin = (IPlugin) loadedClass.newInstance();
plugin.doProc();
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
Thread.sleep(30000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}
}
IPlugin is a simple interface with just one method doProc:
public interface IPlugin {
void doProc();
}
and jartest.TestPlugin is an implementation of this interface where doProc just prints out some statements.
Now, I package the jartest.TestPlugin class into a jar called test.jar and place it under C:\plugins and run this code. The first iteration runs smoothly and the class loads without issues.
When the program is executing the sleep statement, I replace C:\plugins\test.jar with a new jar containing an updated version of the same class and wait for the next iteration of while. Now here's what I don't understand. Sometimes the updated class gets reloaded without issues i.e. the next iteration runs fine. But sometimes, I see an exception thrown:
java.lang.NullPointerException
at java.io.FilterInputStream.close(FilterInputStream.java:155)
at sun.net.www.protocol.jar.JarURLConnection$JarURLInputStream.close(JarURLConnection.java:90)
at sun.misc.Resource.getBytes(Resource.java:137)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:256)
at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at test.misc.TestJarLoading.main(TestJarLoading.java:22)
I have searched on the net and scratched my head but can't really arrive at any conclusion as to why this exception is thrown and that too - only sometimes, not always.
I need your experience and expertise to understand this. What's wrong with this code? Please help!!
Let me know if you need any more info. Thanks for looking!

For everyone's benefit, let me summarize the real problem and the solution that worked for me.
As Ryan pointed out, there is a bug in JVM, which affects Windows Platform. URLClassLoader does not close the open jar files after it opens them for loading classes, effectively locking the jar files. The jar files can't be deleted or replaced.
The solution is simple: close the open jar files after they've been read. However, to get a handle to the open jar files, we need to use reflection since the properties we need to traverse down are not public. So we traverse down this path
URLClassLoader -> URLClassPath ucp -> ArrayList<Loader> loaders
JarLoader -> JarFile jar -> jar.close()
The code to close the open jar files can be added to a close() method in a class extending URLClassLoader:
public class MyURLClassLoader extends URLClassLoader {
public PluginClassLoader(URL[] urls, ClassLoader parent) {
super(urls, parent);
}
/**
* Closes all open jar files
*/
public void close() {
try {
Class clazz = java.net.URLClassLoader.class;
Field ucp = clazz.getDeclaredField("ucp");
ucp.setAccessible(true);
Object sunMiscURLClassPath = ucp.get(this);
Field loaders = sunMiscURLClassPath.getClass().getDeclaredField("loaders");
loaders.setAccessible(true);
Object collection = loaders.get(sunMiscURLClassPath);
for (Object sunMiscURLClassPathJarLoader : ((Collection) collection).toArray()) {
try {
Field loader = sunMiscURLClassPathJarLoader.getClass().getDeclaredField("jar");
loader.setAccessible(true);
Object jarFile = loader.get(sunMiscURLClassPathJarLoader);
((JarFile) jarFile).close();
} catch (Throwable t) {
// if we got this far, this is probably not a JAR loader so skip it
}
}
} catch (Throwable t) {
// probably not a SUN VM
}
return;
}
}
(This code was taken from the second link that Ryan posted. This code is also posted on the bug report page.)
However, there's a catch: For this code to work and be able to get a handle to the open jar files to close them, the loader used to load the classes from the file by URLClassLoader implementation has to be a JarLoader. Looking at the source code of URLClassPath (method getLoader(URL url)), I noticed that it uses a JARLoader only if the file string used to create the URL does not end in "/". So, the URL must be defined like this:
URL jarUrl = new URL("file:" + file.getAbsolutePath());
The overall class loading code should look something like this:
void loadAndInstantiate() {
MyURLClassLoader cl = null;
try {
File file = new File("C:\\jars\\sample.jar");
String classToLoad = "com.abc.ClassToLoad";
URL jarUrl = new URL("file:" + file.getAbsolutePath());
cl = new MyURLClassLoader(new URL[] {jarUrl}, getClass().getClassLoader());
Class loadedClass = cl.loadClass(classToLoad);
Object o = loadedClass.getConstructor().newInstance();
} finally {
if(cl != null)
cl.close();
}
}
Update: JRE 7 has introduced a close() method in the class URLClassLoader which may have solved this issue. I haven't verified it.

This behaviour is related to a bug in the jvm
2 workarounds are documented here

Starting from Java 7, you indeed have a close() method in URLClassLoader but it is not enough to release completely the jar files if you call directly or indirectly methods of type ClassLoader#getResource(String), ClassLoader#getResourceAsStream(String) or ClassLoader#getResources(String). Indeed by default, the JarFile instances are automatically stored into the cache of JarFileFactory in case we call directly or indirectly one of the previous methods and those instances are not released even if we call java.net.URLClassLoader#close().
So a hack is still needed in this particular case even with Java 1.8.0_74, here is my hack https://github.com/essobedo/application-manager/blob/master/src/main/java/com/github/essobedo/appma/core/util/Classpath.java#L83 that I use here https://github.com/essobedo/application-manager/blob/master/src/main/java/com/github/essobedo/appma/core/DefaultApplicationManager.java#L388. Even with this hack, I still had to call the GC explicitly to fully release the jar files as you can see here https://github.com/essobedo/application-manager/blob/master/src/main/java/com/github/essobedo/appma/core/DefaultApplicationManager.java#L419

This is an update tested on java 7 with success. Now the URLClassLoader works fine for me
MyReloader
class MyReloaderMain {
...
//assuming ___BASE_DIRECTORY__/lib for jar and ___BASE_DIRECTORY__/conf for configuration
String dirBase = ___BASE_DIRECTORY__;
File file = new File(dirBase, "lib");
String[] jars = file.list();
URL[] jarUrls = new URL[jars.length + 1];
int i = 0;
for (String jar : jars) {
File fileJar = new File(file, jar);
jarUrls[i++] = fileJar.toURI().toURL();
System.out.println(fileJar);
}
jarUrls[i] = new File(dirBase, "conf").toURI().toURL();
URLClassLoader classLoader = new URLClassLoader(jarUrls, MyReloaderMain.class.getClassLoader());
// this is required to load file (such as spring/context.xml) into the jar
Thread.currentThread().setContextClassLoader(classLoader);
Class classToLoad = Class.forName("my.app.Main", true, classLoader);
instance = classToLoad.newInstance();
Method method = classToLoad.getDeclaredMethod("start", args.getClass());
Object result = method.invoke(instance, args);
...
}
Close and Restart the ClassReloader
then update your jar and call
classLoader.close();
then you can restart the app with the new version.
Do not include your jar into your base class loader
Do not include your jar into your base class loader "MyReloaderMain.class.getClassLoader()" of the "MyReloaderMain", in other words develop 2 project with 2 jars one for "MyReloaderMain" and the other one for your real application without dependency between the two, or you will not able to understand who i loading what.

The error is still present in jdk1.8.0_25 on Windows. Although #Nicolas' answer helps, I hit a ClassNotFound for sun.net.www.protocol.jar.JarFileFactory when running it on WildFly, and several vm crashes while debugging some box tests...
Therefore I ended up extracting the part of the code which deals with loading and unloading, to an external jar. From the main code I just call this with java -jar.... all looks fine for now.
NOTE: Windows does release the locks on the loaded jar files when the jvm exits, that is why this works.

In principle, a class that has already been loaded cannot be reloaded with the same classloader.
For a new load, it is necessary to create a new classloader and thus load the class.
Using URLClassLoader has one problem and that is that the jar file remains open.
If you have multiple classes loaded from one jar file by different instances of URLClassLoader and you change the jar file at runtime, you will usually get this error: java.util.zip.ZipException: ZipFile invalid LOC header (bad signature). The error may be different.
In order for the above errors not to occur, it is necessary to use the close method on all URLClassLoaders using the given jar file. But this is a solution that actually leads to a restart of the entire application.
A better solution is to modify the URLClassLoader so that the contents of the jar file are loaded into the RAM cache. This no longer affects other URLClassloaders that read data from the same jar file. The jar file can then be freely changed while the application is running. For example, you can use this modification of URLClassLoader for this purpose: in-memory URLClassLoader