I work on a project where we use a library that is not guaranteed thread-safe (and isn't) and single-threaded in a Java 8 streams scenario, which works as expected.
We would like to use parallel streams to get the low hanging scalability fruit.
Unfortunately this cause the library to fail - most likely because one instance interferes with variables shared with the other instance - hence we need isolation.
I was considering using a separate classloader for each instance (possibly thread local) which to my knowledge should mean that for all practical purposes that I get the isolation needed but I am unfamiliar with deliberately constructing classloaders for this purpose.
Is this the right approach? How shall I do this in order to have proper production quality?
Edit: I was asked for additional information about the situation triggering the question, in order to understand it better. The question is still about the general situation, not fixing the library.
I have full control over the object created by the library (which is https://github.com/veraPDF/) as pulled in by
<dependency>
<groupId>org.verapdf</groupId>
<artifactId>validation-model</artifactId>
<version>1.1.6</version>
</dependency>
using the project maven repository for artifacts.
<repositories>
<repository>
<snapshots>
<enabled>true</enabled>
</snapshots>
<id>vera-dev</id>
<name>Vera development</name>
<url>http://artifactory.openpreservation.org/artifactory/vera-dev</url>
</repository>
</repositories>
For now it is unfeasible to harden the library.
EDIT: I was asked to show code. Our core adapter is roughly:
public class VeraPDFValidator implements Function<InputStream, byte[]> {
private String flavorId;
private Boolean prettyXml;
public VeraPDFValidator(String flavorId, Boolean prettyXml) {
this.flavorId = flavorId;
this.prettyXml = prettyXml;
VeraGreenfieldFoundryProvider.initialise();
}
#Override
public byte[] apply(InputStream inputStream) {
try {
return apply0(inputStream);
} catch (RuntimeException e) {
throw e;
} catch (ModelParsingException | ValidationException | JAXBException | EncryptedPdfException e) {
throw new RuntimeException("invoking VeraPDF validation", e);
}
}
private byte[] apply0(InputStream inputStream) throws ModelParsingException, ValidationException, JAXBException, EncryptedPdfException {
PDFAFlavour flavour = PDFAFlavour.byFlavourId(flavorId);
PDFAValidator validator = Foundries.defaultInstance().createValidator(flavour, false);
PDFAParser loader = Foundries.defaultInstance().createParser(inputStream, flavour);
ValidationResult result = validator.validate(loader);
// do in-memory generation of XML byte array - as we need to pass it to Fedora we need it to fit in memory anyway.
ByteArrayOutputStream baos = new ByteArrayOutputStream();
XmlSerialiser.toXml(result, baos, prettyXml, false);
final byte[] byteArray = baos.toByteArray();
return byteArray;
}
}
which is a function that maps from an InputStream (providing a PDF-file) to a byte array (representing the XML report output).
(Seeing the code, I've noticed that there is a call to the initializer in the constructor, which may be the culprit here in my particular case. I'd still like a solution to the generic problem.
We have faced similar challenges. Issues usually came from from static properties which became unwillingly "shared" between the various threads.
Using different classloaders worked for us as long as we could guarantee that the static properties were actually set on classes loaded by our class loader. Java may have a few classes which provide properties or methods which are not isolated among threads or are not thread-safe ('System.setProperties() and Security.addProvider() are OK - any canonical documentation on this matter is welcomed btw).
A potentially workable and fast solution - that at least can give you a chance to test this theory for your library - is to use a servlet engine such as Jetty or Tomcat.
Build a few wars that contain your library and start processes in parallel (1 per war).
When running code inside a servlet thread, the WebappClassLoaders of these engines attempt to load a classes from the parent class loader first (the same as the engine) and if it does not find the class, attempts to load it from the jars/classes packaged with the war.
With jetty you can programmatically hot deploy wars to the context of your choice and then theoretically scale the number of processors (wars) as required.
We have implemented our own class loader by extending URLClassLoader and have taken inspiration from the Jetty Webapp ClassLoader. It is not as hard a job as as it seems.
Our classloader does the exact opposite: it attempts to load a class from the jars local to the 'package' first , then tries to get them from the parent class loader. This guarantees that a library accidentally loaded by the parent classloader is never considered (first). Our 'package' is actually a jar that contains other jars/libraries with a customized manifest file.
Posting this class loader code "as is" would not make a lot of sense (and create a few copyright issues). If you want to explore that route further, I can try coming up with a skeleton.
Source of the Jetty WebappClassLoader
The answer actually depends on what your library relies on:
If your library relies on at least one native library, using ClassLoaders to isolate your library's code won't help because according to the JNI Specification, it is not allowed to load the same JNI native library into more than one class loader such that you would end up with a UnsatisfiedLinkError.
If you library relies on at least one external resource that is not meant to be shared like for example a file and that is modified by your library, you could end up with complex bugs and/or the corruption of the resource.
Assuming that you are not in the cases listed above, generally speaking if a class is known as non thread safe and doesn't modify any static fields, using a dedicated instance of this class per call or per thread is good enough as the class instance is then no more shared.
Here as your library obviously relies and modifies some static fields that are not meant to be shared, you indeed need to isolate the classes of your library in a dedicated ClassLoader and of course make sure that your threads don't share the same ClassLoader.
For this you could simply create an URLClassLoader to which you would provide the location of your library as URL (using URLClassLoader.newInstance(URL[] urls, ClassLoader parent)), then by reflection you would retrieve the class of your library corresponding to the entry point and invoke your target method. To avoid building a new URLClassLoader at each call, you could consider relying on a ThreadLocal to store the URLClassLoader or the Class or the Method instance to be used for a given thread.
So here is how you could proceed:
Let's say that the entry point of my library is the class Foo that looks like this:
package com.company;
public class Foo {
// A static field in which we store the name of the current thread
public static String threadName;
public void execute() {
// We print the value of the field before setting a value
System.out.printf(
"%s: The value before %s%n", Thread.currentThread().getName(), threadName
);
// We set a new value
threadName = Thread.currentThread().getName();
// We print the value of the field after setting a value
System.out.printf(
"%s: The value after %s%n", Thread.currentThread().getName(), threadName
);
}
}
This class is clearly not thread safe and the method execute modifies the value of a static field that is not meant to be modified by concurrent threads just like your use case.
Assuming that to launch my library I simply need to create an instance of Foo and invoke the method execute. I could store the corresponding Method in a ThreadLocal to retrieve it by reflection only once per thread using ThreadLocal.withInitial(Supplier<? extends S> supplier) as next:
private static final ThreadLocal<Method> TL = ThreadLocal.withInitial(
() -> {
try {
// Create the instance of URLClassLoader using the context
// CL as parent CL to be able to retrieve the potential
// dependencies of your library assuming that they are
// thread safe otherwise you will need to provide their
// URL to isolate them too
URLClassLoader cl = URLClassLoader.newInstance(
new URL[]{/* Here the URL of my library*/},
Thread.currentThread().getContextClassLoader()
);
// Get by reflection the class Foo
Class<?> myClass = cl.loadClass("com.company.Foo");
// Get by reflection the method execute
return myClass.getMethod("execute");
} catch (Exception e) {
// Here deal with the exceptions
throw new IllegalStateException(e);
}
}
);
And finally let's simulate a concurrent execution of my library:
// Launch 50 times concurrently my library
IntStream.rangeClosed(1, 50).parallel().forEach(
i -> {
try {
// Get the method instance from the ThreadLocal
Method myMethod = TL.get();
// Create an instance of my class using the default constructor
Object myInstance = myMethod.getDeclaringClass().newInstance();
// Invoke the method
myMethod.invoke(myInstance);
} catch (Exception e) {
// Here deal with the exceptions
throw new IllegalStateException(e);
}
}
);
You will get an output of the next type that shows that we have no conflicts between threads and the threads properly reuse its corresponding class/field's value from one call of execute to another:
ForkJoinPool.commonPool-worker-7: The value before null
ForkJoinPool.commonPool-worker-7: The value after ForkJoinPool.commonPool-worker-7
ForkJoinPool.commonPool-worker-7: The value before ForkJoinPool.commonPool-worker-7
ForkJoinPool.commonPool-worker-7: The value after ForkJoinPool.commonPool-worker-7
main: The value before null
main: The value after main
main: The value before main
main: The value after main
...
Since this approach will create one ClassLoader per thread, make sure to apply this approach using a thread pool with a fixed number of threads and the number of threads should be chosen wisely to prevent running out of memory because a ClassLoader is not free in term of memory footprint so you need to limit the total amount of instances according to your heap size.
Once you are done with your library, you should cleanup the ThreadLocal for each thread of your thread pool to prevent memory leaks and to do so here is how you could proceed:
// The size of your the thread pool
// Here as I used for my example the common pool, its size by default is
// Runtime.getRuntime().availableProcessors()
int poolSize = Runtime.getRuntime().availableProcessors();
// The cyclic barrier used to make sure that all the threads of the pool
// will execute the code that will cleanup the ThreadLocal
CyclicBarrier barrier = new CyclicBarrier(poolSize);
// Launch one cleanup task per thread in the pool
IntStream.rangeClosed(1, poolSize).parallel().forEach(
i -> {
try {
// Wait for all other threads of the pool
// This is needed to fill up the thread pool in order to make sure
// that all threads will execute the cleanup code
barrier.await();
// Close the URLClassLoader to prevent memory leaks
((URLClassLoader) TL.get().getDeclaringClass().getClassLoader()).close();
} catch (Exception e) {
// Here deal with the exceptions
throw new IllegalStateException(e);
} finally {
// Remove the URLClassLoader instance for this thread
TL.remove();
}
}
);
I found the question interesing and created a little tool for you:
https://github.com/kriegaex/ThreadSafeClassLoader
Currently it is not available as an official release on Maven Central yet, but you can get a snapshot like this:
<dependency>
<groupId>de.scrum-master</groupId>
<artifactId>threadsafe-classloader</artifactId>
<version>1.0-SNAPSHOT</version>
</dependency>
<!-- (...) -->
<repositories>
<repository>
<snapshots>
<enabled>true</enabled>
</snapshots>
<id>ossrh</id>
<name>Sonatype OSS Snapshots</name>
<url>https://oss.sonatype.org/content/repositories/snapshots</url>
</repository>
</repositories>
Class ThreadSafeClassLoader:
It uses JCL (Jar Class Loader) under the hood because it already offers class-loading, object instantiation and proxy generation features discussed in other parts of this thread. (Why re-invent the wheel?) What I added on top is a nice interface for exactly what we need here:
package de.scrum_master.thread_safe;
import org.xeustechnologies.jcl.JarClassLoader;
import org.xeustechnologies.jcl.JclObjectFactory;
import org.xeustechnologies.jcl.JclUtils;
import org.xeustechnologies.jcl.proxy.CglibProxyProvider;
import org.xeustechnologies.jcl.proxy.ProxyProviderFactory;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
public class ThreadSafeClassLoader extends JarClassLoader {
private static final JclObjectFactory OBJECT_FACTORY = JclObjectFactory.getInstance();
static {
ProxyProviderFactory.setDefaultProxyProvider(new CglibProxyProvider());
}
private final List<Class> classes = new ArrayList<>();
public static ThreadLocal<ThreadSafeClassLoader> create(Class... classes) {
return ThreadLocal.withInitial(
() -> new ThreadSafeClassLoader(classes)
);
}
private ThreadSafeClassLoader(Class... classes) {
super();
this.classes.addAll(Arrays.asList(classes));
for (Class clazz : classes)
add(clazz.getProtectionDomain().getCodeSource().getLocation());
}
public <T> T newObject(ObjectConstructionRules rules) {
rules.validate(classes);
Class<T> castTo = rules.targetType;
return JclUtils.cast(createObject(rules), castTo, castTo.getClassLoader());
}
private Object createObject(ObjectConstructionRules rules) {
String className = rules.implementingType.getName();
String factoryMethod = rules.factoryMethod;
Object[] arguments = rules.arguments;
Class[] argumentTypes = rules.argumentTypes;
if (factoryMethod == null) {
if (argumentTypes == null)
return OBJECT_FACTORY.create(this, className, arguments);
else
return OBJECT_FACTORY.create(this, className, arguments, argumentTypes);
} else {
if (argumentTypes == null)
return OBJECT_FACTORY.create(this, className, factoryMethod, arguments);
else
return OBJECT_FACTORY.create(this, className, factoryMethod, arguments, argumentTypes);
}
}
public static class ObjectConstructionRules {
private Class targetType;
private Class implementingType;
private String factoryMethod;
private Object[] arguments;
private Class[] argumentTypes;
private ObjectConstructionRules(Class targetType) {
this.targetType = targetType;
}
public static ObjectConstructionRules forTargetType(Class targetType) {
return new ObjectConstructionRules(targetType);
}
public ObjectConstructionRules implementingType(Class implementingType) {
this.implementingType = implementingType;
return this;
}
public ObjectConstructionRules factoryMethod(String factoryMethod) {
this.factoryMethod = factoryMethod;
return this;
}
public ObjectConstructionRules arguments(Object... arguments) {
this.arguments = arguments;
return this;
}
public ObjectConstructionRules argumentTypes(Class... argumentTypes) {
this.argumentTypes = argumentTypes;
return this;
}
private void validate(List<Class> classes) {
if (implementingType == null)
implementingType = targetType;
if (!classes.contains(implementingType))
throw new IllegalArgumentException(
"Class " + implementingType.getName() + " is not protected by this thread-safe classloader"
);
}
}
}
I tested my concept with several unit and integration tests, among them one showing how to reproduce and solve the veraPDF problem.
Now this is what your code looks like when using my special classloader:
Class VeraPDFValidator:
We are just adding a static ThreadLocal<ThreadSafeClassLoader> member to our class, telling it which classes/libraries to put into the new classloader (mentioning one class per library is enough, subsequently my tool identifies the library automatically).
Then via threadSafeClassLoader.get().newObject(forTargetType(VeraPDFValidatorHelper.class)) we instantiate our helper class inside the thread-safe classloader and create a proxy object for it so we can call it from outside.
BTW, static boolean threadSafeMode only exists to switch between the old (unsafe) and new (thread-safe) usage of veraPDF so as to make the original problem reproducible for the negative integration test case.
package de.scrum_master.app;
import de.scrum_master.thread_safe.ThreadSafeClassLoader;
import org.verapdf.core.*;
import org.verapdf.pdfa.*;
import javax.xml.bind.JAXBException;
import java.io.InputStream;
import java.lang.reflect.InvocationTargetException;
import java.util.function.Function;
import static de.scrum_master.thread_safe.ThreadSafeClassLoader.ObjectConstructionRules.forTargetType;
public class VeraPDFValidator implements Function<InputStream, byte[]> {
public static boolean threadSafeMode = true;
private static ThreadLocal<ThreadSafeClassLoader> threadSafeClassLoader =
ThreadSafeClassLoader.create( // Add one class per artifact for thread-safe classloader:
VeraPDFValidatorHelper.class, // - our own helper class
PDFAParser.class, // - veraPDF core
VeraGreenfieldFoundryProvider.class // - veraPDF validation-model
);
private String flavorId;
private Boolean prettyXml;
public VeraPDFValidator(String flavorId, Boolean prettyXml)
throws ClassNotFoundException, NoSuchMethodException, InvocationTargetException, IllegalAccessException {
this.flavorId = flavorId;
this.prettyXml = prettyXml;
}
#Override
public byte[] apply(InputStream inputStream) {
try {
VeraPDFValidatorHelper validatorHelper = threadSafeMode
? threadSafeClassLoader.get().newObject(forTargetType(VeraPDFValidatorHelper.class))
: new VeraPDFValidatorHelper();
return validatorHelper.validatePDF(inputStream, flavorId, prettyXml);
} catch (ModelParsingException | ValidationException | JAXBException | EncryptedPdfException e) {
throw new RuntimeException("invoking veraPDF validation", e);
}
}
}
Class VeraPDFValidatorHelper:
In this class we isolate all access to the broken library. Nothing special here, just code copied from the OP's question. Everything done here happens inside the thread-safe classloader.
package de.scrum_master.app;
import org.verapdf.core.*;
import org.verapdf.pdfa.*;
import org.verapdf.pdfa.flavours.PDFAFlavour;
import org.verapdf.pdfa.results.ValidationResult;
import javax.xml.bind.JAXBException;
import java.io.ByteArrayOutputStream;
import java.io.InputStream;
public class VeraPDFValidatorHelper {
public byte[] validatePDF(InputStream inputStream, String flavorId, Boolean prettyXml)
throws ModelParsingException, ValidationException, JAXBException, EncryptedPdfException
{
VeraGreenfieldFoundryProvider.initialise();
PDFAFlavour flavour = PDFAFlavour.byFlavourId(flavorId);
PDFAValidator validator = Foundries.defaultInstance().createValidator(flavour, false);
PDFAParser loader = Foundries.defaultInstance().createParser(inputStream, flavour);
ValidationResult result = validator.validate(loader);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
XmlSerialiser.toXml(result, baos, prettyXml, false);
return baos.toByteArray();
}
}
By isolating a library on a class loader per thread, you can guarantee any classes concurrency properties as you suggest. The only exception are libraries that explicitly interact with the bootstrap class loader or the system class loader. It is possible to inject classes into these class loaders by either reflection or the Instrumentation API. One example for such functionality would be Mockito's inline mock maker that does however not suffer a concurrency constraint as of my knowledge.
Implementing a class loader with this behavior is not all too difficult. The easiest solution would be to explicitly include the required jars in your project, e.g. as a resource. This way, you could use a URLClassLoader for loading your classes:
URL url = getClass().getClassLoader().getResource("validation-model-1.1.6.jar");
ClassLoader classLoader = new URLClassLoader(new URL[] {url}, null);
By referencing null as the super class loader of the URLClassLoader (second argument), you guarantee that there are no shared classes outside of the bootstrap classes. Note that you cannot use any classes of this created class loader from outside of it. However, if you add a second jar containing a class that triggers your logic, you can offer an entry point that becomes accessible without reflection:
class MyEntryPoint implements Callable<File> {
#Override public File call() {
// use library code.
}
}
Simply add this class to its own jar and supply it as a second element to the above URL array. Note that you cannot reference a library type as a return value as this type will not be available to the consumer that lives outside the class loader that makes use of the entry point.
By wrapping the class loader creation into a ThreadLocal, you can guarantee the class loaders uniqunes:
class Unique extends ThreadLocal<ClassLoader> implements Closable {
#Override protected ClassLoader initialValue() {
URL validation = Unique.class.getClassLoader()
.getResource("validation-model-1.1.6.jar");
URL entry = Unique.class.getClassLoader()
.getResource("my-entry.jar");
return new URLClassLoader(new URL[] {validation, entry}, null);
}
#Override public void close() throws IOException {
get().close(); // If Java 7+, avoid handle leaks.
set(null); // Make class loader eligable for GC.
}
public File doSomethingLibrary() throws Exception {
Class<?> type = Class.forName("pkg.MyEntryPoint", false, get());
return ((Callable<File>) type.newInstance()).call();
}
}
Note that class loaders are expensive objects and should be dereferenced when you do no longer need them even if a thread continues to live. Also, to avoid file leaks, you should close a URLClassLoader previously to dereferencing.
Finally, in order to continue using Maven's dependency resolution and in order to simplify your code, you can create a seperate Maven module where you define your entry point code and declare your Maven library dependencies. Upon packaging, use the Maven shade plugin to create an Uber jar that includes everything you need. This way, you only need to provide a single jar to your URLClassLoader and do not need to ensure all (transitive) dependencies manually.
This answer is based on my original "plugin" comment. And it starts with a class loader that inherits from boot and extensions class loaders only.
package safeLoaderPackage;
import java.net.URL;
import java.net.URLClassLoader;
public final class SafeClassLoader extends URLClassLoader{
public SafeClassLoader(URL[] paths){
super(paths, ClassLoader.getSystemClassLoader().getParent());
}
}
This is the only class that needs to be included in the user's class path. This url class loader inherits from the parent of ClassLoader.getSystemClassLoader(). It just includes the boot and the extensions class loader. It has no notion of the class path used by the user.
Next
package safeLoaderClasses;
import java.net.URL;
import java.util.ArrayList;
import java.util.Collection;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
public class SecureClassLoaderPlugin <R> {
private URL[] paths;
private Class[] args;
private String method;
private String unsafe;
public void setMethodData(final String u, final URL[] p, String m, Class[] a){
method = m;
args = a;
paths = p;
unsafe = u;
}
public Collection<R> processUnsafe(Object[][] p){
int i;
BlockingQueue<Runnable> q;
ArrayList<R> results = new ArrayList<R>();
try{
i = p.length;
q = new ArrayBlockingQueue<Runnable>(i);
ThreadPoolExecutor tpe = new ThreadPoolExecutor(i, i, 0, TimeUnit.NANOSECONDS, q);
for(Object[] params : p)
tpe.execute(new SafeRunnable<R>(unsafe, paths, method, args, params, results));
while(tpe.getActiveCount() != 0){
Thread.sleep(10);
}
for(R r: results){
System.out.println(r);
}
tpe.shutdown();
}
catch(Throwable t){
}
finally{
}
return results;
}
}
and
package safeLoaderClasses;
import java.io.IOException;
import java.lang.reflect.Method;
import java.net.URL;
import java.util.ArrayList;
import safeLoaderInterface.SafeClassLoader;
class SafeRunnable <R> implements Runnable{
final URL[] paths;
final private String unsafe;
final private String method;
final private Class[] args;
final private Object[] processUs;
final ArrayList<R> result;
SafeRunnable(String u, URL[] p, String m, Class[] a, Object[] params, ArrayList<R> r){
unsafe = u;
paths = p;
method = m;
args = a;
processUs = params;
result = r;
}
public void run() {
Class clazz;
Object instance;
Method m;
SafeClassLoader sl = null;
try{
sl = new SafeClassLoader(paths);
System.out.println(sl);
clazz = sl.loadClass(unsafe);
m = clazz.getMethod(method, args);
instance = clazz.newInstance();
synchronized(result){
result.add((R) m.invoke(instance, processUs));
}
}
catch(Throwable t){
t.printStackTrace();
}
finally{
try {
sl.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
are the plugin jar. No lambdas. Just a thread pool executor. Each thread just adds to a result list after execution.
The generics need polishing but I have tested these against this class (resides in a different jar)
package stackoverflow4;
public final class CrazyClass {
static int i = 0;
public int returnInt(){
System.out.println(i);
return 8/++i;
}
}
This would be the way to connect from one's code. The path to the class loader needs to be included because it is lost with the getParent() call
private void process(final String plugin, final String unsafe, final URL[] paths) throws Exception{
Object[][] passUs = new Object[][] {{},{}, {},{}, {},{},{},{},{},{}};
URL[] pathLoader = new URL[]{new File(new String(".../safeLoader.jar")).toURI().toURL(),
new File(new String(".../safeLoaderClasses.jar")).toURI().toURL()};
//instantiate the loader
SafeClassLoader sl = new SafeClassLoader(pathLoader);
System.out.println(sl);
Class clazz = sl.loadClass("safeLoaderClasses.SecureClassLoaderPlugin");
//Instance of the class that loads the unsafe jar and launches the thread pool executor
Object o = clazz.newInstance();
//Look up the method that set ups the unsafe library
Method m = clazz.getMethod("setMethodData",
new Class[]{unsafe.getClass(), paths.getClass(), String.class, new Class[]{}.getClass()});
//invoke it
m.invoke(o, new Object[]{unsafe,paths,"returnInt", new Class[]{}});
//Look up the method that invokes the library
m = clazz.getMethod("processUnsafe", new Class[]{ passUs.getClass()});
//invoke it
o = m.invoke(o, passUs);
//Close the loader
sl.close();
}
with up to 30+ threads and it seems to work. The plugin uses a separate class loader and each of the threads use their own class loader. After leaving the method everything is gc'ed.
I believe you should try to fix the problem before seeking for workaround.
You can always run you code in two threads, classloaders, processes, containers, VM, or machines. But they are none of the ideal.
I saw two defaultInstance() from the code. Do the instance threadsafe? If not, can we have two instance? Is it a factory or a singleton?
Second, where the conflicts happen? If it was about initialization/cache problem, a pre warming should fix.
Last but not least, if the library was open-source, fork it fix it and pull request.
"It is unfeasible to harden the library" but it is feasible to introduce such a bloody workaround like a custom class loader?
OK. I am the first who dislikes the replies which are not a reply to the original question. But I honestly believe that patching the library is much easier to do and to mantain than introducing a custom class loader.
The blocker is class org.verapdf.gf.model.impl.containers.StaticContainers which static fields can be easily changed to work per thread as shown below. This impacts six other classes
org.verapdf.gf.model.GFModelParser
org.verapdf.gf.model.factory.colors.ColorSpaceFactory
org.verapdf.gf.model.impl.cos.GFCosFileSpecification
org.verapdf.gf.model.impl.external.GFEmbeddedFile
org.verapdf.gf.model.impl.pd.colors.GFPDSeparation
org.verapdf.gf.model.tools.FileSpecificationKeysHelper
You can still have only one PDFAParser per thread. But the fork takes ten minutes to do and worked for me in a basic multithread smoke test. I'd test this and contact the original author of the library. Maybe he is happy to merge and you can just keep a Maven reference to the updated and mantained library.
package org.verapdf.gf.model.impl.containers;
import org.verapdf.as.ASAtom;
import org.verapdf.cos.COSKey;
import org.verapdf.gf.model.impl.pd.colors.GFPDSeparation;
import org.verapdf.gf.model.impl.pd.util.TaggedPDFRoleMapHelper;
import org.verapdf.model.pdlayer.PDColorSpace;
import org.verapdf.pd.PDDocument;
import org.verapdf.pdfa.flavours.PDFAFlavour;
import java.util.*;
public class StaticContainers {
private static ThreadLocal<PDDocument> document;
private static ThreadLocal<PDFAFlavour> flavour;
// TaggedPDF
public static ThreadLocal<TaggedPDFRoleMapHelper> roleMapHelper;
//PBoxPDSeparation
public static ThreadLocal<Map<String, List<GFPDSeparation>>> separations;
public static ThreadLocal<List<String>> inconsistentSeparations;
//ColorSpaceFactory
public static ThreadLocal<Map<String, PDColorSpace>> cachedColorSpaces;
public static ThreadLocal<Set<COSKey>> fileSpecificationKeys;
public static void clearAllContainers() {
document = new ThreadLocal<PDDocument>();
flavour = new ThreadLocal<PDFAFlavour>();
roleMapHelper = new ThreadLocal<TaggedPDFRoleMapHelper>();
separations = new ThreadLocal<Map<String, List<GFPDSeparation>>>();
separations.set(new HashMap<String,List<GFPDSeparation>>());
inconsistentSeparations = new ThreadLocal<List<String>>();
inconsistentSeparations.set(new ArrayList<String>());
cachedColorSpaces = new ThreadLocal<Map<String, PDColorSpace>>();
cachedColorSpaces.set(new HashMap<String,PDColorSpace>());
fileSpecificationKeys = new ThreadLocal<Set<COSKey>>();
fileSpecificationKeys.set(new HashSet<COSKey>());
}
public static PDDocument getDocument() {
return document.get();
}
public static void setDocument(PDDocument document) {
StaticContainers.document.set(document);
}
public static PDFAFlavour getFlavour() {
return flavour.get();
}
public static void setFlavour(PDFAFlavour flavour) {
StaticContainers.flavour.set(flavour);
if (roleMapHelper.get() != null) {
roleMapHelper.get().setFlavour(flavour);
}
}
public static TaggedPDFRoleMapHelper getRoleMapHelper() {
return roleMapHelper.get();
}
public static void setRoleMapHelper(Map<ASAtom, ASAtom> roleMap) {
StaticContainers.roleMapHelper.set(new TaggedPDFRoleMapHelper(roleMap, StaticContainers.flavour.get()));
}
}
Re-edited...
I'd like to use a superclass constructor which is hidden by the "#hide" Android tag (or whatever it is).
I'm about to extend a class which has been already extended twice (within the Android OS). I'd like to create my own subclass (i.e. outside the Android OS). Example subclass, taken from Android sources:
public class WifiP2pDnsSdServiceInfo extends WifiP2pServiceInfo {
...
private WifiP2pDnsSdServiceInfo(List<String> queryList) {
super(queryList); // <-- this is what I'm trying to do, too
}
public static WifiP2pDnsSdServiceInfo newInstance(String instanceName,
String serviceType, Map<String, String> txtMap) {
...
ArrayList<String> queries = new ArrayList<String>();
...
return new WifiP2pDnsSdServiceInfo(queries);
}
}
The superclass looks like this:
public class WifiP2pServiceInfo implements Parcelable {
...
// this is marked as #hidden therefore inaccessible!
protected WifiP2pServiceInfo(List<String> queryList) {
if (queryList == null) {
throw new IllegalArgumentException("query list cannot be null");
}
mQueryList = queryList;
}
}
And all I want to do is to make another kind of WifiP2pServiceInfo, similar to the WifiP2pDnsSdServiceInfo example above. I can't just inherit & call super() because the superclass constructor is tagged by Android's "#hide", therefore unusable without reflection for non-system programmers.
So my question is how to access / call the superclass constructor if I can't do it by a plain super() call? Reflection should come handy here but I'm not very experienced in Java programming.
After some research I'm able to answer the question by myself.
Short answer: I can't do this because if the superclass constructor is protected and hidden, the compiler is going to complain even if I found a way how to call the constructor via reflection.
Long answer: it turns out it's not so complicated to "unhide" this stuff. Following this tutorial I'm able to extend the class to my needs.
See? A lot of noise for nothing, this is the answer I was looking for.
You want to find that constructor and set its availability to true.
But this is a dangerous operation that you should not attempt lightly. It's a dirty secret that private need not mean private, but I would still expect you to honor the wishes of the class designer and not circumvent.
Besides, you don't need to. If I understand your requirement, I've posted an example that will do what you want.
This works, because of this:
The protected modifier specifies that the member can only be accessed
within its own package (as with package-private) and, in addition, by
a subclass of its class in another package.
I think you've confused the meaning of the protected modifier.
Parent w/ protected ctor:
package foo;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
/**
* Parent with protected constructor
* User: MDUFFY
* Date: 3/27/14
* Time: 5:18 PM
* #link http://stackoverflow.com/questions/22698501/reflection-how-to-call-superclass-constructor-which-is-hidden/22698543?noredirect=1#comment34586243_22698543
*/
public class Foo {
private List<String> x;
protected Foo(List<String> y) {
this.x = ((y == null) ? new ArrayList<String>() : new ArrayList<String>(y));
}
public List<String> getX() {
return Collections.unmodifiableList(this.x);
}
#Override
public String toString() {
return "Foo{" +
"x=" + x +
'}';
}
}
Child extends Parent:
package bar;
import foo.Foo;
import java.util.Arrays;
import java.util.List;
/**
* Child of parent with protected ctor
* User: MDUFFY
* Date: 3/27/14
* Time: 5:22 PM
* #link http://stackoverflow.com/questions/22698501/reflection-how-to-call-superclass-constructor-which-is-hidden/22698543?noredirect=1#comment34586243_22698543
*/
public class Bar extends Foo {
public static void main(String[] args) {
Bar bar = new Bar(Arrays.asList(args));
System.out.println(bar);
}
public Bar(List<String> y) {
super(y);
}
}
This will compile and run.
I have two DLL's: a directx dll, which exports GetNativeBuffer, and an opengl dll, which does the same.
I use the following Java class to call GetNativeBuffer, to read an image from the loaded dll.
class DllLoader {
private ByteBuffer Buffer = null;
private BufferedImage Image = null;
public boolean LibraryLoaded = false;
private static native void GetNativeBuffer(IntBuffer Buffer);
private int ByteSize = 0, Width = 0, Height = 0;
public DllLoader(ByteBuffer Buffer, int ImageWidth, int ImageHeight) {
this.Buffer = Buffer;
}
}
Problem: If both DLL's are loaded by the program, how do I specify which one to read from?
Do I have to make two separate classes? Do I have to rename the functions and have two native functions?
You should make two classes, one for each DLL. If you are naming your java classes identically, it might be easier to separate them into different subpackages, for example:
package com.stackoverflow.jni.opengl;
/*
* GENERATE HEADER FILE FROM GENERATED CLASS AS NEEDED VIA
* javah com.stackoverflow.jni.opengl.NativeClazz
*/
public class NativeClazz {
/**
* Load C++ Library
*/
static {
// Always fun to do this in a static block!
System.loadLibrary("OpenGL");
}
private static native void GetNativeBuffer(IntBuffer Buffer);
}
and
package com.stackoverflow.jni.directx;
/*
* GENERATE HEADER FILE FROM GENERATED CLASS AS NEEDED VIA
* javah com.stackoverflow.jni.directx.NativeClazz
*/
public class NativeClazz {
/**
* Load C++ Library
*/
static {
// Always fun to do this in a static block!
System.loadLibrary("DirectX");
}
private static native void GetNativeBuffer(IntBuffer Buffer);
}
My personal preference is to keep any class containing JNI methods "utility-only" (private constructor), leaving them lean and mean (no internal variables unless necessary) and transfer data back and forth within beans via function call parameters.
I come across to a strange behavior while trying to override a method with default accessor (ex: void run()).
According to Java spec, a class can use or override default members of base class if classes belongs to the same package.
Everything works correctly while all classes loaded from the same classloader.
But if I try to load a subclass from separate classloader then polymorphism don't work.
Here is sample:
App.java:
import java.net.*;
import java.lang.reflect.Method;
public class App {
public static class Base {
void run() {
System.out.println("error");
}
}
public static class Inside extends Base {
#Override
void run() {
System.out.println("ok. inside");
}
}
public static void main(String[] args) throws Exception {
{
Base p = (Base) Class.forName(Inside.class.getName()).newInstance();
System.out.println(p.getClass());
p.run();
} {
// path to Outside.class
URL[] url = { new URL("file:/home/mart/workspace6/test2/bin/") };
URLClassLoader ucl = URLClassLoader.newInstance(url);
final Base p = (Base) ucl.loadClass("Outside").newInstance();
System.out.println(p.getClass());
p.run();
// try reflection
Method m = p.getClass().getDeclaredMethod("run");
m.setAccessible(true);
m.invoke(p);
}
}
}
Outside.java: should be in separate folder. otherwise classloader will be the same
public class Outside extends App.Base {
#Override
void run() {
System.out.println("ok. outside");
}
}
The output:
class App$Inside
ok. inside
class Outside
error
ok. outside
So then I call Outside#run() I got Base#run() ("error" in output). Reflections works correctly.
Whats wrong? Or is it expected behavior?
Can I go around this problem somehow?
From Java Virtual Machine Specification:
5.3 Creation and Loading
...
At run time, a class or interface is
determined not by its name alone, but
by a pair: its fully qualified name
and its defining class loader. Each
such class or interface belongs to a
single runtime package. The runtime
package of a class or interface is
determined by the package name and
defining class loader of the class or
interface.
5.4.4 Access Control
...
A field or method R is accessible to a class
or interface D if and only if any of
the following conditions is true:
...
R is either protected or package private (that is, neither public nor
protected nor private), and is
declared by a class in the same
runtime package as D.
The Java Language Specification mandates that a class can only override methods that it can access. If the super class method is not accessible, it is shadowed rather than overridden.
Reflection "works" because you ask Outside.class for its run method. If you ask Base.class instead, you'll get the super implementation:
Method m = Base.class.getDeclaredMethod("run");
m.setAccessible(true);
m.invoke(p);
You can verify that the method is deemed inaccessible by doing:
public class Outside extends Base {
#Override
public void run() {
System.out.println("Outside.");
super.run(); // throws an IllegalAccessError
}
}
So, why is the method not accessible? I am not totally sure, but I suspect that just like equally named classes loaded by different class loaders result in different runtime classes, equally named packages loaded by different class loaders result in different runtime packages.
Edit: Actually, the reflection API says that it's the same package:
Base.class.getPackage() == p.getClass().getPackage() // true
I found the (hack) way to load external class in main classloader so this problem is gone.
Read a class as bytes and invoke protected ClassLoader#defineClass method.
code:
URL[] url = { new URL("file:/home/mart/workspace6/test2/bin/") };
URLClassLoader ucl = URLClassLoader.newInstance(url);
InputStream is = ucl.getResourceAsStream("Outside.class");
byte[] bytes = new byte[is.available()];
is.read(bytes);
Method m = ClassLoader.class.getDeclaredMethod("defineClass", new Class[] { String.class, byte[].class, int.class, int.class });
m.setAccessible(true);
Class<Base> outsideClass = (Class<Base>) m.invoke(Base.class.getClassLoader(), "Outside", bytes, 0, bytes.length);
Base p = outsideClass.newInstance();
System.out.println(p.getClass());
p.run();
outputs ok. outside as expected.
This question arise while trying to write test cases. Foo is a class within the framework library which I dont have source access to.
public class Foo{
public final Object getX(){
...
}
}
my applications will
public class Bar extends Foo{
public int process(){
Object value = getX();
...
}
}
The unit test case is unable to initalize as I can't create a Foo object due to other dependencies. The BarTest throws a null pointer as value is null.
public class BarTest extends TestCase{
public testProcess(){
Bar bar = new Bar();
int result = bar.process();
...
}
}
Is there a way i can use reflection api to set the getX() to non-final? or how should I go about testing?
As this was one of the top results for "override final method java" in google. I thought I would leave my solution. This class shows a simple solution using the example "Bagel" class and a free to use javassist library:
/**
* This class shows how you can override a final method of a super class using the Javassist's bytecode toolkit
* The library can be found here: http://jboss-javassist.github.io/javassist/
*
* The basic idea is that you get the super class and reset the modifiers so the modifiers of the method don't include final.
* Then you add in a new method to the sub class which overrides the now non final method of the super class.
*
* The only "catch" is you have to do the class manipulation before any calls to the class happen in your code. So put the
* manipulation as early in your code as you can otherwise you will get exceptions.
*/
package packagename;
import javassist.ClassPool;
import javassist.CtClass;
import javassist.CtMethod;
import javassist.CtNewMethod;
import javassist.Modifier;
/**
* A simple class to show how to use the library
*/
public class TestCt {
/**
* The starting point for the application
*/
public static void main(String[] args) {
// in order for us to override the final method we must manipulate the class using the Javassist library.
// we need to do this FIRST because once we initialize the class it will no longer be editable.
try
{
// get the super class
CtClass bagel = ClassPool.getDefault().get("packagename.TestCt$Bagel");
// get the method you want to override
CtMethod originalMethod = bagel.getDeclaredMethod("getDescription");
// set the modifier. This will remove the 'final' modifier from the method.
// If for whatever reason you needed more than one modifier just add them together
originalMethod.setModifiers(Modifier.PUBLIC);
// save the changes to the super class
bagel.toClass();
// get the subclass
CtClass bagelsolver = ClassPool.getDefault().get("packagename.TestCt$BagelWithOptions");
// create the method that will override the super class's method and include the options in the output
CtMethod overrideMethod = CtNewMethod.make("public String getDescription() { return super.getDescription() + \" with \" + getOptions(); }", bagelsolver);
// add the new method to the sub class
bagelsolver.addMethod(overrideMethod);
// save the changes to the sub class
bagelsolver.toClass();
}
catch (Exception e)
{
e.printStackTrace();
}
// now that we have edited the classes with the new methods, we can create an instance and see if it worked
// create a new instance of BagelWithOptions
BagelWithOptions myBagel = new BagelWithOptions();
// give it some options
myBagel.setOptions("cheese, bacon and eggs");
// print the description of the bagel to the console.
// This should now use our new code when calling getDescription() which will include the options in the output.
System.out.println("My bagel is: " + myBagel.getDescription());
// The output should be:
// **My bagel is: a plain bagel with cheese, bacon and eggs**
}
/**
* A plain bagel class which has a final method which we want to override
*/
public static class Bagel {
/**
* return a description for this bagel
*/
public final String getDescription() {
return "a plain bagel";
}
}
/**
* A sub class of bagel which adds some extra options for the bagel.
*/
public static class BagelWithOptions extends Bagel {
/**
* A string that will contain any extra options for the bagel
*/
String options;
/**
* Initiate the bagel with no extra options
*/
public BagelWithOptions() {
options = "nothing else";
}
/**
* Set the options for the bagel
* #param options - a string with the new options for this bagel
*/
public void setOptions(String options) {
this.options = options;
}
/**
* return the current options for this bagel
*/
public String getOptions() {
return options;
}
}
}
you could create another method which you could override in your test:
public class Bar extends Foo {
protected Object doGetX() {
return getX();
}
public int process(){
Object value = doGetX();
...
}
}
then, you could override doGetX in BarTest.
Seb is correct, and just to ensure that you get an answer to your question, short of doing something in native code (and I am pretty sure that would not work) or modifying the bytecode of the class at runtime, and creating the class that overrides the method at runtime, I cannot see a way to alter the "finalness" of a method. Reflection will not help you here.
If your unit test case can't create Foo due to other dependencies, that might be a sign that you're not making your unit test right in the first place.
Unit tests are meant to test under the same circumstances a production code would run, so I'd suggest recreating the same production environment inside your tests. Otherwise, your tests wouldn't be complete.
If the variable returned by getX() is not final you can use the technique explained in What’s the best way of unit testing private methods? for changing the value of the private variable through Reflection.
public class Bar extends Foo{
public int process(){
Object value = getX();
return process2(value);
}
public int process2(Object value){
...
}
}
public class BarTest extends TestCase{
public testProcess(){
Bar bar = new Bar();
Mockobj mo = new Mockobj();
int result = bar.process2(mo);
...
}
}
what i did eventually was the above. it is a bit ugly... James solution is definitely much better than this...