Develop Reference

How can I list all methods of all imported classes in a file using Java?

My objective is to look at some lines of codes of an external file and count the number of functions of a class are called then.
For example, if I have the following code:
import java.io.BufferedReader;
import whatever.MyClass;
import java.util.ArrayList;
...
...
public void example(){
InputStreamReader isr = new InputStreamReader (whatever);
MyClass object = new MyClass();
someArrayList.add(whatever2)
someArrayList.add(whatever3)
}
In this case, BufferedReader and MyClass functions were called once, and ArrayList functions were called twice.
My solution for that is get a list of all methods inside the used classes and try to match with some string of my code.
For classes created in my project, I can do the following:
jar -tf jarPath
which returns me the list of classes inside a JAR . And doing:
javap -cp jarPath className
I can get a list of all methods inside a JAR whit a specific class name. However, what can I do to get a external methods names, like add(...) of an "external" class java.util.ArrayList?
I can't access the .jar file of java.util.ArrayList correct? Anyone have another suggestion to reach the objective?

The compiler doesn't put the imports into the object file. It throws them away. Import is just a shorthand to the compiler.(Imports are a compile-time feature ).
first step :
use Qdox https://github.com/paul-hammant/qdox to get all the imports in a class :
String fileFullPath = "Your\\java\\ file \\full\\path";
JavaDocBuilder builder = new JavaDocBuilder();
builder.addSource(new FileReader( fileFullPath ));
JavaSource src = builder.getSources()[0];
String[] imports = src.getImports();
for ( String imp : imports )
{
System.out.println(imp);
}
second step :
inspire from that code , loop through your imports (String array) and apply the same code and you will get the methods .
import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
public class Tes {
public static void main(String[] args) {
Class c;
try {
c = Class.forName("java.util.ArrayList");
Arrays.stream(getAccessibleMethods(c)).
forEach(System.out::println);
} catch (ClassNotFoundException e) {
e.printStackTrace();
}
}
public static Method[] getAccessibleMethods(Class clazz) {
List<Method> result = new ArrayList<Method>();
while (clazz != null) {
for (Method method : clazz.getDeclaredMethods()) {
result.add(method);
}
clazz = clazz.getSuperclass();
}
return result.toArray(new Method[result.size()]);
}
}
Output :
public void java.util.ArrayList.add(int,java.lang.Object)
public boolean java.util.ArrayList.add(java.lang.Object)
public boolean java.util.ArrayList.remove(java.lang.Object)
public java.lang.Object java.util.ArrayList.remove(int)
public java.lang.Object java.util.ArrayList.get(int)
public java.lang.Object java.util.ArrayList.clone()
public int java.util.ArrayList.indexOf(java.lang.Object)
public void java.util.ArrayList.clear()
.
.
.
All the code - one class :
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import com.thoughtworks.qdox.JavaDocBuilder;
import com.thoughtworks.qdox.model.JavaSource;
public class Tester {
public static void main(String[] args) {
// put your .java file path
// CyclicB is a class within another project in my pc
String fileFullPath =
"C:\\Users\\OUSSEMA\\Desktop\\dev\\OCP_Preparation\\src\\w\\CyclicB.java";
JavaDocBuilder builder = new JavaDocBuilder();
try {
builder.addSource(new FileReader( fileFullPath ));
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
JavaSource src = builder.getSources()[0];
String[] imports = src.getImports();
for ( String imp : imports )
{
Class c;
try {
c = Class.forName(imp);
Arrays.stream(getAccessibleMethods(c)).
forEach(System.out::println);
} catch (ClassNotFoundException e) {
e.printStackTrace();
}
}
}
public static Method[] getAccessibleMethods(Class clazz) {
List<Method> result = new ArrayList<Method>();
while (clazz != null) {
for (Method method : clazz.getDeclaredMethods()) {
result.add(method);
}
clazz = clazz.getSuperclass();
}
return result.toArray(new Method[result.size()]);
}
}
Output all the methods within the classes imported in the file CyclicB.java :
private void java.lang.Throwable.printStackTrace(java.lang.Throwable$PrintStreamOrWriter)
public void java.lang.Throwable.printStackTrace(java.io.PrintStream)
public void java.lang.Throwable.printStackTrace()
public void java.lang.Throwable.printStackTrace(java.io.PrintWriter)
public synchronized java.lang.Throwable java.lang.Throwable.fillInStackTrace()
.
.
.

You may look into OpenJDK project that has a Java compiler. Learn to build the modified versions. Investigate the syntax analysis layer of this compiler and find where the method calls are handled. Put the logging into these locations and now you only need to build your java file with the modified compiler to get the information about the calls.
The build is complex, but you will likely only need a careful editing in a few files. It is not exactly very low hanging fruit but I think it should be possible to discover these files and make changes in them, and still may be a simpler/cleaner approach than to implement the own Java syntax parser (also doable with JavaCC).
If you also need to track calls from the external libraries, build them with the modified compiler as well and you will have the needed records.
GNU Classpath is another open source project where you can do the similar thing, and it may be easier to build. However, unlike OpenJDK, GNU Classpath java system library is not complete.
This approach may not discover some methods called during reflection. But it would discover that reflection framework methods have been called. If it is a security - related project, the simplest would be to agree that reflection is not allowed. It is uncommon to use reflection in a normal Java application that is not a framework.

How to free all resources after reading a JRT?

I am trying to read the list of modules available in a given Java 9+ installation, given its Java Home, using the method described in How to extract the file jre-9/lib/modules?.
The solution works, but it appears that the resources allocated to read the content of the Java Runtime Image are never freed, causing a memory leak, observable with VisualVM for instance:
How can I fix the memory leak in the following reproduction?
package leak;
import java.net.URI;
import java.net.URL;
import java.net.URLClassLoader;
import java.nio.file.FileSystem;
import java.nio.file.FileSystems;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Collections;
import java.util.Map;
import java.util.stream.Stream;
public class JrtfsLeak {
public static void main(String[] args) throws Exception {
Path javaHome = Paths.get(args[0]);
for (int i = 0; i < 100000; ++i) {
modules(javaHome).close();
}
}
private static Stream<Path> modules(Path javaHome) throws Exception {
Map<String, String> env = Collections.singletonMap("java.home", javaHome.toString());
Path jrtfsJar = javaHome.resolve("lib").resolve("jrt-fs.jar");
try (URLClassLoader classloader = new URLClassLoader(new URL[] { jrtfsJar.toUri().toURL() })) {
try (FileSystem fs = FileSystems.newFileSystem(URI.create("jrt:/"), env, classloader)) {
Path modulesRoot = fs.getPath("modules");
return Files.list(modulesRoot);
}
}
}
}

This is a JDK bug JDK-8260621 that has been fixed in JDK 17.
It was caused by a careless use of thread locals in ImageBufferCache.

Note that when you are running under Java 9 or newer, the underlying implementation will create a new class loader for the jrt-fs.jar when you specify the java.home option. So, these classes are not loaded by your URLClassLoader but a different class loader. When you don’t need support for versions prior to 9, you can omit the creation of a class loader.
In either case, they’re loaded by a custom class loader and could be unloaded when the garbage collector supports it. But the class jdk.internal.jimage.ImageBufferCache contains:
private static final ThreadLocal<BufferReference[]> CACHE =
new ThreadLocal<BufferReference[]>() {
#Override
protected BufferReference[] initialValue() {
// 1 extra slot to simplify logic of releaseBuffer()
return new BufferReference[MAX_CACHED_BUFFERS + 1];
}
};
As explained in How does this ThreadLocal prevent the Classloader from getting GCed, a backreference from the value to the thread local can prevent its garbage collection and when the thread local is stored in a static variable, a reference to one of the classes loaded by the same class loader is enough.
And here, the value is an array of BufferReference, which means even when all entries of that array have been cleared, the array type itself has an implicit reference to the class loader of that filesystem.
But since its a thread local variable, we can work-around it by letting the key thread die. When I change your code to
public static void main(String[] args) throws InterruptedException {
Path javaHome = Paths.get(args[0]);
Runnable r = () -> test(javaHome);
for(int i = 0; i < 1000; ++i) {
Thread thread = new Thread(r);
thread.start();
thread.join();
}
}
static void test(Path javaHome) {
for (int i = 0; i < 1000; ++i) {
try(var s = modules(javaHome)) {}
catch(IOException ex) {
throw new UncheckedIOException(ex);
}
catch(Exception ex) {
throw new IllegalStateException(ex);
}
}
}
the classes get unloaded.

Refer to javadoc for method list (in class java.nio.file.Files). Here is the relevant part.
API Note:
This method must be used within a try-with-resources statement or similar control structure to ensure that the stream's open directory is closed promptly after the stream's operations have completed.
In other words you need to close the Stream returned by your modules method.

How to Implement a Non Springboot Utility into a Springboot Application?

I inherited the maintenance work on a Springboot program that currently dumps a number of debugging logs every time the application experiences an error (which could include incorrect queries within the UI itself). These log files are never deleted by the software, so they continually build up until they eventually fill up the server computer.
I've built a utility to manage these logs by deleting them once the log files reach 7 days of age. Admittedly, I'm still familiarizing myself with Springboot, so I built this utility externally. I looked at the program structure of the software, and it only uses a single main class as I expected, but the class only contains a call to run the Springboot API in addition to a couple of beans. My question is, in general, where in the Springboot structure should I look to implement my code?
Here's what I'm seeking to add for context:
package com.climatedev.test;
//Import packages
import java.io.File;
import java.io.FilenameFilter;
import java.util.Date;
public class CleanLogs {
//Create static array that will hold all files in the path
public static File[] files;
//Give the program the path of the log files to delete
private void getPath() {
File file = new File("C:\\ProgramData\\MySQL\\MySQL Server 8.0\\Data");
String id = "CW-LANDCAST-bin";
files = file.listFiles(new FilenameFilter() {
#Override
public boolean accept(File file, String name) {
return name.startsWith(id);
}
});
}
//Delete all files older than 7 days
private void deleteLogs() {
getPath();
for(File f : files) {
long diff = new Date().getTime() - f.lastModified();
if (diff > 7 * 24 * 60 * 60 * 1000) {
f.delete();
}
}
}
//Call the clean files program (testing purposes only)
public static void main(String[] args) {
CleanLogs obj = new CleanLogs();
obj.deleteLogs();
}
}

Java Security Manager: Restrictions on code from external jar loaded via ServiceLoader

What am I trying to achieve?
I am working on a java application that can be extended by additional jars that get integrated via ServiceLoader. These loaded extensions should run with some restrictions by the SecurityManager, of course simply to improve the security. As an example each Extension shall get one specific directory where it can store whatever, but access to any other file/folder should be restricted. The main application is trusted code and can therefore run without any restrictions. Furthermore the main application provides some api implementations for each extension that shall also run without restrictions. That means an extension mustn't access a file outside of its directory but when the extension is calling an api method that tries to access any other file, the access should be granted.
Question
How can I achieve the mentioned behaviour that only 'direct' calls from extension classes get restricted but not any code from the main application?
Running extensions in different threads/threadGroups might be a good solution anyway but since calls to the api might run under the same thread(group) it might not help to identify whether access should be restricted or not based only on the thread.
Example
I created a simplified test environment. On one hand there are these two interfaces:
public interface Extension {
void doSomethingRestricted();
void doSameViaApi(ExtensionApi api);
}
public interface ExtensionApi {
void doSomethingWithHigherPermissions();
}
For testing I created a jar containing this extension:
public class SomeExtension implements Extension {
public void doSomethingRestricted() {
System.out.println(System.getProperty("user.home"));
}
public void doSameViaApi(final ExtensionApi api) {
api.doSomethingWithHigherPermissions();
}
}
In the main application I would like do something like this:
final ExtensionApi api = () -> System.out.println(System.getProperty("user.home"));
try {
final URLClassLoader urlClassLoader = new URLClassLoader(new URL[] { jarFile.toURI().toURL() });
for(final Extension extension : ServiceLoader.load(Extension.class, urlClassLoader)) {
extension.doSomethingRestricted();
extension.doSameViaApi(api);
}
}
So when I call extension.doSomethingRestricted(); it should result in a SecurityException but calling extension.doSameViaApi(api); should work just fine.
So both methods try to do the same but one does try to do it via the api call. The only approach I could think of is iterating through the call history and checking the classloaders to analyze whether the access request is based on trusted code or extension code. But I feel like this might be a nasty error-prone solution so maybe I missed some better approaches?

First ensure your "main" JAR's classes get to enjoy full privileges. Programmatically this may be accomplished as follows:
package q46991566;
import java.nio.file.Files;
import java.nio.file.Path;
import java.security.Policy;
import java.util.Collections;
public class Main {
public static void main(String... args) throws Exception {
// policy configuration contents: this JAR gets all permissions, others get nothing
StringBuilder sb = new StringBuilder("grant {};\n\ngrant codebase \"")
.append(Main.class.getProtectionDomain().getCodeSource().getLocation())
.append("\" {\n\tpermission java.security.AllPermission;\n};\n");
// temp-save the policy configuration
Path policyPath = Files.createTempFile(null, null);
Files.write(policyPath, Collections.singleton(sb.toString()));
// convey to the default file-backed policy provider where to obtain its configuration from;
// leading equals ensures only the specified config file gets processed
System.setProperty("java.security.policy", "=".concat(policyPath.toUri().toURL().toString()));
// establish a policy; "javaPolicy" is the default provider's standard JCA name
Policy.setPolicy(Policy.getInstance("javaPolicy", null));
// policy loaded; backing config no longer needed
Files.delete(policyPath);
// establish a security manager for enforcing the policy (the default implementation is more than
// sufficient)
System.setSecurityManager(new SecurityManager());
// ...
}
}
Alternatively, you will either have to a) modify the JRE distribution's java.policy (or specify a different configuration via the policy.url.n properties in java.security), or b) replace the implementation of the System ClassLoader with one that statically grants AllPermission to the ProtectionDomain associated with classes loaded from the "main" JAR.
Secondly, when loading Extensions from some JAR, employ a URLClassLoader subclass that a) manages extension-specific directories and b) includes a java.io.FilePermission in the permission collection being statically accorded to the protection domain mapped to its defined classes. Crude sample implementation (note that there is no persistent relationship between an extension JAR and a directory; also note that two Extensions originating from the same JAR (but loaded by different class loaders, of course) will get different directories):
package q46991566;
import java.io.FilePermission;
import java.io.IOException;
import java.net.URL;
import java.net.URLClassLoader;
import java.nio.file.Files;
import java.nio.file.Path;
import java.security.CodeSource;
import java.security.Permission;
import java.security.PermissionCollection;
import java.security.Permissions;
import java.security.cert.Certificate;
import java.util.Enumeration;
import java.util.Objects;
public final class ExtensionLoader extends URLClassLoader {
private static void copyPermissions(PermissionCollection src, PermissionCollection dst) {
for (Enumeration<Permission> e = src.elements(); e.hasMoreElements();) {
dst.add(e.nextElement());
}
}
private final CodeSource origin;
private final PermissionCollection perms = new Permissions();
private final Path baseDir;
public ExtensionLoader(URL extensionOrigin) {
super(new URL[] { extensionOrigin });
origin = new CodeSource(Objects.requireNonNull(extensionOrigin), (Certificate[]) null);
try {
baseDir = Files.createTempDirectory(null);
perms.add(new FilePermission(baseDir.toString().concat("/-"), "read,write,delete"));
copyPermissions(super.getPermissions(origin), perms);
perms.setReadOnly();
}
catch (IOException ioe) {
throw new RuntimeException(ioe);
}
}
#Override
protected PermissionCollection getPermissions(CodeSource cs) {
return (origin.implies(cs)) ? perms : super.getPermissions(cs);
}
// ExtensionApiImpl (or ExtensionImpl directly -- but then ExtensionLoader would have to be relocated
// into a separate, also fully privileged JAR, accessible to the extension) can call this to relay to
// extensions where they can persist their data
public Path getExtensionBaseDir() {
return baseDir;
}
// optionally override close() to delete baseDir early
}
Lastly, for unprivileged Extensions to be able to execute privileged operations via ExtensionApi, the latter's implementation must wrap privileged method (methods issuing SecurityManager::checkXXX requests) invocations within Privileged(Exception)Actions and pass them to AccessController::doPrivileged; e.g.:
ExtensionApi api = () -> {
AccessController.doPrivileged((PrivilegedAction<Void>) () -> {
try {
Files.write(Paths.get("/root/Documents/highly-sensitive.doc"), Collections.singleton("trusted content"),
StandardOpenOption.CREATE, StandardOpenOption.WRITE, StandardOpenOption.APPEND);
return null;
}
catch (IOException ioe) {
throw new RuntimeException(ioe);
}
});
};
For details on the (proper) use of "privileged blocks", refer to the AccessController documentation and the "Secure Coding Guidelines for Java SE" document.

Loading JRuby at runtime and ClassLoader leak

I'm trying to load JRuby dynamically at runtime (so I can execute Ruby code using arbitrary JRuby installations and versions). My plan is roughly to create a ClassLoader that has access to jruby.jar, then use that to load the necessary JRuby runtime etc. All was well until I needed to do this multiple times. If I destroy the first JRuby runtime, the third or fourth will cause an OutOfMemory: PermGen space.
I've reduced this to a minimal example. The example uses both the "direct" API as well as the JRuby Embed API. The "direct" API section is commented out, but both exhibit the same behavior: after a few iterations, PermGen is out of memory. (tested with JRuby 1.6.7 and JRuby 1.6.5.1)
import java.lang.reflect.Method;
import java.net.URL;
import java.net.URLClassLoader;
import org.junit.Test;
public class JRubyInstantiationTeardownTest {
#Test
public void test() throws Exception {
for (int i = 0; i < 100; ++i) {
URL[] urls = new URL[] {
new URL("file://path/to/jruby-1.6.7.jar")
};
ClassLoader cl = new URLClassLoader(urls, this.getClass().getClassLoader());
// "Direct" API
/*
Class<?> klass = cl.loadClass("org.jruby.Ruby");
Method newInstance = klass.getMethod("newInstance");
Method evalScriptlet = klass.getMethod("evalScriptlet", String.class);
Method tearDown = klass.getMethod("tearDown");
Object runtime = newInstance.invoke(null);
System.out.println("have " + runtime);
evalScriptlet.invoke(runtime, "puts 'hello, world'");
tearDown.invoke(runtime);
*/
// JRuby Embed API
Class<?> scriptingContainerClass = cl.loadClass("org.jruby.embed.ScriptingContainer");
Method terminate = scriptingContainerClass.getMethod("terminate");
Method runScriptlet = scriptingContainerClass.getMethod("runScriptlet", String.class);
Object container = scriptingContainerClass.newInstance();
System.out.println("have " + container);
runScriptlet.invoke(container, "puts 'hello, world'");
terminate.invoke(container);
}
}
}
Questions: is this a reasonable thing to try to do with a ClassLoader? If so, is this a bug in JRuby, or am I doing something wrong with my class loading?
Bonus: if this were a bug in JRuby, how might something like Eclipse Memory Analysis tool help find the source? I can open a heap dump and see several Ruby objects (where I'd expect no more than one at any given time), but I'm not sure how to find why these aren't being garbage collected...

Try to look at stackoverflow: loading classes with different classloaders to unload them from the JVM when not needed and references from there. Sources of a mature web-container (like Tomcat) should have answers for your problem somewhere in load/unload stack.
PermGen stores bytecode for loaded classes (and generated dynamic proxies). It should be properly compacted by GC, when all references to the classes and their class loader cleared. But your code proves that something keeps your JRuby classes locked and accessible from the main class loader. It could be callback map of somekind the JRuby registers itself on load.

Edit: reported this as a bug: JRUBY-6522, now fixed.
After digging around in the Eclipse Memory Analyzer, I clicked "path to GC" on one of the URLClassLoader instances. It was referenced by org.jruby.RubyEncoding$2 which was referenced by java.lang.ThreadLocal$ThreadLocalMap$Entry.
Looking inside that source file, I see a static ThreadLocal variable being created: RubyEncoding.java:266. ThreadLocals are presumably hanging around forever, referencing my ClassLoader and leaking memory.
This code example succeeds:
import java.lang.reflect.Method;
import java.net.URL;
import java.net.URLClassLoader;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import org.junit.Test;
public class JRubyInstantiationTeardownTest {
public static int i;
#Test
public void test() throws Exception {
for (i = 0; i < 100; ++i) {
URL[] urls = new URL[] {
new URL("file:///home/pat/jruby-1.6.7/lib/jruby.jar")
};
final ClassLoader cl = new URLClassLoader(urls, this.getClass().getClassLoader());
final Class<?> rubyClass = cl.loadClass("org.jruby.Ruby");
final Method newInstance = rubyClass.getMethod("newInstance");
final Method evalScriptlet = rubyClass.getMethod("evalScriptlet", String.class);
final Method tearDown = rubyClass.getMethod("tearDown");
// "Direct" API
Callable<Void> direct = new Callable<Void>() {
public Void call() throws Exception {
// created inside thread because initialization happens immediately
final Object ruby = newInstance.invoke(null);
System.out.println("" + i + ": " + ruby);
evalScriptlet.invoke(ruby, "puts 'hello, world'");
tearDown.invoke(ruby);
return null;
}
};
// JRuby Embed API
final Class<?> scriptingContainerClass = cl.loadClass("org.jruby.embed.ScriptingContainer");
final Method terminate = scriptingContainerClass.getMethod("terminate");
final Method runScriptlet = scriptingContainerClass.getMethod("runScriptlet", String.class);
// created outside thread because ruby instance not created immediately
final Object container = scriptingContainerClass.newInstance();
Callable<Void> embed = new Callable<Void>() {
public Void call() throws Exception {
System.out.println(i + ": " + container);
runScriptlet.invoke(container, "puts 'hello, world'");
terminate.invoke(container);
return null;
}
};
// separate thread for each loop iteration so its ThreadLocal vars are discarded
final ExecutorService executor = Executors.newSingleThreadExecutor();
executor.submit(direct).get();
executor.submit(embed).get();
executor.shutdown();
}
}
}
Now I'm wondering if this is acceptable behavior of JRuby, or what JRuby-Rack does in the context of a servlet container where the servlet container is managing its own thread pool to process requests. It seems like one would need to maintain a completely separate thread pool, only execute Ruby code in those threads, and then ensure they get destroyed when the servlet is undeployed...
This is very relevant: Tomcat Memory Leak Protection
See also JVM bug report: Provide reclaimable thread local values without Thread termination