Develop Reference

Soot: Reload class after source-file changed

I'm writing an IntelliJ-Plugin to analyse java-program code. Thus i use Soot to write static analyses. Every time a user triggers the analyse-action of my plugin, I take the current VirtualFile of the current context like this:
FileEditorManager manager = FileEditorManager.getInstance(e.getProject());
VirtualFile files[] = manager.getSelectedFiles();
toAnalyse = files[0]; [...]
When I check the content of this file all changes are applied. After this I'm loading the class I want to analyse in Soot.
String dir = toAnalyse.getParent().getPath() ;
Options.v().setPhaseOption("jb", "use-original-names");
Options.v().set_soot_classpath( System.getProperty("java.home")+";"+ dir);
c = Scene.v().loadClassAndSupport(name);
/*no analyse c*/
This works perfectly for me. But now to my issue:
If i change sth. in test instance of my plugin and trigger the same analysis again, nothing changes.
What have i tried so far?
I set following options:
Options.v().set_dump_body( Arrays.asList("jb"));
Options.v().set_dump_cfg( Arrays.asList("jb"));
Options.v().set_allow_phantom_refs(true);
Options.v().set_whole_program(true);
I also removed all classes by hand
like this:
Chain<SootClass> classes = Scene.v().getClasses();
Stack<SootClass> stack = new Stack<>();
for(SootClass s : classes)
stack.push(s);
while(!stack.empty())
Scene.v().removeClass(stack.pop());
and started the program again.

I solved this issue.
SootClass c = Scene.v().loadClassAndSupport(name);
// ...
c.setResolvingLevel(0);
G.reset();
G.reset() resets all singleton instances.
Therefore all cached results will be overwritten by calling this action again.
public static Scene v() {
return G.v().soot_Scene();
}
this.instance_soot_Scene is null after calling G.reset().
Therefore the following code:
public Scene soot_Scene() {
if(this.instance_soot_Scene == null) {
synchronized(this) {
if(this.instance_soot_Scene == null) {
this.instance_soot_Scene = new Scene(this.g);
}
}
}
return this.instance_soot_Scene;
}
returns a new instance with an empty result cache.

Groovy Extension Module Method - No Signature of Method

I've created two groovy extension modules / methods on java.util.ArrayList(). It's all working very well inside my IDE. I use gradle to build the jar, and deploy it to a remote JVM. When it reaches the remote JVM, it fails.
Here is the extension method:
static Map sumSelectedAttributes(final List self, List attributes) {
Map resultsMap = [:]
attributes.each { attr ->
resultsMap[attr] = self.inject(0) { sum, obj ->
sum + obj[attr]
}
}
return resultsMap
Here is the code that invokes it:
outputMap[processName][iName] << kBeanList.sumSelectedAttributes([
"messageCount", "userCount", "outstandingRequests",
"cpuUsage", "memoryUsage", "threadCount", "cacheCount", "huserCount",
"manualHuserCount", "dataPointerCount", "tableHandleCount",
"jdbCacheRecordCount", "dbConnectionCount"])
Here is the error:
No signature of method: java.util.ArrayList.sumSelectedAttributes() is
applicable for argument types: (java.util.ArrayList) values:
[[messageCount, incomingConnectionsCount, outgoingConnectionsCount,
...]]
Again, it works fine in intellij with test cases. What is different on the remote JVM that would prevent this from working? Here are some things that came to my mind:
The remote JVM uses Groovy 2.3 while I'm on 2.4.5
We use a custom classloader on the remote JVM to load classes
Other than that, I could not find any other documentation about anything special I need to do to make extensions work on remote JVM's.
Any help is greatly appreciated.
Per a comment, seems like an issue with custom classloader, here is the class that handles the manipulation of a few classloaders.
class CustomLoader {
static Map loaders = [:]
static File loaderRoot = new File("../branches")
static URLClassLoader getCustomLoader(String branchName) {
if (!loaders[branchName]) {
loaders[branchName] = new URLClassLoader(getUrls(branchName))
} else {
loaders[branchName]
}
}
static URLClassLoader updateClassLoader(String branchName) {
loaders[branchName] = null
loaders[branchName] = new URLClassLoader(getUrls(branchName))
}
private static URL[] getUrls(String branchName) {
def loaderDir = new File(loaderRoot, branchName)
List<File> files = []
loaderDir.eachFileRecurse{
if (it.name.endsWith('.jar')) {
files << it
}
}
List urls = files.sort{ it.name }.reverse().collect{it.toURI().toURL()}
return urls
}
}

To manually register an extension method module you can use code similar to what is used in GrapeIvy. Because grapes have the same problem in that they make a jar visible in the wrong loader but still want to enable extension methods. The piece of code in question is this here:
JarFile jar = new JarFile(file)
def entry = jar.getEntry(ExtensionModuleScanner.MODULE_META_INF_FILE)
if (entry) {
Properties props = new Properties()
props.load(jar.getInputStream(entry))
Map<CachedClass, List<MetaMethod>> metaMethods = new HashMap<CachedClass, List<MetaMethod>>()
mcRegistry.registerExtensionModuleFromProperties(props, loader, metaMethods)
// add old methods to the map
metaMethods.each { CachedClass c, List<MetaMethod> methods ->
// GROOVY-5543: if a module was loaded using grab, there are chances that subclasses
// have their own ClassInfo, and we must change them as well!
Set<CachedClass> classesToBeUpdated = [c]
ClassInfo.onAllClassInfo { ClassInfo info ->
if (c.theClass.isAssignableFrom(info.cachedClass.theClass)) {
classesToBeUpdated << info.cachedClass
}
}
classesToBeUpdated*.addNewMopMethods(methods)
}
}
In this code file is a File representing the jar. In your case you will need to have something else here. Basically we first load the descriptor file into Properties, call registerExtensionModuleFromProperties to fill the map with MetaMethods depending on a given class loader. And that is the key part for the solution of your problem, the right class loader here is one that can load all the classes in your extension module and the groovy runtime!. After this any new meta class will know about the extension methods. The code that follows is needed only if there are already existing meta classes, you want to know about those new methods.

How can I get the complete Call Hierarchy of a Java source code?

This is a bit tricky to explain. I have a class A:
public class A {
private Integer a1;
private Integer a2;
// getters and setters.
}
There is a static class B that returns my class A:
public static class B {
public static A getCurrentA() {
return a;
}
}
I need to find all usages of class A returned by B. So let's say class C calls c.setA(B.getCurrentA()) and then further along there's a call to c.getA().getA2();, I'd want to find all of these.
In the real scenario, I have 217 different classes that call B.getCurrentA(). I can't manually follow all the calls in Eclipse and find out which methods are getting called.
Eclipse call hierarchy view only shows me all calls to B.getCurrentA().
How can I achieve this?
EDIT
Chris Hayes understood what I want to do. In order to refactor some really bad legacy code without breaking the whole system, I need to first fine-tune some queries using Hibernate's projections (every mapped entity in the system is eagerly loaded, and many entities are related, so some queries take a LONG time fetching everything). But first I need to find which properties are used so that I don't get a NullPointerException somewhere...
Here's an example of what I'd have to do manually:
Use Eclipse's Search to find all calls to B.getCurrentA();
Open the first method found, let's say it's the one below:
public class CController {
C c = new C();
CFacade facade = new CFacade();
List<C> Cs = new ArrayList<C>();
public void getAllCs() {
c.setA(B.getCurrentA()); // found it!
facade.search(c);
}
}
Open the search method in the CFacade class:
public class CFacade {
CBusinessObject cBo = new CBusinessObject();
public List<C> search(C c) {
// doing stuff...
cBo.verifyA(c);
cBo.search(c); // yes, the system is that complicated
}
}
Open the verifyA method in the CBusinessObject class and identify that field a2 is used:
public class CBusinessObject {
public void verifyA(c) {
if (Integer.valueOf(1).equals(c.getA().getA2())) {
// do stuff
else {
// something else
}
}
}
Repeat steps 2-4 for the next 216 matches... Yay.
Please help.

If you want to make any source code changes/refactoring you will have to manually find all usages and apply your code changes;
Any way, I have two different aproach
Static search
You can simply do Text Search in eclipse to find the occurance of getA2() . It will directly take you to the Caller method (here CBusinessObject.verifyA()) -but it will give you every getA2() occurances, may be from different class
Run time search
Use java instrumentation API to change the byte code at run time on your required method to find invoking class and run as java agent - Enable you to identify the caller with out touching the existing code base and very useful especially when you don't have access to source code.
Here you go how to implement
Step 1- Write Agent main class to initiate instrumentation
public class BasicAgent {
public static void premain(String agentArguments, Instrumentation instrumentation){
System.out.println("Simple Agent");
FindUsageTransformer transformer = new FindUsageTransformer ();
instrumentation.addTransformer(transformer,true);
}
}
Step 2 -Write a ClassFileTransformer implementation and capture the method
public class FindUsageTransformer implements ClassFileTransformer{
Class clazz = null;
public byte[] transform(ClassLoader loader,String className,Class<?> classBeingRedefined, ProtectionDomain protectionDomain,
byte[] classfileBuffer) throws IllegalClassFormatException {
if(className.equals("A")){
doClass(className, classBeingRedefined, classfileBuffer);
}
return classfileBuffer;
}
private byte[] doClass(String name, Class clazz, byte[] b) {
ClassPool pool = ClassPool.getDefault();
CtClass cl = null;
try {
cl = pool.makeClass(new java.io.ByteArrayInputStream(b));
CtMethod method = cl.getDeclaredMethod("getA2");
// here you have lot of options to explore
method.insertBefore("System.out.println(Thread.currentThread().getStackTrace()[0].getClassName()+ Thread.currentThread().getStackTrace()[0].getMethodName());");
b = cl.toBytecode();
} catch (Exception e) {
System.err.println("Could not instrument " + name
+ ", exception : " + e.getMessage());
} finally {
if (cl != null) {
cl.detach();
}
}
return b;
}
Step 3- create jar file for agent classes ( you have to set manifest file with premain class, and add javaassit jar) snippet of build file is given - you can do it by manually as well
<jar destfile="build/jar/BasicAgent.jar" basedir="build/classes">
<manifest>
<attribute name="Manifest-Version" value="1.0"/>
<attribute name="Premain-Class" value="com.sk.agent.basic.BasicAgent"/>
<attribute name="Boot-Class-Path" value="../lib/javassist.jar"/>
</manifest>
</jar>
Step 4- Run your main application with java agent - before that set VM arguments to load agent
-`javaagent:D:\softwares\AgentProject\AgentLib\build\jar\BasicAgent.jar`
Pre requisite : you would need javassist.jar in the class path.

Depending on the IDE you are using this problem is simpler to find.
Eclipse IDE has one of the most potential Call Hierarchy modules existing, you just need to put the mouse in the method declaration that you want to find and execute Ctrl + Alt + H
This will give you the entire hierarchy of which method is using the method you want to analyze.
Also the Call Hierarchy module offers a mode where you can find the methods that your method is calling.
Some extra info: http://help.eclipse.org/indigo/index.jsp?topic=%2Forg.eclipse.cdt.doc.user%2Freference%2Fcdt_u_call_hierarchy_view.htm

In IntelliJ IDEA, if you want to find usages of c.getA().getA2(); right-click on A.a2 and choose "find usages." Similarly for A.a1 and B.getCurrentA(). Unused fields and methods show up in a different color in IDEA. I've heard that IntelliJ has more refactoring power than Eclipse, but I bet Eclipse does the same thing, just slightly differently.
Also, using grep, find, and sed, you can search for the appropriate methods, just in files that are in the same package as A or that import A, or spell it out by name.

I hope I understood your question correctly. I think you can use grep -Irns function to find the calls. You can grep for getA().getA2(). That will return lines from where functions are called along with line numbers.

Rather than scanning for all references to the method getCurrentA do a scan for all references to the Class A.
This will show you everywhere that class is used within your program and you will probably find it is easier to go through and scan that list by hand and decide if you need to act on each result found than trying to do anything fancy.

The easiest way to find Call Usage is using references in eclipse,but there is a funny way
:
Change method name to B.getCurrentAA()
Build your Project
Your Project compiles with error
Go to Marks Part and see usage Error And Find Usage Of your method

I think IntelliJ can solve your problem. It have an "Analyze dataflow" feature and I think it is doing what you are looking for:
Here is my sample code:
public class Main {
private static A a = new A(); //nevermind the way it is initialized
public static A getA(){
return a;
}
public void method(){
A myA = getA();
Integer a1 = myA.getA1(); //this line is found
Integer a2 = myA.getA2(); //this line is found
}
public void anotherMethod(){
A myA = new A();
Integer a1 = myA.getA1(); //this line is NOT found
Integer a2 = myA.getA2(); //this line is NOT found
}
}
Running the "Analyze dataflow from here" (with cursor on return a; line) give me this:
Sorry to provide you only a solution with IntelliJ (tested with IntelliJ-13 Ultimate Edition)

Getting list of fully qualified names from a simple name

I would like to get a list of classes that are available at runtime and that match a simple name.
For example:
public List<String> getFQNs(String simpleName) {
...
}
// Would return ["java.awt.List","java.util.List"]
List<String> fqns = getFQNs("List")
Is there a library that would do this efficiently, or do I have to manually go through all classes in each classloader? What would be the correct way of doing that?
Thanks!
UPDATE
One responder asked me why I wanted to do this. Essentially, I want to implement a feature that is similar to "organize imports/auto import", but available at runtime. I don't mind if the solution is relatively slow (especially if I can then build a cache so subsequent queries become faster) and if it is a best-effort only. For example, I don't mind if I do not get dynamically generated classes.
UPDATE 2
I had to devise my own solution (see below): it uses some hints provided by the other responders, but I came to realize that it needs to be extensible to handle various environments. It is not possible to automatically traverse all classloaders at runtime so you have to rely on general and domain-specific strategies to get a useful list of classes.

I mixed the the answers from #Grodriguez and #bemace and added my own strategy to come up with a best-effort solution. This solution imitates at runtime the auto-import feature available at compile time.
The full code of my solution is here. Given a simple name, the main steps are:
Get a list of packages accessible from the current classloader.
For each package, try to load the fully qualified name obtained from package + simple name.
Step 2 is easy:
public List<String> getFQNs(String simpleName) {
if (this.packages == null) {
this.packages = getPackages();
}
List<String> fqns = new ArrayList<String>();
for (String aPackage : packages) {
try {
String fqn = aPackage + "." + simpleName;
Class.forName(fqn);
fqns.add(fqn);
} catch (Exception e) {
// Ignore
}
}
return fqns;
}
Step 1 is harder and is dependent on your application/environment so I implemented various strategies to get different lists of packages.
Current Classloader (may be useful to detect dynamically generated classes)
public Collection<String> getPackages() {
Set<String> packages = new HashSet<String>();
for (Package aPackage : Package.getPackages()) {
packages.add(aPackage.getName());
}
return packages;
}
Classpath (good enough for applications that are entirely loaded from the classpath. Not good for complex applications like Eclipse)
public Collection<String> getPackages() {
String classpath = System.getProperty("java.class.path");
return getPackageFromClassPath(classpath);
}
public static Set<String> getPackageFromClassPath(String classpath) {
Set<String> packages = new HashSet<String>();
String[] paths = classpath.split(File.pathSeparator);
for (String path : paths) {
if (path.trim().length() == 0) {
continue;
} else {
File file = new File(path);
if (file.exists()) {
String childPath = file.getAbsolutePath();
if (childPath.endsWith(".jar")) {
packages.addAll(ClasspathPackageProvider
.readZipFile(childPath));
} else {
packages.addAll(ClasspathPackageProvider
.readDirectory(childPath));
}
}
}
}
return packages;
}
Bootstrap classpath (e.g., java.lang)
public Collection<String> getPackages() {
// Even IBM JDKs seem to use this property...
String classpath = System.getProperty("sun.boot.class.path");
return ClasspathPackageProvider.getPackageFromClassPath(classpath);
}
Eclipse bundles (domain-specific package provider)
// Don't forget to add "Eclipse-BuddyPolicy: global" to MANIFEST.MF
public Collection<String> getPackages() {
Set<String> packages = new HashSet<String>();
BundleContext context = Activator.getDefault().getBundle()
.getBundleContext();
Bundle[] bundles = context.getBundles();
PackageAdmin pAdmin = getPackageAdmin(context);
for (Bundle bundle : bundles) {
ExportedPackage[] ePackages = pAdmin.getExportedPackages(bundle);
if (ePackages != null) {
for (ExportedPackage ePackage : ePackages) {
packages.add(ePackage.getName());
}
}
}
return packages;
}
public PackageAdmin getPackageAdmin(BundleContext context) {
ServiceTracker bundleTracker = null;
bundleTracker = new ServiceTracker(context,
PackageAdmin.class.getName(), null);
bundleTracker.open();
return (PackageAdmin) bundleTracker.getService();
}
Examples of queries and answers in my Eclipse environment:
File: [java.io.File, org.eclipse.core.internal.resources.File]
List: [java.awt.List, org.eclipse.swt.widgets.List, com.sun.xml.internal.bind.v2.schemagen.xmlschema.List, java.util.List, org.hibernate.mapping.List]
IResource: [org.eclipse.core.resources.IResource]

You probably cannot do this at all. There is no way for the JVM to know whether a class List in an arbitrarily named package a.b.c.d is available without attempting to load a.b.c.d.List first. You would need to test for all possible package names.

Without loading them all you could get the classpath property
String class_path = System.getProperty("java.class.path");
And then you could create a function to search the filesystem for classes in those locations. And you'd have to code it to also search inside jar files, and some of the classes may not actually be available due to incompatibilities that would only be revealed when you load them. But if you just want a best-guess of what's available, this might be viable. Maybe you should tell us why you want to do this so you can get some alternative suggestions?
Edit:
Ok, sounds like you should check out this thread and the ones linked in it: How do I read all classes from a Java package in the classpath?
In particular it appears the Spring framework does something similar, maybe you can look at that code: http://static.springsource.org/spring/docs/2.0.x/api/org/springframework/core/io/support/PathMatchingResourcePatternResolver.html

How to find classes when running an executable jar

I am running an executable jar and wish to find a list of classes WITHIN the jar so that I can decide at run-time which to run. It's possible that I don't know the name of the jar file so cannot unzip it

You can not enumerate classes from a package of jar using Reflection API. This is also made clear in the related questions how-can-i-enumerate-all-classes-in-a-package and
can-i-list-the-resources-in-a-given-package. I once wrote a tool that lists all classes found in a certain classpath. It's too long to paste here, but here is the general approach:
find the used classpath. This is shown nicely by eirikma in another answer.
add other places where the ClassLoader might search for classes, e.g. bootclasspath, endorsed lib in JRE etc. (If you just have a simple app, then 1 + 2 are easy, just take the class path from property.)
readAllFromSystemClassPath("sun.boot.class.path");
readAllFromSystemClassPath("java.endorsed.dirs");
readAllFromSystemClassPath("java.ext.dirs");
readAllFromSystemClassPath("java.class.path");
Scan the classpath and split folders from JARs.
StringTokenizer pathTokenizer = new StringTokenizer(pPath, File.pathSeparator);
Scan the folders with File.listFiles and open the JARs with ZipFile.entries. Pay attention to inner classes and package access classes, you propably do not want them.
isInner = (pClassName.indexOf('$') > -1);
Convert the file name or path in the JAR to a proper classname (/ -> .)
final int i = fullName.lastIndexOf(File.separatorChar);
path = fullName.substring(0, i).replace(File.separatorChar, '.');
name = fullName.substring(i + 1);
Now you can use Reflection to load that class and have a look into it. If you just want to know stuff of about the class you can load it without resolving, or use a byte code engineering library like BCEL to open the class without loading it into the JVM.
ClassLoader.getSystemClassLoader().loadClass(name).getModifiers() & Modifier.PUBLIC

I am not sure if there is a way to list all classes visible to the current classloader.
Lacking that, you could
a) try to find out the name of the jar file from the classpath, and then look at its contents.
or
b) supposing that you have a candidate list of classes you are looking for, try each of them with Class.forName().

you can use a simple program to get a list of all the class files from jar and dump it in a property file on runtime and then in your program you can load req. class as and when req.; without using reflections.

You can get the actual classpath from the classloader. this must include the jar file, otherwise the program wouldn't run. Look throug the classpath URLs to find a URL that ends with ".jar" and contains something that is never changing in the name of you jar file (preferably after the last "/"). After that you open it as a regular jar (or zip) file and read the contents.
There are several methods available for obtaining the classpath. None of them works in every context and with every setup, so you must try them one by one until you find one that works in all the situations you need it to work. Also, sometimes you might need to tweak the runtime context, like (often needed) substituting maven surefire-plugin's classloading mechanism to one of optional (non-default) ones.
Obtaining the classpath 1: from system property:
static String[] getClasspathFromProperty() {
return System.getProperty("java.class.path").split(File.pathSeparator);
}
Obtaining the classpath 2: from classloader (with maven warning):
String[] getClasspathFromClassloader() {
URLClassLoader classLoader = (URLClassLoader) (getClass().getClassLoader());
URL[] classpath = classLoader.getURLs();
if (classpath.length == 1
&& classpath[0].toExternalForm().indexOf("surefirebooter") >= 0)
{
// todo: read classpath from manifest in surefireXXXX.jar
System.err.println("NO PROPER CLASSLOADER HERE!");
System.err.println(
"Run maven with -Dsurefire.useSystemClassLoader=false "
+"-Dsurefire.useManifestOnlyJar=false to enable proper classloaders");
return null;
}
String[] classpathLocations = new String[classpath.length];
for (int i = 0; i < classpath.length; i++) {
// you must repair the path strings: "\.\" => "/" etc.
classpathLocations[i] = cleanClasspathUrl(classpath[i].toExternalform());
}
return classpathLocations;
}
Obtaining the classpath 3: from current thread context: This is similar to method 2, except the first line of the method should read like this:
URLClassLoader classLoader
= (URLClassLoader)(Thread.currentThread().getContextClassLoader());
Good luck!

I would use a bytecode inspector library like ASM. This ClassVisitor can be used to look for the main method:
import org.objectweb.asm.*;
import org.objectweb.asm.commons.EmptyVisitor;
public class MainFinder extends ClassAdapter {
private String name;
private boolean isMainClass;
public MainFinder() {
super(new EmptyVisitor());
}
#Override
public void visit(int version, int access, String name,
String signature, String superName,
String[] interfaces) {
this.name = name;
super.visit(version, access, name, signature,
superName, interfaces);
}
#Override
public MethodVisitor visitMethod(int access, String name,
String desc, String signature, String[] exceptions) {
if ((access & Opcodes.ACC_PUBLIC) != 0
&& (access & Opcodes.ACC_STATIC) != 0
&& "main".equals(name)
&& "([Ljava/lang/String;)V".equals(desc)) {
isMainClass = true;
}
return super.visitMethod(access, name, desc, signature,
exceptions);
}
public String getName() {
return name;
}
public boolean isMainClass() {
return isMainClass;
}
}
Note that you might want to alter the code to confirm that classes are public, etc.
This sample app uses the above class on a command-line-specified JAR:
import java.io.*;
import java.util.Enumeration;
import java.util.jar.*;
import org.objectweb.asm.ClassReader;
public class FindMainMethods {
private static void walk(JarFile jar) throws IOException {
Enumeration<? extends JarEntry> entries = jar.entries();
while (entries.hasMoreElements()) {
MainFinder visitor = new MainFinder();
JarEntry entry = entries.nextElement();
if (!entry.getName().endsWith(".class")) {
continue;
}
InputStream stream = jar.getInputStream(entry);
try {
ClassReader reader = new ClassReader(stream);
reader.accept(visitor, ClassReader.SKIP_CODE);
if (visitor.isMainClass()) {
System.out.println(visitor.getName());
}
} finally {
stream.close();
}
}
}
public static void main(String[] args) throws IOException {
JarFile jar = new JarFile(args[0]);
walk(jar);
}
}
You may also want to look at the "java.class.path" system property.
System.getProperty("java.class.path");
It is possible to use reflection to obtain similar results, but that approach may have some unfortunate side-effects - like causing static initializers to be run, or keeping unused classes in memory (they will probably stay loaded until their ClassLoader is garbage collected).