I just started using Wala Java Slicer to do some source code analysis tasks. I have a question about the proper use of the library. Assuming I have the following example code:
public void main(String[] args) {
...
UserType ut = userType;
int i = ut.getInt();
...
System.out.println(i);
}
Calculating a slice for the println statement with Wala gives the following statements:
NORMAL_RET_CALLER:Node: < Application, LRTExecutionClass, main([Ljava/lang/String;)V > Context: Everywhere[15]13 = invokevirtual < Application, LUserType, getInt()I > 11 #27 exception:12
NORMAL main:23 = getstatic < Application, Ljava/lang/System, out, <Application,Ljava/io/PrintStream> > Node: < Application, LRTExecutionClass, main([Ljava/lang/String;)V > Context: Everywhere
NORMAL main:invokevirtual < Application, Ljava/io/PrintStream, println(I)V > 23,13 #63 exception:24 Node: < Application, LRTExecutionClass, main([Ljava/lang/String;)V > Context: Everywhere
The code I am using to create the slice with Wala is shown below:
AnalysisScope scope = AnalysisScopeReader.readJavaScope("...",
null, WalaJavaSlicer.class.getClassLoader());
ClassHierarchy cha = ClassHierarchy.make(scope);
Iterable<Entrypoint> entrypoints = Util.makeMainEntrypoints(scope, cha);
AnalysisOptions options = new AnalysisOptions(scope, entrypoints);
// Build the call graph
CallGraphBuilder cgb = Util.makeZeroCFABuilder(options, new AnalysisCache(),cha, scope, null, null);
CallGraph cg = cgb.makeCallGraph(options, null);
PointerAnalysis pa = cgb.getPointerAnalysis();
// Find seed statement
Statement statement = findCallTo(findMainMethod(cg), "println");
// Context-sensitive thin slice
Collection<Statement> slice = Slicer.computeBackwardSlice(statement, cg, pa, DataDependenceOptions.NO_BASE_NO_HEAP, ControlDependenceOptions.NONE);
dumpSlice(slice);
There are a number of statements that I expect to find in the slice but are not present:
The assign statement ut = userType is not included even though the dependent method call ut.getInt(), IS included in the slice
No statements from the implementation of getInt() are included. Is there an option to activate "inter-procedural" slicing? I should mention here that the .class file is included in the path used to create the AnalysisScope.
As you can see, I am using DataDependenceOptions.NO_BASE_NO_HEAP and ControlDependenceOptions.NONE for the dependence options. But even when I use FULL for both, the problem persists.
What am I doing wrong?
The assign statement ut = userType is not included even though the
dependent method call ut.getInt(), IS included in the slice
I suspect that assignment never makes it into the byte code since it's an un-required local variable and hence will not be visible to WALA:
Because the SSA IR has already been somewhat optimized, some
statements such as simple assignments (x=y, y=z) do not appear in the
IR, due to copy propagation optimizations done automatically during
SSA construction by the SSABuilder class. In fact, there is no SSA
assignment instruction; additionally, a javac compiler is free to do
these optimizations, so the statements may not even appear in the
bytecode. Thus, these Java statements will never appear in the slice.
http://wala.sourceforge.net/wiki/index.php/UserGuide:Slicer#Warning:_exclusion_of_copy_statements_from_slice
Related
I'm trying to find a concise example which shows auto vectorization in java on a x86-64 system.
I've implemented the below code using y[i] = y[i] + x[i] in a for loop. This code can benefit from auto vectorization, so I think java should compile it at runtime using SSE or AVX instructions to speed it up.
However, I couldn't find the vectorized instructions in the resulting native machine code.
VecOpMicroBenchmark.java should benefit from auto vectorization:
/**
* Run with this command to show native assembly:<br/>
* java -XX:+UnlockDiagnosticVMOptions
* -XX:CompileCommand=print,VecOpMicroBenchmark.profile VecOpMicroBenchmark
*/
public class VecOpMicroBenchmark {
private static final int LENGTH = 1024;
private static long profile(float[] x, float[] y) {
long t = System.nanoTime();
for (int i = 0; i < LENGTH; i++) {
y[i] = y[i] + x[i]; // line 14
}
t = System.nanoTime() - t;
return t;
}
public static void main(String[] args) throws Exception {
float[] x = new float[LENGTH];
float[] y = new float[LENGTH];
// to let the JIT compiler do its work, repeatedly invoke
// the method under test and then do a little nap
long minDuration = Long.MAX_VALUE;
for (int i = 0; i < 1000; i++) {
long duration = profile(x, y);
minDuration = Math.min(minDuration, duration);
}
Thread.sleep(10);
System.out.println("\n\nduration: " + minDuration + "ns");
}
}
To find out if it gets vectorized, I did the following:
open eclipse and create the above file
right-click the file and from the dropdown menu, choose Run > Java Application (ignore the output for now)
in the eclipse menu, click Run > Run Configurations...
in the opened window, find VecOpMicroBenchmark, click it and choose the Arguments tab
in the Arguments tab, under VM arguments: put in this: -XX:+UnlockDiagnosticVMOptions -XX:CompileCommand=print,VecOpMicroBenchmark.profile
get libhsdis and copy (possibly rename) the file hsdis-amd64.so (.dll for windows) to java/lib directory. In my case, this was /usr/lib/jvm/java-11-openjdk-amd64/lib .
run VecOpMicroBenchmark again
It should now print lots of information to the console, part of it being the disassembled native machine code, which was produced by the JIT compiler. If you see lots of messages, but no assembly instructions like mov, push, add, etc, then maybe you can somewhere find the following message:
Could not load hsdis-amd64.so; library not loadable; PrintAssembly is disabled
This means that java couldn't find the file hsdis-amd64.so - it's not in the right directory or it doesn't have the right name.
hsdis-amd64.so is the disassembler which is required for showing the resulting native machine code. After the JIT compiler compiles the java bytecode to native machine code, hsdis-amd64.so is used to disassemble the native machine code to make it human readable. You can find more infos on how to get/install it at How to see JIT-compiled code in JVM? .
After finding assembly instructions in the output, I skimmed through it (too much to post all of it here) and looked for line 14. I found this:
0x00007fac90ee9859: nopl 0x0(%rax)
0x00007fac90ee9860: cmp 0xc(%rdx),%esi ; implicit exception: dispatches to 0x00007fac90ee997f
0x00007fac90ee9863: jnb 0x7fac90ee9989
0x00007fac90ee9869: movsxd %esi,%rbx
0x00007fac90ee986c: vmovss 0x10(%rdx,%rbx,4),%xmm0 ;*faload {reexecute=0 rethrow=0 return_oop=0}
; - VecOpMicroBenchmark::profile#16 (line 14)
0x00007fac90ee9872: cmp 0xc(%rdi),%esi ; implicit exception: dispatches to 0x00007fac90ee9997
0x00007fac90ee9875: jnb 0x7fac90ee99a1
0x00007fac90ee987b: movsxd %esi,%rbx
0x00007fac90ee987e: vmovss 0x10(%rdi,%rbx,4),%xmm1 ;*faload {reexecute=0 rethrow=0 return_oop=0}
; - VecOpMicroBenchmark::profile#20 (line 14)
0x00007fac90ee9884: vaddss %xmm1,%xmm0,%xmm0
0x00007fac90ee9888: movsxd %esi,%rbx
0x00007fac90ee988b: vmovss %xmm0,0x10(%rdx,%rbx,4) ;*fastore {reexecute=0 rethrow=0 return_oop=0}
; - VecOpMicroBenchmark::profile#22 (line 14)
So it's using the AVX instruction vaddss. But, if I'm correct here, vaddss means
add scalar single-precision floating-point values and this only adds one float value to another one (here, scalar means just one, whereas here single means 32 bit, i.e. float and not double).
What I expect here is vaddps, which means add packed single-precision floating-point values and which is a true SIMD instruction (SIMD = single instruction, multiple data = vectorized instruction). Here, packed means multiple floats packed together in one register.
About the ..ss and ..ps, see http://www.songho.ca/misc/sse/sse.html :
SSE defines two types of operations; scalar and packed. Scalar operation only operates on the least-significant data element (bit 0~31), and packed operation computes all four elements in parallel. SSE instructions have a suffix -ss for scalar operations (Single Scalar) and -ps for packed operations (Parallel Scalar).
Queston:
Is my java example incorrect, or why is there no SIMD instruction in the output?
In the main() method, put in i < 1000000 instead of just i < 1000. Then the JIT also produces AVX vector instructions like below, and the code runs faster:
0x00007f20c83da588: vmovdqu 0x10(%rbx,%r11,4),%ymm0
0x00007f20c83da58f: vaddps 0x10(%r13,%r11,4),%ymm0,%ymm0
0x00007f20c83da596: vmovdqu %ymm0,0x10(%rbx,%r11,4) ;*fastore {reexecute=0 rethrow=0 return_oop=0}
; - VecOpMicroBenchmark::profile#22 (line 14)
The code from the question is actually optimizable by the JIT compiler using auto-vectorization. However, as Peter Cordes pointed out in a comment, the JIT needs quite some processing, thus it is rather reluctant to decide that it should fully optimize some code.
The solution is simply to execute the code more often during one execution of the program, not just 1000 times, but 100000 times or a million times.
When executing the profile() method this many times, the JIT compiler is convinced that the code is very important and the overall runtime will benefit from full optimization, thus it optimizes the code again and then it also uses true vector instructions like vaddps.
More details in Auto Vectorization in Java
What am I doing?
I am writing a data analysis program in Java which relies on R´s arulesViz library to mine association rules.
What do I want?
My purpose is to store the rules in a String variable in Java so that I can process them later.
How does it work?
The code works using a combination of String.format and eval Java and RJava instructions respectively, being its behavior summarized as:
Given properly formatted Java data structures, creates a data frame in R.
Formats the recently created data frame into a transaction list using the arules library.
Runs the apriori algorithm with the transaction list and some necessary values passed as parameter.
Reorders the generated association rules.
Given that the association rules cannot be printed, they are written to the standard output with R´s write method, capture the output and store it in a variable. We have converted the association rules into a string variable.
We return the string.
The code is the following:
// Step 1
Rutils.rengine.eval("dataFrame <- data.frame(as.factor(c(\"Red\", \"Blue\", \"Yellow\", \"Blue\", \"Yellow\")), as.factor(c(\"Big\", \"Small\", \"Small\", \"Big\", \"Tiny\")), as.factor(c(\"Heavy\", \"Light\", \"Light\", \"Heavy\", \"Heavy\")))");
//Step 2
Rutils.rengine.eval("transList <- as(dataFrame, 'transactions')");
//Step 3
Rutils.rengine.eval(String.format("info <- apriori(transList, parameter = list(supp = %f, conf = %f, maxlen = 2))", supportThreshold, confidenceThreshold));
// Step 4
Rutils.rengine.eval("orderedRules <- sort(info, by = c('count', 'lift'), order = FALSE)");
// Step 5
REXP res = Rutils.rengine.eval("rulesAsString <- paste(capture.output(write(orderedRules, file = stdout(), sep = ',', quote = TRUE, row.names = FALSE, col.names = FALSE)), collapse='\n')");
// Step 6
return res.asString().replaceAll("'", "");
What´s wrong?
Running the code in Linux Will work perfectly, but when I try to run it in Windows, I get the following error referring to the return line:
Exception in thread "main" java.lang.NullPointerException
This is a common error I have whenever the R code generates a null result and passes it to Java. There´s no way to syntax check the R code inside Java, so whenever it´s wrong, this error message appears.
However, when I run the R code in brackets in the R command line in Windows, it works flawlessly, so both the syntax and the data flow are OK.
Technical information
In Linux, I am using R with OpenJDK 10.
In Windows, I am currently using Oracle´s latest JDK release, but trying to run the program with OpenJDK 12 for Windows does not solve anything.
Everything is 64 bits.
The IDE used in both operating systems is IntelliJ IDEA 2019.
Screenshots
Linux run configuration:
Windows run configuration:
At this moment I participate in big legacy project with many huge classes and generated code.
I wish to find all methods that have bytecode length bigger than 8000 bytes (because OOTB java will not optimize it).
I found manual way like this: How many bytes of bytecode has a particular method in Java?
, however my goal is to scan many files automatically.
I tried to use jboss-javassist, but AFAIK getting bytecode length is available only on class level.
Huge methods might indeed never get inlined, however, but I have my doubts regarding the threshold of 8000. This comment suggests a much smaller limit, though it is platform and configuration dependent anyway.
You are right that getting bytecode length needs to process classes on that low level, however, you didn’t specify what actual obstacle you encountered when trying to do that with Javassist. A simple program doing that with Javassist, would be
try(InputStream is=javax.swing.JComponent.class.getResourceAsStream("JComponent.class")) {
ClassFile cf = new ClassFile(new DataInputStream(is));
for(MethodInfo mi: cf.getMethods()) {
CodeAttribute ca = mi.getCodeAttribute();
if(ca == null) continue; // abstract or native
int bLen = ca.getCode().length;
if(bLen > 300)
System.out.println(mi.getName()+" "+mi.getDescriptor()+", "+bLen+" bytes");
}
}
This has been written and tested with a recent version of Javassist that uses Generics in the API. If you have a different/older version, you have to use
try(InputStream is=javax.swing.JComponent.class.getResourceAsStream("JComponent.class")) {
ClassFile cf = new ClassFile(new DataInputStream(is));
for(Object miO: cf.getMethods()) {
MethodInfo mi = (MethodInfo)miO;
CodeAttribute ca = mi.getCodeAttribute();
if(ca == null) continue; // abstract or native
int bLen = ca.getCode().length;
if(bLen > 300)
System.out.println(mi.getName()+" "+mi.getDescriptor()+", "+bLen+" bytes");
}
}
I am trying bcel to modify a method by inserting invoke before specific instructions.
It seems that my instrumentation would result in a different stackmap table, which can not be auto-generated by the bcel package itself.
So, my instrumented class file contains the old stackmap table, which would cause error with jvm.
I haved tried with removeCodeAttributes, the method of MethodGen, that can remove all the code attributes. It can work in simple cases, a wrapped function, for example. And it can not work in my case now.
public class Insert{
public static void main(String[] args) throws ClassFormatException, IOException{
Insert isrt = new Insert();
String className = "StringBuilder.class";
JavaClass jclzz = new ClassParser(className).parse();
ClassGen cgen = new ClassGen(jclzz);
ConstantPoolGen cpgen = cgen.getConstantPool();
MethodGen mgen = new MethodGen(jclzz.getMethods()[1], className, cpgen);
InstructionFactory ifac = new InstructionFactory(cgen);
InstructionList ilist = mgen.getInstructionList();
for (InstructionHandle ihandle : ilist.getInstructionHandles()){
System.out.println(ihandle.toString());
}
InstructionFinder f = new InstructionFinder(ilist);
InstructionHandle[] insert_pos = (InstructionHandle[])(f.search("invokevirtual").next());
Instruction inserted_inst = ifac.createInvoke("java.lang.System", "currentTimeMillis", Type.LONG, Type.NO_ARGS, Constants.INVOKESTATIC);
System.out.println(inserted_inst.toString());
ilist.insert(insert_pos[0], inserted_inst);
mgen.setMaxStack();
mgen.setMaxLocals();
mgen.removeCodeAttributes();
cgen.replaceMethod(jclzz.getMethods()[1], mgen.getMethod());
ilist.dispose();
//output the file
FileOutputStream fos = new FileOutputStream(className);
cgen.getJavaClass().dump(fos);
fos.close();
}
}
Removing a StackMapTable is not a proper solution for fixing a wrong StackMapTable. The important cite is:
4.7.4. The StackMapTable Attribute
In a class file whose version number is 50.0 or above, if a method's Code attribute does not have a StackMapTable attribute, it has an implicit stack map attribute (§4.10.1). This implicit stack map attribute is equivalent to a StackMapTable attribute with number_of_entries equal to zero.
Since a StackMapTable must have explicit entries for every branch target, such an implicit StackMapTable will work with branch-free methods only. But in these cases, the method usually doesn’t have an explicit StackMapTable anyway, so you wouldn’t have that problem then (unless the method had branches which your instrumentation removed).
Another conclusion is that you can get away with removing the StackMapTable, if you patch the class file version number to a value below 50. Of course, this is only a solution if you don’t need any class file feature introduced in version 50 or newer…
There was a grace period in which JVMs supported a fall-back mode for class files with broken StackMapTables just for scenarios like yours, where the tool support is not up-to-date. (See -XX:+FailoverToOldVerifier or -XX:-UseSplitVerifier) But the grace period is over now and that support has been declined, i.e. Java 8 JVMs do not support the fall-back mode anymore.
If you want to keep up with the Java development and instrument newer class files which might use features of these new versions you have only two choices:
Calculate the correct StackMapTable manually
Use a tool which supports calculating the correct StackMapTable attributes, e.g. ASM, (see java-bytecode-asm) does support it
I am getting expected ClassVerifyErrors when attempting to load a class i have generated using ASM. On further inspection i can see that the jvm is correct and that the method is talking about has an invalid MAX_STACK value. THe strange thing is am using the auto calculate the stack and max local options so this should not be a problem...
The method with the invalid option is very simple and yet the result is bad bytecode.
I have written a class with the intended method and compared my asm generated class against what javac produces and the byte codes matchup with the only error being the max stack is 0 which is wrong while javac sets a value of 2.
Id like to avoid having to calculate tha max stack/locals myself.
Max stack and variable calculation can produce the wrong results if bytecode is not valid. You can verify that by running generated code trough the CheckClassAdapter.
For example,
ClassWriter cw = new ClassWriter(ClassWriter.COMPUTE_MAXS);
// generate code into cw instance...
PrintWriter pw = new PrintWriter(System.out);
CheckClassAdapter.verify(new ClassReader(cw.toByteArray()), true, pw);