Develop Reference

Interpreted language in Java and calls to Java methods

I have implemented simple interpreted language with dynamic typing in Java. Unfortunately I ran into the following problem. When testing the following code:
def main() {
def ks = Map[[1, 2]].keySet();
return ks.size();
}
I stumbled upon the following exception:
java.lang.IllegalAccessException: class is not public: java.util.HashMap$KeySet.size()int/invokeSpecial
Of course This is true and caused by the fact that HashMap$KeySet class has "package" visibility. This means that when I call it's "size()" method, I call method from class that is not visible to my code. Java avoids this problem easily - method keySet() returns value of type Set, so method size() used is declared in public and abstract class "Set".
My question is: does anyone has an idea, how this should be handled in generic way? By "general" case I mean not only this simple case, where I can walk through whole inheritance chain and find "first declaration" of this method, but also pathological cases like the following:
interface I1 {
public void foo();
}
interface I2 {
public void foo();
}
interface I3 {
public void foo();
}
class C implements I1, I2, I3 {
public void foo() { .... }
}
My current impression is that I could ignore those pathological cases and select any matching method on the grounds that if such object exists, then it's creation was successful, so it's compilation was successful, so all these methods have identical signatures and since in Java there is no way to specify different implementations of these methods depending on how object is viewed (as I1, I2 or I3), then result will be always the same.
Any help will be appreciated.

Ok, so here is my solution. It isn't very nice, but hey, whatever works:
public static Method findMethod(Class<?> cls, String name, Class<?>[] fa) {
System.out.println("Checking class " + cls + " for method " + name);
// since it is called recursively, we want to stop some day, and when we are
// passed null (so most getSuperclass was called on Object.class or something similar)
if (cls == null) {
return null;
}
Method m = null;
if ((m = findMethod(cls.getSuperclass(), name, fa)) != null) {
return m;
}
// ok, if we're here, then m is null. so check if cls is public. it must be public, because
// otherwise we won't be able to call it - we are definitely in different package. if class
// isn't public, then check interfaces.
if (!Modifier.isPublic(cls.getModifiers())) {
System.out.println("Class is not public, and superclasses do not contain method " + name);
System.out.println("Checking all interfaces");
for (Class<?> iface: cls.getInterfaces()) {
if ((m = findMethod(iface, name, fa)) != null) {
return m;
}
}
}
return findMethodInClass(cls, name, fa);
}
private static Method findMethodInClass(Class<?> cls, String name, Class<?>[] fa) {
Method m = null;
// scan all methods and move plausible candidates to the start of an array
Method[] mm = cls.getMethods();
int n = 0;
for (int i = 0 ; i < mm.length ; ++i) {
if (checkMethod(mm[i], name, fa)) {
mm[n++] = mm[i];
}
}
if (n > 1) {
System.out.println("Caveat: we have to perform more specific test. n == " + n);
System.out.println("class: " + cls + "\nname: " + name);
for (int i = 0 ; i < n ; ++i) {
System.out.println(mm[i]);
}
}
if (n > 0) {
m = mm[0];
}
return m;
}
Method checkMethod() called in findMethodInClass simply checks if name is correct and if arguments with which method will be called more or less match formal argument list. It's implementation is left as a simple exercise for the reader. Any comments?

java+spark: org.apache.spark.SparkException: Job aborted: Task not serializable: java.io.NotSerializableException

I'm new to spark, and was trying to run the example JavaSparkPi.java, it runs well, but because i have to use this in another java s I copy all things from main to a method in the class and try to call the method in main, it saids
org.apache.spark.SparkException: Job aborted: Task not serializable:
java.io.NotSerializableException
the code looks like this:
public class JavaSparkPi {
public void cal(){
JavaSparkContext jsc = new JavaSparkContext("local", "JavaLogQuery");
int slices = 2;
int n = 100000 * slices;
List<Integer> l = new ArrayList<Integer>(n);
for (int i = 0; i < n; i++) {
l.add(i);
}
JavaRDD<Integer> dataSet = jsc.parallelize(l, slices);
System.out.println("count is: "+ dataSet.count());
dataSet.foreach(new VoidFunction<Integer>(){
public void call(Integer i){
System.out.println(i);
}
});
int count = dataSet.map(new Function<Integer, Integer>() {
#Override
public Integer call(Integer integer) throws Exception {
double x = Math.random() * 2 - 1;
double y = Math.random() * 2 - 1;
return (x * x + y * y < 1) ? 1 : 0;
}
}).reduce(new Function2<Integer, Integer, Integer>() {
#Override
public Integer call(Integer integer, Integer integer2) throws Exception {
return integer + integer2;
}
});
System.out.println("Pi is roughly " + 4.0 * count / n);
}
public static void main(String[] args) throws Exception {
JavaSparkPi myClass = new JavaSparkPi();
myClass.cal();
}
}
anyone have idea on this? thanks!

The nested functions hold a reference to the containing object (JavaSparkPi). So this object will get serialized. For this to work, it needs to be serializable. Simple to do:
public class JavaSparkPi implements Serializable {
...

The main problem is that when you create an Anonymous Class in java it is passed a reference of the enclosing class.
This can be fixed in many ways
Declare the enclosing class Serializable
This works in your case but will fall flat in case your enclosing class has some field that is not serializable. I would also say that serializing the parent class is a total waste.
Create the Closure in a static function
Creating the closure by invoking some static function doesn't pass the reference to the closure and hence no need to make serializable this way.

This error comes because you have multiple physical CPUs in your local or cluster and spark engine try to send this function to multiple CPUs over network.
Your function
dataSet.foreach(new VoidFunction<Integer>(){
public void call(Integer i){
***System.out.println(i);***
}
});
uses println() which is not serialize. So the exception thrown by Spark Engine.
The solution is you can use below:
dataSet.collect().forEach(new VoidFunction<Integer>(){
public void call(Integer i){
System.out.println(i);
}
});

Big execution time difference between java Lambda vs Anonymous class

I was curious about performance of creation of java8 lambda instances against the same anonymous class. (Measurement performed on win32 java build 1.8.0-ea-b106). I've created very simple example and measured if java propose some optimization of new operator while create lambda expression:
static final int MEASURES = 1000000;
static interface ICallback{
void payload(int[] a);
}
/**
* force creation of anonymous class many times
*/
static void measureAnonymousClass(){
final int arr[] = {0};
for(int i = 0; i < MEASURES; ++i){
ICallback clb = new ICallback() {
#Override
public void payload(int[] a) {
a[0]++;
}
};
clb.payload(arr);
}
}
/**
* force creation of lambda many times
*/
static void measureLambda(){
final int arr[] = {0};
for(int i = 0; i < MEASURES; ++i){
ICallback clb = (a2) -> {
a2[0]++;
};
clb.payload(arr);
}
}
(Full code can be taken there: http://codepad.org/Iw0mkXhD) The result is rather predictable - lambda wins 2 times.
But really little shift to make closure shows very bad time for lambda. Anonymous class wins 10 times!
So now anonymous class looks like:
ICallback clb = new ICallback() {
#Override
public void payload() {
arr[0]++;
}
};
And lambda does as follow:
ICallback clb = () -> {
arr[0]++;
};
(Full code can be taken there: http://codepad.org/XYd9Umty )
Can anybody explain me why exists so big (bad) difference in handling of closure?

UPDATE
A few comments wondering if my benchmark at the bottom was flawed - after introducing a lot of randomness (to prevent the JIT from optimising too much stuff), I still get similar results so I tend to think it is ok.
In the meantime, I have come across this presentation by the lambda implementation team. Page 16 shows some performance figures: inner classes and closures have similar performance / non-capturing lambda are up to 5x times faster.
And #StuartMarks posted this JVMLS 2013 talk from Sergey Kuksenko on lambda performance. The bottom line is that post JIT compilation, lambdas and anonymous classes perform similarly on current Hostpot JVM implementations.
YOUR BENCHMARK
I have also run your test, as you posted it. The problem is that it runs for as little as 20 ms for the first method and 2 ms for the second. Although that is a 10:1 ratio, it is in no way representative because the measurement time is way too small.
I have then taken modified your test to allow for more JIT warmup and I get similar results as with jmh (i.e. no difference between anonymous class and lambda).
public class Main {
static interface ICallback {
void payload();
}
static void measureAnonymousClass() {
final int arr[] = {0};
ICallback clb = new ICallback() {
#Override
public void payload() {
arr[0]++;
}
};
clb.payload();
}
static void measureLambda() {
final int arr[] = {0};
ICallback clb = () -> {
arr[0]++;
};
clb.payload();
}
static void runTimed(String message, Runnable act) {
long start = System.nanoTime();
for (int i = 0; i < 10_000_000; i++) {
act.run();
}
long end = System.nanoTime();
System.out.println(message + ":" + (end - start));
}
public static void main(String[] args) {
runTimed("as lambdas", Main::measureLambda);
runTimed("anonymous class", Main::measureAnonymousClass);
runTimed("as lambdas", Main::measureLambda);
runTimed("anonymous class", Main::measureAnonymousClass);
runTimed("as lambdas", Main::measureLambda);
runTimed("anonymous class", Main::measureAnonymousClass);
runTimed("as lambdas", Main::measureLambda);
runTimed("anonymous class", Main::measureAnonymousClass);
}
}
The last run takes about 28 seconds for both methods.
JMH MICRO BENCHMARK
I have run the same test with jmh and the bottom line is that the four methods take as much time as the equivalent:
void baseline() {
arr[0]++;
}
In other words, the JIT inlines both the anonymous class and the lambda and they take exactly the same time.
Results summary:
Benchmark Mean Mean error Units
empty_method 1.104 0.043 nsec/op
baseline 2.105 0.038 nsec/op
anonymousWithArgs 2.107 0.028 nsec/op
anonymousWithoutArgs 2.120 0.044 nsec/op
lambdaWithArgs 2.116 0.027 nsec/op
lambdaWithoutArgs 2.103 0.017 nsec/op

Class.forName() caching

In one of my applications I am using the following:
public void calculate (String className)
{
...
Class clazz = Class.forName(className);
...
}
This function is called several times / second.
There are about 10 possible class names.
And while I do realize there is some internal caching inside this function,
I think this caching is only available on native level.
For this reason I am starting to wonder if I should add my own caching.
private static Map<String,Class> classMap;
public void calculate (String className)
{
...
Class clazz = classMap.get(className);
if (clazz == null)
{
clazz = Class.forName(className);
if (classMap == null) classMap = new HashMap<String, Class>(40);
classMap.put(className, clazz);
}
...
}
Will this be a performance gain or does it really make no difference ?
Thank you in advance

I wrote a little script to calculate the execution time of both functions.
This is the Main class that I used.
public class Benchmark
{
public static void main(String... pArgs)
{
// prepare all data as much as possible.
// we don't want to do this while the clock is running.
Class[] classes = {Object.class, Integer.class, String.class, Short.class, Long.class, Double.class,
Float.class, Boolean.class, Character.class, Byte.class};
int cycles = 1000000;
String[] classNames = new String[cycles];
for (int i = 0; i < cycles; i++)
{
classNames[i] = classes[i % classes.length].getName();
}
// THERE ARE 2 IMPLEMENTATIONS - CLASSIC vs CACHING
Implementation impl = new Caching(); // or Classic();
// Start the clocks !
long startTime = System.currentTimeMillis();
for (int i = 0; i < cycles; i++)
{
impl.doStuff(classNames[i]);
}
long endTime = System.currentTimeMillis();
// calculate and display result
long totalTime = endTime - startTime;
System.out.println(totalTime);
}
}
Here is the classic implementation that uses Class.forName
private interface Implementation
{
Class doStuff(String clzName);
}
private static class Classic implements Implementation
{
#Override
public Class doStuff(String clzName)
{
try
{
return Class.forName(clzName);
}
catch (Exception e)
{
return null;
}
}
}
Here is the second implementation that uses a HashMap to cache the Class objects.
private static class Caching implements Implementation
{
private Map<String, Class> cache = new HashMap<String, Class>();
#Override
public Class doStuff(String clzName)
{
Class clz = cache.get(clzName);
if (clz != null) return clz;
try
{
clz = Class.forName(clzName);
cache.put(clzName, clz);
}
catch (Exception e)
{
}
return clz;
}
}
The results:
1100 ms without caching.
only 15 ms with caching.
Conclusion:
Is it a significant difference --> yes !
Does it matter for my application --> not at all.

Will this be a performance gain or does it really make no difference?
I would be astonished if it made a significant difference - and if you're only calling it "several times per second" (rather than, say, a million) it's really not worth optimizing.
You should at least try this in isolation in a benchmark before committing to this more complicated design. I would strongly expect Class.forName to be caching this anyway, and adding more complexity into your app does no good.

Class.forName() does two things:
it fetches a loaded class from the classloader
if no such class is found, it tries to load it.
Part #1 is pretty quick. #2 is where the real work starts (where the JVM might hit the hard disk or even the network, depending on the classloader). And if you pass the same parameters in, then all but the first invocations will never get to step #2.
So no: it's probably not worth optimizing.

No you shouldn't. Class.forName will not load the same class twice but will try to find the class among the loaded classes. It's done at native level and supposed to be very efficient.

Why are interface method invocations slower than concrete invocations?

This is question comes in mind when I finding difference between abstract class and interface.
In this post I came to know that interfaces are slow as they required extra indirection.
But I am not getting what type of indirection required by the interface and not by the abstract class or concrete class.Please clarify on it.
Thanks in advance

There are many performance myths, and some were probably true several years ago, and some might still be true on VMs that don't have a JIT.
The Android documentation (remember that Android don't have a JVM, they have Dalvik VM) used to say that invoking a method on an interfaces was slower than invoking it on a class, so they were contributing to spreading the myth (it's also possible that it was slower on the Dalvik VM before they turned on the JIT). The documentation does now say:
Performance Myths
Previous versions of this document made various misleading claims. We
address some of them here.
On devices without a JIT, it is true that invoking methods via a
variable with an exact type rather than an interface is slightly more
efficient. (So, for example, it was cheaper to invoke methods on a
HashMap map than a Map map, even though in both cases the map was a
HashMap.) It was not the case that this was 2x slower; the actual
difference was more like 6% slower. Furthermore, the JIT makes the two
effectively indistinguishable.
Source: Designing for performance on Android
The same thing is probably true for the JIT in the JVM, it would be very odd otherwise.

If in doubt, measure it. My results showed no significant difference. When run, the following program produced:
7421714 (abstract)
5840702 (interface)
7621523 (abstract)
5929049 (interface)
But when I switched the places of the two loops:
7887080 (interface)
5573605 (abstract)
7986213 (interface)
5609046 (abstract)
It appears that abstract classes are slightly (~6%) faster, but that should not be noticeable; These are nanoseconds. 7887080 nanoseconds are ~7 milliseconds. That makes it a difference of 0.1 millis per 40k invocations (Java version: 1.6.20)
Here's the code:
public class ClassTest {
public static void main(String[] args) {
Random random = new Random();
List<Foo> foos = new ArrayList<Foo>(40000);
List<Bar> bars = new ArrayList<Bar>(40000);
for (int i = 0; i < 40000; i++) {
foos.add(random.nextBoolean() ? new Foo1Impl() : new Foo2Impl());
bars.add(random.nextBoolean() ? new Bar1Impl() : new Bar2Impl());
}
long start = System.nanoTime();
for (Foo foo : foos) {
foo.foo();
}
System.out.println(System.nanoTime() - start);
start = System.nanoTime();
for (Bar bar : bars) {
bar.bar();
}
System.out.println(System.nanoTime() - start);
}
abstract static class Foo {
public abstract int foo();
}
static interface Bar {
int bar();
}
static class Foo1Impl extends Foo {
#Override
public int foo() {
int i = 10;
i++;
return i;
}
}
static class Foo2Impl extends Foo {
#Override
public int foo() {
int i = 10;
i++;
return i;
}
}
static class Bar1Impl implements Bar {
#Override
public int bar() {
int i = 10;
i++;
return i;
}
}
static class Bar2Impl implements Bar {
#Override
public int bar() {
int i = 10;
i++;
return i;
}
}
}

An object has a "vtable pointer" of some kind which points to a "vtable" (method pointer table) for its class ("vtable" might be the wrong terminology, but that's not important). The vtable has pointers to all the method implementations; each method has an index which corresponds to a table entry. So, to call a class method, you just look up the corresponding method (using its index) in the vtable. If one class extends another, it just has a longer vtable with more entries; calling a method from the base class still uses the same procedure: that is, look up the method by its index.
However, in calling a method from an interface via an interface reference, there must be some alternative mechanism to find the method implementation pointer. Because a class can implement multiple interfaces, it's not possible for the method to always have the same index in the vtable (for instance). There are various possible ways to resolve this, but no way that is quite as efficient as simple vtable dispatch.
However, as mentioned in the comments, it probably won't make much difference with a modern Java VM implementation.

This is variation on Bozho example. It runs longer and re-uses the same objects so the cache size doesn't matter so much. I also use an array so there is no overhead from the iterator.
public static void main(String[] args) {
Random random = new Random();
int testLength = 200 * 1000 * 1000;
Foo[] foos = new Foo[testLength];
Bar[] bars = new Bar[testLength];
Foo1Impl foo1 = new Foo1Impl();
Foo2Impl foo2 = new Foo2Impl();
Bar1Impl bar1 = new Bar1Impl();
Bar2Impl bar2 = new Bar2Impl();
for (int i = 0; i < testLength; i++) {
boolean flip = random.nextBoolean();
foos[i] = flip ? foo1 : foo2;
bars[i] = flip ? bar1 : bar2;
}
long start;
start = System.nanoTime();
for (Foo foo : foos) {
foo.foo();
}
System.out.printf("The average abstract method call was %.1f ns%n", (double) (System.nanoTime() - start) / testLength);
start = System.nanoTime();
for (Bar bar : bars) {
bar.bar();
}
System.out.printf("The average interface method call was %.1f ns%n", (double) (System.nanoTime() - start) / testLength);
}
prints
The average abstract method call was 4.2 ns
The average interface method call was 4.1 ns
if you swap the order the tests are run you get
The average interface method call was 4.2 ns
The average abstract method call was 4.1 ns
There is more difference in how you run the test than which one you chose.
I got the same result with Java 6 update 26 and OpenJDK 7.
BTW: If you add a loop which only call the same object each time, you get
The direct method call was 2.2 ns

I tried to write a test that would quantify all of the various ways methods might be invoked. My findings show that it is not whether a method is an interface method or not that matters, but rather the type of the reference through which you are calling it. Calling an interface method through a class reference is much faster (relative to the number of calls) than calling the same method on the same class via an interface reference.
The results for 1,000,000 calls are...
interface method via interface reference: (nanos, millis) 5172161.0, 5.0
interface method via abstract reference: (nanos, millis) 1893732.0, 1.8
interface method via toplevel derived reference: (nanos, millis) 1841659.0, 1.8
Concrete method via concrete class reference: (nanos, millis) 1822885.0, 1.8
Note that the first two lines of the results are calls to the exact same method, but via different references.
And here is the code...
package interfacetest;
/**
*
* #author rpbarbat
*/
public class InterfaceTest
{
static public interface ITest
{
public int getFirstValue();
public int getSecondValue();
}
static abstract public class ATest implements ITest
{
int first = 0;
#Override
public int getFirstValue()
{
return first++;
}
}
static public class TestImpl extends ATest
{
int second = 0;
#Override
public int getSecondValue()
{
return second++;
}
}
static public class Test
{
int value = 0;
public int getConcreteValue()
{
return value++;
}
}
static int loops = 1000000;
/**
* #param args the command line arguments
*/
public static void main(String[] args)
{
// Get some various pointers to the test classes
// To Interface
ITest iTest = new TestImpl();
// To abstract base
ATest aTest = new TestImpl();
// To impl
TestImpl testImpl = new TestImpl();
// To concrete
Test test = new Test();
System.out.println("Method call timings - " + loops + " loops");
StopWatch stopWatch = new StopWatch();
// Call interface method via interface reference
stopWatch.start();
for (int i = 0; i < loops; i++)
{
iTest.getFirstValue();
}
stopWatch.stop();
System.out.println("interface method via interface reference: (nanos, millis)" + stopWatch.getElapsedNanos() + ", " + stopWatch.getElapsedMillis());
// Call interface method via abstract reference
stopWatch.start();
for (int i = 0; i < loops; i++)
{
aTest.getFirstValue();
}
stopWatch.stop();
System.out.println("interface method via abstract reference: (nanos, millis)" + stopWatch.getElapsedNanos() + ", " + stopWatch.getElapsedMillis());
// Call derived interface via derived reference
stopWatch.start();
for (int i = 0; i < loops; i++)
{
testImpl.getSecondValue();
}
stopWatch.stop();
System.out.println("interface via toplevel derived reference: (nanos, millis)" + stopWatch.getElapsedNanos() + ", " + stopWatch.getElapsedMillis());
// Call concrete method in concrete class
stopWatch.start();
for (int i = 0; i < loops; i++)
{
test.getConcreteValue();
}
stopWatch.stop();
System.out.println("Concrete method via concrete class reference: (nanos, millis)" + stopWatch.getElapsedNanos() + ", " + stopWatch.getElapsedMillis());
}
}
package interfacetest;
/**
*
* #author rpbarbat
*/
public class StopWatch
{
private long start;
private long stop;
public StopWatch()
{
start = 0;
stop = 0;
}
public void start()
{
stop = 0;
start = System.nanoTime();
}
public void stop()
{
stop = System.nanoTime();
}
public float getElapsedNanos()
{
return (stop - start);
}
public float getElapsedMillis()
{
return (stop - start) / 1000;
}
public float getElapsedSeconds()
{
return (stop - start) / 1000000000;
}
}
This was using the Oracles JDK 1.6_24. Hope this helps put this question to bed...
Regards,
Rodney Barbati

Interfaces are slower than abstract class as run time decision of method invocation would add little penalty of time,
However as JIT comes in picture which will take care of repeated calls of same method hence you may see the performance lag only in first call which is also very minimal,
Now for Java 8, they almost made abstract class useless by adding default & static function,