Develop Reference

Using scala's ParHashMap in Java's project instead of ConcurrentHashMap

I've got a fairly complicated project, which heavily uses Java's multithreading. In an answer to one of my previous questions I have described an ugly hack, which is supposed to overcome inherent inability to iterate over Java's ConcurrentHashMap in parallel. Although it works, I don't like ugly hacks, and I've had a lot of trouble trying to introduce proposed proof of concept in the real system. Trying to find an alternative solution I have encountered Scala's ParHashMap, which claims to implement a foreach method, which seems to operate in parallel. Before I start learning a new language to implement a single feature I'd like to ask the following:
1) Is foreach method of Scala's ParHashMap scalable?
2) Is it simple and straightforward to call Java's code from Scala and vice versa? I'll just remind that the code is concurrent and uses generics.
3) Is there going to be a performance penalty for switching a part of codebase to Scala?
For reference, this is my previous question about parallel iteration of ConcurrentHashMap:
Scalable way to access every element of ConcurrentHashMap<Element, Boolean> exactly once
EDIT
I have implemented the proof of concept, in probably very non-idiomatic Scala, but it works just fine. AFAIK it is IMPOSSIBLE to implement a corresponding solution in Java given the current state of its standard library and any available third-party libraries.
import scala.collection.parallel.mutable.ParHashMap
class Node(value: Int, id: Int){
var v = value
var i = id
override def toString(): String = v toString
}
object testParHashMap{
def visit(entry: Tuple2[Int, Node]){
entry._2.v += 1
}
def main(args: Array[String]){
val hm = new ParHashMap[Int, Node]()
for (i <- 1 to 10){
var node = new Node(0, i)
hm.put(node.i, node)
}
println("========== BEFORE ==========")
hm.foreach{println}
hm.foreach{visit}
println("========== AFTER ==========")
hm.foreach{println}
}
}

I come to this with some caveats:
Though I can do some things, I consider myself relatively new to Scala.
I have only read about but never used the par stuff described here.
I have never tried to accomplish what you are trying to accomplish.
If you still care what I have to say, read on.
First, here is an academic paper describing how the parallel collections work.
On to your questions.
1) When it comes to multi-threading, Scala makes life so much easier than Java. The abstractions are just awesome. The ParHashMap you get from a par call will distribute the work to multiple threads. I can't say how that will scale for you without a better understanding of your machine, configuration, and use case, but done right (particularly with regard to side effects) it will be at least as good as a Java implementation. However, you might also want to look at Akka to have more control over everything. It sounds like that might be more suitable to your use case than simply ParHashMap.
2) It is generally simple to convert between Java and Scala collections using JavaConverters and the asJava and asScala methods. I would suggest though making sure that the public API for your method calls "looks Java" since Java is the least common denominator. Besides, in this scenario, Scala is an implementation detail, and you never want to leak those anyway. So keep the abstraction at a Java level.
3) I would guess there will actually be a performance gain with Scala--at runtime. However, you will find much slower compile time (which can be worked around. ish). This Stack Overflow post by the author of Scala is old but still relevant.
Hope that helps. That's quite a problem you got there.

Since Scala compiles to the same bytecode as Java, doing the same in both languages is very well possible, no matter the task. There are however some things which are easier to solve in Scala, but if this is worth learning a new language is a different question. Especially since Java 8 will include exactly what you ask for: simple parallel execution of functions on lists.
But even now you can do this in Java, you just need to write what Scala already has on your own.
final ExecutorService executor = Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors());
//...
final Entry<String, String>[] elements = (Entry<String, String>[]) myMap.entrySet().toArray();
final AtomicInteger index = new AtomicInteger(elements.length);
for (int i = Runtime.getRuntime().availableProcessors(); i > 0; --i) {
executor.submit(new Runnable() {
public void run() {
int myIndex;
while ((myIndex = index.decrementAndGet()) >= 0) {
process(elements[myIndex]);
}
}
});
}
The trick is to pull those elements into a temporary array, so threads can take out elements in a thread-safe way. Obviously doing some caching here instead of re-creating the Runnables and the array each time is encouraged, because the Runnable creation might already take longer than the actual task.
It is as well possible to instead copy the elements into a (reusable) LinkedBlockingQueue, then have the threads poll/take on it instead. This however adds more overhead and is only reasonable for tasks that require at least some calculation time.
I don't know how Scala actually works, but given the fact that it needs to run on the same JVM, it will do something similar in the background, it just happens to be easily accessible in the standard library.

Will java compiler optimize a method call on a final static variable ? And what happens when it becomes dalvik code?

I have the following code
float square(float val) { return val*val;}
boolean isInCircle(final float x,final float y) {
float squareDistance = square(cx - x) + square(cy - y);
return squareDistance < square(RADIUS);
}
where RADIUS is a static final float.
Will the java compiler optimize the call square(RADIUS) ?
What happens when this converted to dalvik code for android ? Will it remain optimized ?

The Java compiler won't do anything with that code.
The HotSpot JVM will almost certainly precompute square(RADIUS).
Android doesn't have that particular JVM.
I personally wouldn't write the square() method at all, just return (cx-x)*(cx-x)+(cy-y)*(cy-y) < RADIUS*RADIUS; and let the compiler/JVM battle it out from there. But then I'm a mathematician ;-)

The Dalvik JIT compiler does in fact inline short functions, such as square() as defined in the question (though probably better to declare it static). However, I couldn't tell you off-hand whether it would definitely get inlined.
Profile the code if it matters!

Optimizations in Java are done (as far as I know) by the HotSpot compiler at runtime (bytecode is optimized when translated to machine code). So the answer is yes and no.
The transformed code will be equally optimized, but it depends on JVM, what will do with it. According my experience, it is highly dependent on the JVM and probably in its setting (agressivity of the optimizer). I have tried to compare running of SHA1 with loops and without on Windows JVM and Linux one. In one case the code without loops was many times faster, in the second (I think on Linux) there was only a difference about 40% of the time taken...
So it is a magic, you might give HotSpot good hints to optimize, or configure JVM, but still, it will depend on the current algorithm of JVM...

The only optimization that will happen is that the compiler will most likely "intern" the value of the static final field as a constant where it is accessed, rather than performing a field lookup at runtime.

Why does Scala implement for as a closure?

Recent events on the blogosphere have indicated that a possible performance problem with Scala is its use of closures to implement for.
What are the reasons for this design decision, as opposed to a C or Java-style "primitive for" - that is one which will be turned into a simple loop?
(I'm making a distinction between Java's for and its "foreach" construct here, as the latter involves an implicit Iterator).
More detail, following up from Peter. This bit of Scala:
object ScratchFor {
def main(args : Array[String]) : Unit = {
for (val s <- args) {
println(s)
}
}
}
creates 3 classes: ScratchFor$$anonfun$main$1.class ScratchFor$.class ScratchFor.class
ScratchFor::main just forwards to the companion object, ScratchFor$.MODULE$::main which spins up an ScratchFor$$anonfun$main$1 (which is an implementation of AbstractFunction1).
It's in the apply() method of this anonymous inner impl of AbstractFunction1 that the actual code lives, which is effectively the loop body.
I don't see HotSpot being able to rewrite this into a simple loop. Happy to be proved wrong on this, though.

Traditional for loops are clumsy, verbose and error-prone. I think it is proof enough of this that "for-each" loops where added to Java, C# and C++, but if you want more details you may check item 46 of Effective Java.
Now, for-each loops are still much faster than Scala for-comprehension, but they are also much less powerful (and more clumsy) because they cannot return values. If you want to transform or filter a collection (or do both to a group of collections), you'll still have to handle all the mechanical details of constructing the result collection in addition to computing the values. Not to mention it inevitably uses some mutable state.
Finally, even though for-each loops are adequate enough for collections, they are not suited to other monadic classes (of which collections are a subset of).
So Scala has a general method which takes care of all of the above. Yes, it is slower, but the goal is to have the compiler effectively optimise it well enough so that this doesn't become a hindrance (and, of course, JIT could help here as well).
That has not been accomplished to this date, but -optimise has reduced a lot of ground between common for-each loops and for-comprehensions on the latest versions of Scala. If performance is essential, you can always use while or tail recursion.
Now, it would be possibly for Scala to have common for loops or for-each loops as special cases specifically targeted at performance issues (since for-comprehensions can do everything they do). However, that violates two principles that guide Scala's design:
Reduce complexity. Yes, contrary to what some say, that is a design goal, and special cases that serve no other purpose other than optimise performance -- even though a workable solution exists for performance cases -- would needlessly increase the complexity of the language.
Scalability. This is in the sense that the use can scale the language for any size of problem by writing libraries. The point here is that having the compiler optimise one particular class, such as Range, would make it impossible for the user to create a replacement class that would perform just as well.

The for comprehension in Scala is a powerful general-purpose looping and pattern-matching construct. Look at what it can do:
case class Person(first: String, last: String) {}
val people = List(Person("Isaac","Newton"), Person("Michael","Jordan"))
val lastfirst = for (Person(f,l) <- people) yield l+", "+f
for (n <- lastfirst) println(n)
The second case looks pretty straightforward--take each item in a collection and print it. But the first takes apart a list containing a custom data structure and transforms it into a different collection type!
The first for there highlights only a small portion of the capability of the construct; it is both extremely powerful and extremely general. In order to maintain this power, the for must be able to turn into something very general, which means closures. Then the question is: do you also introduce special cases that operate on known collections in simple ways with improved performance? The answer thus far has been mostly no, instead preferring solutions that optimize the general closure-taking methods that for turns into.
Whether this is useful for you in particular depends on whether you are using the general capabilities a lot (in which case you will be glad) or not (in which case you may wish progress was faster).
Still, try -optimize. It often usefully speeds up simple for-comprehensions these days.

The for-comprehension is much more than a simple loop.
If you need an imperative loop, use while. If you want to write performant code in Scala, you need to know this. Just like you have to know about language implementation when you want to write fast code in every other language.
So, since the for-comprehension is not a simple loop, I hope you understand that it's not compiled down to a simple loop.

I would assume using a closure is a general solution. A more optimal solution in some cases would be to "inline" the closure as a loop and eliminate the need to create an object. Perhaps the Scala designers feel the JIT should do this, rather having the compiler do this.
Let's say in Java this is the same as writing
public static void main(String... args) {
for_loop(args, new Function<String>() {
public void apply(String s) {
System.out.println(s);
}
});
}
interface Function<T> {
void apply(T s);
}
public static <T> void for_loop(T... ts, Function<T> tFunc) {
for(T t: ts) tFunc.apply(t);
}
This is fairly easy to inline (if you're a human). What is surprising is that Scala doesn't have an intrinsic to perform the optimisation to eliminate the need for a new object. Certainly the JIT could do it in theory, but in practise, it might be a while before it handles this specific case.

I'm surprised that no one has mentioned one of the pitfalls you can get into if for does not create a closure.
In Python for example:
ls = [None] * 3
for i in [0, 1, 2]:
ls[i] = lambda: i
print(ls[0]())
print(ls[1]())
print(ls[2]())
This prints 2 2 2, because i has a longer lifetime than the for loop. I run into this trap all the time in Python and R.
So even in the very simplest of cases, it is important for for in Scala to be implemented using an anonymous function, because it creates an environment to store variables.

Complexity of Java 7's current Lambda proposal? (August 2010)

Some people say that every programming language has its "complexity budget" which it can use to accomplish its purpose. But if the complexity budget is depleted, every minor change becomes increasingly complicated and hard to implement in a backward-compatible way.
After reading the current provisional syntax for Lambda (≙ Lambda expressions, exception transparency, defender methods and method references) from August 2010 I wonder if people at Oracle completely ignored Java's complexity budget when considering such changes.
These are the questions I'm thinking about - some of them more about language design in general:
Are the proposed additions comparable in complexity to approaches other languages chose?
Is it generally possible to add such additions to a language and protecting the developer from the complexity of the implementation ?
Are these additions a sign of reaching the end of the evolution of Java-as-a-language or is this expected when changing a language with a huge history?
Have other languages taken a totally different approach at this point of language evolution?
Thanks!

I have not followed the process and evolution of the Java 7 lambda
proposal, I am not even sure of what the latest proposal wording is.
Consider this as a rant/opinion rather than statements of truth. Also,
I have not used Java for ages, so the syntax might be rusty and
incorrect at places.
First, what are lambdas to the Java language? Syntactic sugar. While
in general lambdas enable code to create small function objects in
place, that support was already preset --to some extent-- in the Java
language through the use of inner classes.
So how much better is the syntax of lambdas? Where does it outperform
previous language constructs? Where could it be better?
For starters, I dislike the fact that there are two available syntax
for lambda functions (but this goes in the line of C#, so I guess my
opinion is not widespread. I guess if we want to sugar coat, then
#(int x)(x*x) is sweeter than #(int x){ return x*x; } even if the
double syntax does not add anything else. I would have preferred the
second syntax, more generic at the extra cost of writting return and
; in the short versions.
To be really useful, lambdas can take variables from the scope in
where they are defined and from a closure. Being consistent with
Inner classes, lambdas are restricted to capturing 'effectively
final' variables. Consistency with the previous features of the
language is a nice feature, but for sweetness, it would be nice to be
able to capture variables that can be reassigned. For that purpose,
they are considering that variables present in the context and
annotated with #Shared will be captured by-reference, allowing
assignments. To me this seems weird as how a lambda can use a variable
is determined at the place of declaration of the variable rather than
where the lambda is defined. A single variable could be used in more
than one lambda and this forces the same behavior in all of them.
Lambdas try to simulate actual function objects, but the proposal does
not get completely there: to keep the parser simple, since up to now
an identifier denotes either an object or a method that has been kept
consistent and calling a lambda requires using a ! after the lambda
name: #(int x)(x*x)!(5) will return 25. This brings a new syntax
to use for lambdas that differ from the rest of the language, where
! stands somehow as a synonim for .execute on a virtual generic
interface Lambda<Result,Args...> but, why not make it complete?
A new generic (virtual) interface Lambda could be created. It would
have to be virtual as the interface is not a real interface, but a
family of such: Lambda<Return>, Lambda<Return,Arg1>,
Lambda<Return,Arg1,Arg2>... They could define a single execution
method, which I would like to be like C++ operator(), but if that is
a burden then any other name would be fine, embracing the ! as a
shortcut for the method execution:
interface Lambda<R> {
R exec();
}
interface Lambda<R,A> {
R exec( A a );
}
Then the compiler need only translate identifier!(args) to
identifier.exec( args ), which is simple. The translation of the
lambda syntax would require the compiler to identify the proper
interface being implemented and could be matched as:
#( int x )(x *x)
// translated to
new Lambda<int,int>{ int exec( int x ) { return x*x; } }
This would also allow users to define Inner classes that can be used
as lambdas, in more complex situations. For example, if lambda
function needed to capture a variable annotated as #Shared in a
read-only manner, or maintain the state of the captured object at the
place of capture, manual implementation of the Lambda would be
available:
new Lambda<int,int>{ int value = context_value;
int exec( int x ) { return x * context_value; }
};
In a manner similar to what the current Inner classes definition is,
and thus being natural to current Java users. This could be used,
for example, in a loop to generate multiplier lambdas:
Lambda<int,int> array[10] = new Lambda<int,int>[10]();
for (int i = 0; i < 10; ++i ) {
array[i] = new Lambda<int,int>{ final int multiplier = i;
int exec( int x ) { return x * multiplier; }
};
}
// note this is disallowed in the current proposal, as `i` is
// not effectively final and as such cannot be 'captured'. Also
// if `i` was marked #Shared, then all the lambdas would share
// the same `i` as the loop and thus would produce the same
// result: multiply by 10 --probably quite unexpectedly.
//
// I am aware that this can be rewritten as:
// for (int ii = 0; ii < 10; ++ii ) { final int i = ii; ...
//
// but that is not simplifying the system, just pushing the
// complexity outside of the lambda.
This would allow usage of lambdas and methods that accept lambdas both
with the new simple syntax: #(int x){ return x*x; } or with the more
complex manual approach for specific cases where the sugar coating
interferes with the intended semantics.
Overall, I believe that the lambda proposal can be improved in
different directions, that the way it adds syntactic sugar is a
leaking abstraction (you have deal externally with issues that are
particular to the lambda) and that by not providing a lower level
interface it makes user code less readable in use cases that do not
perfectly fit the simple use case.
:

Modulo some scope-disambiguation constructs, almost all of these methods follow from the actual definition of a lambda abstraction:
λx.E
To answer your questions in order:
I don't think there are any particular things that make the proposals by the Java community better or worse than anything else. As I said, it follows from the mathematical definition, and therefore all faithful implementations are going to have almost exactly the same form.
Anonymous first-class functions bolted onto imperative languages tend to end up as a feature that some programmers love and use frequently, and that others ignore completely - therefore it is probably a sensible choice to give it some syntax that will not confuse the kinds of people who choose to ignore the presence of this particular language feature. I think hiding the complexity and particulars of implementation is what they have attempted to do by using syntax that blends well with Java, but which has no real connotation for Java programmers.
It's probably desirable for them to use some bits of syntax that are not going to complicate existing definitions, and so they are slightly constrained in the symbols they can choose to use as operators and such. Certainly Java's insistence on remaining backwards-compatible limits the language evolution slightly, but I don't think this is necessarily a bad thing. The PHP approach is at the other end of the spectrum (i.e. "let's break everything every time there is a new point release!"). I don't think that Java's evolution is inherently limited except by some of the fundamental tenets of its design - e.g. adherence to OOP principles, VM-based.
I think it's very difficult to make strong statements about language evolution from Java's perspective. It is in a reasonably unique position. For one, it's very, very popular, but it's relatively old. Microsoft had the benefit of at least 10 years worth of Java legacy before they decided to even start designing a language called "C#". The C programming language basically stopped evolving at all. C++ has had few significant changes that found any mainstream acceptance. Java has continued to evolve through a slow but consistent process - if anything I think it is better-equipped to keep on evolving than any other languages with similarly huge installed code bases.

It's not much more complicated then lambda expressions in other languages.
Consider...
int square(x) {
return x*x;
}
Java:
#(x){x*x}
Python:
lambda x:x*x
C#:
x => x*x
I think the C# approach is slightly more intuitive. Personally I would prefer...
x#x*x

Maybe this is not really an answer to your question, but this may be comparable to the way objective-c (which of course has a very narrow user base in contrast to Java) was extended by blocks (examples). While the syntax does not fit the rest of the language (IMHO), it is a useful addition and and the added complexity in terms of language features is rewarded for example with lower complexity of concurrent programming (simple things like concurrent iteration over an array or complicated techniques like Grand Central Dispatch).
In addition, many common tasks are simpler when using blocks, for example making one object a delegate (or - in Java lingo - "listener") for multiple instances of the same class. In Java, anonymous classes can already be used for that cause, so programmers know the concept and can just spare a few lines of source code using lambda expressions.
In objective-c (or the Cocoa/Cocoa Touch frameworks), new functionality is now often only accessible using blocks, and it seems like programmers are adopting it quickly (given that they have to give up backwards compatibility with old OS versions).

This is really really close to Lambda functions proposed in the new generation of C++ (C++0x)
so I think, Oracle guys have looked at the other implementations before cooking up their own.
http://en.wikipedia.org/wiki/C%2B%2B0x
[](int x, int y) { return x + y; }

Automatic Java to C++ conversion [duplicate]

This question already has answers here:
Does a Java to C++ converter/tool exist? [closed]
(10 answers)
Closed 7 years ago.
Has anyone tried automatic Java to C++ conversion for speed improvements? Is it a maintenance nightmare in the long run?
Just read that is used to generate the HTML5 parsing engine in Gecko http://ejohn.org/blog/html-5-parsing/

In general, automatic conversions from one language to another will not be an improvement. Different languages have different idioms that affect performance.
The simplest example is with loops and variable creation. In a Java GC world, creating objects with new is almost free, and they dive into oblivion just as easily. In C++ memory allocation is (generally speaking) expensive:
// Sample java code
for ( int i = 0; i < 10000000; ++i )
{
String str = new String( "hi" ); // new is free, GC is almost free for young objects
}
Direct conversion to C++ will result in bad performance (use of TR1 shared_ptr as memory handler instead of GC):
for ( int i = 0; i < 10000000; ++i )
{
std::shared_ptr< std::string > str( new std::string( "hi" ) );
}
The equivalent loop written in C++ would be:
for ( int i = 0; i < 10000000; ++i )
{
std::string str( "hi" );
}
Direct translation from a language to another usually ends with the worst of both worlds and harder to maintain code.

The positive point to that conversion is that you will need a proper object oriented design in order to switch from java to C++ (paradigm intersection).
However some people say that coding C++ does not bring speed improvement compared to java code.

Even if that worked, I am not so sure that you will see much speed improvement. Java's Hotspot JIT compiler has become pretty good.

It's nearly impossible to replace the automatic memory-management of Java with a manual memory-management via a program. So you most likely will end up with a program, that has memory leaks or with C++-code that uses a garbage-collector. But a garbage-collector in Java have much more things to rely on (for instance no pointer-arithm,etics), so a garbage-collector in C++ to be safe it has a decreased performance. So your automatic conversion will most likely decrease performance.
Instead try to port it by hand to C++ or optimize the Java-code.

There is hardly a chance that this type of conversion shell lead to better performances. Usually when the JVM is working, it converts most of the code to native machine code. what you are suggesting is converting the Jave code to C++ and from there to native machine code, that is to add an extra phase.
However there are some trivial cases in which some gain may be achived due to the fact that:
1) It takes some time to load the JVM from scratch.
2) It takes some time to do the Jit, when running a very short program, a lot of times, you may prefer to waste this time before running.
3) You may not got the same level of machine code on Java, if you are not running on Server mode. (On server Mode you are expected to get top notch machine code, and also one that is best suited for your own CPU as detected on run-time, that is usually lacking on most C/C++ implantations of programs, and moreover a machine code optimized at run-time)

The languages have such different styles in usage that a mindless conversion would be of little use. An intelligent converter would be next to imposable to write because of the different styles used.
Some Problem areas:
Resource allocation is controlled by 'try{} finally{}' blocks in Java while C++ uses RAII.
Java does exception checking at compile time C++ run-time.
Exceptions are handled differently. Two exceptions:
In Java (last throw is propagated)
In C++ application terminated.
Java has a massive standard library
C++ has all the same functionality you just need to go find it on the web [which is a pain].
Java uses pointers for everything.
A straight unthinking conversion will leave you with a program consisting of just shared_ptr objects.
Anyway with JIT Compilation Java is comparable to C++ in speed.

Speaking of such converters in general, they can't be expected to produce either maintainable or high-performance code, since those generally require having an actual human who understands the language writing the code. What they are very useful for is in making it easy to port languages. Any language with a converter to C, for example, can be swiftly implemented on a wide range of languages. I used f2c in the mid-90s to run a Fortran routine on a Macintosh.
You may or may not have performance improvements if you rewrite the code in C++. You'll probably get a slowdown if you use an automatic converter.