Develop Reference

parsing a Python expression from Java

I've got a bit of an interesting challenge
To the point:
I want to allow a user to enter an expression in a text field, and have that string treated as a python expression. There are a number of local variables I would like to make available to this expression.
I do have a solution though it will be cumbersome to implement. I was thinking of keeping a Python class source file, with a function that has a single %s in it. When the user enters his expression, we simply do a string format, and then call Jython's interpreter, to spit out something we can execute. There would have to be a number of variable declaration statements in front of that expression to make sure the variables we want to expose to the user for his expression.
So the user would be presented with a text field, he would enter
x1 + (3.5*x2) ** x3
and we would do our interpreting process to come up with an open delegate object. We then punch the values into this object from a map, and call execute, to get the result of the expression.
Any objections to using Jython, or should I be doing something other than modifying source code? I would like to think that some kind of mutable object akin to C#'s Expression object, where we could do something like
PythonExpression expr = new PythonExpression(userSuppliedText)
expr.setDefaultNamespace();
expr.loadLibraries("numPy", /*other libraries?*/);
//comes from somewhere else in the flow, but effectively we get
Map<String, Double> symbolValuesByName = new HashMap<>(){{
put("x1", 3.0);
put("x2", 20.0);
put("x3", 2.0);
}};
expr.loadSymbols(symbolValuesByName);
Runnable exprDelegate = expr.compile();
//sometime later
exprDelegate.run();
but, I'm hoping for a lot, and it looks like Jython is as good as it gets. Still, modifying source files and then passing them to an interpreter seems really heavy-handed.
Does that sound like a good approach? Do you guys have any other libraries you'd suggest?
Update: NumPy does not work with Jython
I should've discovered this one on my own.
So now my question shifts: Is there any way that from a single JVM process instance (meaning, without ever having to fork) I can compile and run some Python code?

If you simply want to parse the expressions, you ought to be able to put something together with a Java parser generator.
If you want to parse, error check and evaluate the expressions, then you will need a substantial subset of the functionality a full Python interpreter.
I'm not aware of a subset implementation.
If such a subset implementation exists, it is unclear that it would be any easier to embed / call than to use a full Python interpreter ... like Jython.
If the powers that be dictate that "thou shalt use python", then they need to pay for the extra work it is going to cause you ... and the next guy who is going to need to maintain a hybrid system across changes in requirements, and updates to the Java and Python / Jython ecosystems. Factor it into the project estimates.
The other approach would be to parse the full python expression grammar, but limit what your evalutor can handle ... based on what it actually required, and what is implementable in your project's time-frame. Limit the types supported and the operations on the types. Limit the built-in functions supported. Etcetera.
Assuming that you go down the Java calling Jython route, there is a lot of material on how to implement it here: http://www.jython.org/jythonbook/en/1.0/JythonAndJavaIntegration.html

Using scala's ParHashMap in Java's project instead of ConcurrentHashMap

I've got a fairly complicated project, which heavily uses Java's multithreading. In an answer to one of my previous questions I have described an ugly hack, which is supposed to overcome inherent inability to iterate over Java's ConcurrentHashMap in parallel. Although it works, I don't like ugly hacks, and I've had a lot of trouble trying to introduce proposed proof of concept in the real system. Trying to find an alternative solution I have encountered Scala's ParHashMap, which claims to implement a foreach method, which seems to operate in parallel. Before I start learning a new language to implement a single feature I'd like to ask the following:
1) Is foreach method of Scala's ParHashMap scalable?
2) Is it simple and straightforward to call Java's code from Scala and vice versa? I'll just remind that the code is concurrent and uses generics.
3) Is there going to be a performance penalty for switching a part of codebase to Scala?
For reference, this is my previous question about parallel iteration of ConcurrentHashMap:
Scalable way to access every element of ConcurrentHashMap<Element, Boolean> exactly once
EDIT
I have implemented the proof of concept, in probably very non-idiomatic Scala, but it works just fine. AFAIK it is IMPOSSIBLE to implement a corresponding solution in Java given the current state of its standard library and any available third-party libraries.
import scala.collection.parallel.mutable.ParHashMap
class Node(value: Int, id: Int){
var v = value
var i = id
override def toString(): String = v toString
}
object testParHashMap{
def visit(entry: Tuple2[Int, Node]){
entry._2.v += 1
}
def main(args: Array[String]){
val hm = new ParHashMap[Int, Node]()
for (i <- 1 to 10){
var node = new Node(0, i)
hm.put(node.i, node)
}
println("========== BEFORE ==========")
hm.foreach{println}
hm.foreach{visit}
println("========== AFTER ==========")
hm.foreach{println}
}
}

I come to this with some caveats:
Though I can do some things, I consider myself relatively new to Scala.
I have only read about but never used the par stuff described here.
I have never tried to accomplish what you are trying to accomplish.
If you still care what I have to say, read on.
First, here is an academic paper describing how the parallel collections work.
On to your questions.
1) When it comes to multi-threading, Scala makes life so much easier than Java. The abstractions are just awesome. The ParHashMap you get from a par call will distribute the work to multiple threads. I can't say how that will scale for you without a better understanding of your machine, configuration, and use case, but done right (particularly with regard to side effects) it will be at least as good as a Java implementation. However, you might also want to look at Akka to have more control over everything. It sounds like that might be more suitable to your use case than simply ParHashMap.
2) It is generally simple to convert between Java and Scala collections using JavaConverters and the asJava and asScala methods. I would suggest though making sure that the public API for your method calls "looks Java" since Java is the least common denominator. Besides, in this scenario, Scala is an implementation detail, and you never want to leak those anyway. So keep the abstraction at a Java level.
3) I would guess there will actually be a performance gain with Scala--at runtime. However, you will find much slower compile time (which can be worked around. ish). This Stack Overflow post by the author of Scala is old but still relevant.
Hope that helps. That's quite a problem you got there.

Since Scala compiles to the same bytecode as Java, doing the same in both languages is very well possible, no matter the task. There are however some things which are easier to solve in Scala, but if this is worth learning a new language is a different question. Especially since Java 8 will include exactly what you ask for: simple parallel execution of functions on lists.
But even now you can do this in Java, you just need to write what Scala already has on your own.
final ExecutorService executor = Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors());
//...
final Entry<String, String>[] elements = (Entry<String, String>[]) myMap.entrySet().toArray();
final AtomicInteger index = new AtomicInteger(elements.length);
for (int i = Runtime.getRuntime().availableProcessors(); i > 0; --i) {
executor.submit(new Runnable() {
public void run() {
int myIndex;
while ((myIndex = index.decrementAndGet()) >= 0) {
process(elements[myIndex]);
}
}
});
}
The trick is to pull those elements into a temporary array, so threads can take out elements in a thread-safe way. Obviously doing some caching here instead of re-creating the Runnables and the array each time is encouraged, because the Runnable creation might already take longer than the actual task.
It is as well possible to instead copy the elements into a (reusable) LinkedBlockingQueue, then have the threads poll/take on it instead. This however adds more overhead and is only reasonable for tasks that require at least some calculation time.
I don't know how Scala actually works, but given the fact that it needs to run on the same JVM, it will do something similar in the background, it just happens to be easily accessible in the standard library.

Java for Clojure users

I've been using Lisp on and off, and I'm catching up with clojure.
The good thing about clojure is that I can use all the java functions naturally, and the bad thing about clojure is also that I have to know java function naturally.
For example, I had to spend some time (googling) to find square function in Java (Math/sqrt in clojure notation).
Could you recommend me some good information resource for Java functions (libraries) for clojure users that are not so familiar with Java?
It can be anything - good books, webpages, forums or whatever.

I had similar problems when I first started using Clojure. I had done some Java development years ago, but was still pretty unfamiliar with the libraries out there.
Intro
I find the easiest way to use Java is to not really use it. I think a book would be a little bit much to just get started using Java from Clojure. There isn't that much you really need to know, unless you really start getting down into the JVM/Java libraries. Let me explain.
Spend more time learning how to use Clojure inside and out, and become familiar with Clojure-Contrib. For instance, sqrt is in generic.math-functions in clojure.contrib.
Many of the things you'll need are in fact already in Clojure–but still plenty are not.
Become familiar with calling conventions and syntactic sugar in Clojure for using Java. e.g. Math/sqrt, as per your example, is calling the static method (which just a function, basically) sqrt from the class Math.
Anyway, here's a guide that should help you get started if you find yourself really needing to use Java. I'm going to assume you've done some imperative OO programming, but not much else. And even if you haven't, you should be okay.
Isaac's Clojurist's Guide to Java
Classes
A class is a bundle of methods (functions which act on the class) that
can also be a data type: e.g. to create a new class of the type Double : (Double. 1.2) which initializes the class Double (the period is the syntactic sugar for calling the class constructor methods, which initialize the class with the values you provide) with the value 1.2.
Now, look at the Double class in the Java 6 API:
Double
public Double(double value)
Constructs a newly allocated Double object that represents the
primitive double argument.
Parameters:
value - the value to be represented by the Double.
So you can see what happened there. You "built" a new Double with value 1.2, which is a double. A little confusing there, but really a Double is a class that represents a Double and can do things relating to doubles.
Static Methods
For instance, to parse a Double value out of a string, we can use the static method (meaning we don't need a particular instance of Double, we can just call it like we called sqrt) parseDouble(String s):
(Double/parseDouble "1.2") => 1.2
Not to tricky there.
Nonstatic Methods
Say we want to use a Java class that we initialized to something. Not too difficult:
(-> (String. "Hey there") ;; make a new String object
(.toUpperCase)) ;; pass it to .toUpperCase (look up -> to see what it does)
;; toUpperCase is a non-static method
=> "HEY THERE"
So now we've used a method which is not static, and which requires a real, live String object to deal with. Let's look at how the docs say it works:
toUpperCase
public String toUpperCase()
Converts all of the characters in this String to upper case using
the rules of the default locale. This method is equivalent to
toUpperCase(Locale.getDefault()).
Returns:
the String, converted to uppercase.
So here we have a method which returns a string (as shown by the "String" after the public in the definition, and takes no parameters. But wait! It does take a parameter. In Python, it'd be the implicit parameter self: this is called this in Java.
We could also use the method like this: (.toUpper (String. "Hey there")) and get the same result.
More on Methods
Since you deal with mutable data and classes in Java, you need to be able to apply functions to Classes (instances of Classes, really) and not expect a return value.
For instance, say we're dealing with a JFrame from the javax.swing library. We might need to do a number of things to it, not with it (you generally operate with values, not on them in functional languages). We can, like this:
(doto (JFrame. "My Frame!");; clever name
(.setContentPane ... here we'd add a JPanel or something to the JFrame)
(.pack) ;; this simply arranges the stuff in the frame–don't worry about it
(.setVisibleTrue)) ;; this makes the Frame visible
doto just passes its first argument to all the other functions you supply it, and passes it as the first argument to them. So here we're just doing a lot of things to the JFrame that don't return anything in particular. All these methods are listed as methods of the JFrame in the documentation (or its superclasses… don't worry about those yet).
Wrapping up
This should prepare you for now exploring the JavaDocs yourself. Here you'll find everything that is available to you in a standard Java 1.6 install. There will be new concepts, but a quick Google search should answer most of your questions, and you can always come back here with specific ones.
Be sure to look into the other important Clojure functions like proxy and reify as well as extend-type and its friends. I don't often use them, but when I need to, they can be invaluable. I still am understanding them myself, in fact.
There's a ton out there, but it's mostly a problem of volume rather than complexity. It's not a bad problem to have.
Additional reading:
Static or Nonstatic? ;; a guide to statis vs. nonstatic methods
The Java Class Library ;; an overview of what's out there, with a nice picture
The JavaDocs ;; linked above
Clojure Java Interop Docs ;; from the Clojure website
Best Java Books ;; as per clartaq's answer

Really, any good Java book can get you started. See for example the answer to the question about the
best Java book people have read so far. There are lots of good sources there.
Once you have a little Java under you belt, using it is all just a matter of simple Clojure syntax.
Mastering the content of the voluminous Java libraries is a much bigger task than figuring out how to use them in Clojure.

My first question would be: what do you exactly need? There are many Java libraries out there. Or do you just need the standard libraries? In that case the answer given by dbyrne should be enough.
Keep in mind that in general you are better of using the Clojure data structures like sequences instead of the Java equivalents.

Start with the Sun (now Oracle) Java Tutorials: http://download.oracle.com/javase/tutorial/index.html
Then dive into the Java 6 API docs:
http://download-llnw.oracle.com/javase/6/docs/
Then ask questions on #clojure IRC or the mailing list, and read blogs.
For a deep dive into Java the language, I recommend Bruce Eckel's free Thinking in Java:
http://www.mindview.net/Books/TIJ/

I think the plain old Java 6
API Specification should be pretty much all you need.

how to count all Operators and Operands in java class file

how to count all Operators and Operands in java class file? Does anyone have an idea?

Doing this kind of thing using regexes is unreliable. The syntax of Java is sufficiently complex that there are bound to be tricky corner cases that will cause your regexes to miscount.
Similarly using a bytecode analyser is liable to give you incorrect results because there isn't necessarily a one-to-one correspondence between source code operators / operands and bytecode instructions. The Java compiler may reorganize and rewrite the code in non-obvious ways.
The best way to do this sort of thing is to find a decent Java AST library, use that to parse your source code, and then traverse the AST to extract the information you need. (In this case, you need to count the operator and operand nodes.)

Forget regex (you'll never get that right without getting false positives like operators in comments etc), you're going to have to run a visitor over your code that counts operators. Now you can either use a source code parser or a byte code parser to do that.
For source code parsing I'd suggest the javaparser project. There, you'd create a custom Visitor extending VoidVisitorAdapter and overriding several relevant methods like this:
public void visit(AssignExpr n, A arg) {
// track the operator here
super.visit(n, arg); // resume visitor
}
On the byte code side, you'd probably use ASM and extend ClassAdapter to create your visitor. Both versions should work equally well. Or maybe not, as Stephen C writes (the compiler may have added or removed some operations).

You could try to analyze the bytecode of your class using a library like bcel.
Or use the sourceforge project lachesis (I haven't tried it):
Lachesis Analysis is a Software Complexity Measurement program for Object-Oriented source code. Analysis for Java source code and Java byte-code only is currently available.

Standard API way to check if one array is contained in another

I have two byte[] arrays in a method like this:
private static boolean containsBytes(byte[] body, byte[] checker){
//Code you do not want to ever see here.
}
I want to, using the standard API as much as possible, determine if the series contained in the checker array exists anywhere in the body array.
Right now I'm looking at some nasty code that did a hand-crafted algorithm. The performance of the algorithm is OK, which is about all you can say for it. I'm wondering if there is a more standard api way to accomplish it. Otherwise, I know how to write a readable hand-crafted one.
To get a sense of scale here, the checker array would not be larger than 48 (probably less) and the body might be a few kb large at most.

Not in the standard library (like Jon Skeet said, probably nothing there that does this) but Guava could help you here with its method Bytes.indexOf(byte[] array, byte[] target).
boolean contained = Bytes.indexOf(body, checker) != -1;
Plus, the same method exists in the classes for the other primitive types as well.

I don't know of anything in the standard API to help you here. There may be something in a third party library, although it would potentially need to be implemented repeatedly, once for each primitive type :(
EDIT: I was going to look for Boyer-Moore, but this answer was added on my phone, and I ran out of time :)
Depending on the data and your requirements, you may find that a brute force approach is absolutely fine - and a lot simpler to implement than any of the fancier algorithms available. The simple brute force approach is generally my first port of call - it often turns out to be perfectly adequate :)

You probably already know this, but what you're trying to (re-)implement is basically a string search:
http://en.wikipedia.org/wiki/String_searching_algorithm
The old code might in fact be an implementation of one of the string search algorithms; for better performance, it might be good to implement one of the other algorithms. You didn't mention how often this method is going to be called, which would help to decide whether it's worth doing that.

The collections framework can both cheaply wrap an array in the List interface and search for a sublist. I think this would work reasonably well:
import java.util.Arrays;
import java.util.Collections;
boolean found = Collections.indexOfSubList(Arrays.asList(body), Arrays.asList(checker) >= 0;