parsing a Python expression from Java - java

I've got a bit of an interesting challenge
To the point:
I want to allow a user to enter an expression in a text field, and have that string treated as a python expression. There are a number of local variables I would like to make available to this expression.
I do have a solution though it will be cumbersome to implement. I was thinking of keeping a Python class source file, with a function that has a single %s in it. When the user enters his expression, we simply do a string format, and then call Jython's interpreter, to spit out something we can execute. There would have to be a number of variable declaration statements in front of that expression to make sure the variables we want to expose to the user for his expression.
So the user would be presented with a text field, he would enter
x1 + (3.5*x2) ** x3
and we would do our interpreting process to come up with an open delegate object. We then punch the values into this object from a map, and call execute, to get the result of the expression.
Any objections to using Jython, or should I be doing something other than modifying source code? I would like to think that some kind of mutable object akin to C#'s Expression object, where we could do something like
PythonExpression expr = new PythonExpression(userSuppliedText)
expr.setDefaultNamespace();
expr.loadLibraries("numPy", /*other libraries?*/);
//comes from somewhere else in the flow, but effectively we get
Map<String, Double> symbolValuesByName = new HashMap<>(){{
put("x1", 3.0);
put("x2", 20.0);
put("x3", 2.0);
}};
expr.loadSymbols(symbolValuesByName);
Runnable exprDelegate = expr.compile();
//sometime later
exprDelegate.run();
but, I'm hoping for a lot, and it looks like Jython is as good as it gets. Still, modifying source files and then passing them to an interpreter seems really heavy-handed.
Does that sound like a good approach? Do you guys have any other libraries you'd suggest?
Update: NumPy does not work with Jython
I should've discovered this one on my own.
So now my question shifts: Is there any way that from a single JVM process instance (meaning, without ever having to fork) I can compile and run some Python code?

If you simply want to parse the expressions, you ought to be able to put something together with a Java parser generator.
If you want to parse, error check and evaluate the expressions, then you will need a substantial subset of the functionality a full Python interpreter.
I'm not aware of a subset implementation.
If such a subset implementation exists, it is unclear that it would be any easier to embed / call than to use a full Python interpreter ... like Jython.
If the powers that be dictate that "thou shalt use python", then they need to pay for the extra work it is going to cause you ... and the next guy who is going to need to maintain a hybrid system across changes in requirements, and updates to the Java and Python / Jython ecosystems. Factor it into the project estimates.
The other approach would be to parse the full python expression grammar, but limit what your evalutor can handle ... based on what it actually required, and what is implementable in your project's time-frame. Limit the types supported and the operations on the types. Limit the built-in functions supported. Etcetera.
Assuming that you go down the Java calling Jython route, there is a lot of material on how to implement it here: http://www.jython.org/jythonbook/en/1.0/JythonAndJavaIntegration.html

Related

Is there a way in Java to Create or emulate your own syntax similar to "+="? Not Overloading

Edit: I have rewritten the question to hopefully make it more understandable.
I do not want to overload!
If you have the following code:
ImmutableObject mutableReference = new ImuttableObject();
mutableReference = mutableReference.doStuff(args);
Can a compile time or pre-compile time process replace defined text formats? For example:
DEFINE X.=Y AS X = X.Y
could replace
mutableReference .= doStuff(args) with mutableReference = mutableReference.doStuff(args);
So some process knows that the code before ".=" is X and after is Y. Similar to syntactic sugar, before compiling or during, just replace X.=Y with X = X.Y.
Below is the old version of the question.
I have the following "form" of code for lack of a better word.
turnStates = turnStates.add(currentState); // log end of turn state.
//turnStates.=add(currentState);
//turnStates=.add(currentState);
Where turnStates can be a reference to any immutable object.
I would like it to look like the code commented out or similar.
Much like integers that have ++ and += I'd like a way to write my own for my immutables.
I think I recall some pre-processor stuff from C++ that I think could replace predefined text for code snippets. I was wondering if there was a way in java to define a process for replacing my desired code for the working code at compile time.
I'm sure you could make the IDE do it, but then you can't share the code with others not running a pre-configured IDE.
Edit:
turnStates is immutable and returns a different object on a call to add. It is test code and I have my reasons why a list, or as it is at the moment acting more like a stack, is immutable. Irrelevant for the question as I could simply replace it with
player = player.doSomething(args) where doSomething(args) returns a Player instance. Player is just a small part of the model and is costless to be immutable.
I know Overloads and syntax can't be changed in Java. As I tried to portray originally, sorry if it didn't come across this way is:
I was hoping that I wasn't aware of a syntax to do with maybe the # sign that could replace text before compiling. So for example:
DEFINE X.=Y AS X = X.Y where X = turnStates and Y = add() in my example.
But as the answer I upvoted said. I'll check out Scala as the answer seems to be no.
No. Java explicitly does not support operator overloading for user defined data types. However, scala is a JVM hosted language and does.
Unlike C++,Java doesn't support operator overloading.But Scala or Groovy does.
Scala can be integrated into Java but the operator overloading integration part is still not directly supported by Java as you will not be able to use the operator itself but something like #eq(...) for the "=" operator.
Check this link out for a little more detail if you want to know about Scala integration into java
Bottom line:
operator overloading​ is not supported by Java
And if your project requires a lot of vector addition, substraction,etc. i.e. lot of custom operators then a good suggestion would be using C# as your choice of language which is a Java like language

How can I get function names defined in Lisp script?

I have a Lisp file with some functions defined [ (defun ... ].
Is there an easy way to get all function names defined in the script using Java?
Perhaps you can use some of the code from A Small Lisp interpreter in Java
If you want to know, which functions are defined in a certain package, you cane use with-package-iterator macro. Like this:
(with-package-iterator (next (find-package 'test) :internal)
(loop :for (more? sym) := (multiple-value-list (next))
:if (fboundp sym) :collect sym :into rez
:else :unless more? :do (return rez)))
If you want to just extract functions from a script(-file), the easiest way seems to be scanning it with a regex "\\(defun (.+)\\s" or something similar.
In very simple cases, you can use regular expressions. In slightly less simple cases, implementing an S-expression reader might be a reasonable approach (you might even get away with ignoring reader macros altogether). In the general case, though, especially if macrology is involved, consider embedding Armed Bear Common Lisp, which is a Java implementation of Common Lisp, and using the with-package-iterator form Vsevolod mentioned.

Best sandboxed expression language for JVM

I want an expression language that runs on the JVM and includes support for
math expressions, including operator priority
string expressions, like substring, etc
supports named functions
this allows me to decorate and control exactly who and what functions can be executed.
read/write variables that are "typeless" / allow type conversion in a controlled manner.
does not allow arbitary java scriptlets.
it should not be possible to include constructs like new Someclass()
cannot execute arbitrary static or otherwise method
does not allow any OGNL like expressions.
I only want to functions I map to be available.
support for control constructs like if this then that is for the moment optional.
must be embeddable.
This previous stackoverflow question is similar, but:
does not really answer "how" or "what" as does the above,
allows java object expressions, throwing an exception from a SecurityManager to stop method execution, which is nasty and wrong.
java object like expressions should be an error at parse time.
jexel seem to be closest possible match, but License is a bit horrible (GPL/Commercial).
If you only want the scripts to output text, then Apache Velocity fit's your constraints quite well. It runs in an environment where it only has access to the objects you give it, but can do things like basic math.
The Apache license is a bit friendlier than GPL too.

Java source refactoring of 7000 references

I need to change the signature of a method used all over the codebase.
Specifically, the method void log(String) will take two additional arguments (Class c, String methodName), which need to be provided by the caller, depending on the method where it is called. I can't simply pass null or similar.
To give an idea of the scope, Eclipse found 7000 references to that method, so if I change it the whole project will go down. It will take weeks for me to fix it manually.
As far as I can tell Eclipse's refactoring plugin of Eclipse is not up to the task, but I really want to automate it.
So, how can I get the job done?
Great, I can copy a previous answer of mine and I just need to edit a tiny little bit:
I think what you need to do is use a source code parser like javaparser to do this.
For every java source file, parse it to a CompilationUnit, create a Visitor, probably using ModifierVisitor as base class, and override (at least) visit(MethodCallExpr, arg). Then write the changed CompilationUnit to a new File and do a diff afterwards.
I would advise against changing the original source file, but creating a shadow file tree may me a good idea (e.g. old file: src/main/java/com/mycompany/MyClass.java, new file src/main/refactored/com/mycompany/MyClass.java, that way you can diff the entire directories).
Eclipse is able to do that using Refactor -> Change Method signature and provide default values for the new parameters.
For the class parameter the defaultValue should be this.getClass() but you are right in your comment I don't know how to do for the method name parameter.
IntelliJ IDEA shouldn't have any trouble with this.
I'm not a Java expert, but something like this could work. It's not a perfect solution (it may even be a very bad solution), but it could get you started:
Change the method signature with IntelliJ's refactoring tools, and specify default values for the 2 new parameters:
c: self.getClass()
methodName: Thread.currentThread().getStackTrace()[1].getMethodName()
or better yet, simply specify null as the default values.
I think that there are several steps to dealing with this, as it is not just a technical issue but a 'situation':
Decline to do it in short order due to the risk.
Point out the issues caused by not using standard frameworks but reinventing the wheel (as Paul says).
Insist on using Log4j or equivalent if making the change.
Use Eclipse refactoring in sensible chunks to make the changes and deal with the varying defaults.
I have used Eclipse refactoring on quite large changes for fixing old smelly code - nowadays it is fairly robust.
Maybe I'm being naive, but why can't you just overload the method name?
void thing(paramA) {
thing(paramA, THE_DEFAULT_B, THE_DEFAULT_C)
}
void thing(paramA, paramB, paramC) {
// new method
}
Do you really need to change the calling code and the method signature? What I'm getting at is it looks like the added parameters are meant to give you the calling class and method to add to your log data. If the only requirement is just adding the calling class/method to the log data then Thread.currentThread().getStackTrace() should work. Once you have the StackTraceElement[] you can get the class name and method name for the caller.
If the lines you need replaced fall into a small number of categories, then what you need is Perl:
find -name '*.java' | xargs perl -pi -e 's/log\(([^,)]*?)\)/log(\1, "foo", "bar")/g'
I'm guessing that it wouldn't be too hard to hack together a script which would put the classname (derived from the filename) in as the second argument. Getting the method name in as the third argument is left as an exercise to the reader.
Try refactor using intellij. It has a feature called SSR (Structural Search and Replace). You can refer classes, method names, etc for a context. (seanizer's answer is more promising, I upvoted it)
I agree with Seanizer's answer that you want a tool that can parse Java. That's necessary but not sufficient; what you really want is a tool that can carry out a reliable mass-change.
To do this, you want a tool that can parse Java, can pattern match against the parsed code, install the replacement call, and spit out the answer without destroying the rest of the source code.
Our DMS Software Reengineering Toolkit can do all of this for a variety of languages, including Java. It parses complete java systems of source, builds abstract syntax trees (for the entire set of code).
DMS can apply pattern-directed, source-to-source transformations to achieve the desired change.
To achieve the OP's effect, he would apply the following program transformation:
rule replace_legacy_log(s:STRING): expression -> expression
" log(\s) " -> " log( \s, \class\(\), \method\(\) ) "
What this rule says is, find a call to log which has a single string argument, and replace it with a call to log with two more arguments determined by auxiliary functions class and method.
These functions determine the containing method name and containing class name for the AST node root where the rule finds a match.
The rule is written in "source form", but actually matches against the AST and replaces found ASTs with the modified AST.
To get back the modified source, you ask DMS to simply prettyprint (to make a nice layout) or fidelity print (if you want the layout of the old code preserved). DMS preserves comments, number radixes, etc.\
If the exisitng application has more than one defintion of the "log" function, you'll need to add a qualifier:
... if IsDesiredLog().
where IsDesiredLog uses DMS's symbol table and inheritance information to determine if the specific log refers to the definition of interest.
Il fact your problem is not to use a click'n'play engine that will allow you to replace all occurences of
log("some weird message");
by
log(this.getClass(), new Exception().getStackTrace()[1].getMethodName());
As it has few chances to work on various cases (like static methods, as an example).
I would tend to suggest you to take a look at spoon. This tool allows source code parsing and transformation, allowing you to achieve your operation in a -obviously code based- slow, but controlled operation.
However, you could alos consider transforming your actual method with one exploring stack trace to get information or, even better, internally use log4j and a log formatter that displays the correct information.
I would search and replace log( with log(#class, #methodname,
Then write a little script in any language (even java) to find the class name and the method names and to replace the #class and #method tokens...
Good luck
If the class and method name are required for "where did this log come from?" type data, then another option is to print out a stack trace in your log method. E.g.
public void log(String text)
{
StringWriter sw = new StringWriter();
PrintWriter pw = new PrintWriter(sw, true);
new Throwable.printStackTrace(pw);
pw.flush();
sw.flush();
String stackTraceAsLog = sw.toString();
//do something with text and stackTraceAsLog
}

Java for Clojure users

I've been using Lisp on and off, and I'm catching up with clojure.
The good thing about clojure is that I can use all the java functions naturally, and the bad thing about clojure is also that I have to know java function naturally.
For example, I had to spend some time (googling) to find square function in Java (Math/sqrt in clojure notation).
Could you recommend me some good information resource for Java functions (libraries) for clojure users that are not so familiar with Java?
It can be anything - good books, webpages, forums or whatever.
I had similar problems when I first started using Clojure. I had done some Java development years ago, but was still pretty unfamiliar with the libraries out there.
Intro
I find the easiest way to use Java is to not really use it. I think a book would be a little bit much to just get started using Java from Clojure. There isn't that much you really need to know, unless you really start getting down into the JVM/Java libraries. Let me explain.
Spend more time learning how to use Clojure inside and out, and become familiar with Clojure-Contrib. For instance, sqrt is in generic.math-functions in clojure.contrib.
Many of the things you'll need are in fact already in Clojure–but still plenty are not.
Become familiar with calling conventions and syntactic sugar in Clojure for using Java. e.g. Math/sqrt, as per your example, is calling the static method (which just a function, basically) sqrt from the class Math.
Anyway, here's a guide that should help you get started if you find yourself really needing to use Java. I'm going to assume you've done some imperative OO programming, but not much else. And even if you haven't, you should be okay.
Isaac's Clojurist's Guide to Java
Classes
A class is a bundle of methods (functions which act on the class) that
can also be a data type: e.g. to create a new class of the type Double : (Double. 1.2) which initializes the class Double (the period is the syntactic sugar for calling the class constructor methods, which initialize the class with the values you provide) with the value 1.2.
Now, look at the Double class in the Java 6 API:
Double
public Double(double value)
Constructs a newly allocated Double object that represents the
primitive double argument.
Parameters:
value - the value to be represented by the Double.
So you can see what happened there. You "built" a new Double with value 1.2, which is a double. A little confusing there, but really a Double is a class that represents a Double and can do things relating to doubles.
Static Methods
For instance, to parse a Double value out of a string, we can use the static method (meaning we don't need a particular instance of Double, we can just call it like we called sqrt) parseDouble(String s):
(Double/parseDouble "1.2") => 1.2
Not to tricky there.
Nonstatic Methods
Say we want to use a Java class that we initialized to something. Not too difficult:
(-> (String. "Hey there") ;; make a new String object
(.toUpperCase)) ;; pass it to .toUpperCase (look up -> to see what it does)
;; toUpperCase is a non-static method
=> "HEY THERE"
So now we've used a method which is not static, and which requires a real, live String object to deal with. Let's look at how the docs say it works:
toUpperCase
public String toUpperCase()
Converts all of the characters in this String to upper case using
the rules of the default locale. This method is equivalent to
toUpperCase(Locale.getDefault()).
Returns:
the String, converted to uppercase.
So here we have a method which returns a string (as shown by the "String" after the public in the definition, and takes no parameters. But wait! It does take a parameter. In Python, it'd be the implicit parameter self: this is called this in Java.
We could also use the method like this: (.toUpper (String. "Hey there")) and get the same result.
More on Methods
Since you deal with mutable data and classes in Java, you need to be able to apply functions to Classes (instances of Classes, really) and not expect a return value.
For instance, say we're dealing with a JFrame from the javax.swing library. We might need to do a number of things to it, not with it (you generally operate with values, not on them in functional languages). We can, like this:
(doto (JFrame. "My Frame!");; clever name
(.setContentPane ... here we'd add a JPanel or something to the JFrame)
(.pack) ;; this simply arranges the stuff in the frame–don't worry about it
(.setVisibleTrue)) ;; this makes the Frame visible
doto just passes its first argument to all the other functions you supply it, and passes it as the first argument to them. So here we're just doing a lot of things to the JFrame that don't return anything in particular. All these methods are listed as methods of the JFrame in the documentation (or its superclasses… don't worry about those yet).
Wrapping up
This should prepare you for now exploring the JavaDocs yourself. Here you'll find everything that is available to you in a standard Java 1.6 install. There will be new concepts, but a quick Google search should answer most of your questions, and you can always come back here with specific ones.
Be sure to look into the other important Clojure functions like proxy and reify as well as extend-type and its friends. I don't often use them, but when I need to, they can be invaluable. I still am understanding them myself, in fact.
There's a ton out there, but it's mostly a problem of volume rather than complexity. It's not a bad problem to have.
Additional reading:
Static or Nonstatic? ;; a guide to statis vs. nonstatic methods
The Java Class Library ;; an overview of what's out there, with a nice picture
The JavaDocs ;; linked above
Clojure Java Interop Docs ;; from the Clojure website
Best Java Books ;; as per clartaq's answer
Really, any good Java book can get you started. See for example the answer to the question about the
best Java book people have read so far. There are lots of good sources there.
Once you have a little Java under you belt, using it is all just a matter of simple Clojure syntax.
Mastering the content of the voluminous Java libraries is a much bigger task than figuring out how to use them in Clojure.
My first question would be: what do you exactly need? There are many Java libraries out there. Or do you just need the standard libraries? In that case the answer given by dbyrne should be enough.
Keep in mind that in general you are better of using the Clojure data structures like sequences instead of the Java equivalents.
Start with the Sun (now Oracle) Java Tutorials: http://download.oracle.com/javase/tutorial/index.html
Then dive into the Java 6 API docs:
http://download-llnw.oracle.com/javase/6/docs/
Then ask questions on #clojure IRC or the mailing list, and read blogs.
For a deep dive into Java the language, I recommend Bruce Eckel's free Thinking in Java:
http://www.mindview.net/Books/TIJ/
I think the plain old Java 6
API Specification should be pretty much all you need.

Categories

Resources