Has anyone attempted to "link" in the Rascal command line jar in a java executable and call REPL commands from this java executable?
I found a similar question on stackoverflow (Running a Rascal program from outside the REPL), but that doesn't go into details unfortunately.
I also checked the Rascal tutor site, but couldn't find any examples on how to do this. Tijs told me that it's something along the lines of "instantiate an interpreter and then call the import() function, after which the call() function can be called to inject REPL commands).
Is there any example code on how to do, e.g. the following from the tutor site on the REPL but from a java programming context instead of on the command line:
rascal>import demo::lang::Exp::Concrete::NoLayout::Syntax;
ok
rascal>import ParseTree;
ok
rascal>parse(#Exp, "2+3");
sort("Exp"): `2+3`
The following would do the trick; a utility class for the same can be found in rascal/src/org/rascalmpl/interpreter/JavaToRascal.java:
GlobalEnvironment heap = new GlobalEnvironment();
IValueFactory vf = ValueFactoryFactory.getValueFactory();
TypeFactory TF = TypeFactory.getInstance();
IRascalMonitor mon = new NullRascalMonitor();
Evaluator eval = new Evaluator(vf, new PrintWriter(System.err), new PrintWriter(System.out), new ModuleEnvironment(ModuleEnvironment.SHELL_MODULE, heap), heap);
eval.doImport(mon, "demo::lang::Exp::Concrete::NoLayout::Syntax");
eval.doImport(mon, "ParseTree");
eval.eval(mon, "parse(#Exp, \"2+3\");", URIUtil.rootLocation("unknown"));
There is also more efficient ways of interacting with the evaluator, via the pdb.values IValue interfaces to build data and ICalleableValue to call Rascal functions. You can use the above heap object to query its environments to get references to functions and you can use the low level pdb.values API to construct values to pass to these functions.
Caveat emptor: this code is "internal" API with no guarantee for backward compatibility. I can guarantee that something like this will always be possible.
Related
in order to make use of Machine Learning in Java, I'm trying to train a model in TensorFlow, save it as ONNX file and then use the file for inference in Java. While this works fine with simple models, it's getting more complicated using pre-processing layers, as they seem to depend on custom operators.
https://www.tensorflow.org/tutorials/keras/text_classification
As an example, this Colab deals with text classification and uses an TextVectorization layer this way:
#tf.keras.utils.register_keras_serializable()
def custom_standardization2(input_data):
lowercase = tf.strings.lower(input_data)
stripped_html = tf.strings.regex_replace(lowercase, '<br />',' ')
return tf.strings.regex_replace(stripped_html, '[%s]' % re.escape(string.punctuation), '')
vectorize_layer = layers.TextVectorization(
standardize=custom_standardization2,
max_tokens=max_features,
output_mode='int',
output_sequence_length=sequence_length
)
It is used as pre-processing layer in the compiled model:
export_model = tf.keras.Sequential([
vectorize_layer,
model,
layers.Activation('sigmoid')
])
export_model.compile(loss=losses.BinaryCrossentropy(from_logits=False), optimizer="adam", metrics=['accuracy'])
In order to create the ONNX file I save the model as protobuf and then convert it to ONNX:
export_model.save("saved_model")
python -m tf2onnx.convert --saved-model saved_model --output saved_model.onnx --extra_opset ai.onnx.contrib:1 --opset 11
Using onnxruntime-extensions it is now possible to register the custom ops and to run the model in Python for inference.
import onnxruntime
from onnxruntime import InferenceSession
from onnxruntime_extensions import get_library_path
so = onnxruntime.SessionOptions()
so.register_custom_ops_library(get_library_path())
session = InferenceSession('saved_model.onnx', so)
res = session.run(None, { 'text_vectorization_2_input': example_new })
This raises the question if it's possible to use the same model in Java in a similar way. Onnxruntime for Java does have a SessionOptions#registerCustomOpLibrary function, so I thought of something like this:
OrtEnvironment env = OrtEnvironment.getEnvironment();
OrtSession.SessionOptions options = new OrtSession.SessionOptions();
options.registerCustomOpLibrary(""); // reference the library
OrtSession session = env.createSession("...", options);
Does anyone have an idea if the use case described is feasable or how to use models with pre-processing layers in Java (without using TensorFlow Java)?
UPDATE:
Spotted a potential solution. If I understand the comments in this GitHub Issue correctly, one possibility is to build the ONNXRuntime Extensions package from source (see this explanation) and reference the generated library file by calling registerCustomOpLibrary in the ONNX Runtime Library for Java. However, as I have no experience with tools like cmake this might become a challenge for me.
The solution you propose in your update is correct, you need to compile the ONNX Runtime extension package from source to get the dll/so/dylib, and then you can load that into ONNX Runtime in Java using the session options. The Python whl doesn't distribute the binary in a format that can be loaded outside of Python, so compiling from source is the only option. I wrote the ONNX Runtime Java API, so if this approach fails open an issue on Github and we'll fix it.
From this answer, I learnt that, every Groovy script compiles to a class that extends groovy.lang.Script class
Below is a test groovy script written for Jenkins pipeline in Jenkins editor.
node('worker_node'){
print "***1. DRY principle***"
def list1 = [1,2,3,4]
def list2 = [10,20,30,40]
def factor = 2
def applyFactor = {e -> e * factor}
print(list1.each(applyFactor))
print(list2.each(applyFactor))
print "***2. Higher order function***"
def foo = { value, f -> f(value *2) }
foo(3, {print "Value is $it"})
foo(3){
print "Value is $it"
}
}
How to compile this groovy script to see the class generated(source code)?
The class generated is bytecode, not source code. The source code is the Groovy script.
If you want to see something similar to what the equivalent Java source code would look like, use groovyc to compile the script as usual, and then use a Java decompiler to produce Java source (this question's answers lists a few).
That's subject to the usual caveats on decompiled code, of course. High-level information is lost in the process of compiling. Decompilers have to guess a bit to figure out the best way to represent what might have been in the original source. For instance, what was a for loop in the original code may end up being decompiled as a while loop instead.
groovy in jenkins pipeline is a Domain Specific Language.
It's not a plain groovy.
However if you remove node(){ } then it seems to be groovy in your case.
and you can run it in groovyconsole or compile to class with groovyc
just download a stable groovy binary and extract it.
if you have java7 or java8 on your computer - you can run groovyconsole and try your code there.
with Ctrl+T you can see the actual class code generated for your script.
I am using LibSVM library on Weka 3.6 and experiencing similar problem as in here (for Java) and here (for Python)
The libSVM library generates a lot of logs simliar to this
optimization finished, #iter = 399
nu = 0.9503376170384973
obj = -124.54791151883072, rho = 0.0528133707297996
nSV = 257, nBSV = 97
I followed the solution using -q parameters by setting this parameter in my code:
LibSVM svm = new LibSVM();
String[] options = {"-q"};
svm.setOptions(options);
Although this solution seems to work in Python but I doesn't work in my Java code.
Another solution suggests using Log4j and disable some level of logs, however, I don't want to add another library to my code.
Now, I'm wondering is there any clean and simple solution to disable libSVM logs?
LibSVM library for Weka with FQN of "weka.classifiers.functions.LibSVM" is a wrapper around svm algorithm to create a common interface for Java programmers are using Weka API.
Inside "LibSVM.jar" there is another jar file which named "libsvm.jar" which is the main algorithm. Contrary to LibSVM which use common Java naming conventions, the naming convention inside "libsvm.jar" is different. Inside "libsvm" package there is a class named "svm". Because I had used "svm" as my variable name, the "svm" class was invisible.
After knowing that, I followed the instruction in here and changed the "svm" to "libsvm.svm" and this is the code which is working for me. In addition, I put this code in a static block of my code to have it for all my usages.
static{
libsvm.svm.svm_set_print_string_function(new libsvm.svm_print_interface() {
#Override
public void print(String s) {
} // Disables svm output
});
}
Finally, I am using LibSVM without annoying logs.
I got a little project where I have to compute a list. The computation depends on serveal factors.
The point is that these factors change from time to time and the user should be allowed to change this by it's self.
Up to now, the factors are hard-coded and no changes can be done without recompiling the code.
At the moment the code looks like this:
if (someStatement.equals("someString")) {
computedList.remove("something");
}
My idea is to make an editable and human readable textfile, configfile, etc. which is loaded at runtime/ at startup? This file should hold the java code from above.
Any ideas how to do that? Please note: The targeted PCs do not have the JDK installed, only an JRE.
An effective way of going about this is using a static initializer. Static Block in Java A good and concise explanation can be found under this link.
One option here that would allow this would be to use User Input Dialogs from the swing API - then you could store the users answer's in variables and export them to a text file/config file, or just use them right in the program without saving them. You would just have the input dialogs pop up at the very beginning of the program before anything else happens, and then the program would run based off those responses.
You could use Javascript for the configuration file language, instead of java. Java 7 SE and later includes a javascript interpreter that you can call from Java. it's not difficult to use, and you can inject java objects into the javascript environment.
Basically, you'd create a Javascript environment, insert the java objects into it which the config file is expected to configure, and then run the config file as javascript.
Okay, here we go... I found an quite simple solution for my problem.
I am using Janino by Codehaus (Link). This library has an integrated Java compiler and seems to work like the JavaCompiler class in Java 7.
BUT without having the JDK to be installed.
Through Janino you can load and compile *.java files(which are human readable) at runtime, which was exactly what I needed.
I think the examples and code-snippets on their homepage are just painful, so here's my own implementation:
Step one is to implement an interface with the same methods your Java file has which is loaded at runtime:
public interface ZuordnungInterface {
public ArrayList<String> Zuordnung(ArrayList<String> rawList);}
Then you call the Janino classloader when you need the class:
File janinoSourceDir = new File(PATH_TO_JAVAFILE);
File[] srcDir = new File[] { janinoSourceDir };
String encoding = null;
ClassLoader parentClassLoader = this.getClass().getClassLoader();
ClassLoader cl = new JavaSourceClassLoader(parentClassLoader, srcDir,
encoding);
And create an new instance
ZuordnungsInterface myZuordnung = (ZuordnungInterface) cl.loadClass("zuordnung")
.newInstance();
Note: The class which is loaded is named zuordnung.java, so there is no extension needed in the call cl.loadClass("zuordnung").
And finaly the class I want to load and compile at runtime of my program, which can be located wherever you want it to be (PATH_TO_JAVAFILE):
public class zuordnung implements ZuordnungInterface {
public ArrayList<String> Zuordnung(ArrayList<String> rawList){
ArrayList<String> computedList = (ArrayList<String>) rawList.clone();
if (Model.getSomeString().equals("Some other string")) {
computedList.add("Yeah, I loaded an external Java class");
}
return computedList;
}}
That's it. Hope it helps others with similar problems!
Is there a way in using externally stored sourcecode and loading it into a Java programm, so that it can be used by it?
I would like to have a program that can be altered without editing the complete source code and that this is even possible without compiling this every time. Another advantage is, that I can change parts of the code like I want.
Of course I have to have interfaces so that it is possible to send data into this and getting it back into the fixed source program again.
And of course it should be faster than a pure interpreting system.
So is there a way in doing this like an additional compiling of these external source code parts and a start of the programm after this is done?
Thank you in advance, Andreas :)
You need the javax.tools API for this. Thus, you need to have at least the JDK installed to get it to work (and let your IDE point to it instead of the JRE). Here's a basic kickoff example (without proper exception and encoding handling just to make the basic example less opaque, cough):
public static void main(String... args) throws Exception {
String source = "public class Test { static { System.out.println(\"test\"); } }";
File root = new File("/test");
File sourceFile = new File(root, "Test.java");
Writer writer = new FileWriter(sourceFile);
writer.write(source);
writer.close();
JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
compiler.run(null, null, null, sourceFile.getPath());
URLClassLoader classLoader = URLClassLoader.newInstance(new URL[] { root.toURI().toURL() });
Class<?> cls = Class.forName("Test", true, classLoader);
}
This should print test in stdout, as done by the static initializer in the test source code. Further use would be more easy if those classes implements a certain interface which is already in the classpath. Otherwise you need to involve the Reflection API to access and invoke the methods/fields.
In Java 6 or later, you can get access to the compiler through the javax.tools package. ToolProvider.getSystemJavaCompiler() will get you a javax.tools.JavaCompiler, which you can configure to compile your source. If you are using earlier versions of Java, you can still get at it through the internal com.sun.tools.javac.Main interface, although it's a lot less flexible.
Java6 has a scripting API. I've used it with Javascript, but I believe you can have it compile external Java code as well.
http://java.sun.com/developer/technicalArticles/J2SE/Desktop/scripting/
Edit: Here is a more relevant link:
"Dynamic source" code in Java applications