I'm using JavaCompiler to dyamically create a Java class, compile it and load in my application.
My problem is the following: the execution time with JavaCompiler is much slower than the standard way to instantiate the same class.
Here an example:
static void function() {
long startTime = System.currentTimeMillis();
String source = "package myPackage; import java.util.BitSet; public class MyClass{ static {";
while (!OWLMapping.axiomStack.isEmpty()) {
source += OWLMapping.axiomStack.pop() + ";";
}
source += "} }";
File root = new File("/java");
File sourceFile = new File(root, "myPackage/MyClass.java");
sourceFile.getParentFile().mkdirs();
Files.write(sourceFile.toPath(), source.getBytes(StandardCharsets.UTF_8));
// Compile source file.
JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
compiler.run(null, null, null, sourceFile.getPath());
// Load and instantiate compiled class.
URLClassLoader classLoader = URLClassLoader.newInstance(new URL[] { root.toURI().toURL() });
Class<?> cls = Class.forName("myPackage.MyClass", true, classLoader);
long stopTime = System.currentTimeMillis();
long elapsedTime = stopTime - startTime;
System.out.println("EXECUTION TIME: " + elapsedTime);
}
After measuring this code, I created a new java Class with the same content of the var source to test the performance: it is much faster than the JavaCompiler way. (I cannot use a standard class because in my application I need to create it dynamically).
So, is it possible to improve the performance of this code? Or this low performance is normal?
EDIT: the generated code I also tested is a simple sequence of OWLAPI axioms:
package myPackage;
public class myClass{
static {
myPackage.OWLMapping.manager.addAxiom(myPackage.OWLMapping.ontology, myPackage.OWLMapping.factory.getOWLSubClassOfAxiom(/*whatever*/);
myPackage.OWLMapping.manager.addAxiom(myPackage.OWLMapping.ontology,myPackage.OWLMapping.factory.getOWLSubClassOfAxiom(/*whatever*/);
myPackage.OWLMapping.manager.addAxiom(myPackage.OWLMapping.ontology,myPackage.OWLMapping.factory.getOWLSubClassOfAxiom(/*whatever*/);
}
}
and this exactly what the variable source contains.
The number of axioms depends on the user's input.
You have two areas which are likely to be slow (but your benchmarks combine the two areas).
The first is in building the Java String which contains your source code. When appending Strings across different statements, the JVM can't optimize them into StringBuilders which means that first it creates the string on one side of the append, then the String on the other, then it creates a third String resulting from the two being appended. This puts a lot of pressure on the heap and garbage collection, generating lots of objects which are nearly immediately garbage collected.
To fix the first problem, create a StringBuilder and call it's .append(...).
The second problem is that you are instantiating a JavaCompiler. The compiler used to compile Java programs may have one class driving it at the top level, but it will source in tons of supporting classes to fill out its private fields and the embedded includes. Finally, when running that, more objects will be created to hold the code, the Lexer, the Parser, the AST of the CompilationUnit, and eventually the byte-code emitter. This means that the one lines of code
JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
compiler.run(null, null, null, sourceFile.getPath());
Are likely (again they are not independently benchmarked) to take some time.
Finally, the class loader lines interact with the class loading system, and might be poorly adapted for performance. While it's a smaller chance it's a big performance hit, I'd benchmark that line independently too.
Related
We want to enable our customers to customize certain aspects of their requests processing, by letting them write something (currently looking at Groovy scripts), then have those scripts saved in a DB and applied when necessary, this way we won't have to maintain all those tiny aspects of processing details that might apply to certain customers only.
So, with Groovy, a naive implementation would go like this:
GroovyShell shell = new GroovyShell(); // prepare execution engine - probably once per thread
(retrieve script body from the DB, when necessary)
Script script = shell.parse(scriptBody); // parse/compile execution unit
Binding binding = prepareBinding(..); script.setBinding(binding); // provide script instance with execution context
script.run(); doSomething(binding);
When run one after the other, step 1 takes approx. 800 msec, step 3 takes almost 2000 msec, and step 5 takes about 150 msec. Absolute numbers will vary, but the relative numbers are quite stable. Assuming that step 1 is not going to be executed per-request, and step 5 number execution time is quite tolerable, I am very much concerned with step 3: parsing the Groovy script instance from the source code. I did some reading across the documentation and code, and some googling as well, but has not thus far discovered any solution, so here's the question:
Can we somehow pre-compile groovy code ONCE, then persist it in the DB and then re-hydrate whenever necessary, to obtain an executable Script instance (that we could also cache when necessary) ?
Or (as I am just thinking now) we could just compile Java code to bytecode and persist it in the Db?? Anyway, I am not so much concerned about particular language used for the scripts, but sub-second execution time is a must.. Thanks for any hints!
NB: I am aware that GroovyShellEngine will likely cache the compiled script; that still risks too long of a delay for first time execution, also risks memory overconsumption...
UPD1: based on excellent suggestion by #daggett, I've modified a solution to look as follows:
GroovyShell shell = new GroovyShell();
final Class<? extends MetaClass> theClass = shell.parse(scriptBody).getMetaClass().getTheClass();
Script script = InvokerHelper.createScript(theClass, binding);
script.run();
this works all fine and well! Now, we need to de-couple metaclass creation and usage; for that, I've created a helper method:
private Class dehydrateClass(Class theClass) throws IOException, ClassNotFoundException {
final ByteArrayOutputStream stream = new ByteArrayOutputStream();
ObjectOutputStream outputStream = new ObjectOutputStream(stream);
outputStream.writeObject(theClass);
InputStream in = new ByteArrayInputStream(stream.toByteArray());
final ObjectInputStream inputStream = new ObjectInputStream(in);
return (Class) inputStream.readObject();
}
which I've dested as follows:
#Test
void testDehydratedClass() throws IOException, ClassNotFoundException, IllegalAccessException, InstantiationException {
RandomClass instance = (RandomClass) dehydrateClass(RandomClass.class).newInstance();
assertThat(instance.getName()).isEqualTo("Test");
}
public static class RandomClass {
private final String name;
public RandomClass() {
this("Test");
}
public RandomClass(String name) {
this.name = name;
}
public String getName() {
return this.name;
}
}
which passes OK, which means that, in general, this approach is OK.
However, when I try to apply this dehydrateClass approach to theClass, returned by compile phase, I get this exception:
java.lang.ClassNotFoundException: Script1
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:686)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1866)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1749)
at java.io.ObjectInputStream.readClass(ObjectInputStream.java:1714)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1554)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
so, my impression is, that this de-serialization trick will not do any good, if the ClassLoader in question does not already have knowledge of what constitutes a Script1.. seems like the only way to make this kind of approach work is to save those pre-compiled classes somehow somewhere .. or may be learn to serialize them differently
you can parse/compile scripts/classes during editing and store compiled version somewhere - in database, file system, memory, ...
here is a groovy code snippet to compile script/class to a bytecode and then define/load classes from the bytecode.
import org.codehaus.groovy.control.BytecodeProcessor
import org.codehaus.groovy.control.CompilerConfiguration
//bytecode processor that could be used to store bytecode to cache(file,db,...)
#groovy.transform.CompileStatic
class BCP implements BytecodeProcessor{
Map<String,byte[]> bytecodeMap = [:]
byte[] processBytecode(String name, byte[] original){
println "$name >> ${original.length}"
bytecodeMap[name]=original //here we could store bytecode to a database or file system instead of memory map...
return original
}
}
def bcp = new BCP()
//------ COMPILE PHASE
def cc1 = new CompilerConfiguration()
cc1.setBytecodePostprocessor(bcp)
def gs1 = new GroovyShell(new GroovyClassLoader(), cc1)
//the next line will define 2 classes: MyConst and MyAdd (extends Script) named after the filename
gs1.parse("class MyConst{static int cnt=0} \n x+y+(++MyConst.cnt)", "MyAdd.groovy")
//------ RUN PHASE
// let's create another classloader that has no information about classes MyAdd and MyConst
def cl2 = new GroovyClassLoader()
//this try-catch just to test that MyAdd fails to load at this point
// because unknown for 2-nd class loader
try {
cl2.loadClass("MyAdd")
assert 1==0: "this should not happen because previous line should throw exception"
}catch(ClassNotFoundException e){}
//now define previously compiled classes from the bytecode
//you can load bytecode from filesystem or from database
//for test purpose let's take them from map
bcp.bytecodeMap.each{String name, byte[] bytes->
cl2.defineClass(name, bytes)
}
def myAdd = cl2.loadClass("MyAdd").newInstance()
assert myAdd instanceof groovy.lang.Script //it's a script
myAdd.setBinding([x: 1000, y: 2000] as Binding)
assert myAdd.run() == 3001 // +1 because we have x+y+(++MyConst.cnt)
myAdd.setBinding([x: 1100, y: 2200] as Binding)
assert myAdd.run() == 3302
println "OK"
This question already has answers here:
Java: Load class from string
(5 answers)
Closed 8 years ago.
Here I have found some good example:
// Prepare source somehow.
String source = "package test; public class Test { static { System.out.println(\"hello\"); } public Test() { System.out.println(\"world\"); } }";
// Save source in .java file.
File root = new File("/java"); // On Windows running on C:\, this is C:\java.
File sourceFile = new File(root, "test/Test.java");
sourceFile.getParentFile().mkdirs();
new FileWriter(sourceFile).append(source).close();
// Compile source file.
JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
compiler.run(null, null, null, sourceFile.getPath());
// Load and instantiate compiled class.
URLClassLoader classLoader = URLClassLoader.newInstance(new URL[] { root.toURI().toURL() });
Class<?> cls = Class.forName("test.Test", true, classLoader); // Should print "hello".
Object instance = cls.newInstance(); // Should print "world".
System.out.println(instance); // Should print "test.Test#hashcode".
Question: Is it possible to achieve exactly the same thing without writing to a file?
#Edit:
To be more exact: I know how to compile from string (overloading JavaFileObject). But after doing this, I have no idea how to load the class. I probably missed the output-write part, but this also a thing I would like not to do.
#Edit2
For anyone interested, I created this small project to implement discussed feature: https://github.com/Krever/JIMCy
OK, so here is an example of using JavaCompiler to compile from a String input. this file is the core of it.
In this file, I load the class which is compiled. You will notice that I also detect the package and classname using regexes.
As to the output, if you want to do that in memory, it appears that you can do so if you implement your own JavaFileManager; however I have never tried that!
Note that you can debug what happens quite easily by extending ForwardingJavaFileManager
Not exactly what you are asking for, but Groovy's library for Java makes it easy to compile and evaluate expressions from String. The syntax is not identical but very similar to Java, so if you can change the strings to be compiled so they are in Groovy instead of Java, the task will be very easy. See Embedding Groovy for code samples.
In my main program I am allowing users to create Java classes and storing them in a .java file within the package UserInputs. I am now looking for a way to instantiate the user created class within my main program and also running the public methods within the class.
Here is the code which gets executed when the user presses a JButton to finish creating their class.
public void actionPerformed(ActionEvent e) {
if(e.getSource() == inputButt.getButtons()){
try{
PrintWriter writer = new PrintWriter("C:/Users/human/Desktop/UserInputTest/src/UserInputs/UserCreatedClass.java", "UTF-8");
writer.println(textArea.getText());
writer.close();
}catch(Exception except){
except.printStackTrace();
}
}
}
You need to compile the file at runtime. Maybe this or this post here on SO helps you.
What the first link says is that you should use the Java 6 Compiler API. What you need to do is this:
JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
int compilationResult = compiler.run(null, null, null, fileToCompile);
Where fileToCompile is the path to your file, in your case "C:/Users/human/Desktop/UserInputTest/src/UserInputs/UserCreatedClass.java". Then you can execute the code via Reflection.
I would be very carefully with letting people create and execute their own Java code, though. I don't know what you plan to do but if you are running this code on a server, I would not recommend doing such things. In case this application should run locally on the clients computer (so they can only harm themselves) this should not be a problem. Otherwise I would not let them program what they want.
You might also want to consider Groovy compiler which is almost fully compatible with Java syntax, more functional and has simpler API. Example from a Groovy page:
ClassLoader parent = getClass().getClassLoader();
GroovyClassLoader loader = new GroovyClassLoader(parent);
Class groovyClass = loader.parseClass(new File("src/test/groovy/script/HelloWorld.groovy"));
// let's call some method on an instance
GroovyObject groovyObject = (GroovyObject) groovyClass.newInstance();
Object[] args = {};
groovyObject.invokeMethod("run", args);
Is there a way in using externally stored sourcecode and loading it into a Java programm, so that it can be used by it?
I would like to have a program that can be altered without editing the complete source code and that this is even possible without compiling this every time. Another advantage is, that I can change parts of the code like I want.
Of course I have to have interfaces so that it is possible to send data into this and getting it back into the fixed source program again.
And of course it should be faster than a pure interpreting system.
So is there a way in doing this like an additional compiling of these external source code parts and a start of the programm after this is done?
Thank you in advance, Andreas :)
You need the javax.tools API for this. Thus, you need to have at least the JDK installed to get it to work (and let your IDE point to it instead of the JRE). Here's a basic kickoff example (without proper exception and encoding handling just to make the basic example less opaque, cough):
public static void main(String... args) throws Exception {
String source = "public class Test { static { System.out.println(\"test\"); } }";
File root = new File("/test");
File sourceFile = new File(root, "Test.java");
Writer writer = new FileWriter(sourceFile);
writer.write(source);
writer.close();
JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
compiler.run(null, null, null, sourceFile.getPath());
URLClassLoader classLoader = URLClassLoader.newInstance(new URL[] { root.toURI().toURL() });
Class<?> cls = Class.forName("Test", true, classLoader);
}
This should print test in stdout, as done by the static initializer in the test source code. Further use would be more easy if those classes implements a certain interface which is already in the classpath. Otherwise you need to involve the Reflection API to access and invoke the methods/fields.
In Java 6 or later, you can get access to the compiler through the javax.tools package. ToolProvider.getSystemJavaCompiler() will get you a javax.tools.JavaCompiler, which you can configure to compile your source. If you are using earlier versions of Java, you can still get at it through the internal com.sun.tools.javac.Main interface, although it's a lot less flexible.
Java6 has a scripting API. I've used it with Javascript, but I believe you can have it compile external Java code as well.
http://java.sun.com/developer/technicalArticles/J2SE/Desktop/scripting/
Edit: Here is a more relevant link:
"Dynamic source" code in Java applications
I'm constructing a framework in Java that will listen for events and then process them in Jython. Different event types will be sent to different scripts.
Since jython takes quite some time to compile the script when PythonInterpreter.exec() is called, I will have to pre-compile the scripts. I'm doing it the following way:
// initialize the script as string (would load it from file in final version)
String script = "print 'foo'";
// get the compiled code object
PyCode compiled = org.python.core.__builtin__.compile( script, "<>", "exec" );
The PyCode compiled object would be pushed to repository and used as events come in
PythonInterpreter pi = new PythonInterpreter();
pi.set( "variable_1", "value_1");
pi.set( "variable_x", "value_x");
pi.exec( compiled );
Now for my conundrum - it might happen that there are multiple events of certain type happening at the same time - thus multiple instances of script running at the same time.
Almost all scripts would probably remain short-lived - up to 100 lines, no loops. Number and frequency is completely random (user generated events) and could be from 0 to about 200 per second per event type.
What would the best way to do it be? I'm looking at a few possibilities:
use synchronization at trigger event point - this would prevent multiple instances of same script but also events wouldn't be processed as quickly as they should be
create a pool of same type scripts somehow populated by cloning original PyCode object - the biggest problem would probably be optimizing pool sizes
dynamically clone the script object from the parent whenever needed and then discard it when exec() finishes - this way the lag is removed from compile but it is still present in clone method
Probably the combination of number 2 and 3 would be the best - creating dynamic pool sizes?
So, any thoughts? ;)
It is a pity that PyCode instances aren't immutable (there are a lot of public members on the classes).
You can precompile a reusable script using this code:
// TODO: generate this name
final String name = "X";
byte[] scriptBytes = PyString.to_bytes(script);
CompilerFlags flags = Py.getCompilerFlags();
ByteArrayOutputStream ostream = new ByteArrayOutputStream();
Module.compile(parser.parse(new ByteArrayInputStream(scriptBytes), "exec",
"<>", flags), ostream, name, "<>", false, false, false, flags);
byte[] buffer = ostream.toByteArray();
Class<PyRunnable> clazz = BytecodeLoader.makeClass(name, null, buffer);
final Constructor<PyRunnable> constructor = clazz
.getConstructor(new Class[] { String.class });
You can then use the constructor to produce PyCode instances for the script whenever you need one:
PyRunnable r = constructor.newInstance(name);
PyCode pc = r.getMain();
I would be the first to admit that this is not a good way of doing things and probably speaks volumes about my inexperience with Jython. However, it is significantly faster than compiling every time. The code works under Jython 2.2.1, but won't compile under Jython 2.5 (nor will yours).
PythonInterpreter is expensive, this code will use only one.
#action.py
def execute(filename, action_locals):
#add caching of compiled scripts here
exec(compile(open(filename).read(), filename, 'exec'), action_locals)
//class variable, only one interpreter
PythonInterpreter pi;
//run once in init() or constructor
pi = new PythonInterpreter();//could do more initialization here
pi.exec("import action");
//every script execution
PyObject pyActionRunner = pi.eval("action.execute");
PyString pyActionName = new PyString(script_path);
PyDictionary pyActionLocals = new PyDictionary();
pyActionLocals.put("variable_1", "value_1");
pyActionLocals.put("variable_x", "value_x")
pyActionRunner.__call__(pyActionName, pyActionLocals);
#example_script.py
print variable_1, variable_x