I am trying to create a serializable interface implementation in groovy dynamically which could be send over the wire where it can be deserialized and executed with args. I have created anonymous interface implementation using map but it fails on serialization.
gcloader = new GroovyClassLoader()
script = "class X { public def x = [call: {y -> y+1}] as MyCallable }"
gclass = gcloader.parseClass(script)
x = gclass.newInstance().x
// serialzing x fails
I am not sure if a groovy closure is compiled to a random class name, which would make it impossible to deserialized even if it gets serialized. Is there a way to do this?
Here's a piece of code that might be helpful:
import groovy.lang.GroovyClassLoader
def gcLoader = new GroovyClassLoader()
def script = """
class X implements Serializable {
public def x = [
call: { y -> y + 1 }
]
}"""
def cls = gcLoader.parseClass(script)
def inst = cls.newInstance().x
def baos = new ByteArrayOutputStream()
def oos = new ObjectOutputStream(baos)
oos.writeObject(inst)
def serialized = baos.toByteArray()
def bais = new ByteArrayInputStream(serialized)
def ois = new CustomObjectInputStream(bais, gcLoader)
inst = ois.readObject()
assert 2 == inst.call(1)
public class CustomObjectInputStream extends ObjectInputStream {
private ClassLoader classLoader
public CustomObjectInputStream(InputStream ins, ClassLoader classLoader) throws IOException {
super(ins)
this.classLoader = classLoader
}
protected Class<?> resolveClass(ObjectStreamClass desc) throws ClassNotFoundException {
return Class.forName(desc.getName(), false, classLoader)
}
}
Basically, You need an instance of ObjectInputStream with custom ClassLoader.
According to my own limited research I came to conclusion that, In jvm there is no standard/popular library which could pickle code like in python which was the requirement i was primarily after. There are some ways to do it through URL classloaders etc but comes with some inherent complexities. I ended up just simply sending the code string and recompiling it whenever required on multiple machines.
Related
I want to instatiate a groovy class and i have some concerns
My first choice is to use GroovyShell :
groovy-script:
class Foo {
public String doStuff(String stuff) {
return stuff + "_utils";
}
}
new Foo(); // ??
main-script :
GroovyShell shell = new GroovyShell();
Script script = shell.parse(new File(path));
def clazz = script.run();
String result = clazz.doStuff("test");
print(result); // test_utils;
The second option is to use GroovyClassLoader :
groovy-script
class Foo {
public String doStuff(String stuff) {
return stuff + "_utils";
}
}
main-script
GroovyClassLoader loader = new GroovyClassLoader();
Script script = loader.parseClass(new File(path))
Object clazz = script.newInstance();
Object[] args = new Object[1];
args[0] = "test";
String result = clazz.invokeMethod("doStuff", args);
print(result) // test_utils
Both will run fine, i would prefer to use GroovyShell because i use it everywhere in my current code, but i don't know if new Foo() inside my scripts can cause any memory leaks. Is it possible?
GroovyShell uses the default GroovyClassLoader. So if you don't need any extra specific features that are provided by GroovyClassLoader, you should stick to GroovyShell to keep it simple. GroovyShell and GroovyClassLoader are instances of garbage collected and I don't believe there are memory leaks for neither of them.
After #daggett's help i managed to find the solution that fits my needs.
I will not use groovy classes at all.
A simple example
utility groovy script :
String doStuff() {
return "doStuff";
}
String doStuff2(){
return "doStuff2";
}
calling the utility methods from the main groovy script
GroovyShell shell = new GroovyShell();
Script utilsScript = shell.parse(new File(PATH_TO_UTIL_SCRIPT));
String result = utilsScript.doStuff();
println(result); // doStuff;
String result2 = utilsScript.doStuff2();
println(result2); // doStuff2;
This is not the answer to the original question but since it fits my need i am fine.
I am using Tensorflow java API (1.8.0) where I load multiple models (in different sessions). Those models are loaded from .pb files using the SavedModelBundle.load(...) method. Those .pb files were obtained by saving Keras' models.
Let's say that I want to load 3 models A, B, C.
To do that, I implemented a java Model class :
public class Model implements Closeable {
private String inputName;
private String outputName;
private Session session;
private int inputSize;
public Model(String modelDir, String input_name, String output_name, int inputSize) {
SavedModelBundle bundle = SavedModelBundle.load(modelDir, "serve");
this.inputName = input_name;
this.outputName = output_name;
this.inputSize = inputSize;
this.session = bundle.session();
}
public void close() {
session.close();
}
public Tensor predict(Tensor t) {
return session.runner().feed(inputName, t).fetch(outputName).run().get(0);
}
}
Then I easily can instantiate 3 Model objects corresponding to my A, B and C models with this class and make predictions with those 3 models in the same java program.
I also noticed that if I have a GPU, the 3 models are loaded on it.
However, I would like only model A to be running on GPU and force the 2 others to be running on CPU.
By reading documentation and diving into the source code I didn't find a way to do so. I tried to define a new ConfigProto setting visible devices to None and instantiate a new Session with the graph but it didn't work (see code below).
public Model(String modelDir, String input_name, String output_name, int inputSize) {
SavedModelBundle bundle = SavedModelBundle.load(modelDir, "serve");
this.inputName = input_name;
this.outputName = output_name;
this.inputSize = inputSize;
ConfigProto configProto = ConfigProto.newBuilder().setAllowSoftPlacement(false).setGpuOptions(GPUOptions.newBuilder().setVisibleDeviceList("").build()).build();
this.session = new Session(bundle.graph(),configProto.toByteArray());
}
When I load the model, it uses the available GPU. Do you have any solution to this problem ?
Thank you for your answer.
According to this issue , the new source code fixed this problem. Unfortunately you will have to build from source following these instructions
Then you can test :
ConfigProto configProto = ConfigProto.newBuilder()
.setAllowSoftPlacement(true) // allow less GPUs than configured
.setGpuOptions(GPUOptions.newBuilder().setPerProcessGpuMemoryFraction(0.01).build())
.build();
SavedModelBundle bundle = SavedModelBundle.loader(modelDir).withTags("serve").withConfigProto(configProto.toByteArray()).load();
You can set the device configuration of your tensorflow graph. Here is some relevant code [source].
...
byte[] config = ConfigProto.newBuilder()
.setLogDevicePlacement(true)
.setAllowSoftPlacement(true)
.build()
.toByteArray()
Session sessions[] = new Session[numModels];
// function to move the graph definition to a new device
public static byte[] modifyGraphDef(byte[] graphDef, String device) throws Exception {
GraphDef.Builder builder = GraphDef.parseFrom(graphDef).toBuilder();
for (int i = 0; i < builder.getNodeCount(); ++i) {
builder.getNodeBuilder(i).setDevice(device);
}
return builder.build().toByteArray();
}
graphA.importGraphDef(modifyGraphDef(graphDef, "/gpu:0"));
graphB.importGraphDef(modifyGraphDef(graphDef, "/cpu:0"));
This would probably be cleaner than to do the more obvious setting of the CUDA_VISIBLE_DEVICES environment variable to "" after loading the first model.
Above given answers did not work for me.Using putDeviceCount("GPU", 0) makes TF use CPU . It is working in version 1.15.0.You can load same model to both cpu and gpu and if gpu throws Resource exhausted: OOM when allocating tensor, use the CPU model to do prediction.
ConfigProto configProtoCpu = ConfigProto.newBuilder().setAllowSoftPlacement(true).putDeviceCount("GPU", 0)
.build();
SavedModelBundle modelCpu=SavedModelBundle.loader(modelPath).withTags("serve")
.withConfigProto(configProtoCpu.toByteArray()).load();
ConfigProto configProtoGpu = ConfigProto.newBuilder().setAllowSoftPlacement(true)
.setGpuOptions(GPUOptions.newBuilder().setAllowGrowth(true).build()).build();
SavedModelBundle modelgpu = SavedModelBundle.loader(modelPath).withTags("serve")
.withConfigProto(configProtoGpu.toByteArray()).load();
I am successfully able to compile Groovy in Java at runtime and store it in a database and pull it out. I can't compile a Groovy class if it has inner classes or an inner enum. Has anyone successfully compiled Groovy code like this and included inner classes/enums and able to pull the script out by classname?
For example, I want to load the "Test" script shown below that contains inner classes and run the script at run time.
Compiler code:
public byte[] compileGroovyScript(final String className, final String script) {
byte[] compiledScriptBytes = null;
CompilationUnit compileUnit = new CompilationUnit();
compileUnit.addSource(className, script);
compileUnit.compile(Phases.CLASS_GENERATION);
for (Object compileClass : compileUnit.getClasses()) {
GroovyClass groovyClass = (GroovyClass) compileClass;
compiledScriptBytes = groovyClass.getBytes();
}
return compiledScriptBytes;
}
Code to pull script out:
public Class getGroovyScript(final String className, final byte[] script) {
Class clazz = null;
try (GroovyClassLoader classLoader = new GroovyClassLoader(this.getClass().getClassLoader())) {
clazz = classLoader.defineClass(className, script);
} catch (IOException e) {
} catch (Exception e) {
}
return clazz;
}
Code to run the script:
Class groovyClass = app.getGroovyScript(className, compiledScript);
TestScript script = (TestScript) groovyClass.newInstance();
System.out.println(script.getMessage());
Groovy script:
import com.groovy.groovy.TestScript
class Test implements TestScript {
String getMessage() {
[1..10].each(){
println it
}
return "Jello"
}
}
It isn't clear from the description why you are doing the compiling yourself. If you can just let Groovy do it for you then the whole thing can just be simplified to something like this:
String script = // string containing the script you want to parse
GroovyClassLoader groovyClassLoader = new GroovyClassLoader();
Class theParsedClass = groovyClassLoader.parseClass(script);
Ok this may be a little late but hopefully it helps the next person. I think you need to save a List for each groovy class and then cl.defineClass and finally cl.loadClass. I think groovy sometimes compile to a list of classes basically as in below when I addSource(), I add one class and then loop over all the generated classes from that one file.
This is the code I am currently running(though I have not tried saving and reloading at a later time)
GroovyClassLoader cl = new GroovyClassLoader();
CompilationUnit compileUnit = new CompilationUnit();
compileUnit.addSource(scriptCode.getClassName(), scriptCode.getScriptSourceCode());
compileUnit.compile(Phases.CLASS_GENERATION);
compileUnit.setClassLoader(cl);
GroovyClass target = null;
for (Object compileClass : compileUnit.getClasses()) {
GroovyClass groovyClass = (GroovyClass) compileClass;
cl.defineClass(groovyClass.getName(), groovyClass.getBytes());
if(groovyClass.getName().equals(scriptCode.getClassName())) {
target = groovyClass;
}
}
if(target == null)
throw new IllegalStateException("Could not find proper class");
return cl.loadClass(target.getName());
take note of the cl.defineClass call which puts the class in the classloader so when it is looked up(the enum or innerclass), it will be there.
and so now I think you do not need to create your own class loader(though you avoid useless defineClass until it is needed with your own classloader which can be useful and more performant).
This forgoes any error handling for the sake of simplicity here, but this is probably what you want:
public byte[] compileGroovyScript(final String className, final String script) {
byte[] compiledScriptBytes = null;
CompilationUnit compileUnit = new CompilationUnit();
compileUnit.addSource(className, script);
compileUnit.compile(Phases.CLASS_GENERATION);
List classes = compileUnit.getClasses();
GroovyClass firstClass = (GroovyClass)classes.get(0);
compiledScriptBytes = firstClass.getBytes();
return compiledScriptBytes;
}
Depending on your requirements, you might want to provide access to the inner classes and you could do that with something like this which finds the class with the matching name instead of assuming the first class:
public byte[] compileGroovyScript(final String className, final String script) {
byte[] compiledScriptBytes = null;
CompilationUnit compileUnit = new CompilationUnit();
compileUnit.addSource(className, script);
compileUnit.compile(Phases.CLASS_GENERATION);
for (Object compileClass : compileUnit.getClasses()) {
GroovyClass groovyClass = (GroovyClass) compileClass;
if(className.equals(groovyClass.getName())) {
compiledScriptBytes = groovyClass.getBytes();
break;
}
}
return compiledScriptBytes;
}
I am running into this myself but having just done an on-demand java compiler at runtime, I believe you are running into the same issue I solved in this code
https://github.com/deanhiller/webpieces/tree/master/runtimecompile/src/main/java/org/webpieces/compiler/api
webpieces/runtimecompile is a re-usable on-demand java compiler using the eclipse compiler.
Now, for groovy, I think you are running into this case
1. you compile ONE script
2. this results in 'multiple' class file objects (I think) just like mine did
3. This is where you need to store EACH in the database SEPARATELY
4. Then you need a classloader that tries to lookup the 'inner classes' when jvm asks for it
5. finally you do a yourclassLoader.loadApplicationClass (much like the one in CompileOnDemandImpl.java in the project above
6. To be clear, step 5 causes step 4 to happen behind the scenes (and that is what is confusing).
If you step through the test case AnonymousByteCacheTest, it pretty much is doing something like that.
you don't need to install ANYTHING to run the build on that project, just clone it and "./gradlew test" and will pass and "./gradlew eclipse" or "./gradlew idea" and it generates IDE files so you can step through it.
It is very very similar. I am trying to get the groovy version working next myself.
Let's say we have a String like this:
String string2code = "variable = 'hello';";
How could we convert that String to a piece of code like this?:
variable = "hello";
GroovyShell is the answer:
String string2code = "variable = 'hello'; return variable.toUpperCase()";
def result = new GroovyShell().evaluate string2code
assert result == "HELLO"
If you're into more complex stuff later, you can compile whole classes using GroovyClassLoader.
private static Class loadGroovyClass( File file ) throws MigrationException {
try {
GroovyClassLoader gcl = new GroovyClassLoader( ExternalMigratorsLoader.class.getClassLoader() );
GroovyCodeSource src = new GroovyCodeSource( file );
Class clazz = gcl.parseClass( src );
return clazz;
}
catch( CompilationFailedException | IOException ex ){
...
}
}
Maybe you can take a look a Janino
Janino is a small java compiler than not only can compile source files, it can compile expressions like the one you have.
I am beginning with Java and testng test cases.
I need to write a class, which reads data from a file and makes an in-memory data structure and uses this data structure for further processing. I would like to test, if this DS is being populated correctly. This would call for dumping the DS into a file and then comparing the input file with the dumped file. Is there any testNG assert available for file matching? Is this a common practice?
I think it would be better to compare the data itself not the written out data.
So I would write a method in the class to return this data structure (let's call it getDataStructure()) and then write a unit test to compare with the correct data.
This only needs a correct equals() method in your data structure class and do:
Assert.assertEquals(yourClass.getDataStructure(), correctData);
Of course if you need to write out the data structure to a file, then you can test the serialization and deserialization separately.
File compare/matching can be extracted to a utility method or something like that.
If you need it only for testing there are addons for jUnit
http://junit-addons.sourceforge.net/junitx/framework/FileAssert.html
If you need file compare outside the testing environment you can use this simple function
public static boolean fileContentEquals(String filePathA, String filePathB) throws Exception {
if (!compareFilesLength(filePathA, filePathB)) return false;
BufferedInputStream streamA = null;
BufferedInputStream streamB = null;
try {
File fileA = new File(filePathA);
File fileB = new File(filePathB);
streamA = new BufferedInputStream(new FileInputStream(fileA));
streamB = new BufferedInputStream(new FileInputStream(fileB));
int chunkSizeInBytes = 16384;
byte[] bufferA = new byte[chunkSizeInBytes];
byte[] bufferB = new byte[chunkSizeInBytes];
int totalReadBytes = 0;
while (totalReadBytes < fileA.length()) {
int readBytes = streamA.read(bufferA);
streamB.read(bufferB);
if (readBytes == 0) break;
MessageDigest digestA = MessageDigest.getInstance(CHECKSUM_ALGORITHM);
MessageDigest digestB = MessageDigest.getInstance(CHECKSUM_ALGORITHM);
digestA.update(bufferA, 0, readBytes);
digestB.update(bufferB, 0, readBytes);
if (!MessageDigest.isEqual(digestA.digest(), digestB.digest()))
{
closeStreams(streamA, streamB);
return false;
}
totalReadBytes += readBytes;
}
closeStreams(streamA, streamB);
return true;
} finally {
closeStreams(streamA, streamB);
}
}
public static void closeStreams(Closeable ...streams) {
for (int i = 0; i < streams.length; i++) {
Closeable stream = streams[i];
closeStream(stream);
}
}
public static boolean compareFilesLength(String filePathA, String filePathB) {
File fileA = new File(filePathA);
File fileB = new File(filePathB);
return fileA.length() == fileB.length();
}
private static void closeStream(Closeable stream) {
try {
stream.close();
} catch (IOException e) {
// ignore exception
}
}
Your choice, but having an utility class with that functionality that can be reused is better imho.
Good luck and have fun.
Personally I would do the opposite. Surely you need a way to compare two of these data structure in the Java world - so the test would read from the file, build the DS, do its processing, and then assert it's equal to an "expected" DS you set up in your test.
(using JUnit4)
#Test
public void testProcessingDoesWhatItShould() {
final DataStructure original = readFromFile(filename);
final DataStructure actual = doTheProcessingYouNeedToDo(original);
final DataStructure expected = generateMyExpectedResult();
Assert.assertEquals("data structure", expected, actual);
}
If this DS is a simple Java Bean. then you can use EqualsBuilder from Apache Commons to compare 2 objects.
compare bytes loaded from file system and bytes you are going to write file system
pseudo code
byte[] loadedBytes = loadFileContentFromFile(file) // maybe apache commons IOUtils.toByteArray(InputStream input)
byte[] writeBytes = constructBytesFromDataStructure(dataStructure)
Assert.assertTrue(java.util.Arrays.equals(writeBytes ,loadedBytes));