I have a spark Scala library and I am building a python wrapper on top of it. One class of my library provides the following method
package com.example
class F {
def transform(df: DataFrame): DataFrame
}
and I am using py4j in the following way to create a wrapper for F
def F():
return SparkContext.getOrCreate()._jvm.com.example.F()
which allows me to call the method transform
The problem is that the python Dataframe object is obviously different from the Java Dataframe object. For this purpose, I need a way to convert a python df to a java one, for which I use the following code from py4j docs
class DataframeConverter(object):
def can_convert(self, object):
from pyspark.sql.dataframe import DataFrame
return isinstance(object, DataFrame)
def convert(self, object, gateway_client):
from pyspark.ml.common import _py2java
return _py2java(SparkContext.getOrCreate(), object)
protocol.register_input_converter(DataframeConverter())
My problem is that now I want to do the inverse: getting a java dataframe from the transform and continue to use it in python. I tried to use protocol.register_output_converter but I couldn't find any useful example, apart for code dealing with java collections.
How can I do that? An obvious solution would be to create a python class F which defines all methods present in java F, forwards all the python calls to the jvm, get back the result and convert it accordingly. This approach works but it implies that I have to redefine all methods of F thus generating code duplication and a lot more maintainance
Related
I have a Java library and I have to build a Python wrapper to it.
I am using py4j, and it's pretty easy to get any instance and any class, complete with method.
The problem is that the type of an object doesn't correspond to its class.
From python:
>>> gw = JavaGateway()
>>> a_circle = gw.jvm.com.geometry.Circle(15)
>>> a_circle.getDiameter()
30
>>> type(a_circle)
<class 'py4j.java_gateway.JavaObject'>
This is almost ok for basic usage, but what if I want to create custom constructors? Should I subclass the JavaObject class? Or it's better to create my classes from scratch and for every method and constructor invoke the corresponding Java method?
Any suggestion for a good way to achieve it? Should i try something different than py4j?
Thank you!
EDIT: for example, I have to wrap a Java class that has 3 methods, one of those methods takes an array as parameter, so I have to inject some code in order to convert.
class Service:
def __init__(self, javaService):
'''
Create a new instance of Service, assigning Java methods as attributes.
Accepts a working instance of Service from Java
This constructor is not meant to be called by the user.
'''
self.subscribe = javaService.subscribe
self.unsubscribe = javaService.unsubscribe
def publish(values):
'''
Wraps the java implementation of this method, converting the list of value from a Python iterable to a Java list.
'''
global java_gateway
parameterValues = ListConverter().convert(values, java_gateway._gateway_client)
return javaService.publish(values)
self.publish = publish
This works, but I am doing only when it is necessary. If the class just works directly, I am not writing anything to wrap it.
I'm writing a Gradle plugin in Scala.
To allow me to configure the plugin and associated tasks via a Gradle DSL I need to be able to invoke methods dynamically (i.e. methods that don't actually exist). If I were writing the plugin in Groovy I'd use methodMissing or propertyMissing as described here.
In Scala I can invoke functions dynamically using dynamic member lookup. This works when invoking the functions from Scala code but I can't seem to find a way to invoke these dynamic functions from Groovy (i.e. when the Scala plugin is used as part of another Gradle project).
Is it possible to utilise Scala's dynamic member lookup from Groovy? Or is this a dead end?
Just figured out the answer to my question. In Scala I need to implement the same methodMissing and propertyMissing functions that I'd implement if I were writing a Groovy class and have the Scala class extend GroovyObjectSupport (needed for Gradle integration).
If anyone else is trying to do something similar below are the function signatures in Scala you need to implement:
class Foo extends GroovyObjectSupport {
def methodMissing(m: String, arg: Any): Any = ???
def propertyMissing(p: String): Any = ???
def propertyMissing(p: String, v: Any): Any = ???
}
I wrote a set of C++ classes and created a DLL that exports one of these C++ classes. I need to instantiate the exported C++ class in a Java class. Is that possible?
I searched the web for a possible solution, but all I have found where solutions using JNA or JNI that import C++ functions only.
Yes, you can instantiate a C++ class from Java.
One way is with SWIG, which can generate Java wrappers for C++ classes.
For example, given a C++ class like this:
class MyClass {
public:
MyClass();
int myMethod( int arg );
}
SWIG allows you to write Java code like this:
MyClass myclass = new MyClass();
int val = myClass.myMethod( 42 );
If you want to instantiate a C++ class from Java, you'll have to write a little glue code (in C++) that instantiates the desired object. Further, you'll need a Java class that corresponds to the C++ class, and you need to have the glue code convert the C++ object into an object of the Java class aforementioned, and keeps them together (i.e., changes to the C++ object should reflect to the Java object, and the other way around).
This tutorial seems to have some pointers how you could do that. Specifically, it tells you how to instantiate a Java object, which is what you will need for the above approach.
I am quite a newbie to Clojure. I am trying to build my computational units (pure functions) in Clojure and bind all these functions into a program using Java.
For accessing Clojure in Java, I have done one thing i.e. Ahead-of-time compilation and class generation. But it looks cumbersome and weakens the idea of using Clojure into my application. So my question is have anyone tried to access Clojure functions in Java (excluding class generation and AOT compilation)? If not, then how to interlink these computational units (or Clojure functions) into a program (where there are several methods interlinked with each other) using purely Clojure?
Just as an overview the general process is:
include the clojure runtime:
import clojure.lang.RT;
use the runtime to load your namespace (which will compile it):
RT.loadResourceScript("path/core.clj");
get the iFn Object for the function you would like to call:
RT.var("mynamespace.core", "main")
and call the invoke method on that object
Have a look at my clojure-utils library. There are a lot of handy tools here for calling Clojure code from Java.
Here's a trivial demonstration:
import mikera.cljutils.Clojure;
public class DemoApp {
public static void main(String [] args) {
String s = "(+ 1 2)";
System.out.println("Evaluating Clojure code: "+s);
Object result=Clojure.eval(s);
System.out.println("=> "+ result);
}
}
I prefer to avoid AOT compilation: instead use the utilities in mikera.cljutils.Clojure to load, compile and execute Clojure code dynamically at runtime.
I'm trying to figure out the difference between
Groovy:
def name = "stephanie"
Java:
Object name = "stephanie"
as both seem to act as objects in that to interact with them i have to cast them to their original intended type.
I was originally on a search for a java equivalent of C#'s dynamic class ( Java equivalent to C# dynamic class type? ) and it was suggested to look at Groovy's def
for example my impression of groovy's def is that I could do the following:
def DOB = new Date(1998,5,23);
int x = DOB.getYear();
however this wont build
thanks,steph
Solution edit:
Turns out the mistake iw as making is I had a groovy class wtih public properties (in my example above DOB) defined with def but then was attemping to access them from a .java class(in my example above calling .getYear() on it). Its a rookie mistake but the problem is once the object leaves a Groovy file it is simply treated as a Object. Thanks for all your help!
Per se, there is not much difference between those two statements; but since Groovy is a dynamic language, you can write
def name = "Stephanie"
println name.toUpperCase() // no cast required
while you would need an explicit cast in the Java version
Object name = "Stephanie";
System.out.println(((String) name).toUpperCase());
For that reason, def makes much more sense in Groovy than unfounded use of Object in Java.
You can experiment with groovy in the groovy web console http://groovyconsole.appspot.com/
Your initial groovy date example works.