it is clear to me how to extend Python with C++, but what if I want to write a function in Java to be used with numpy?
Here is a simple scenario: I want to compute the average of a numpy array using a Java class. How do I pass the numpy vector to the Java class and gather the result?
Thanks for any help!
I spent some time on my own question and would like to share my answer as I feel there is not much information on this topic on stackoverflow. I also think Java will become more relevant in scientific computing (e.g. see WEKA package for data mining) because of the improvement of performance and other good software development features of Java.
In general, it turns out that using the right tools it is much easier to extend Python with Java than with C/C++!
Overview and assessment of tools to call Java from Python
http://pypi.python.org/pypi/JCC: because of no proper
documentation this tool is useless.
Py4J: requires to start the Java process before using python. As
remarked by others this is a possible point of failure. Moreover, not many examples of use are documented.
JPype: although development seems to be death, it works well and there are
many examples on it on the web (e.g. see http://kogs-www.informatik.uni-hamburg.de/~meine/weka-python/ for using data mining libraries written in Java) . Therefore I decided to focus
on this tool.
Installing JPype on Fedora 16
I am using Fedora 16, since there are some issues when installing JPype on Linux, I describe my approach.
Download JPype, then modify setup.py script by providing the JDK path, in line 48:
self.javaHome = '/usr/java/default'
then run:
sudo python setup.py install
Afters successful installation, check this file:
/usr/lib64/python2.7/site-packages/jpype/_linux.py
and remove or rename the method getDefaultJVMPath() into getDefaultJVMPath_old(), then add the following method:
def getDefaultJVMPath():
return "/usr/java/default/jre/lib/amd64/server/libjvm.so"
Alternative approach: do not make any change in the above file _linux.py, but never use the method getDefaultJVMPath() (or methods which call this method). At the place of using getDefaultJVMPath() provide directly the path to the JVM. Note that there are several paths, for example in my system I also have the following paths, referring to different versions of the JVM (it is not clear to me whether the client or server JVM is better suited):
/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre/lib/x86_64/client/libjvm.so
/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre/lib/x86_64/server/libjvm.so
/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/amd64/server/libjvm.so
Finally, add the following line to ~/.bashrc (or run it each time before opening a python interpreter):
export JAVA_HOME='/usr/java/default'
(The above directory is in reality just a symbolic link to my last version of JDK, which is located at /usr/java/jdk1.7.0_04).
Note that all the tests in the directory where JPype has been downloaded, i.e. JPype-0.5.4.2/test/testsuite.py will fail (so do not care about them).
To see if it works, test this script in python:
import jpype
jvmPath = jpype.getDefaultJVMPath()
jpype.startJVM(jvmPath)
# print a random text using a Java class
jpype.java.lang.System.out.println ('Berlusconi likes women')
jpype.shutdownJVM()
Calling Java classes from Java also using Numpy
Let's start implementing a Java class containing some functions which I want to apply to numpy arrays. Since there is no concept of state, I use static functions so that I do not need to create any Java object (creating Java objects would not change anything).
/**
* Cookbook to pass numpy arrays to Java via Jpype
* #author Mannaggia
*/
package test.java;
public class Average2 {
public static double compute_average(double[] the_array){
// compute the average
double result=0;
int i;
for (i=0;i<the_array.length;i++){
result=result+the_array[i];
}
return result/the_array.length;
}
// multiplies array by a scalar
public static double[] multiply(double[] the_array, double factor) {
int i;
double[] the_result= new double[the_array.length];
for (i=0;i<the_array.length;i++) {
the_result[i]=the_array[i]*factor;
}
return the_result;
}
/**
* Matrix multiplication.
*/
public static double[][] mult_mat(double[][] mat1, double[][] mat2){
// find sizes
int n1=mat1.length;
int n2=mat2.length;
int m1=mat1[0].length;
int m2=mat2[0].length;
// check that we can multiply
if (n2 !=m1) {
//System.err.println("Error: The number of columns of the first argument must equal the number of rows of the second");
//return null;
throw new IllegalArgumentException("Error: The number of columns of the first argument must equal the number of rows of the second");
}
// if we can, then multiply
double[][] the_results=new double[n1][m2];
int i,j,k;
for (i=0;i<n1;i++){
for (j=0;j<m2;j++){
// initialize
the_results[i][j]=0;
for (k=0;k<m1;k++) {
the_results[i][j]=the_results[i][j]+mat1[i][k]*mat2[k][j];
}
}
}
return the_results;
}
/**
* #param args
*/
public static void main(String[] args) {
// test case
double an_array[]={1.0, 2.0,3.0,4.0};
double res=Average2.compute_average(an_array);
System.out.println("Average is =" + res);
}
}
The name of the class is a bit misleading, as we do not only aim at computing the average of a numpy vector (using the method compute_average), but also multiply a numpy vector by a scalar (method multiply), and finally, the matrix multiplication (method mult_mat).
After compiling the above Java class we can now run the following Python script:
import numpy as np
import jpype
jvmPath = jpype.getDefaultJVMPath()
# we to specify the classpath used by the JVM
classpath='/home/mannaggia/workspace/TestJava/bin'
jpype.startJVM(jvmPath,'-Djava.class.path=%s' % classpath)
# numpy array
the_array=np.array([1.1, 2.3, 4, 6,7])
# build a JArray, not that we need to specify the Java double type using the jpype.JDouble wrapper
the_jarray2=jpype.JArray(jpype.JDouble, the_array.ndim)(the_array.tolist())
Class_average2=testPkg.Average2
res2=Class_average2.compute_average(the_jarray2)
np.abs(np.average(the_array)-res2) # ok perfect match!
# now try to multiply an array
res3=Class_average2.multiply(the_jarray2,jpype.JDouble(3))
# convert to numpy array
res4=np.array(res3) #ok
# matrix multiplication
the_mat1=np.array([[1,2,3], [4,5,6], [7,8,9]],dtype=float)
#the_mat2=np.array([[1,0,0], [0,1,0], [0,0,1]],dtype=float)
the_mat2=np.array([[1], [1], [1]],dtype=float)
the_mat3=np.array([[1, 2, 3]],dtype=float)
the_jmat1=jpype.JArray(jpype.JDouble, the_mat1.ndim)(the_mat1.tolist())
the_jmat2=jpype.JArray(jpype.JDouble, the_mat2.ndim)(the_mat2.tolist())
res5=Class_average2.mult_mat(the_jmat1,the_jmat2)
res6=np.array(res5) #ok
# other test
the_jmat3=jpype.JArray(jpype.JDouble, the_mat3.ndim)(the_mat3.tolist())
res7=Class_average2.mult_mat(the_jmat3,the_jmat2)
res8=np.array(res7)
res9=Class_average2.mult_mat(the_jmat2,the_jmat3)
res10=np.array(res9)
# test error due to invalid matrix multiplication
the_mat4=np.array([[1], [2]],dtype=float)
the_jmat4=jpype.JArray(jpype.JDouble, the_mat4.ndim)(the_mat4.tolist())
res11=Class_average2.mult_mat(the_jmat1,the_jmat4)
jpype.java.lang.System.out.println ('Goodbye!')
jpype.shutdownJVM()
I consider Jython to be one of the best options - which makes it seamless to use java objects in python. I actually integrated weka with my python programs, and it was super easy. Just import the weka classes and call them as you would in java within the python code.
http://www.jython.org/
I'm not sure about numpy support, but the following might be helpful:
http://pypi.python.org/pypi/JCC/
Related
From this answer, I learnt that, every Groovy script compiles to a class that extends groovy.lang.Script class
Below is a test groovy script written for Jenkins pipeline in Jenkins editor.
node('worker_node'){
print "***1. DRY principle***"
def list1 = [1,2,3,4]
def list2 = [10,20,30,40]
def factor = 2
def applyFactor = {e -> e * factor}
print(list1.each(applyFactor))
print(list2.each(applyFactor))
print "***2. Higher order function***"
def foo = { value, f -> f(value *2) }
foo(3, {print "Value is $it"})
foo(3){
print "Value is $it"
}
}
How to compile this groovy script to see the class generated(source code)?
The class generated is bytecode, not source code. The source code is the Groovy script.
If you want to see something similar to what the equivalent Java source code would look like, use groovyc to compile the script as usual, and then use a Java decompiler to produce Java source (this question's answers lists a few).
That's subject to the usual caveats on decompiled code, of course. High-level information is lost in the process of compiling. Decompilers have to guess a bit to figure out the best way to represent what might have been in the original source. For instance, what was a for loop in the original code may end up being decompiled as a while loop instead.
groovy in jenkins pipeline is a Domain Specific Language.
It's not a plain groovy.
However if you remove node(){ } then it seems to be groovy in your case.
and you can run it in groovyconsole or compile to class with groovyc
just download a stable groovy binary and extract it.
if you have java7 or java8 on your computer - you can run groovyconsole and try your code there.
with Ctrl+T you can see the actual class code generated for your script.
I have been reading about the Panama Project recently.
I understand that it will be the next generation replacement to JNI - it will allow java developers to code on the native layer using Java (which is amazing IMHO).
The usage is simple from what I can tell looking at jnr-posix, for example:
public class FileTest {
private static POSIX posix;
#BeforeClass
public static void setUpClass() throws Exception {
posix = POSIXFactory.getPOSIX(new DummyPOSIXHandler(), true);
}
#Test
public void utimesTest() throws Throwable {
// FIXME: On Windows this is working but providing wrong numbers and therefore getting wrong results.
if (!Platform.IS_WINDOWS) {
File f = File.createTempFile("utimes", null);
int rval = posix.utimes(f.getAbsolutePath(), new long[]{800, 200}, new long[]{900, 300});
assertEquals("utimes did not return 0", 0, rval);
FileStat stat = posix.stat(f.getAbsolutePath());
assertEquals("atime seconds failed", 800, stat.atime());
assertEquals("mtime seconds failed", 900, stat.mtime());
// The nano secs part is available in other stat implementations. We really just want to verify that the
// nsec portion of the timeval is passed through to the POSIX call.
// Mac seems to fail this test sporadically.
if (stat instanceof NanosecondFileStat && !Platform.IS_MAC) {
NanosecondFileStat linuxStat = (NanosecondFileStat) stat;
assertEquals("atime useconds failed", 200000, linuxStat.aTimeNanoSecs());
assertEquals("mtime useconds failed", 300000, linuxStat.mTimeNanoSecs());
}
f.delete();
}
}
// ....
// ....
// ....
}
My question is this - having worked with JNI, and knowing how cumbersome it is, will there be a solution for porting existing JNI solutions to the Panama format?
IE - go over the generated (via the deprecated javah) C header file and given implementation in C of the header file, identify functions which can be replaced by the Panama API, then generate a java output file?
Or will existing JNI solutions need to be refactored by hand?
Additional links :
OpenJDK: Panama
Working with Native Libraries in Java
JEP 191: Foreign Function Interface thanks to a comment made by Holger
The JNI format is as follows:
Java -> JNI glue-code library -> Native code
One of the goals of project panama is to remove this middle layer and get:
Java -> Native code
The idea is that you can use a command line tool to process a native header (.h) file to generate a Java interface for calling the native code, and the JDK code will do the rest at runtime as far as connecting the 2 together goes.
If your current JNI code does a lot of stuff in this glue-code layer, then that might have to be re-written on the Java side when porting to panama. (this depends on how much could be done automatically by the used interface extraction tool).
But if you are using something like JNA or JNR then moving to panama should be relatively easy, since those 2 have very similar APIs, where you bind an interface to a native library as well.
But questions like:
will there be a solution for porting existing JNI solutions to the Panama format?
Are difficult to answer, since nobody can predict the future. I feel that there are enough differences between panama and JNI that an automatic 1-to-1 conversion between the 2 will probably not be possible. Though if your glue-code is not doing much besides forwarding arguments then the interface extraction tool will probably be able to do all the work for you.
If you're interested you could take a look at the early access builds of panama that started shipping recently: https://jdk.java.net/panama/
Or watch a recent talk about it: https://youtu.be/cfxBrYud9KM
I am using LibSVM library on Weka 3.6 and experiencing similar problem as in here (for Java) and here (for Python)
The libSVM library generates a lot of logs simliar to this
optimization finished, #iter = 399
nu = 0.9503376170384973
obj = -124.54791151883072, rho = 0.0528133707297996
nSV = 257, nBSV = 97
I followed the solution using -q parameters by setting this parameter in my code:
LibSVM svm = new LibSVM();
String[] options = {"-q"};
svm.setOptions(options);
Although this solution seems to work in Python but I doesn't work in my Java code.
Another solution suggests using Log4j and disable some level of logs, however, I don't want to add another library to my code.
Now, I'm wondering is there any clean and simple solution to disable libSVM logs?
LibSVM library for Weka with FQN of "weka.classifiers.functions.LibSVM" is a wrapper around svm algorithm to create a common interface for Java programmers are using Weka API.
Inside "LibSVM.jar" there is another jar file which named "libsvm.jar" which is the main algorithm. Contrary to LibSVM which use common Java naming conventions, the naming convention inside "libsvm.jar" is different. Inside "libsvm" package there is a class named "svm". Because I had used "svm" as my variable name, the "svm" class was invisible.
After knowing that, I followed the instruction in here and changed the "svm" to "libsvm.svm" and this is the code which is working for me. In addition, I put this code in a static block of my code to have it for all my usages.
static{
libsvm.svm.svm_set_print_string_function(new libsvm.svm_print_interface() {
#Override
public void print(String s) {
} // Disables svm output
});
}
Finally, I am using LibSVM without annoying logs.
after having looked at many discussions like for example :
Install commons math library for java in Ubuntu
Correctly Importing Apache Commons Math Package
I am still stuck and I am not able to make the following code to work :
import org.apache.commons.math3.linear;
class linearAlgebraLearning{
public static void main(String[] args){
// Create a real matrix with two rows and three columns, using a factory
// method that selects the implementation class for us.
double[][] matrixData = { {1d,2d,3d}, {2d,5d,3d}};
RealMatrix m = MatrixUtils.createRealMatrix(matrixData);
// One more with three rows, two columns, this time instantiating the
// RealMatrix implementation class directly.
double[][] matrixData2 = { {1d,2d}, {2d,5d}, {1d, 7d}};
RealMatrix n = new Array2DRowRealMatrix(matrixData2);
// Note: The constructor copies the input double[][] array in both cases.
// Now multiply m by n
RealMatrix p = m.multiply(n);
System.out.println(p.getRowDimension()); // 2
System.out.println(p.getColumnDimension()); // 2
// Invert p, using LU decomposition
RealMatrix pInverse = new LUDecomposition(p).getSolver().getInverse();
}
}
So here are what I have done step by step.
First I installed Apache using
sudo apt-get install libcommons-math3-java
Then I have looked where commons-math3-java has been installed.
dpkg -L libcommons-math3-java
/.
/usr
/usr/share
/usr/share/maven-repo
/usr/share/maven-repo/org
/usr/share/maven-repo/org/apache
/usr/share/maven-repo/org/apache/commons
/usr/share/maven-repo/org/apache/commons/commons-math3
/usr/share/maven-repo/org/apache/commons/commons-math3/3.2
/usr/share/maven-repo/org/apache/commons/commons-math3/3.2/commons-math3-3.2.pom
/usr/share/maven-repo/org/apache/commons/commons-math3/debian
/usr/share/maven-repo/org/apache/commons/commons-math3/debian/commons-math3-debian.pom
/usr/share/doc
/usr/share/doc/libcommons-math3-java
/usr/share/doc/libcommons-math3-java/changelog.Debian.gz
/usr/share/doc/libcommons-math3-java/copyright
/usr/share/java
/usr/share/java/commons-math3.jar
/usr/share/maven-repo/org/apache/commons/commons-math3/3.2/commons-math3-3.2.jar
/usr/share/maven-repo/org/apache/commons/commons-math3/debian/commons-math3-debian.jar
/usr/share/java/commons-math3-3.2.jar
then I used ( as told in the Install commons math library for java in Ubuntu )
javac -cp .:/usr/share/java/commons-math3-3.2.jar linearAlgebraLearning.java
however I still an import error message :
linearAlgebraLearning.java:1: error: cannot find symbol
import org.apache.commons.math3.linear;
And additional errors since the compiler does not find the classes (like RealMatrix). I know that this kind question has been asked many times. People here might be tired of seeing this question... But I would be really happy if you could help me.
Ps : Because there is some bug with Eclipse on my linux distribution I am not using IDE and use gedit and the terminal.
I installed every libcommons package first:
sudo apt install libcommons\*
then set the classpath:
export CLASSPATH="/usr/share/java/commons-math3.jar:/usr/share/java/commons-lang3.jar"
Your java should pick it up automatically. I tested it with jshell and it was able to autocomplete/import BlockRealMatrix for example:
jshell> import org.apache.commons.math3.linear.BlockRealMatrix
jshell> BlockRealMatrix foo = new BlockRealMatrix(2, 2);
foo ==> BlockRealMatrix{{0.0,0.0},{0.0,0.0}}
Installation instructions: http://docs.opencv.org/doc/tutorials/introduction/desktop_java/java_dev_intro.html
I have downloaded everything. Everything seems to be working except when I run this sample program:
import org.opencv.core.Core;
import org.opencv.core.CvType;
import org.opencv.core.Mat;
public class Main {
public static void main(String[] args) {
System.out.println("Welcome to OpenCV " + Core.VERSION);
System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
Mat m = Mat.eye(3, 3, CvType.CV_8UC1);
System.out.println("m = " + m.dump());
}
}
I get the output:
m = [1, 0, 0;
0, 1, 0;
0, 0, 1]
(which I hope is right).
But I also get these error messages:
objc[63784]: Class CVWindow is implemented in both /Users/.../cv2.so and /Users/... /libopencv_java246.dylib. One of the two will be used. Which one is undefined.
objc[63784]: Class CVView is implemented in both /Users/.../cv2.so and /Users/.../libopencv_java246.dylib. One of the two will be used. Which one is undefined.
objc[63784]: Class CVSlider is implemented in both /Users/.../cv2.so and /Users/.../libopencv_java246.dylib. One of the two will be used. Which one is undefined.
objc[63784]: Class CaptureDelegate is implemented in both /Users/... /cv2.so and /Users/jsuit/opencv/lib/libopencv_java246.dylib. One of the two will be used. Which one is undefined.
I have tried moving the cv2.so file to another folder, but then the program won't compile.
The problem has to do, as far as I can make out, with a reference to the Python libraries (the .so version) that ends up included within the Java libraries themselves. This would seem to be a build configuration error (following the instructions does produce it).
I was able to eliminate the double-definition error by re-building the Java version without support for the Python libraries in it, using the following in the cmake step (all else the same):
cmake -D BUILD_SHARED_LIBS=OFF -D BUILD_NEW_PYTHON_SUPPORT=NO ..
The newly produced Java libraries and .jar work just as before, but without the error message.
(Note that I can't guarantee this won't cause other problems, especially if you want to do some sort of mixed-language programming, but it does produce libraries useful for Java and Eclipse. You can always build multiple versions of OpenCV, too, some with support for Python, some without, and use whichever one you like if you switch languages at some point.)
Hat tip: http://answers.opencv.org/question/16015/mac-opencv-246-java-exception/