Develop Reference

GraalVM - embedding python multi-file project in java

I couldn't find a solution create a polyglot source out of multiple files in GraalVM.
What exactly I want to achieve:
I have a python project:
my-project:
.venv/
...libs
__main__.py
src/
__init__.py
Service.py
Example sourcecode:
# __main__.py
from src.Service import Service
lambda url: Service(url)
# src/Service.py
import requests
class Service:
def __init__(self, url):
self.url = url
def invoke(self):
return requests.get(self.url)
This is very simple example, where we've got an entry-point script, project is structured in packages and there is one external library (requests).
It works, when I run it from command-line with python3 __main__.py, but I can't get it work, when embedding it in Java (it can't resolve imports).
Example usage in java:
import org.graalvm.polyglot.Context;
import org.graalvm.polyglot.Source;
import org.graalvm.polyglot.Value;
import java.io.File;
import java.io.IOException;
public class Runner {
public static void main(String[] args) throws IOException {
Context context = Context.newBuilder("python")
.allowExperimentalOptions(true)
.allowAllAccess(true)
.allowIO(true)
.build();
try (context) {
// load lambda reference:
Value reference = context.eval(Source.newBuilder("python", new File("/path/to/my-project/__main__.py")).build());
// invoke lambda with `url` argument (returns `Service` object)
Value service = reference.execute("http://google.com");
// invoke `invoke` method of `Service` object and print response
System.out.println("Response: " + service.getMember("invoke").execute());
}
}
}
It fails with Exception in thread "main" ModuleNotFoundError: No module named 'src'.
The solution works for javascript project (having similar index.js to __main__.py, its able to resolve imports - GraalVM "sees" other project's files, but somehow it doesn't, when using python.
I found out, that python is able to run zip package with project inside, but this also doesn't work with GraalVM.
Is there any chance to accomplish it? If not, maybe there is a similar tool to webpack for python (if I could create a single-file bundle, it should also work).
Btw, I don't know python at all, so I may missing something.
Thanks for any help!

calling clojure from Java (Clojure Interop)

Calling Java from Clojoure is quite simple and straightforward but the inverse has proven to be unpredictable.
They seem to be two ways of doing it:
1)the following classes
i) import clojure.java.api.Clojure; ,
ii) import clojure.lang.IFn;
2)compile your clojure into an uberjar then import it into the java
code.
I have opted for the 2nd option as it's more straight forward.
Here is the clojure code
(ns com.test.app.service
(:gen-class
:name com.test.app.service
:main false
:methods [^{:static true} [returned [int] int]]))
(defn returned
[number]
(* 2 number))
(defn -returned
[number]
(returned number))
Here is the Java code.
package com.s.profile;
import java.util.*;
import com.microsoft.azure.serverless.functions.annotation.*;
import com.microsoft.azure.serverless.functions.*;
import com.test.app.service;
/**
* Azure Functions with HTTP Trigger.
*/
public class Function {
/**
* This function listens at endpoint "/api/hello". Two ways to invoke it using "curl" command in bash:
* 1. curl -d "HTTP Body" {your host}/api/hello
* 2. curl {your host}/api/hello?name=HTTP%20Query
*/
#FunctionName("hello")
public HttpResponseMessage<String> hello(
#HttpTrigger(name = "req", methods = {"get", "post"}, authLevel = AuthorizationLevel.ANONYMOUS) HttpRequestMessage<Optional<String>> request,
final ExecutionContext context) {
context.getLogger().info("Java HTTP trigger processed a request.");
// Parse query parameter
String query = request.getQueryParameters().get("name");
String name = request.getBody().orElse(query);
if (name == null) {
return request.createResponse(400, "Please pass a name on the query string or in the request body");
} else {
service.returned(4);
context.getLogger().info("process data" );
return request.createResponse(200, "Hellos, " + name );
}
}
}
When ever I make the "service.returned(4);" the system never returns. I can't quite figure out why to me it comes off like the function doesn't return from Clojure but I can't see the cause.
Just to add some context I have tried it when its a simple hello world java app which just prints out the result and it works. It's when I try implement it in the Azure functions.

Please see this question for a running example:
How to invoke Clojure function directly from Java
I would suggest simplifying your code at first, then adding back in the Azure stuff one line at a time in case some interaction there is causing the problem.

I followed these instructions and it seemed to resolve the error of class not found. It seems as though when running the command
mvn azure-functions:run
It doesn't automatically find all imported libraries. You either have to use
maven-assembly-plugin
maven-shade-plugin

How can I export a Java class (.jar) from Clojure?

I am relatively novice at Clojure and Java. I have an existing Clojure project someone else wrote that I am trying to embed in NodeJS using node-java.
Clojure
The project defines a namespace that provides certain public functions, like so:
(ns my.namespace
(:require ...etc...))
(defn dosomething ...)
(defn dosomethingelse ...)
I have built the project with leiningen (lein jar and lein uberjar).
Questions
The #import() docs on node-java say I need to import a java class like so:
const java = require('java');
var Test = java.import('Test');
How can I access these functions (presumably as Java class static methods?)
Am I approaching this all wrong? =)
Update
Thanks to Magos (answer below) I made a little more progress. It turns out I can use (:gen-class :name my.name) in the (ns ...) scope to tell it to generate a class. If I add a profile to the project.clj like so:
...
:profiles {
...
:uberjar {:aot :all}
}
...
It will compile and I can now see the class in Node. I still haven't figured out how to export the methods, though. Working on that part now.

Since someone else wrote the Clojure, I'll assume you aren't in control of it. The recommended approach for using Clojure code from another JVM language is bootstrapping from the class clojure.java.api.Clojure. This class allows you to resolve Vars from Clojure, and by resolving and invoking the other core Clojure functions you can load your own code. Based on your example it might look something like this:
const java = require('java');
var clojure = java.import('clojure.java.api.Clojure');
IFn require = clojure.var("clojure.core", "require");
require.invoke(clojure.read("my.namespace"));
IFn doSomething = clojure.var("my.namespace","dosomething");
//doSomething.invoke(....
If you do control the Clojure, :gen-class allows you to export functions as methods of the namespace's generated class.

Note: I arrived at this through a combination of Magos's answer and clartaq's comment to the question.
Here are simple instructions for how to do it. Let's assume you have this (simple) clojure code:
(ns my.namespace
"a trivial library"
(:require [something.else :as other]))
(defn excite
"make things more exciting"
[mystr]
(print-str mystr "!"))
Use these steps to expose the excite method.
Create an exposed version of the method with the same signature by prefixing it with -. It should simply call the function you wish to expose.
(defn -excite [mystr] (excite mystr))
Declare in (ns ...) that you want to generate a class and export methods.
(ns my.namespace
"a trivial library"
(:require [something.else :as other])
(:gen-class
:name my.classname
:methods [
; metadata mtd.name signature returns
#^{:static true} [excite [String] void]
]))
Note that you may optionally remove the #^{:static true} if you do not wish to provide this as a static method.
In your project.clj (assuming you are using leiningen), add ahead-of-time compilation instructions to your :profiles entry:
:profiles {:uberjar {:aot :all}}
Compile your uberjar:
lein uberjar
The resulting .jar file will have a class my.classname with a static method excite.

JNI native call *prototypes* in their own class/namespace?

I have written JNI wrappers to export the API of a C application (G-WAN) which embeds a JVM. The native calls are implemented in the C application and exported with RegisterNatives().
Ideally I would have a 'gwan' class for the G-WAN API:
import gwan // G-WAN API
public class hello {
public static int jmain(long env, String[] args) {
gwan.xbuf_cat(gwan.get_reply(env), "Hello World");
return 200; // HTTP status (200:'OK')
}
}
I would like to do something like the "#import gwan" above to import the native call prototypes, but currently I only have the following (which works):
public class hello {
public static int jmain(long env, String[] args) {
gwan_xbuf_cat(gwan_get_reply(env), "Hello World");
return 200; // HTTP status (200:'OK')
}
public static native long gwan_get_reply(long env);
public static native void gwan_xbuf_cat(long ctx, String str);
}
Again, the implementation of the native calls in made in the G-WAN executable (not in a Java class stored on disk).
Because the G-WAN API is quite large, I would like to have the native call prototypes in their own 'gwan' class (or namespace) if possible (like in the first hello example above).
Any suggestion about how to do that?
(please post Java or JNI code because I am not a Java expert)
Disclamer: I am involved in the development of this project.

I would suggest that you read following paper on JNI from Sun now Oracle
http://java.sun.com/docs/books/jni/html/jniTOC.html
And after that it should be understandable but some pseudocode and its not tested would be to move the two gwanapi calls into its own file named gwanapi.java
public class gwanapi {
public static native long get_reply(long answer);
public static native void xbuf_cat(long ctx,String str);
}
then you compile that file with javac gwanapi.java -> output: gwanapi.class
you type javah -jni for the c/c++ header implementation:
javah -jni gwanapi
the next you should in your hello java class is to call static{ System.loadLibrary("gwanapi");}
Pseudo code and NOT tested
public class hello{
static{
System.loadLibrary("gwanapi");
}
public static int jmain(long env,String args[]){
gwanapi.xbuf_cat(gwanapi.get_reply(env),"Hello World!");
return 200;
}
}
and you should be good to go.
But I might have missed a point or two but I think this is the smallest amount of work you should do.
Oh by the way http://en.wikipedia.org/wiki/Java_Native_Interface is also some form of source for JNI calls and how it works and lead you to more sites with more information.
Thanks

Being a C programmer, I have had to read the C source code of the JVM to find that Java remaps Class (path) names with dots instead of slashes.
Since the G-WAN directory hierarchy uses IP addresses to define listeners and virtual hosts (192.168.10.10_80/#domain.com/csp), those dots were confusing the FindClass() JNI call, making it fail to find the classes.
I also found that the classpath path separator is a ":" for Unix and a ";" for Windows. That was not the cause of my problem, but it might cause the same issue.
Finally, I stopped using the GCJ compiler's JVM because it does not support formating doubles (since at least 2006). Using either OpenJDK or the SUN/ORACLE JVM works as expected.
All works fine now. I post all this here just in case it may help others.

Embedding swank-clojure in java program

Based on the Embedding section of http://github.com/technomancy/swank-clojure,
I'm using the following to test it out. Is there a better way to do
this that doesn't use Compiler? Is there a way to programmatically
stop swank? It seems start-repl takes control of the thread. What
would be a good way to spawn off another thread for it and be able to
kill that thread programatically.
import clojure.lang.Compiler;
import java.io.StringReader;
public class Embed {
public static void main(String[] args) throws Exception {
final String startSwankScript =
"(ns my-app\n" +
" (:use [swank.swank :as swank]))\n" +
"(swank/start-repl) ";
Compiler.load(new StringReader(startSwankScript));
}
}
Any help much appreciated,
hhh

Would it be acceptable to you to implement the Embed class in Clojure? You could do that with gen-class (see Meikel Brandmeyer's tutorial for details) and AOT compilation.
The code could go something like
(ns your-app.Embed
(:require [swank.swank :as swank])
(:gen-class
:methods [[startSwank [] void]]))
(defn -startSwank []
(swank/start-repl))
(add anything else you require); then in the Java part of your application, you could import your Clojure-prepared class, instantiate it and call .startSwank() on the instance.
Not sure about programmatically stopping Swank... I'd be curious to know of a good way to do that myself. (And I'll be back with an update if I figure it out; otherwise, I'd love to read somebody else's answer detailing how to go about that.)