Develop Reference

How to Run a jar from another jar and show the console output from the first runnable during the execution

I'm running a jar file from another jar like here somebody answers but waiting for the process.
Process proc = Runtime.getRuntime().exec("java -jar A.jar" + stringParams);
try {
proc.waitFor();
} catch (InterruptedException e) {
e.printStackTrace();
}
InputStream in = proc.getInputStream();
InputStream err = proc.getErrorStream();
My problem comes when i have no feedback on the status of the program that is called, but i don't want my program continues beyond those lines. I would need the standard and error outputs but the results are shown when the execution is over. Is there any way of executing and getting those streams while the jar is still running?

Buffered/unbuffered
It seems like an issue with buffered output.
Executed process (in this case java -jar <path>) buffers output and writes it only when it's done (in big chunks, we don't like that!)
So one way to go is execute process through unbuffering (very hacky tools):
unbuffered <command>
stdbuf -i0 -o0 -e0 <command>
use terminal emulation
Hacking
stdbuf is part of GNU tools.
https://www.gnu.org/software/coreutils/manual/html_node/stdbuf-invocation.html
unbuffered is part of expect package.
https://wiki.tcl.tk/3548
The key thing is making the program thinking that it's in interactive mode (like you are launching it from console).
The first two options are very hacky and do not work in all cases (idk if java command works with them?)
Emulation
The third option is most promising.
We launch a program (terminal emulator) that emulates interactive terminal making program think it's working in real active session!
Pty4j
You might use pty4j too:
From there: https://github.com/traff/pty4j
// The command to run in a PTY...
String[] cmd = { "java", "-jar", "path_to_jar" };
// The initial environment to pass to the PTY child process...
String[] env = { "TERM=xterm" };
PtyProcess pty = PtyProcess.exec(cmd, env);
OutputStream os = pty.getOutputStream();
InputStream is = pty.getInputStream();
// ... work with the streams ...
// wait until the PTY child process terminates...
int result = pty.waitFor();
// free up resources.
pty.close();
Zt-exec
Maybe it's worth trying zt-exec?
I have no idea how it executes commands.
But it may be it (I didn't test that).
Using https://github.com/zeroturnaround/zt-exec
new ProcessExecutor().command("java", "-jar path_to_jar")
.redirectOutput(new LogOutputStream() {
#Override
protected void processLine(String line) {
...
}
})
.execute();
That should work, but I didn't test that.
In general, there are no ways to nicely resolve your problem.
Depending on what platforms you want to target consider using unbuffered, stdbuff or the (slowest) terminal emulation...
Please let me know if that helps and good luck! :)

How to write a custom Protobuf CodeGenerator in Java

I'm trying to write a custom code generator for an in-house proprietary programming language. I figured I could write the generator in Java, using the protoc plugin guide. My main() does something like this:
public static void main(String[] args) throws IOException {
CodeGenerator gen = new CodeGenerator();
PluginProtos.CodeGeneratorRequest codeGeneratorRequest = PluginProtos.CodeGeneratorRequest.parseFrom(args[0].getBytes());
codeGeneratorRequest.getProtoFileList().forEach(gen::handleFile);
// get the response and do something with it
//PluginProtos.CodeGeneratorResponse response = PluginProtos.CodeGeneratorResponse.newBuilder().build();
//response.writeTo(System.out);
}
(Obviously I've only just started; wanted to get something stubby working first before actually writing the generation logic)
Problem is: how do I invoke protoc with the --plugin argument to generate code in my custom language, using my plugin? I tried writing a shell script to do it like this:
#!/bin/bash
java -cp ./codegen.jar CodeGeneratorMain "$#"
And I tried invoking protoc like this: protoc --plugin=protoc-gen-code --code_out=./build hello.proto however, when I run that, I get this error:
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 0
at CodeGeneratorMain.main(CodeGeneratorMain.java:12)
--code_out: protoc-gen-code: Plugin failed with status code 1.
As though it's not passing the CodeGeneratorRequest on stdin at all. How would I verify that? Am I doing something obviously wrong?

So after reading and re-reading the docs I realized my very silly error: protoc passes the parsed input via stdin not via argv. That means that if I change this: PluginProtos.CodeGeneratorRequest codeGeneratorRequest = PluginProtos.CodeGeneratorRequest.parseFrom(args[0].getBytes()); to this: PluginProtos.CodeGeneratorRequest codeGeneratorRequest = PluginProtos.CodeGeneratorRequest.parseFrom(System.in);
it works.

pyspark: call a custom java function from pyspark. Do I need Java_Gateway?

I wrote the following MyPythonGateway.java so that I can call my custom java class from Python:
public class MyPythonGateway {
public String findMyNum(String input) {
return MyUtiltity.parse(input).getMyNum();
}
public static void main(String[] args) {
GatewayServer server = new GatewayServer(new MyPythonGateway());
server.start();
}
}
and here is how I used it in my Python code:
def main():
gateway = JavaGateway() # connect to the JVM
myObj = gateway.entry_point.findMyNum("1234 GOOD DAY")
print(myObj)
if __name__ == '__main__':
main()
Now I want to use MyPythonGateway.findMyNum() function from PySpark, not just a standalone python script. I did the following:
myNum = sparkcontext._jvm.myPackage.MyPythonGateway.findMyNum("1234 GOOD DAY")
print(myNum)
However, I got the following error:
... line 43, in main:
myNum = sparkcontext._jvm.myPackage.MyPythonGateway.findMyNum("1234 GOOD DAY")
File "/home/edamameQ/spark-1.5.2/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 726, in __getattr__
py4j.protocol.Py4JError: Trying to call a package.
So what did I miss here? I don't know if I should run a separate JavaApplication of MyPythonGateway to start a gateway server when using pyspark. Please advice. Thanks!
Below is exactly what I need:
input.map(f)
def f(row):
// call MyUtility.java
// x = MyUtility.parse(row).getMyNum()
// return x
What would be the best way to approach this? Thanks!

First of all the error you see usually means the class you're trying to use is not accessible. So most likely it is a CLASSPATH issue.
Regarding general idea there are two important issues:
you cannot access SparkContext inside an action or transformation so using PySpark gateway won't work (see How to use Java/Scala function from an action or a transformation? for some details)). If you want to use Py4J from the workers you'll have to start a separate gateways on each worker machine.
you really don't want to pass data between Python an JVM this way. Py4J is not designed for data intensive tasks.

In PySpark before start calling the method -
myNum = sparkcontext._jvm.myPackage.MyPythonGateway.findMyNum("1234 GOOD DAY")
you have to import MyPythonGateway java class as follows
java_import(sparkContext._jvm, "myPackage.MyPythonGateway")
myPythonGateway = spark.sparkContext._jvm.MyPythonGateway()
myPythonGateway.findMyNum("1234 GOOD DAY")
specify the jar containing myPackage.MyPythonGateway with --jars option in spark-submit

If input.map(f) has inputs as an RDD for example, this might work, since you can't access the JVM variable (attached to spark context) inside the executor for a map function of an RDD (and to my knowledge there is no equivalent for #transient lazy val in pyspark).
def pythonGatewayIterator(iterator):
results = []
jvm = py4j.java_gateway.JavaGateway().jvm
mygw = jvm.myPackage.MyPythonGateway()
for value in iterator:
results.append(mygw.findMyNum(value))
return results
inputs.mapPartitions(pythonGatewayIterator)

all you need to do is compile jar and add to pyspark classpath with --jars or --driver-class-path spark submit options. Then access class and method with below code-
sc._jvm.com.company.MyClass.func1()
where sc - spark context
Tested with Spark 2.3. Keep in mind, you can call JVM class method only from driver program and not executor.

Calling KeyTool, redirecting System.out has no effect

So we want to use the bog-standard keytool utility that ships with a JRE. But rather than going through the trouble of finding the correct path and executable extension, spawning a subprocess, and running the executable, we collectively had the bright idea ("remember, none of us is as dumb as all of us!") to just call KeyTool's main() directly. It's implemented in Java code and also shipped with the JRE, and contains the standard "classpath" exception to the GPL so we can link against it.
Looking at the KeyTool source, there's even some provision made for this sort of thing: there are comments like "if you're calling KeyTool.main() directly in your own Java program, then [helpful reminder]" and the top-level main() is capable of propagating exceptions to calling code instead of just dying with System.exit(). Being able to just build the same command-line argument array and run KeyTool.main(stuff) instead of having to mess with platform differences seems like a very Java-esque thing to do, right?
In practice, weird things happen and we don't know why.
We want to capture any output from running KeyTool, which starts off like this:
// jdk/src/share/classes/sun/security/tools/KeyTool.java, line 331:
public static void main(String[] args) throws Exception {
KeyTool kt = new KeyTool();
kt.run(args, System.out);
}
private void run(String[] args, PrintStream out) throws Exception {
// real code here, sends to 'out'
}
The KeyTool entry points don't allow us to pass a PrintStream, it's hardcoded to use System.out. That should be okay thanks to System.setOut. We have an OutputStream subclass which feeds to a JTextComponent, but for initial coding, redirecting to a text file is fine. So our code does
PrintStream orig = System.out;
try {
System.out.println("This is the last visible console line");
System.setOut(new PrintStream("redirect_test.txt"));
System.out.println("This is now redirected!");
KeyTool.main(keytool_argv); // "-help" and "-debug" for now
}
catch all the myriad ways things might go wrong { ... }
finally {
System.setOut(orig);
System.out.println("Back to normal console output");
}
But when we run the code, the redirect_test.txt file contains only "This is now redirected!". The output from keytool's "-help" still shows up on the console, along with the before-and-after println calls.
There are some other oddities in calling KeyTool directly, like the package and class name has changed between Java 7 and Java 8, but that's easy to deal with via reflection. (The comments in the KeyTool source in Java 8 still refer to the Java 7 name, heh.) The only thing just freaky weird is how its "System.out" is strangely not affected by the same redirection that works everywhere else. (No, there are no weird import statements bringing in a special System replacement.)
Here's an online copy of Java 7's KeyTool.java if you don't happen to have OpenJDK sitting around.

You just need to redirect both System.out and System.err, since the usage instructions get printed to the standard error stream instead of the standard output stream. Try this:
PrintStream original = System.out;
PrintStream redirected = new PrintStream("redirect_test.txt")
try {
System.out.println("This is the last visible console line");
System.setOut(redirected);
System.setErr(redirected);
System.out.println("This is now redirected!");
KeyTool.main(keytool_argv); // "-help" and "-debug" for now
}
catch all the myriad ways things might go wrong { ... }
finally {
System.setOut(original);
System.setErr(original);
System.out.println("Back to normal console output");
}

Any way to delay process execution?

I'm writing a wrapper script using Groovy (but the question is really a Java one) and would like to know if it's possible to create a Process without letting it run first. The problem is that the Process starts running and generating output on stdout and stderr. I would like to forward those to their appropriate destinations and at the same time create a merged stream for processing within the script. The problem I'm running into, however, is that the Process generates output too quickly and the output on the first two lines is a bit garbled. I would like to setup the streams before the process starts running. Any way to do that?

This consumes the output and error streams into two separate StringWriters, but I don't see anything "garbled"
new StringWriter().with { out ->
new StringWriter().with { err ->
'ls /tmp'.execute().with { proc ->
consumeProcessOutput( out, err )
waitFor()
}
println "OUT: $out"
println "ERR: $err"
}
}