I have a python application which I cant edit its a black box from my point of view. The python application knows how to process text and return processed text.
I have another application written in Java which knows how to collect non processed texts.
Current state, the python app works in batch mode every x minutes.
I want to make the python
processing part of the process: Java app collects text and request the python app to process and return processed text as part of a flow.
What do you think is the simplest solution for this?
Thanks,
Rod
I don't know nothing about Jython and the like. I guess it's the best solution if you can execute two programs without executing a new process each time the Java app needs to transform text. Anyway a simple proof of concept is to execute a separate process from the Java App to make it work. Next you can enhance the execution with all that tools.
Executing a separate process from Java
String[] envprops = new String[] {"PROP1=VAL1", "PROP2=VAL2" };
Process pythonProc = Runtime.getRuntime().exec(
"the command to execute the python app",
envprops,
new File("/workingdirectory"));
// get an outputstream to write into the standard input of python
OutputStream toPython = pythonProc.getOutputStream();
// get an inputstream to read from the standard output of python
InputStream fromPython = pythonProc.getInputStream();
// send something
toPython.write(.....);
// receive something
fromPython.read(....);
Important: chars are NOT bytes
A lot of people understimate this.
Be careful with char to byte conversions (remember Writers/Readers are for chars, Input/OutputStreams are for bytes, encoding is necesary for convertir one to another, you can use OuputStreamWriter to convert string to bytes and send, InputStreamReader to convert bytes to chars and read them).
Look into Jython - you can run Python programs directly from Java code, and interact seamlessly back and forth.
Use ProcessBuilder to execute your Python code as a filter:
import java.io.BufferedReader;
import java.io.InputStreamReader;
public class PBTest {
public static void main(String[] args) {
ProcessBuilder pb = new ProcessBuilder("python", "-c", "print 42");
pb.redirectErrorStream(true);
try {
Process p = pb.start();
String s;
BufferedReader stdout = new BufferedReader (
new InputStreamReader(p.getInputStream()));
while ((s = stdout.readLine()) != null) {
System.out.println(s);
}
System.out.println("Exit value: " + p.waitFor());
p.getInputStream().close();
p.getOutputStream().close();
p.getErrorStream().close();
} catch (Exception ex) {
ex.printStackTrace();
}
}
}
Expose one of the two as a service of some kind, web service maybe. Another option is to port the python code to Jython
One possible solution is jpype. This allows you to launch a JVM from Python and pass data back and forth between them.
Another solution may be to write the Python program as a filter (reading data from stdin and writing result to stdout) then run it as a pipe. However I do not know how well Java supports this - according to the Sun docs their concept of pipes only supports communication between threads on the same JVM.
An option is making the python application work as a server, listens for request via sockets (TCP).
Related
I am pretty new to both Java and Python, although I have some experience in programming. For an assignement, I need to create a program which uses Java in some way. My project would use Java as an UX, and Python for signal processing and feature extraction, since it has some good tools for that.
However, my question is how to establish communication between both programse. Maybe this question has been asked before, but since I do not know the best terms, I could not find answers.
In my Java Program, I can get the file path to a .csv file, send it to Python, and Python returns the original signals and processed signals. For that, I wrote:
private static void sendPython(String path, JTextField console)
{
String pathPython = "C:\\Users\\gonca\\Desktop\\untitled0.py";
String [] cmd = new String[3];
cmd[0] = "python";
cmd[1] = pathPython;
cmd[2] = path;
Runtime r = Runtime.getRuntime();
try
{
Process p = r.exec(cmd);
BufferedReader in = new BufferedReader(new InputStreamReader(p.getInputStream()));
String s = "";
while((s = in.readLine()) != null)
{
console.setText(s);
}
}
catch (IOException e)
{
console.setText("Unable to run python script");
}
}
I was thinking of having the py script output the signals in separated lines, with values separated by "," or ";", and using the BufferedRead to read each line, separate the values and create a new ArrayList from the separated values.
However, before starting working harder to do that, I would like to know if that is the best way to proceed, or is there a more efficient way to do it.
There are more ways to do that:
Solution 1:
Use the python library from java with System.loadLibrary, and call the python method. (here is an example using C/C++: Calling a python method from C/C++, and extracting its return value)
Solution 2:
Launch python as another process, and use D-Bus (or something similar) to communicate with it.
Since you have not mentioned how robust your application is I can think of a solution which can be used if you are planning for a higher level architecture.
Create a python based web application (HTTP server) with all logic to process your files.
Create a java app which can communicate via HTTP python server to get the CSV processed information.
Try to avoid Runtime execution of commands with in your codes that faces user as if it not is properly managed there is always a chance for security breach
I have problems sending data from java to a (linux)-subprocess created by ProcessBuilder/Process.
A shell-only based basic example would look like as follows and works fine.
echo "hello world" | cat - >/tmp/receive.dat
Now, I want to substituted the echo "hello world" by a java program which should internally create a new process (the cat - >/tmp/receive.dat) and then send data to it.
I tried the following, but the file /tmp/receive.dat remains untouched:
String[] cmdArray = { "/bin/cat", "-", ">/tmp/receive.dat" };
ProcessBuilder builder = new ProcessBuilder (cmdArray);
builder.directory (new File ("/tmp"));
Process p = builder.start ();
OutputStream pos = p.getOutputStream ();
byte [] bytes = "hello world".getBytes ();
pos.write (bytes);
pos.close ();
p.waitFor ();
The same happens under Windows, of course with an adapted cmdArray:
cmd /c type con >c:\tmp\receive.dat
Printing directly to system.out from java is no alternative as many subprocesses should be called within the livecycle of the java program.
thx for any help!
Tombo
The issue here is that /bin/cat does not actually write to files, it merely writes to standard out. The output redirection >/tmp/receive.dat is actually performed by the shell, but you are bypassing the shell by invoking cat in this manner.
If what you are trying to achieve is merely an OutputStream that writes to a file, then doing that via standard java I/O (e.g., FileOutputStream and friends) is what you want. It is also cross-platform (i.e., Windows-friendly), unlike anything that depends on the shell.
Regarding the comment about not being able to merely write to standard out from java because of subprocesses - any subprocess you invoke can inherit standard out from their parent process (the java process - look at ProcessBuilder.Redirect.INHERIT). So even if you are invoking subprocesses from java, redirecting all the output to a file should still work in the same way as your initial example (where the java program replaces the echo command).
You probably want ProcessBuilder#redirectOutput(File), as the > functionality is not of cat, rather what is calling cat (in our sense, the process builder).
I'm launching wkhtmltopdf from within my Java app (part of a Tomcat server, running in debug mode within Eclipse Helios on Win7 64-bit): I'd like to wait for it to complete, then Do More Stuff.
String cmd[] = {"wkhtmltopdf", htmlPathIn, pdfPathOut};
Process proc = Runtime.getRuntime().exec( cmd, null );
proc.waitFor();
But waitFor() never returns. I can still see the process in the Windows Task Manager (with the command line I passed to exec(): looks fine). AND IT WORKS. wkhtmltopdf produces the PDF I'd expect, right where I'd expect it. I can open it, rename it, whatever, even while the process is still running (before I manually terminate it).
From the command line, everything is fine:
c:\wrk>wkhtmltopdf C:\Temp\foo.html c:\wrk\foo.pdf
Loading pages (1/6)
Counting pages (2/6)
Resolving links (4/6)
Loading headers and footers (5/6)
Printing pages (6/6)
Done
The process exits just fine, and life goes on.
So what is it about runtime.exec() that's causing wkhtmltopdf to never terminate?
I could grab proc.getInputStream() and look for "Done", but that's... vile. I want something that is more general.
I've calling exec() with and without a working directory. I've tried with and without an empty "env" array. No joy.
Why is my process hanging, and what can I do to fix it?
PS: I've tried this with a couple other command line apps, and they both exhibit the same behavior.
Further exec woes.
I'm trying to read standard out & error, without success. From the command line, I know there's supposed to be something remarkably like my command line experience, but when I read the input stream returned by proc.getInputStream(), I immediately get an EOL (-1, I'm using inputStream.read()).
I checked the JavaDoc for Process, and found this
The parent process uses these streams to feed input to and get output from the subprocess. Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the [b]subprocess to block, and even deadlock[/b].
Emphasis added. So I tried that. The first 'read()' on the Standard Out inputStream blocked until I killed the process...
WITH WKHTMLTOPDF
With the generic command line ap & no params so it should "dump usage and terminate", it sucks out the appropriate std::out, then terminates.
Interesting!
JVM version issue? I'm using 1.6.0_23. The latest is... v24. I just checked the change log and don't see anything promising, but I'll try updating anyway.
Okay. Don't let the Input Streams fill or they'll block. Check. .close() can also prevent this, but isn't terribly bright.
That works in general (including the generic command line apps I've tested).
In specific however, it falls down. It appears that wkhtmltopdf is using some terminal manipulation/cursor stuff to do an ASCII-graphic progress bar. I believe this is causing the inputStream to immediately return EOF rather than giving me the correct values.
Any ideas? Hardly a deal-breaker, but it would definitely be Nice To Have.
I had the same exact issue as you and I solved it. Here are my findings:
For some reason, the output from wkhtmltopdf goes to STDERR of the process and NOT STDOUT. I have verified this by calling wkhtmltopdf from Java as well as perl
So, for example in java, you would have to do:
//ProcessBuilder is the recommended way of creating processes since Java 1.5
//Runtime.getRuntime().exec() is deprecated. Do not use.
ProcessBuilder pb = new ProcessBuilder("wkhtmltopdf.exe", htmlFilePath, pdfFilePath);
Process process = pb.start();
BufferedReader errStreamReader = new BufferedReader(new InputStreamReader(process.getErrorStream()));
//not "process.getInputStream()"
String line = errStreamReader.readLine();
while(line != null)
{
System.out.println(line); //or whatever else
line = reader.readLine();
}
On a side note, if you spawn a process from java, you MUST read from the stdout and stderr streams (even if you do nothing with it) because otherwise the stream buffer will fill and the process will hang and never return.
To futureproof your code, just in case the devs of wkhtmltopdf decide to write to stdout, you can redirect stderr of the child process to stdout and read only one stream like this:
ProcessBuilder pb = new ProcessBuilder("wkhtmltopdf.exe", htmlFilePath, pdfFilePath);
pb.redirectErrorStream(true);
Process process = pb.start();
BufferedReader inStreamReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
Actually, I do this in all the cases where I have to spawn an external process from java. That way I don't have to read two streams.
You should also read the streams of the spawned process in different threads if you dont want your main thread to block, since reading from streams is blocking.
Hope this helps.
UPDATE: I raised this issue in the project page and was replied that this is by design because wkhtmltopdf supports giving the actual pdf output in STDOUT. Please see the link for more details and java code.
A process has 3 streams: input, output and error. you can read both output and error stream at the same time using separate processes. see this question and its accepted answer and also this one for example.
You should read from the streams in a different thread.
final Semaphore semaphore = new Semaphore(numOfThreads);
final String whktmlExe = tmpwhktmlExePath;
int doccount = 0;
try{
File fileObject = new File(inputDir);
for(final File f : fileObject.listFiles()) {
if(f.getAbsolutePath().endsWith(".html")) {
doccount ++;
if(doccount >500 ) {
LOG.info(" done with conversion of 1000 docs exiting ");
break;
}
System.out.println(" inside for before "+semaphore.availablePermits());
semaphore.acquire();
System.out.println(" inside for after "+semaphore.availablePermits() + " ---" +f.getName());
new java.lang.Thread() {
public void run() {
try {
String F_ = f.getName().replaceAll(".html", ".pdf") ;
ProcessBuilder pb = new ProcessBuilder(whktmlExe , f.getAbsolutePath(), outPutDir + F_ .replaceAll(" ", "_") );//"wkhtmltopdf.exe", htmlFilePath, pdfFilePath);
pb.redirectErrorStream(true);
Process process = pb.start();
BufferedReader errStreamReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
String line = errStreamReader.readLine();
while(line != null)
{
System.err.println(line); //or whatever else
line = errStreamReader.readLine();
}
System.out.println("after completion for ");
} catch (Exception e) {
e.printStackTrace();
}finally {
System.out.println(" in finally releasing ");
semaphore.release();
}
}
}.start();
}
}
}catch (Exception ex) {
LOG.error(" *** Error in pdf generation *** ", ex);
}
while (semaphore.availablePermits() < numOfThreads) {//till all threads finish
LOG.info( " Waiting for all threads to exit "+ semaphore.availablePermits() + " --- " +( numOfThreads - semaphore.availablePermits()));
java.lang.Thread.sleep(10000);
}
I want to execute a external .exe program from within java. The .exe is a CLI application which takes input in runtime( scanf() ) and outputs depending on the input. I can call the program to execute from java using
Process p = Runtime.getRuntime().exec("cmd /c start a.exe");
instead of
Process p = Runtime.getRuntime().exec("cmd /c start a.exe");
But I think it is also possible to call a program from within java. I have my whole program written in C++ just need a GUI which is written in java. There are a few things to notice:=
1) The communication with the .exe should be runtime (not through main(args) )
2) The java program should take the outputs and store in some variable / panel to use for future
3) Program to be executed can differ ( for example user may select a .exe that doesnt take any input at all)
........So basically the java GUI will act as a RuntimeEnv
public void runEXE()
{
String s = null;
try {
Process p = Runtime.getRuntime().exec("cmd /c a.exe");
System.exit(0);
}
catch (IOException e) {
System.out.println("exception happened - here's what I know: ");
e.printStackTrace();
System.exit(-1);
}
}
I know there are a lot of questions about this topic out there. But i cant find any of them much useful.
Rather ugly little function that I use. This takes in the command to be passed to Runtime.getRuntime().exec, then saves the results into a String, and returns the String at the end. You can pick whether you only want the last line (or all output) and whether you want to save the stdout or stderr string from the process.
private static String systemResult(String cmd, boolean append, boolean useErr)
{
String result = "";
try{
// spawn the external process
//printCmd(cmd);
Process proc = Runtime.getRuntime().exec(cmd);
LineNumberReader lnr1 = new LineNumberReader(new InputStreamReader(proc.getErrorStream()));
LineNumberReader lnr2 = new LineNumberReader(new InputStreamReader(proc.getInputStream()));
String line;
int done = 0;
while(lnr1 != null || lnr2 != null){
try{
if(lnr1.ready()){
if((line = lnr1.readLine()) != null){
//System.err.println("A:" +line);
if(useErr){
if(append) result = result + line + "\n";
else result = line;
}
}
}else if(done == 1){
done = 2;
}
}catch(Exception e1){
try{ lnr1.close(); }catch(Exception e2){}
lnr1 = null;
}
try{
if(lnr2.ready()){
if((line = lnr2.readLine()) != null){
//System.err.println("====>Result: " + line);
if(!useErr){
if(append) result = result + line + "\n";
else result = line;
}
}
}else if(done == 2){
break;
}
}catch(Exception e1){
try{ lnr2.close(); }catch(Exception e2){}
lnr2 = null;
}
try{
proc.exitValue();
done = 1;
}catch(IllegalThreadStateException itsa){}
}
if(lnr1 != null) lnr1.close();
if(lnr2 != null) lnr2.close();
try{
proc.waitFor();
}catch(Exception ioe){
}finally{
try{
proc.getErrorStream().close();
proc.getInputStream().close();
proc.getOutputStream().close();
}catch(Exception e){}
proc = null;
}
}catch(Exception ioe){
}
return result;
}
You could use JNI as #linuxuser27 suggests, or you could use SWIG which helps to make the process of communicating from Java --> C++ a little less painful.
Google Protocol Buffers would be a good option for Java/C++ interoperability.
Protocol buffers are Google's
language-neutral, platform-neutral,
extensible mechanism for serializing
structured data – think XML, but
smaller, faster, and simpler. You
define how you want your data to be
structured once, then you can use
special generated source code to
easily write and read your structured
data to and from a variety of data
streams and using a variety of
languages – Java, C++, or Python.
I would look at JNI and use some type of IPC to communicate with the C++ project.
Update:
JNI is a way for Java to interface with the underlying native environment the JRE is running on. This method would require you to create a DLL that is loaded into the JRE when your Java program starts. This JNI DLL would then contain a method which could be called from within your Java program that would pass data into the JNI DLL that could then communicate to the C++ project via a named pipe or shared memory.
The named piped would be create using the CreateNamedPipe Win32 API. Within the JNI DLL you would most likely create a server and in the C++ project you would create the client. Note that the server example is multi-threaded but can easily be converted to a single thread model for simplicity.
Note that this is not a simple task. The other answers offer some approaches that are easier, JNA and passing data to the C++ project via stdin.
A couple of things:
First and foremost, if you haven't done so, read this critically important article: When Runtime.exec won't
Next, there are several ways for Java to communicate to other applications, and probably the easiest is via standard input and output streams. Have you tried using these?
Next there's JNI and the easier JNA, but you stated that your C++ program is running via CLI which suggests to me that you have a .NET dll, not a true Windows dll. Is this so? If so, it would make communication between Java and C++ more difficult.
You can execute .exe programs directly. You just need to supply the full path. Relative path will also be ok.
You can use pipes to interface with an external process: Sending Input to a Command, Reading Output from a Command
The solution can be found here
Problem getting output and passing input to a executing process running under java
I am using Java on Windows XP and want to be able to send commands to another program such as telnet.
I do not want to simply execute another program. I want to execute it, and then send it a sequence of commands once it's running.
Here's my code of what I want to do, but it does not work:
(If you uncomment and change the command to "cmd" it works as expected. Please help.)
This is a simplified example. In production there will be many more commands sent, so please don't suggest calling "telnet localhost".
try
{
Runtime rt = Runtime.getRuntime();
String command = "telnet";
//command = "cmd";
Process pr = rt.exec(command);
BufferedReader processOutput = new BufferedReader(new InputStreamReader(pr.getInputStream()));
BufferedWriter processInput = new BufferedWriter(new OutputStreamWriter(pr.getOutputStream()));
String commandToSend = "open localhost\n";
//commandToSend = "dir\n" + "exit\n";
processInput.write(commandToSend);
processInput.flush();
int lineCounter = 0;
while(true)
{
String line = processOutput.readLine();
if(line == null) break;
System.out.println(++lineCounter + ": " + line);
}
processInput.close();
processOutput.close();
pr.waitFor();
}
catch(Exception x)
{
x.printStackTrace();
}
That looks OK, as it won't be producing that much output, but you should really read and write in separate threads so it doesn't fill up the buffer and block waiting you to read before you reach the next step.
So if it's reaching the point where you flush the command you send to it, find out whether the Windows telnet client supports receiving commands from standard input rather than a console by piping the text you're sending to its standard input to it in a command prompt.
For example, echo dir c:\ | cmd causes cmd to run, list the c: drive contents and exit, much the same behaviour as if you typed dir c:\ into the console. But echo open localhost | telnet causes telnet to clear the screen then exit, rather than behaving the same way as if you typed it into the console. As telnet needs to mask user input for passwords, it's quite likely that it's using the console API rather than reading from standard input. It's help doesn't list any command arguments to tell it to read from standard input, so maybe you need to use a telnet implementation which is better suited to scripting.
It's not directly an answer to your question, but...
Instead of using Runtime.exec() you should use a ProcessBuilder and redirect stderr to stdout (ProcessBuilder.redirectErrorStream(true)). Otherwise your process could block if it writes something to stderr (Windows doesn't like it when the output of a process isn't read).
If you want to control a telnet session programatically from Java, you might be able to use this Java telnet library... you can do the same things (open connections, send username/password, send commands and receive results) but without actually spawning a separate process.
You may take a look at the Telnet Ant task you can call it directly in your code with out having to use a build.xml file.
You can also take a look at the source code and see how they do it.