I'm trying to run an external program in Java and to read the output. The program is a Linux application in C++ that runs a data mining algorithm and prints the patterns found on standard output. I want to be able to read that output from my Java app and to show the patterns using a table. The problem is that the size of the output is quite big (as a test it produces 6.5MB in about 30 seconds). I'm using ProcessBuilder and reading the output using an InputStreamReader buffered using a BufferedReader as you can see in the following code:
String[] cmd = {"./clogen_periodic", selected, support, "-t 4"};
Process p = new ProcessBuilder(cmd).start();
input = new BufferedReader (new InputStreamReader(p.getInputStream()));
while ((line = input.readLine()) != null) {
...
process line;
...
}
The problem is that the output gets corrupted. When I execute the same program on a console the output is correct but when I use the Java app some lines are merged. More precisely output should be like this
TMEmulation log_pseduo_allocation (34985) (2 45 76 89 90)
__divw clock timer (8273) (4 6 67 4 2)
but it is like this
TMEmulation log_pseduo_allocation (34985) (2__divw 45clock 76timer (89 8273) 904) (6 67 4 2)
Any idea about the possible problem?
Thanks a lot in advance,
Patricia
A few possibilities all to do with the called program
1) as #Artefacto says the C++ program output might not be fully buffered so call setvbuf to make it consistant. ie the first output is partially buffered and second is not and so first flushes after the end of the second. In general buffering can differ if called from the command line and from a process.
2) The program is multi-threaded and the output behaves differently when called from java and so the output timing differs.
Basically you need to look at the code for the called program to force logging/output to be all through the same call.
Try calling in C++ program, setvbuf with the option _IOLBF. The end of the pipe exposed to the C++ is probably unbuffered, while when you run the programs from the command line with |, it's line buffered.
If you're doing a System.out.print() or what ever for debugging in every iteration currently, then try putting all lines from all iterations into one String and give that a try.
Maybe your output method prints out asynchronously. Therefore your printed output may be corrupted but not the one you got from input stream.
Just an idea ...
You should be reading stdout and stderr in separate threads to avoid blocking issues.
I can't say for sure if that will fix your problem but it should be done anyway to avoid other problems you may hit (your app may deadlock waiting on stdout for example).
Luckily there's a very good example with sample code that walks you through this.
http://www.javaworld.com/javaworld/jw-12-2000/jw-1229-traps.html
The article states (see bottom of page 2) that you should always read from stderr and stdout even if you don't need the output to prevent possible deadlocks.
Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block, and even deadlock.
Related
try {
ProcessBuilder pb = new ProcessBuilder("C:\\Users\\--------\\PycharmProjects\\--------\\venv\\Scripts\\Python.exe", "---------.py");
Process p = pb.start();
System.out.println(p.getOutputStream());
}
catch(Exception e){
System.out.println("Exception: " + e);
}
">" So I am working on a program that grabs information from Spotify's API. I have a script in python that feeds a java program the data I need. Unfortunately, I am having trouble getting eclipse to run the .py script by itself. I am using ProcessBuilder and for some reason there are no errors but yet the program isn't executing the python script. I am new to integrating multiple languages in a project so any help is appreciated! I have done hours of research trying to get this figured out. I know that there are similar posts on here regarding the same topic but none of the answers seemed to work for me. Thanks!"<"
It is running the script, you just aren't getting the output, because you did two things wrong. First, see the javadoc for Process.getOutputStream:
Returns the output stream connected to the normal input of the process. Output to the stream is piped into the standard input of the process represented by this Process object.
That's not what you want. To get the output from the process USE Process.getInputStream:
Returns the input stream connected to the normal output of the process. The stream obtains data piped from the standard output of the process represented by this Process object. [plus stderr if merged]
Second, System.out.println(stream) (for an input stream) doesn't print the data that can be received on the stream, it prints only the stream object (as internal classname, atsign, hashcode). To display the data from the python process (i.e. the script) you must read it from the stream and then output the data that was read. There are examples of this everywhere; I can't imagine how you could spend hours without finding at least a hundred. Try for example:
read the output from java exec
Reading InputStream from Java Process
java Process, getInputStream, read newest line only
Cannot get the getInputStream from Runtime.getRunTime.exec()
Printing a Java InputStream from a Process
I'm trying to make a java program that commands through cmd.exe and prints their output. To do this, I'm using this code:
cmdLine = Runtime.getRuntime().exec("cmd.exe");
cmdLineOut = new BufferedReader(new InputStreamReader(cmdLine.getInputStream()));
cmdLineIn = new PrintWriter(cmdLine.getOutputStream());
// ...
cmdLineIn.println(command);
cmdLineIn.flush();
String s = null;
while ((s = cmdLineOut.readLine()) != null)
System.out.println(s);
Although, when input is given, the output is never printed.
EDIT: Solved
The cmdLineOut.readLine() doesn't return null when the input is empty, it freezes. Since readLine freezes at the end no other code is executed, I just put the printing of the readLine in a seperate thread.
If somebody wants to answer this better, go ahead.
You never actually execute the user's command, at least in the snippet you posted. Also, nearly all command prompt "commands" are actually just programs that are on the default program search path. You should probably just Runtime.getRuntime().exec(user_command) for each command. This means that you will have to set up the input and output streams like you have already done for each command. You are right to get input in a separate thread, since attempting to read input will block the current thread until there is actually input to read.
However, some commands, even under UNIX or Linux systems, are "built-in" (like cd), meaning that the command prompt (aka "shell") handles the commands internally. Your program will have to test the user input to see if they are calling a built-in, and specially handle calls to built-in commands. Your program should actually be portable to non-Windows computers. Of course, the user would use different commands (cp instead ofcopy), but the only part you would have to add would be handling for other OS' shells' lists of built-ins (or simply have your program implement a "cross-platform" set of built-ins - this is your shell program, you get to make the rules).
Ok, So i am trying to read the output of a c binary from java code and I am unable to figure out whether the communication channel is blocking or non blocking.
The setup is such:
A java class (A.java) is run
A.java runs a c binary (B.o) using Runtime.getRuntime().exec("B.o"). At this point I have the Process object (returned by Runtime.exec)
A.java reads from the input stream of the Process object using a bufferedreader
A.java outputs the data read from the input stream to a file (output.txt)
The B.o binary simply prints random lines using printf function call.
Now, if I run the above setup, I receive all the data sent by B.o flawlessly. Then to test (the blocking / nonblocking thing), I changed the A.java to sleep for 5 milliseconds after every read from the inputstream of the Process object of B.o. As it turned out, now I wasn't receiving the complete data in A.java send by B.o. This indicates that the communication channel being used is non-blocking (as per my weak understanding).
Then just to make sure, I started looking at the source code of java to see if I was right. And I have found the following so far:
Every call to Runtime.getRuntime().exec(...) ends up in forkAndExec() method in ProcessImpl_md.c. In ProcessImpl_md.c the command is executed, a process is created, and PIPES are setup for communication (using the pipe function call in c). I can't find anywhere in the source code where the PIPES are being set to nonblocking mode (as indicated by my code). I am assuming the PIPES are blocking by default.
I know this is a very bad way to check what I want to check. I am way out of my depth here and I am just head-banging uselessly, I think.
Can anyone point me in the right direction or tell me:
Are the PIPES of a process created through java runtime API are blocking or non-blocking?
When I make A.java sleep after reading from the input stream, why all data is not received? (Assumption being that the PIPE is blocking)
Any non-programmatic way (i.e. I don't have to change the source code of java and etc!) to figure out if the PIPES of a process are blocking or non-blocking?
Thank you.
EDIT: (added code)
Following is not the actual (or even compilable) code but it shows what i am trying to do.
Source of "B.o":
#include <stdio.h>
void main(int argc, char*argv[]){
int a = 0;
for(; a<9000000; a++){
printf("%s", argv[1]);
}
}
Source of "A.java":
<java imports>
public class A{
public static void main(String[] args) throws Exception{
Process p = Runtime.getRuntime().exec("./B.o");
BufferedReader br = new
BufferedReader(new InputStreamReader(p.getInputStream()));
int a = 0;
while(br.readLine() != null){
a++;
Thread.sleep(5);//data missed if this line not commented out
}
br.close();
System.out.println(a);
}
}
PLEASE CHECK MY ANSWER. USELESS QUESTION BY ME.
Whether the communication channels between Java and the external program (there are three, one from Java to native, and two coming back) are operating in blocking or non-blocking mode is not directly relevant to whether all data will be successfully transmitted across each. Likewise, delays between read requests are not directly relevant to whether all data will be successfully transmitted, regardless of blocking vs. non-blocking I/O in your particular implementation of java.lang.Process.
Really, your efforts to probe blocking vs. non-blocking inter-process I/O are futile, because the I/O interface provided to your Java program is based on InputStream and OutputStream, which provide only for blocking I/O. Even if non-blocking I/O were involved at some low level of the implementation, I can't think of any way for your program to detect that.
With respect to your specific questions, however:
Are the PIPES of a process created through java runtime API are blocking or non-blocking?
They could be either, but they are more likely blocking because that better matches the interface presented to the Process user.
When I make A.java sleep after reading from the input stream, why all data is not received? (Assumption being that the PIPE is blocking)
I can only speculate, but the problem is likely in the external program. Possibly it goes to sleep when its output buffer fills, and nothing happens to wake it up. It might help to invoke myProcess.getOutputStream().close() if your Java program is not sending data to the external program. It's in any case a good idea to close that stream once you've written to it everything you're ever going to write.
Any non-programmatic way (i.e. I don't have to change the source code of java and etc!) to figure out if the PIPES of a process are blocking or non-blocking?
Potentially you could run the VM under strace or connect a native debugger to it, and analyze the VM's behavior that way. If you mean to do this from inside Java then the answer is a resounding "NO". Your Java program will see blocking behavior under all circumstances because the contracts of InputStream and OutputStream demand it.
I was making a big blunder and was completely off base. Posting this answer to clear things up (though I would like to delete the question altogether). I wanted to know if the communication channels between a C binary run from Java code are blocking or non-blocking. And I mentioned that the data was missing when I made my java code sleep after reading from the input stream of the created process (of C code). The data wasn't missing because of that. I had actually put a timer in Java code after which to terminate the process of the C binary. And since the PIPES are blocking, it wasn't able to receive all the data before the timer expired. I was misinterpreting this loss of data to mean that the PIPES were non-blocking. Confirmed this by running STRACE on the created C binary process. There were no EAGAIN errors on the write syscalls. My bad. But thank you very much to all for taking out the time to respond.
I am monitoring and Minecraft server and I am making a setup file in Python. I need to be able to run two threads, one running the minecraft_server.jar in the console window, while a second thread is constantly checking the output of the minecraft_server. Also, how would I input into the console from Python after starting the Java process?
Example:
thread1 = threading.Thread(target=listener)
thread2 = minecraft_server.jar
def listener():
if minecraft_server.jarOutput == "Server can't keep up!":
sendToTheJavaProccessAsUserInputSomeCommandsToRestartTheServer
It's pretty hard to tell here, but I think what you're asking is how to:
Launch a program in the background.
Send it input, as if it came from a user on the console.
Read its output that it tries to display to a user on the console.
At the same time, run another thread that does other stuff.
The last one is pretty easy; in fact, you've mostly written it, you just need to add a thread1.start() somewhere.
The subprocess module lets you launch a program and control its input and output. It's easiest if you want to just feed in all the input at once, wait until it's done, then process all the output, but obviously that's not your case here, so it's a bit more involved:
minecraft = subprocess.Popen(['java', 'path/to/minecraft_server.jar', '-other', 'args],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
I'm merging stdout and stderr together into one pipe; if you want to read them separately, or send stderr to /dev/null, or whatever, see the docs; it's all pretty simple. While we're making assumptions here, I'm going to assume that minecraft_server uses a simple line-based protocol, where every command, every response, and every info message is exactly one line (that is, under 1K of text ending in a \n).
Now, to send it input, you just do this:
minecraft.stdin.write('Make me a sandwich\n')
Or, in Python 3.x:
minecraft.stdin.write(b'Make me a sandwich\n')
To read its output, you do this:
response = minecraft.stdout.readline()
That works just like a regular file. But note that it works like a binary file. In Python 2.x, the only difference is that newlines don't get automatically converted, but in Python 3.x, it means you can only write bytes (and compatible objects), not strs, and you will receive bytes back. There are good reasons for that, but if you want to get pipes that act like text files instead, see the universal_newlines (and possibly bufsize) arguments under Frequently Used Arguments and Popen Constructor.
Also, it works like a blocking file. With a regular file, this rarely matters, but with a pipe, it's quite possible that there will be data later, but there isn't data yet (because the server hasn't written it yet). So, if there is no output yet (or not a complete line's worth, since I used readline()), your thread just blocks, waiting until there is.
If you don't want that, you probably want to create another thread to service stdout. And its function can actually look pretty similar to what you've got:
def listener():
for line in minecraft.stdout:
if line.strip() == "Server can't keep up!":
minecraft.stdin.write("Restart Universe\n")
Now that thread can block all day and there's no problem, because your other threads are still going.
Well, not quite no problem.
First it's going to be hard to cleanly shut down your program.
More seriously, the pipes between processes have a fixed size; if you don't service stdout fast enough, or the child doesn't service stdin fast enough, the pipe can block. And, the way I've written things, if the stdin pipe blocks, we'll be blocked forever in that stdin.write and won't get to the next read off stdout, so that can block too, and suddenly we're both waiting on each other forever.
You can solve this by having another thread to service stdout. The subprocess module itself includes an example, in the Popen._communicate function used by all the higher-level functions. (Make sure to look at Python 3.3 or later, because earlier versions had bugs.)
If you're in Python 3.4+ (or 3.3 with a backport off PyPI), you can instead use asyncio to rewrite your program around an event loop and handle the input and output the same way you'd write a reactor-based network server. That's what all the cool kids are doing in 2017, but back in late 2014 many people still thought it looked new and scary.
If all of this is sounding like a lot more work than you signed on for, you may want to consider using pexpect, which wraps up a lot of the tedious details, and makes some simplifying assumptions that are probably true in your case.
I have a little executable program in C that produce a lot of output to a file.
When I call this program with Runtime, like this:
Runtime r = Runtime.getRuntime();
Process p = null;
p = r.exec("./my_program -in input.file -out output.file", null, new File(System.getProperty("java.io.tmpdir")));
When the program produce low output everything is ok, but when I call "*my_program*" with a large input it will produce a large quantity of output to the output.file, but in this case my program in Java freeze and nothing happen...
I test "*my_program*" in terminal with a lot of large inputs and everything is ok, but when I call the program in Java with Runtime.exec, the Java program freeze.
--
Thanks in advance
Make sure you're reading from the Process's .getOutputStream() and .getErrorStream() if you aren't already. Looking at your code snippet, it appears that you're just executing .exec(...) (and maybe waiting for it to complete with a call not shown to .waitFor()?).
Per http://download.oracle.com/javase/6/docs/api/java/lang/Process.html (emphasis added):
The parent process uses these streams to feed input to and get output
from the subprocess. Because some native platforms only provide
limited buffer size for standard input and output streams, failure to
promptly write the input stream or read the output stream of the
subprocess may cause the subprocess to block, and even deadlock.