Develop Reference

Get STDOUT from Runtime.getRuntime().exec() line-by-line rather than all at once

I'm executing a script from Java with the following code.
try (BufferedReader input = new BufferedReader(new InputStreamReader(Runtime.getRuntime()
.exec("ruby test.rb").getInputStream()))) {
String line = null;
while ((line = input.readLine()) != null) {
System.out.println("Got line: " + line);
}
}
Where test.rb is simply:
puts "one"
sleep 1
puts "two"
The problem is that all of the input arrives in the BufferedReader at the same time. There should be a 1 second delay between the two lines. Is there a way to flush the Inputstream after every line in the script?
What I'm trying to accomplish
I want to call some Ruby scripts from a Java GUI and have the output of the script show up incrementally in a textpane. Using the code above, STDOUT from the script shows up in the textpane all at once when the script terminates (and the Inputstream is presumably flushed).
Edit:
The issue was that Ruby was buffering STDOUT. Fixed by $stdout.sync = true.

Have you tried Thread.sleep(1000) after System.out.println(). You must catch some InterruptedException. Your editor will tell you for sure.

No input from ProcessBuilder's InputStream until external process is closed

Whilst building an wrapper for an console application I came across this weird issue where the Input Stream connected to the output (stdout) of the external process is completely blank until the external process exits.
My code as below:
import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
public class Example{
public static void main(String args[]) throws IOException{
File executable = ...
ProcessBuilder pb = new ProcessBuilder(executable.getCanonicalPath());
pb.redirectErrorStream(true);
Process p = pb.start();
BufferedReader br = new BufferedReader(new InputStreamReader(p.getInputStream(), Charset.forName("UTF-8")));
String line;
while((line = br.readLine()) != null){
System.out.println(line);
}
}
}
I've tried several variants of reading from the input stream and all resulted in the same behavior.
I've tried:
CharBuffer charBuf = CharBuffer.allocate(1000);
InputStreamReader isr = new InputStreamReader(p.getInputStream(), Charset.forName("UTF-8"));
while(isr.read(charBuf) != -1){
System.out.print(charBuf.flip().toString());
}
and
byte[] buf = new byte[1000];
int r;
while((r = p.getInputStream().read(buf)) != -1){
System.out.print(new String(buf, 0, r));
}
all to no avail.
Somewhere along the line the output from the external process is being buffered (indefinitely) and I can't really figure out where. Loading the process from the command line seems to work fine where I see output coming out instantaneously. The strangest part is where the fact that the termination of the external process results in a flood of all the "buffered" output at once (a lot of stuff).
Unfortunately I don't have access to the source of the external process but given that it writes to stdout fine when in a console shouldn't really make a difference there (as far as I know).
Any ideas are welcome.
Edit:
One answer recommended me to rewrite the reader for the output and error streams to run on a separate thread. My actual implementation is doing that! And yet the problem still exists. The code posted above is a SSCCE of my actual code condensed for readability purposes, the actual code involves a separate thread for reading from the InputStream.
Edit 2:
User FoggyDay seems to have provided the answer which defines how the behavior of output buffering change when outputting between console and non-consoles. Whilst processes which detect that they are writing to a console use line buffering (buffered flushed every new line), writing to non-consoles (everything that it detects to not be a console) may be fully buffered (to a size of something like 8K). If I make the external process spam (8K of lorem ipsum in a for loop) output does indeed appear. I guess my question now is how to make my java program trigger line buffering on the external process.

To your question "how to make my java program trigger line buffering on the external process":
On Linux you can use the "stdbuf" program (coreutils package): stdbuf -oL your_program program_args
You only need to change stdout since stderr is unbuffered by default. The man page of setlinebuf gives additional background information if you're interested: http://linux.die.net/man/3/setlinebuf

Some software checks if it is writing to a terminal and switches behavior to unbuffered output. This could be a reason, why it works in the terminal. Pipe the output to cat and see if the output still appears immediately.
Another reason could be that the program is waiting for input or a close of its stdin before it does something, although this does not really match the symptoms described so far.

ProcessBuilder running processes in foreground

I want to run an executable written in C++ and to see the cmd promt associated with it in foreground, since the executable prints some lines in the cmd.
I have written the following code, but all processes are created and run in background (In this code I open the dummy cmd.exe process, not my process).
Process p = new ProcessBuilder("C:\\Windows\\System32\\cmd.exe").start();
How can i enable foreground running of processes?
Thanks!

The issue is not whether the process is in the foreground or background. When you start a process using Java, you have to use Java to control that process' lifecyle. The Java API provides you access to various attributes of the process. What you're interested in here is the output of the process. That is represented by the process' InputStream. It seems counterintuitive, but it makes sense because from the perspective of your Java program, the process' output is the program's input. Conversely, if you need to send data to the process, you write to the process' OutputStream.
To sum up, access the process' InputStream and print that out to the command-line:
Process process = new ProcessBuilder("C:\\Path\\To\\My\\Application.exe").start();
BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
StringBuilder output = new StringBuilder();
String line;
while ((line = reader.readLine()) != null) {
output.append(line);
}
System.out.println(line);
This code, of course, assumes that your process is not waiting for any input, i.e., it is not interactive.

Vivin Paliath's answer is really the way to go, then you can do whatever you want with the output, display it in your own dialogue, log it, interpret it, check for errors or whatever.
But just in case you really want that command window showing up. Execute cmd.exe and get the process' OutputStream and write the command (application.exe) to it ending with a new line.
Something along the lines of:
Process p = new ProcessBuilder("C:\\Windows\\System32\\cmd.exe").start();
out = p.getOutputStream();
out.write("path\\application.exe\r\n".getBytes());
out.flush();
Should usually drain the input stream too though anyway.

shell process java synchronization

I want to run a shell script from a java program. This shell script invokes a system library which needs a big file as resource.
My java program calls this script for every word in a document. If I call this script again and again using Runtime.exec() the time taken is very high since the resource loading takes lot of time.
To overcome this I thought of writing the shell script as follows (to make it run continuously in background ):
count=0
while count -lt 10 ; do
read WORD
//execute command on this line
done
I need retrieve the output of the command in my java program and process it further.
How should I code the I/O operations for achieving this task?
I have tried writing words in to the process's output stream and reading back output from process's input stream. But this does not work and throws a broken pipe exception.
try {
parseResult = Runtime.getRuntime().exec(parseCommand);
parsingResultsReader = new BufferedReader(new InputStreamReader (parseResult.getInputStream()));
errorReader = new BufferedReader(new InputStreamReader (parseResult.getErrorStream()));
parseResultsWriter = new BufferedWriter(new OutputStreamWriter((parseResult.getOutputStream())));
} catch (IOException e) {
e.printStackTrace();
}
parseResultsWriter.write(word);
parseResultsWriter.flush();
while ((line = parsingResultsReader.readLine()) != null) {
// capture output in list here
}
Kindly help with this issue

//execute command on this line
Is this command a separate program? Then it will be launched for every word, so you'll get rid of only shell process which is lightweight anyway.
You have to learn how to run the heavyweight command for many words at once.

Launching wkhtmltopdf from Runtime.getRuntime().exec(): never terminates?

I'm launching wkhtmltopdf from within my Java app (part of a Tomcat server, running in debug mode within Eclipse Helios on Win7 64-bit): I'd like to wait for it to complete, then Do More Stuff.
String cmd[] = {"wkhtmltopdf", htmlPathIn, pdfPathOut};
Process proc = Runtime.getRuntime().exec( cmd, null );
proc.waitFor();
But waitFor() never returns. I can still see the process in the Windows Task Manager (with the command line I passed to exec(): looks fine). AND IT WORKS. wkhtmltopdf produces the PDF I'd expect, right where I'd expect it. I can open it, rename it, whatever, even while the process is still running (before I manually terminate it).
From the command line, everything is fine:
c:\wrk>wkhtmltopdf C:\Temp\foo.html c:\wrk\foo.pdf
Loading pages (1/6)
Counting pages (2/6)
Resolving links (4/6)
Loading headers and footers (5/6)
Printing pages (6/6)
Done
The process exits just fine, and life goes on.
So what is it about runtime.exec() that's causing wkhtmltopdf to never terminate?
I could grab proc.getInputStream() and look for "Done", but that's... vile. I want something that is more general.
I've calling exec() with and without a working directory. I've tried with and without an empty "env" array. No joy.
Why is my process hanging, and what can I do to fix it?
PS: I've tried this with a couple other command line apps, and they both exhibit the same behavior.
Further exec woes.
I'm trying to read standard out & error, without success. From the command line, I know there's supposed to be something remarkably like my command line experience, but when I read the input stream returned by proc.getInputStream(), I immediately get an EOL (-1, I'm using inputStream.read()).
I checked the JavaDoc for Process, and found this
The parent process uses these streams to feed input to and get output from the subprocess. Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the [b]subprocess to block, and even deadlock[/b].
Emphasis added. So I tried that. The first 'read()' on the Standard Out inputStream blocked until I killed the process...
WITH WKHTMLTOPDF
With the generic command line ap & no params so it should "dump usage and terminate", it sucks out the appropriate std::out, then terminates.
Interesting!
JVM version issue? I'm using 1.6.0_23. The latest is... v24. I just checked the change log and don't see anything promising, but I'll try updating anyway.
Okay. Don't let the Input Streams fill or they'll block. Check. .close() can also prevent this, but isn't terribly bright.
That works in general (including the generic command line apps I've tested).
In specific however, it falls down. It appears that wkhtmltopdf is using some terminal manipulation/cursor stuff to do an ASCII-graphic progress bar. I believe this is causing the inputStream to immediately return EOF rather than giving me the correct values.
Any ideas? Hardly a deal-breaker, but it would definitely be Nice To Have.

I had the same exact issue as you and I solved it. Here are my findings:
For some reason, the output from wkhtmltopdf goes to STDERR of the process and NOT STDOUT. I have verified this by calling wkhtmltopdf from Java as well as perl
So, for example in java, you would have to do:
//ProcessBuilder is the recommended way of creating processes since Java 1.5
//Runtime.getRuntime().exec() is deprecated. Do not use.
ProcessBuilder pb = new ProcessBuilder("wkhtmltopdf.exe", htmlFilePath, pdfFilePath);
Process process = pb.start();
BufferedReader errStreamReader = new BufferedReader(new InputStreamReader(process.getErrorStream()));
//not "process.getInputStream()"
String line = errStreamReader.readLine();
while(line != null)
{
System.out.println(line); //or whatever else
line = reader.readLine();
}
On a side note, if you spawn a process from java, you MUST read from the stdout and stderr streams (even if you do nothing with it) because otherwise the stream buffer will fill and the process will hang and never return.
To futureproof your code, just in case the devs of wkhtmltopdf decide to write to stdout, you can redirect stderr of the child process to stdout and read only one stream like this:
ProcessBuilder pb = new ProcessBuilder("wkhtmltopdf.exe", htmlFilePath, pdfFilePath);
pb.redirectErrorStream(true);
Process process = pb.start();
BufferedReader inStreamReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
Actually, I do this in all the cases where I have to spawn an external process from java. That way I don't have to read two streams.
You should also read the streams of the spawned process in different threads if you dont want your main thread to block, since reading from streams is blocking.
Hope this helps.
UPDATE: I raised this issue in the project page and was replied that this is by design because wkhtmltopdf supports giving the actual pdf output in STDOUT. Please see the link for more details and java code.

A process has 3 streams: input, output and error. you can read both output and error stream at the same time using separate processes. see this question and its accepted answer and also this one for example.

You should read from the streams in a different thread.

final Semaphore semaphore = new Semaphore(numOfThreads);
final String whktmlExe = tmpwhktmlExePath;
int doccount = 0;
try{
File fileObject = new File(inputDir);
for(final File f : fileObject.listFiles()) {
if(f.getAbsolutePath().endsWith(".html")) {
doccount ++;
if(doccount >500 ) {
LOG.info(" done with conversion of 1000 docs exiting ");
break;
}
System.out.println(" inside for before "+semaphore.availablePermits());
semaphore.acquire();
System.out.println(" inside for after "+semaphore.availablePermits() + " ---" +f.getName());
new java.lang.Thread() {
public void run() {
try {
String F_ = f.getName().replaceAll(".html", ".pdf") ;
ProcessBuilder pb = new ProcessBuilder(whktmlExe , f.getAbsolutePath(), outPutDir + F_ .replaceAll(" ", "_") );//"wkhtmltopdf.exe", htmlFilePath, pdfFilePath);
pb.redirectErrorStream(true);
Process process = pb.start();
BufferedReader errStreamReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
String line = errStreamReader.readLine();
while(line != null)
{
System.err.println(line); //or whatever else
line = errStreamReader.readLine();
}
System.out.println("after completion for ");
} catch (Exception e) {
e.printStackTrace();
}finally {
System.out.println(" in finally releasing ");
semaphore.release();
}
}
}.start();
}
}
}catch (Exception ex) {
LOG.error(" *** Error in pdf generation *** ", ex);
}
while (semaphore.availablePermits() < numOfThreads) {//till all threads finish
LOG.info( " Waiting for all threads to exit "+ semaphore.availablePermits() + " --- " +( numOfThreads - semaphore.availablePermits()));
java.lang.Thread.sleep(10000);
}