Develop Reference

Run a process asynchronously and read from stdout and stderr

I have some code that runs a process and reads from the stdout and stderr asynchronously and then handles when the process completes. It looks something like this:
Process process = builder.start();
Thread outThread = new Thread(() -> {
try (BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()))) {
// Read stream here
} catch (Exception e) {
}
});
Thread errThread = new Thread(() -> {
try (BufferedReader reader = new BufferedReader(new InputStreamReader(process.getErrorStream()))) {
// Read stream here
} catch (Exception e) {
}
});
outThread.start();
errThread.start();
new Thread(() -> {
int exitCode = -1;
try {
exitCode = process.waitFor();
outThread.join();
errThread.join();
} catch (Exception e) {
}
// Process completed and read all stdout and stderr here
}).start();
My issue is with the fact that I am using 3 threads to achieve this asynchronous "run-and-get-output" task - I don't know why, but I feel it doesn't feel right using 3 threads. I could allocate the threads out of a thread pool, but that would still be blocking those threads.
Is there anything I can do, maybe with NIO, to reduce this to fewer (1?) thread? Anything I can think of will be constantly spinning a thread (unless I add a few sleeps), which I don't really want to do either...
NOTE: I do need to read as I go (rather than when the process has stopped) and I do need to separate stdin from stderr so can't do a redirect.

Since you've specified that you need to read the output as you go, there is no non-multi-threaded solution.
You can reduce the number of threads to one beyond your main thread though:
Process process = builder.start();
Thread errThread = new Thread(() -> {
try (BufferedReader reader = new BufferedReader(new InputStreamReader(process.getErrorStream()))) {
// Read stream here
} catch (Exception e) {
}
});
errThread.start();
try (BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()))) {
// Read stream here
} catch (Exception e) {
}
// we got an end of file, so there can't be any more input. Now we need to wait for stderr/process exit.
int exitCode = -1;
try {
exitCode = process.waitFor();
errThread.join();
} catch (Exception e) {
}
// Process completed
If you truely don't need to deal with the error/output until after the process ends, you can simplify it a bit and only use your main thread like this:
File stderrFile = File.createTempFile("tmpErr", "out");
File stdoutFile = File.createTempFile("tmpStd", "out");
try {
ProcessBuilder builder = new ProcessBuilder("ls /tmp");
Process p = builder.start();
int exitCode = -1;
boolean done = false;
while (!done) {
try {
exitCode = p.waitFor();
done = true;
} catch (InterruptedException ie) {
System.out.println("Interrupted waiting for process to exit.");
}
}
BufferedReader err = new BufferedReader(new FileReader(stderrFile));
BufferedReader in = new BufferedReader(new FileReader(stdoutFile));
....
} finally {
stderrFile.delete();
stdoutFile.delete();
}
This is probably not a good idea if you generate a lot of output from the process you are calling as it could run out of disk space... but it'll likely be slightly faster since it doesn't have to spin up another Thread.

Assuming you don't mind the input and error streams to be merged, you could only use one thread with:
builder.redirectErrorStream(true); //merge input and error streams
Process process = builder.start();
Thread singleThread = new Thread(() -> {
int exitCode = -1;
//read from the merged stream
try (BufferedReader reader =
new BufferedReader(new InputStreamReader(process.getInputStream()))) {
String line;
//read until the stream is exhausted, meaning the process has terminated
while ((line = reader.readLine()) != null) {
System.out.println(line); //use the output here
}
//get the exit code if required
exitCode = process.waitFor();
} catch (Exception e) { }
}).start();

Have a look at the ExecHelper from OstermillerUtils.
The idea is that the thread waiting for the process to complete, does not just wait but reads input from stdout and stderr if there is input available and regurarly checks if the process has finished.
If you do not do any heavy processing with the input from stdout and stderr, you might not need an extra thread to handle the input. Just copy ExecHelper and add some extra functions/methods to process any new input. I've done this before to show the process output while the process is running, it is not difficult to do (but I lost the source code).
If you do need a separate thread for processing the input, make sure to synchronize the output and error StringBuffers when these buffers are updated or read.
Another thing you might want to consider is adding an abort time-out. It is a little bit harder to implement but was very valuable to me: if a process takes too much time, the process gets destroyed which in turn ensures nothing remains hanging. You can find an old (outdated?) example this gist.

You'll have to compromise. Here are your options:
A. You can do it with 2 threads (instead of 3):
First thread:
read from stdout until readline returns null
call Process.waitFor()
join Thread#2
Second thread:
reads from stderr until readline returns null
B. Merge streams and use Debian's annotate-output to discriminate the 2 streams
http://manpages.debian.org/cgi-bin/man.cgi?query=annotate-output&sektion=1
C. If it's a short-living process just wait for the end of it
D. If it's a long-living process then you can spin between readers with some sleep in between.

java - how to kill a Process if it exceeds an alloted time

I am wondering what the best way is to detect/kill a process if it exceeds a predefined time. I know an old way was to use the watchdog/timeoutobserver class from the ant package. But this is deprecated now, so I am wondering how it should be done now?
Here is the code I have which uses watchdog:
import org.apache.tools.ant.util.Watchdog;
import org.apache.tools.ant.util.TimeoutObserver;
public class executer implements TimeoutObserver {
private int timeOut = 0;
Process process = null;
private boolean killedByTimeout =false;
public executer(int to) {
timeOut = t;
}
public String executeCommand() throws Exception {
Watchdog watchDog = null;
String templine = null;
StringBuffer outputTrace = new StringBuffer();
StringBuffer errorTrace = new StringBuffer();
Runtime runtime = Runtime.getRuntime();
try {
//instantiate a new watch dog to kill the process
//if exceeds beyond the time
watchDog = new Watchdog(getTimeout());
watchDog.addTimeoutObserver(this);
watchDog.start();
process = runtime.exec(command);
//... Code to do the execution .....
InputStream inputStream = process.getInputStream();
InputStreamReader inputStreamReader = new InputStreamReader(inputStream);
bufferedReader = new BufferedReader(inputStreamReader);
while (((templine = bufferedReader.readLine()) != null) && (!processWasKilledByTimeout)) {
outputTrace.append(templine);
outputTrace.append("\n");
}
this.setStandardOut(outputTrace);
int returnCode = process.waitFor();
//Set the return code
this.setReturnCode(returnCode);
if (processWasKilledByTimeout) {
//As process was killed by timeout just throw an exception
throw new InterruptedException("Process was killed before the waitFor was reached.");
}
} finally {
// stop the watchdog as no longer needed.
if (aWatchDog != null) {
aWatchDog.stop();
}
try {
// close buffered readers etc
} catch Exception() {
}
//Destroy process
// Process.destroy() sends a SIGTERM to the process. The default action
// when SIGTERM is received is to terminate, but any process is free to
// ignore the signal or catch it and respond differently.
//
// Also, the process started by Java might have created additional
// processes that don't receive the signal at all.
if(process != null) {
process.destroy();
}
}
public void timeoutOccured(Watchdog arg0) {
killedByTimeout = true;
if (process != null){
process.destroy();
}
arg0.stop();
}
}
}
Any help would be greatly appreciated as I am a bit lost. I am trying to take this up to Java 7, but I am not uptodate with the best way to kill it if it hangs beyond the alloted time.
Thanks,

try
final Process p = ...
Thread t = new Thread() {
public void run() {
try {
Thread.sleep(1000);
p.destroy();
} catch (InterruptedException e) {
}
};
};
p.waitFor();
t.interrupt();

Theoretically Thread has method stop() that totally kills the thread. This method is deprecated since java 1.1 because it may cause resources leak. So, you are really not recommended to use it.
The "right" solution is to implement your thread so that they can gracefully exit when receiving a special "signal". You can use "interruption" mechanism: your watchdog should call "interrupt()" of thread that exceeds the time limit. But thread should call isInterrupted() itself and exit if it is interrupted. The good news is that method like sleep() and wait() already support this, so if your thread is waiting and you interrupt it from outside it InterruptedException will be thrown.

I have written a set of ExecutorServices that will cancel processes after they have been given a certain period of time to execute. This code has been checked into GitHub.
The class to use to create the ExecutorService is CancelingExecutors. There are two main classes:
CancelingListeningExecutorService allows you to specify the timeout for each passed Callable
FixedTimeoutCancelingListeningExecutorService is configured to use a single timeout for all Callables

If you just concern about WatchDog itself is deprecated, it is nothing more difficult for you to make use of TimerTask, and do the process.destroy() after a period of time.

How to write a byte array to OutputStream of process builder (Java)

byte[] bytes = value.getBytes();
Process q = new ProcessBuilder("process","arg1", "arg2").start();
q.getOutputStream().write(bytes);
q.getOutputStream().flush();
System.out.println(q.getInputStream().available());
I'm trying to stream file contents to an executable and capture the output but the output(InputStream) is always empty. I can capture the output if i specify the the file location but not with streamed input.
How might I overcome this?

Try wrapping your streams with BufferedInputStream() and BufferedOutputStream():
http://download.oracle.com/javase/6/docs/api/java/lang/Process.html#getOutputStream%28%29
Implementation note: It is a good idea for the output stream to be buffered.
Implementation note: It is a good idea for the input stream to be buffered.
Even with buffered streams, it is still possible for the buffer to fill if you're dealing with large amounts of data, you can deal with this by starting a separate thread to read from q.getInputStream(), so you can still be reading from the process while writing to the process.

Perhaps the program you execute only starts its work when it detects the end of its input data. This is normally done by waiting for an EOF (end-of-file) symbol. You can send this by closing the output stream to the process:
q.getOutputStream().write(bytes);
q.getOutputStream().close();
Try this together with waiting for the process.

I dont know if something else may also be wrong here, but the other process ("process") does not even have time to respond, you are not waiting for it (the method available() does not block). To try this out you can first insert a sleep(2000) after the flush(), and if that works you should switch to query'ing q.getInputStream().available() multiple times with short pauses in between.

I think, you have to wait, until the process finished.
I implemented something like this this way:
public class ProcessReader {
private static final int PROCESS_LOOP_SLEEP_MILLIS = 100;
private String result;
public ProcessReader(Process process) {
BufferedReader resultReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
StringBuilder resultOutput = new StringBuilder();
try {
while (!checkProcessTerminated(process, resultReader, resultOutput)) {
}
} catch (Exception ex1) {
throw new RuntimeException(ex1);
}
result = resultOutput.toString();
}
public String getResult(){
return result;
}
private boolean checkProcessTerminated(Process process, BufferedReader resultReader, StringBuilder resultOutput) throws Exception {
try {
int exit = process.exitValue();
return true;
} catch (IllegalThreadStateException ex) {
Thread.sleep(PROCESS_LOOP_SLEEP_MILLIS);
} finally {
while (resultReader.ready()) {
String out = resultReader.readLine();
resultOutput.append(out).append("\n");
}
}
return false;
}
}
I just removed now some specific code, that you dont need, but it should work, try it.
Regards

How to interrupt BufferedReader's readLine

I am trying to read input from a socket line by line in multiple threads. How can I interrupt readLine() so that I can gracefully stop the thread that it's blocking?
EDIT (bounty): Can this be done without closing the socket?

Without closing the socket:
The difficult problem isn't the BufferedReader.readLine, but the underlying read. If a thread is blocked reading, the only way to get it going is to supply some actual data or close the socket (interrupting the thread probably should work, but in practice does not).
So the obvious solution is to have two threads. One that reads the raw data, and will remain blocked. The second, will be the thread calling readLine. Pipe data from the first the second. You then have access to a lock than can be used to wakeup the second thread, and have it take appropriate action.
There are variations. You could have the first thread using NIO, with a single thread instance shared between all consumers.
Alternatively you could write a readLine that works with NIO. This could even take a a relatively simple single-threaded form, as Selector.wakeup exists and works.

Close the socket on the interrupting thread. This will cause an exception to be thrown on the interrupted thread.
For more information on this and other concurrency issues, I highly recommend Brian Goetz's book "Java Concurrency in Practice".

Sorry for being over 6 years late ;-) I had a need for some interruptible readLine when reading from the keyboard, for a simple hobby console application. In other words, I couldn't "close the socket".
As you may know, System.in is an InputStream that apparently already does some buffering (you need to press Enter]). However, it seems to be suggested to wrap it in a BufferedReader for better efficiency, so my input is from:
BufferedReader consoleIn = new BufferedReader(new InputStreamReader(System.in));
The other thing one might have discovered is that BufferedReader.readLine() blocks until input is provided (even if the thread is interrupted, which seems to only end the thread once readline() gets its input). It is however possible to predict when BufferedReader.read() will not block, by calling BufferedReader.ready() == true. (However, == false does not guarantee a block, so beware.)
So I have incorporated the above ideas into a method that reads the BufferedReader character by character, checking in between each character if the thread has been interrupted, and also checks for end-of-line, at which point the line of text is returned.
You may find this code useful, pass the consoleIn variable as declared above. (Criticism may be welcomed too...):
private String interruptibleReadLine(BufferedReader reader)
throws InterruptedException, IOException {
Pattern line = Pattern.compile("^(.*)\\R");
Matcher matcher;
boolean interrupted = false;
StringBuilder result = new StringBuilder();
int chr = -1;
do {
if (reader.ready()) chr = reader.read();
if (chr > -1) result.append((char) chr);
matcher = line.matcher(result.toString());
interrupted = Thread.interrupted(); // resets flag, call only once
} while (!interrupted && !matcher.matches());
if (interrupted) throw new InterruptedException();
return (matcher.matches() ? matcher.group(1) : "");
}
... And in the thread that is calling this, catch the exceptions and end the thread appropriately.
This was tested in Java 8 on Linux.

I was playing around with this recently (using Scala), and I didn't like the accepted answer of closing the socket and getting an exception.
Eventually I discovered that it's possible to call socket.shutdownInput() in the interrupting thread to get out of the readLine call without an exception. I make this call in a SIGINT handler so that I can clean up and close the socket in the main thread.
Note, that the equivalent exists for the outputstream with socket.shutdownOutput()

you can design a Timer class around the read() block.
you need to set a timeout for your timer.
on timeout just interrupt your thread.

Without closing the socket, no question the best solution with the least overhead is to simply avoid using the blocking read methods until the BufferedReader is ready, or a timeout is reached.
public String readLineTimeout(BufferedReader reader, long timeout) throws TimeoutException, IOException {
long start = System.currentTimeMillis();
while (!reader.ready()) {
if (System.currentTimeMillis() - start >= timeout)
throw new TimeoutException();
// optional delay between polling
try { Thread.sleep(50); } catch (Exception ignore) {}
}
return reader.readLine(); // won't block since reader is ready
}

If you want to use readLine on a server socket within a client-server tcp architecture, for instance, you can use setSoTimeout(int timeout) of java.net.Socket.
From the Socket#setSoTimeout(int timeout) Documentation:
Enable/disable SO_TIMEOUT with the specified timeout, in milliseconds. With this option set to a non-zero timeout, a read() call on the InputStream associated with this Socket will block for only this amount of time. If the timeout expires, a java.net.SocketTimeoutException is raised, though the Socket is still valid.
public class MainApp {
public static void main(String[] args) throws Exception {
ExecutorService executorService = Executors.newFixedThreadPool(10);
ServerSocket serverSocket = new ServerSocket(11370);
Socket clientSocket = serverSocket.accept();
clientSocket.setSoTimeout(2000);
executorService.execute(new ReadingThread(clientSocket));
// ... some async operations
executorService.shutdown();
}
}
public class ReadingThread implements Runnable {
private final Socket clientSocket;
public ReadingThread(Socket clientSocket) {
this.clientSocket = clientSocket;
}
#Override
public void run() {
BufferedReader socketReader = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
String readInput = null;
while (!Thread.currentThread().isInterrupted()) {
try {
readInput = socketReader.readLine();
} catch (SocketTimeoutException e) {
continue;
}
}
// operations with readInput
}
}
The main application implements a server socket which listens to connections and has a thread pool. If an incoming client communication is accepted, then a new Thread from the pool is assigned and the run function is invoked in ReadingThread (can be adjusted to allow multiple threads).
On the socket used for communicating to the client the property setSoTimeout(int timeout) has been set. Therefore if readLine does not return within the specified timeout a SocketTimeoutException is thrown.
You can check in a loop whether the ReadingThread has been interrupted by the main application, and if so stop reading from the socket.

When the buffered reader is being used to read the input stream from a socket then you can achieve this by having the read call timeout. Once this timeout is triggered you will be able to check if your thread should be stopped. To do this call setSoTimeout on the socket. The read call will then have a SocketTimeoutException and you can use that to stop the thread.
#Override
public void run() {
running = true;
try {
socket.setSoTimeout(1000); // This will determine how quick your thread responds to the shutdown call
var inputStream = socket.getInputStream();
bufferedReader = new BufferedReader(new InputStreamReader(inputStream, StandardCharsets.UTF_8));
} catch (IOException e) {
Logger.error("IOException while setting up input stream");
Logger.error(e);
return;
}
StringBuilder stringBuilder = null;
while (running) {
try {
int singleChar = bufferedReader.read();
// Do something with the data
} catch (SocketTimeoutException e) {
// SocketTimeoutException is expected periodically as we do setSoTimeout on the socket,
// this makes the above read call not block for ever and allows the loop to be interrupted
// cleanly when we want to shut the thread down.
Logger.trace("Socket timeout exception");
Logger.trace(e);
} catch (IOException e) {
Logger.error("IOException while reading from socket stream");
Logger.error(e);
return;
}
}
}
public void stopThread() {
running = false;
try {
bufferedReader.close();
} catch (IOException e) {
Logger.error("IOException while closing BufferedReader in SocketThread");
Logger.error(e);
}
}
Answer found here: Any way of using java.nio.* to interrupt a InputStream#read() without closing socket?

I think that you might have to use something other than readLine(). You could use read() and at every loop iteration check to see if the thread was interrupted and break out of the loop if it was.
BufferedReader reader = //...
int c;
while ((c = reader.read()) != -1){
if (Thread.isInterrupted()){
break;
}
if (c == '\n'){
//newline
}
//...
}

A sketch for a solution might be this: NIO provides methods for nonblocking IO, so you have to implement something called Foo that uses nonblocking NIO on the socket end but also provides a InputStream or Reader interface on the other end. If the BufferedReader enters its own read, it will call Foo, which will call Selector.select with read intent. select will either return indicating the presence of more data or it will block until more data is available.
If another thread wants to unblock the reader, it must call Selector.wakeup and the selector can return gracefully by throwing an exception the by BufferedReader.
The socket should be still open after that.
Variation A: call Selector.select(timeout) to do busy polling light.

design for a wrapper around command-line utilities

im trying to come up with a design for a wrapper for use when invoking command line utilities in java. the trouble with runtime.exec() is that you need to keep reading from the process' out and err streams or it hangs when it fills its buffers. this has led me to the following design:
public class CommandLineInterface {
private final Thread stdOutThread;
private final Thread stdErrThread;
private final OutputStreamWriter stdin;
private final History history;
public CommandLineInterface(String command) throws IOException {
this.history = new History();
this.history.addEntry(new HistoryEntry(EntryTypeEnum.INPUT, command));
Process process = Runtime.getRuntime().exec(command);
stdin = new OutputStreamWriter(process.getOutputStream());
stdOutThread = new Thread(new Leech(process.getInputStream(), history, EntryTypeEnum.OUTPUT));
stdOutThread.setDaemon(true);
stdOutThread.start();
stdErrThread = new Thread(new Leech(process.getErrorStream(), history, EntryTypeEnum.ERROR));
stdErrThread.setDaemon(true);
stdErrThread.start();
}
public void write(String input) throws IOException {
this.history.addEntry(new HistoryEntry(EntryTypeEnum.INPUT, input));
stdin.write(input);
stdin.write("\n");
stdin.flush();
}
}
And
public class Leech implements Runnable{
private final InputStream stream;
private final History history;
private final EntryTypeEnum type;
private volatile boolean alive = true;
public Leech(InputStream stream, History history, EntryTypeEnum type) {
this.stream = stream;
this.history = history;
this.type = type;
}
public void run() {
BufferedReader reader = new BufferedReader(new InputStreamReader(stream));
String line;
try {
while(alive) {
line = reader.readLine();
if (line==null) break;
history.addEntry(new HistoryEntry(type, line));
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
my issue is with the Leech class (used to "leech" the process' out and err streams and feed them into history - which acts like a log file) - on the one hand reading whole lines is nice and easy (and what im currently doing), but it means i miss the last line (usually the prompt line). i only see the prompt line when executing the next command (because there's no line break until that point).
on the other hand, if i read characters myself, how can i tell when the process is "done" ? (either complete or waiting for input)
has anyone tried something like waiting 100 millis since the last output from the process and declaring it "done" ?
any better ideas on how i can implement a nice wrapper around things like runtime.exec("cmd.exe") ?

Use PlexusUtils it is used by Apache Maven 2 to execute all external processes.

I was looking for the same thing myself, and I found a Java port of Expect, called ExpectJ. I haven't tried it yet, but it looks promising

I would read the input in with the stream and then write it into a ByteArrayOutputStream. The byte array will continue to grow until there are no longer any available bytes to read. At this point you will flush the data to history by converting the byte array into a String and splitting it on the platform line.separator. You can then iterate over the lines to add history entries. The ByteArrayOutputStream is then reset and the while loop blocks until there is more data or the end of stream is reached (probably because the process is done).
public void run() {
ByteArrayOutputStream out = new ByteArrayOutputStream();
int bite;
try {
while((bite = stream.read()) != -1) {
out.write(bite);
if (stream.available() == 0) {
String string = new String(out.toByteArray());
for (String line : string.split(
System.getProperty("line.separator"))) {
history.addEntry(new HistoryEntry(type, line));
}
out.reset();
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
This will make sure you pick up that last line of input and it solves your problem of knowing when the stream is ended.