I am using a thread to capture stream output from a process, and then outputting that stream to the eclipse console. The question I have is when to terminate the thread that is doing the stream output.
Thread t = new Thread(new Runnable(){
private boolean isProcessDone(Process p)
{
//not sure what to do here
}
public void run()
{
Process p = Runtime.getRuntime().exec("executable with output");
BufferedReader input = new BufferedReader(new InputStreamReader(p.getInputStream()));
BufferedReader error = new BufferedReader (new InputStreamReader(p.getErrorStream()));
while ( !isProcessDone(p) ) {
String line;
if( (line = input.readLine()) != null )
{
System.out.println(line);
}
if( (line = error.readLine()) != null )
{
System.out.println(line);
}
}
input.close();
error.close();
}
});
t.start();
My question is what belongs in the isProcessDone() function. The example I am basing this off of uses the stream's ready() function, but I am unclear whether this will work for programs that either std::err and std::out, but not both. I also tried using
try{
p.exitValue();
return true;
}catch(IllegalThreadStateException e){}
return false;
but then the thread finishes before the while loop has a chance to act on the streams, and the output is lost.
You need to use Process.waitFor() to wait for process completion.
Additionally, you need to consume stdout and stderr concurrently in order to avoid blocking and a possible process hang. Consequently you need two threads to read these streams, and to continue reading whilst the streams are available.
See this Javaworld article for more info and a StreamGobbler implementation to consume the stdout/err.
You can use VerboseProcess from jcabi-log (I'm a developer):
String name = new VerboseProcess(
new ProcessBuilder("executable with output")
).stdout();
The only dependency you need:
<dependency>
<groupId>com.jcabi</groupId>
<artifactId>jcabi-log</artifactId>
<version>0.7.5</version>
</dependency>
You will need to have two threads. One to handle the I/O and another to wait for process completion (Process.waitFor()) and set a flag telling the I/O thread to quit when it exhausts the data.
You need to process reading the output in a separate thread, theres an example here
Related
What is the proper way to produce and consume the streams (IO) of external process from Java? As far as I know, java end input streams (process output) should be consumed in threads parallel to producing the process input due the possibly limited buffer size.
But I'm not sure if I eventually need to synchronize with those consumer threads, or is it enough just to wait for process to exit with waitFor method, to be certain that all the process output is actually consumed? I.E is it possible, even if the process exits (closes it's output stream), there is still unread data in the java end of the stream? How does the waitFor actually even know when the process is done? For the process in question, EOF (closing the java end of it's input stream) signals it to exit.
My current solution to handle the streams is following
public class Application {
private static final StringBuffer output = new StringBuffer();
private static final StringBuffer errOutput = new StringBuffer();
private static final CountDownLatch latch = new CountDownLatch(2);
public static void main(String[] args) throws IOException, InterruptedException {
Process exec = Runtime.getRuntime().exec("/bin/cat");
OutputStream procIn = exec.getOutputStream();
InputStream procOut = exec.getInputStream();
InputStream procErrOut = exec.getErrorStream();
new Thread(new StreamConsumer(procOut, output)).start();
new Thread(new StreamConsumer(procErrOut, errOutput)).start();
PrintWriter printWriter = new PrintWriter(procIn);
printWriter.print("hello world");
printWriter.flush();
printWriter.close();
int ret = exec.waitFor();
latch.await();
System.out.println(output.toString());
System.out.println(errOutput.toString());
}
public static class StreamConsumer implements Runnable {
private InputStream input;
private StringBuffer output;
public StreamConsumer(InputStream input, StringBuffer output) {
this.input = input;
this.output = output;
}
#Override
public void run() {
BufferedReader reader = new BufferedReader(new InputStreamReader(input));
String line;
try {
while ((line = reader.readLine()) != null) {
output.append(line + System.lineSeparator());
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} finally {
try {
reader.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} finally {
latch.countDown();
}
}
}
}
}
Is it necessary to use the latch here, or does the waitFor implicate all the output is already consumed? Also, if the output doesn't end/contain new line, will the readLine miss the output, or still read all that is left? Does reading null mean process has closed it's end of the stream - is there any other scenario where null could be read?
What is the correct way to handle streams, could I do something better than in my example?
waitFor signals that the process ended, but you cannot be sure the threads which collect strings from its stdout and stderr finished also, so using a latch is a step in the right direction, but not an optimal one.
Instead of waiting for the latch, you can wait for the threads directly:
Thread stdoutThread = new Thread(new StreamConsumer(procOut, output)).start();
Thread stderrThread = ...
...
int ret = exec.waitFor();
stdoutThread.join();
stderrThread.join();
BTW, storing lines in StringBuffers is useless work. Use ArrayList<String> instead, put lines there without any conversion, and finally retrieve them in a loop.
Your appapproach is right, but is't better to remove CountDownLatch and use ThreadPool, and not create new Thread directly. From ThreadPool you will get two futures, which you can wait after to completion.
But I'm not sure if I eventually need to synchronize with those consumer threads, or is it enough just to wait for process to exit with waitFor method, to be certain that all the process output is actually consumed? I.E is it possible, even if the process exits (closes it's output stream), there is still unread data in the java end of the stream?
Yes, this situation may occurs. Termination and reading IO streams is unrelated processes.
I have some code that runs a process and reads from the stdout and stderr asynchronously and then handles when the process completes. It looks something like this:
Process process = builder.start();
Thread outThread = new Thread(() -> {
try (BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()))) {
// Read stream here
} catch (Exception e) {
}
});
Thread errThread = new Thread(() -> {
try (BufferedReader reader = new BufferedReader(new InputStreamReader(process.getErrorStream()))) {
// Read stream here
} catch (Exception e) {
}
});
outThread.start();
errThread.start();
new Thread(() -> {
int exitCode = -1;
try {
exitCode = process.waitFor();
outThread.join();
errThread.join();
} catch (Exception e) {
}
// Process completed and read all stdout and stderr here
}).start();
My issue is with the fact that I am using 3 threads to achieve this asynchronous "run-and-get-output" task - I don't know why, but I feel it doesn't feel right using 3 threads. I could allocate the threads out of a thread pool, but that would still be blocking those threads.
Is there anything I can do, maybe with NIO, to reduce this to fewer (1?) thread? Anything I can think of will be constantly spinning a thread (unless I add a few sleeps), which I don't really want to do either...
NOTE: I do need to read as I go (rather than when the process has stopped) and I do need to separate stdin from stderr so can't do a redirect.
Since you've specified that you need to read the output as you go, there is no non-multi-threaded solution.
You can reduce the number of threads to one beyond your main thread though:
Process process = builder.start();
Thread errThread = new Thread(() -> {
try (BufferedReader reader = new BufferedReader(new InputStreamReader(process.getErrorStream()))) {
// Read stream here
} catch (Exception e) {
}
});
errThread.start();
try (BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()))) {
// Read stream here
} catch (Exception e) {
}
// we got an end of file, so there can't be any more input. Now we need to wait for stderr/process exit.
int exitCode = -1;
try {
exitCode = process.waitFor();
errThread.join();
} catch (Exception e) {
}
// Process completed
If you truely don't need to deal with the error/output until after the process ends, you can simplify it a bit and only use your main thread like this:
File stderrFile = File.createTempFile("tmpErr", "out");
File stdoutFile = File.createTempFile("tmpStd", "out");
try {
ProcessBuilder builder = new ProcessBuilder("ls /tmp");
Process p = builder.start();
int exitCode = -1;
boolean done = false;
while (!done) {
try {
exitCode = p.waitFor();
done = true;
} catch (InterruptedException ie) {
System.out.println("Interrupted waiting for process to exit.");
}
}
BufferedReader err = new BufferedReader(new FileReader(stderrFile));
BufferedReader in = new BufferedReader(new FileReader(stdoutFile));
....
} finally {
stderrFile.delete();
stdoutFile.delete();
}
This is probably not a good idea if you generate a lot of output from the process you are calling as it could run out of disk space... but it'll likely be slightly faster since it doesn't have to spin up another Thread.
Assuming you don't mind the input and error streams to be merged, you could only use one thread with:
builder.redirectErrorStream(true); //merge input and error streams
Process process = builder.start();
Thread singleThread = new Thread(() -> {
int exitCode = -1;
//read from the merged stream
try (BufferedReader reader =
new BufferedReader(new InputStreamReader(process.getInputStream()))) {
String line;
//read until the stream is exhausted, meaning the process has terminated
while ((line = reader.readLine()) != null) {
System.out.println(line); //use the output here
}
//get the exit code if required
exitCode = process.waitFor();
} catch (Exception e) { }
}).start();
Have a look at the ExecHelper from OstermillerUtils.
The idea is that the thread waiting for the process to complete, does not just wait but reads input from stdout and stderr if there is input available and regurarly checks if the process has finished.
If you do not do any heavy processing with the input from stdout and stderr, you might not need an extra thread to handle the input. Just copy ExecHelper and add some extra functions/methods to process any new input. I've done this before to show the process output while the process is running, it is not difficult to do (but I lost the source code).
If you do need a separate thread for processing the input, make sure to synchronize the output and error StringBuffers when these buffers are updated or read.
Another thing you might want to consider is adding an abort time-out. It is a little bit harder to implement but was very valuable to me: if a process takes too much time, the process gets destroyed which in turn ensures nothing remains hanging. You can find an old (outdated?) example this gist.
You'll have to compromise. Here are your options:
A. You can do it with 2 threads (instead of 3):
First thread:
read from stdout until readline returns null
call Process.waitFor()
join Thread#2
Second thread:
reads from stderr until readline returns null
B. Merge streams and use Debian's annotate-output to discriminate the 2 streams
http://manpages.debian.org/cgi-bin/man.cgi?query=annotate-output&sektion=1
C. If it's a short-living process just wait for the end of it
D. If it's a long-living process then you can spin between readers with some sleep in between.
byte[] bytes = value.getBytes();
Process q = new ProcessBuilder("process","arg1", "arg2").start();
q.getOutputStream().write(bytes);
q.getOutputStream().flush();
System.out.println(q.getInputStream().available());
I'm trying to stream file contents to an executable and capture the output but the output(InputStream) is always empty. I can capture the output if i specify the the file location but not with streamed input.
How might I overcome this?
Try wrapping your streams with BufferedInputStream() and BufferedOutputStream():
http://download.oracle.com/javase/6/docs/api/java/lang/Process.html#getOutputStream%28%29
Implementation note: It is a good idea for the output stream to be buffered.
Implementation note: It is a good idea for the input stream to be buffered.
Even with buffered streams, it is still possible for the buffer to fill if you're dealing with large amounts of data, you can deal with this by starting a separate thread to read from q.getInputStream(), so you can still be reading from the process while writing to the process.
Perhaps the program you execute only starts its work when it detects the end of its input data. This is normally done by waiting for an EOF (end-of-file) symbol. You can send this by closing the output stream to the process:
q.getOutputStream().write(bytes);
q.getOutputStream().close();
Try this together with waiting for the process.
I dont know if something else may also be wrong here, but the other process ("process") does not even have time to respond, you are not waiting for it (the method available() does not block). To try this out you can first insert a sleep(2000) after the flush(), and if that works you should switch to query'ing q.getInputStream().available() multiple times with short pauses in between.
I think, you have to wait, until the process finished.
I implemented something like this this way:
public class ProcessReader {
private static final int PROCESS_LOOP_SLEEP_MILLIS = 100;
private String result;
public ProcessReader(Process process) {
BufferedReader resultReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
StringBuilder resultOutput = new StringBuilder();
try {
while (!checkProcessTerminated(process, resultReader, resultOutput)) {
}
} catch (Exception ex1) {
throw new RuntimeException(ex1);
}
result = resultOutput.toString();
}
public String getResult(){
return result;
}
private boolean checkProcessTerminated(Process process, BufferedReader resultReader, StringBuilder resultOutput) throws Exception {
try {
int exit = process.exitValue();
return true;
} catch (IllegalThreadStateException ex) {
Thread.sleep(PROCESS_LOOP_SLEEP_MILLIS);
} finally {
while (resultReader.ready()) {
String out = resultReader.readLine();
resultOutput.append(out).append("\n");
}
}
return false;
}
}
I just removed now some specific code, that you dont need, but it should work, try it.
Regards
How might I get the output from a CMD process to display in my GUI? This is the code I'm using to run the process:
try {
String line;
Process p = Runtime.getRuntime().exec("cmd /c \"e:\\folder\\someCommands.cmd\"");
BufferedReader input =
new BufferedReader(new InputStreamReader(p.getInputStream()));
while ((line = input.readLine()) != null) {
System.out.println(line);
}
input.close();
} catch (Exception err) {
err.printStackTrace();
}
I've tried doing this:
jLabel1.setText(line);
...but the GUI is completely locked up while the process is running, so nothing updates until the very end, which isn't very useful. Other than that the CMD works fine. I just want to display the output in real-time.
Did you repaint() after setting the text of the label?
Anyway, you should generally be hesitant to execute a long operation on the GUI event thread. Look into using a SwingWorker instead.
You'll need to start a separate thread to run the process. The code you're using to run it can mostly just be inserted into the thread's (or Runnable's) run() method as is, but to set the text in the JLabel, you should use something like this:
...
while ((line = input.readLine()) != null) {
SwingUtilities.invokeLater(new SetTextRunnable(jLabel1, line));
}
...
class SetTextRunnable implements Runnable {
private String line;
private JLabel jLabel1
public SetTextRunnable(JLabel jLabel1, String line) {
this.jLabel1 = jLabel1;
this.line = line;
}
public void run() {
jLabel1.setText(line);
}
}
EDIT: just noticed something: apparently the class SwingWorker was designed for this sort of thing, so that's another option for you (if it exists in your version of Java).
EDIT to the EDIT: so silly me, I didn't notice SwingWorker was already mentioned in another answer.
In addition to what others have said about multithreading this - you'll also want to read the child process error stream. I believe that (in some instances) if you don't drain the error stream for the process it could cause it to hang.
I have a class that manages the creation of RTF documents and a method in that class that calls the RTF editor with a XML file for display.
All but one user can access this editor without any issues. This one user consistently runs into an issue where their application just hangs. There are no errors in any logs. Normally this kind of problem is easily identified, reproduced and corrected, however, I can't for the life of my reproduce it so my attempts at debugging are failing.
Basically the code is as follows:
int exitVal = CUBSRTFEditor.runRTFEditor("c:\\tmp\\control"+ap_doc_id+".xml", xml,"I:\\AppealsLetters.exe /process \"c:\\tmp\\control"+ap_doc_id+".xml\"");
public static int runRTFEditor(String xmlLocation, String xmlContent, String executePath)
{
int exitVal = 0;
createLocalFile(xmlLocation, xmlContent);
try
{
System.out.println("executePath must = "+executePath);
Runtime rt = Runtime.getRuntime();
Process proc = rt.exec(executePath);
System.out.println("after executePath runs");
//exhaust that stream before waiting for the process to exit
InputStream inputstream = proc.getInputStream();
InputStreamReader inputstreamreader = new InputStreamReader(inputstream);
BufferedReader bufferedreader = new BufferedReader(inputstreamreader);
// read the ls output
String line;
while ((line = bufferedreader.readLine())!= null)
{
System.out.println(line);
}
exitVal = proc.waitFor();
}
catch (Throwable t)
{
t.printStackTrace();
}
CUBSRTFEditor.deleteTempFile(xmlLocation);
return exitVal;
}
The last output is the first System.out. When I take the xml file and execute this on any other PC it executes without issue. I see no useful info in proc.getErrorStream() or proc.getOutputStream().
The JDK's Javadoc documentation on this problem (exec hanging):
Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block, and even deadlock.
I try exhausting that stream before waiting for the process to exit and that does not seem to help as it never seems to get to that point (the second System.out is not displayed)
Have I implemented this incorrectly? Am I missing something important? Any ideas on how to get more info out of the process would be great.
I am stuck....
Runtime.exec() is a deceptively nasty little spud to work with. I found this article (old, but still relevant) to be quite helpful. You can always skip to Page 4 for some highly gankable sample code. :-)
At a glance, your code needs to handle both proc.getOutputStream() and proc.getErrorStream(), which is a good reason to handle those streams in separate threads.
I wanted to update this because the change went into Production today and worked. Based off of BlairHippo's suggestions I got it to work with an anonymous inner class to create a separate thread to exhaust both the Error and Input streams.
new Thread(new Runnable(){
public void run()
{
try
{
BufferedReader br = new BufferedReader(new InputStreamReader(proc.getErrorStream()));
String line;
while ((line = br.readLine())!= null)
{
System.out.println(line);
}
}
catch (Throwable t)
{
t.printStackTrace();
}
}
}).start();