In my Java application, I need to execute some scripts as subprocesses and monitor the output on stdout from Java so that I can react when necessary to some output.
I am using apache commons-exec to spawn the subprocess and redirect stdout of the executed script to an input stream.
The problem that I am having is that when reading from the stream, the Java process is blocked until the subprocess is finished execution. I cannot wait until the end of the subprocess to react to the output, but I need to read it asynchronously as it becomes available.
Below is my Java code:
public class SubProcessReact {
public static class LogOutputStreamImpl extends LogOutputStream {
#Override
protected void processLine(String line, int logLevel) {
System.out.println("R: " + line);
}
}
public static void main (String[] args) throws IOException, InterruptedException {
CommandLine cl = CommandLine.parse("python printNumbers.py");
DefaultExecutor e = new DefaultExecutor();
ExecuteStreamHandler sh = new PumpStreamHandler(new LogOutputStreamImpl());
e.setStreamHandler(sh);
Thread th = new Thread(() -> {
try {
e.execute(cl);
} catch (IOException e1) {
e1.printStackTrace();
}
});
th.start();
}
}
For this example, the subprocess is a python script which counts upwards with a one second delay between outputs so that I can verify that the Java code is responding as data comes in.
Python Code:
import time
for x in range(0,10):
print x
time.sleep(1)
I would expect LogOutputStreamImpl to print each line as it comes, but what is actually happening is that it reading the stream blocks until the subprocess is completed, and then all of the output is printed.
Is there something I could do to make this work as I intend?
Why use a third-party library to do something Java SE already does well? Personally, I prefer to depend on as few external libraries as possible, in order to make my programs easily portable and to reduce the points of failure:
ProcessBuilder builder = new ProcessBuilder("python", "printNumbers.py");
builder.inheritIO().redirectOutput(ProcessBuilder.Redirect.PIPE);
Process process = builder.start();
try (BufferedReader reader = new BufferedReader(
new InputStreamReader(process.getInputStream()))) {
reader.lines().forEach(line -> System.out.println("R: " + line));
}
process.waitFor();
Related
I've got some code that uses Runtime.exec() to run an external .jar (built as an IzPack installer).
If I run this external.jar from the command line like so:
java -jar external.jar
Then the command prompt does not return control until the application is finished. However, if I run external.jar from within some java class, using:
Process p = Runtime.getRuntime().exec("java -jar external.jar");
int exitCode = p.waitFor();
System.out.println("Process p returned: " + exitCode);
Then p returns almost instantly with a success code of 0, despite external.jar having not yet completed execution (i've also tried this via the ProcessBuilder route of external file execution).
Why does it wait to return from the command line, but not when executed from within another java program?
I've also set up 3 jars, A, B and C where A calls B which calls C (using Runtime.exec()), where C Thread.sleeps for 10 seconds, as a simple test, and as expected, A doesn't return until 10 seconds after it runs.
I figure this is probably some kind of a threading issue with external.jar where execution is being handed over from one thing to another, but given that it works directly from the command line i kind of expected to see the same behaviour (perhaps naively) when called from within another java program.
I've tested this on Windows and Ubuntu with Java 6.
Thanks!
another possible way to achieve this might be to capture the output of the process and wait for it to finish.
For example:
Process tr = Runtime.getRuntime().exec( new String[]{"wkhtmltopdf",mainPage,mainPagePDF});
BufferedReader stdOut=new BufferedReader(new InputStreamReader(tr.getInputStream()));
String s;
while((s=stdOut.readLine())!=null){
//nothing or print
}
Normally the output stream is tr.getInputStream() but depending on the program you are executing the process output stream migh be:
tr.getInputStream()
tr.getErrorStream()
tr.getOutputStream()
By doing this while loop you force your program to wait the process to finish.
You can use Process Builder....
ProcessBuilder pb = new ProcessBuilder("java", "-jar", "/fielname.jar");
Process p = pb.start();
p.waitFor();
Are you spawning a new thread to handle the spawning of the process? If so the origional program will continue to operate independently of the spawned process and therefore waitFor() will only work on the new process and not the parent.
Process.waitFor() is useless for some native system command.
You need to get the process's output to determine if it is returned.
I wrote a sample code for you
/**
*
* #param cmdarray command and parameter of System call
* #param dir the directory execute system call
* #param returnImmediately true indicate return after system call immediately;
* false otherwise.
* if set true, the returned call result does not have reference value
* #return the return code of system call , default is -1
*/
public static int systemCall(String[] cmdarray,File dir,boolean returnImmediately)
{
int result = -1;
try {
Process p = Runtime.getRuntime().exec(cmdarray,null,dir);
if(!returnImmediately)
{
java.io.InputStream stdin = p.getInputStream();
java.io.InputStreamReader isr = new java.io.InputStreamReader(stdin);
java.io.BufferedReader br = new java.io.BufferedReader(isr);
String line = null;
while ( (line = br.readLine()) != null)
System.out.println(line);
}
try{result = p.exitValue();}
catch(Exception ie){;}
} catch (IOException e) {
e.printStackTrace();}
return result;
}
public static void main(String[] argc){
String[] cmdarray = {"jar","cvf","s2.jar","*"};
File dir = new File("D:\\src\\struts-2.3.1");
int k = systemCall(cmdarray,dir,true);
System.out.println("k="+k);
}
I had the same problem using processs to execute some software using the console, and i just solved it using process.waitFor()
For me it worked perfectly.
try{
Process tr = Runtime.getRuntime().exec( new String[]{ "wkhtmltopdf",frontPage,frontPagePDF});
tr.waitFor();
} catch (Exception ex) {
EverLogger.logEntry("Error al pasar a PDF la portada", "error", "activity");
return;
}
some more code here.
I am trying to run a batch file with Runtime.exec() and then output its InputStream into a JTextArea. What I have works, but only partially. What happens is the batch file runs, but if it executes a command other than something like "echo" that command immediately terminates and the next line executes. For example, let's say I try to run a simple batch file like this:
#echo off
echo hello. waiting 5 seconds.
timeout /t 5 /nobreak > NUL
echo finished. goodbye.
The batch file executes, and the JTextArea says
hello. waiting 5 seconds.
finished. goodbye.
but it doesn't wait for 5 seconds in the middle.
I can't figure out why it's doing this. Here's what I use to run the batch file and read its InputStream.
private class ScriptRunner implements Runnable {
private final GUI.InfoGUI gui; // the name of my GUI class
private final String script;
public ScriptRunner(final GUI.InfoGUI gui, final File script) {
this.gui = gui;
this.script = script.getAbsolutePath();
}
#Override
public void run() {
try {
final Process p = Runtime.getRuntime().exec(script);
StreamReader output = new StreamReader(p.getInputStream(), gui);
Thread t = new Thread(output);
t.start();
int exit = p.waitFor();
output.setComplete(true);
while (t.isAlive()) {
sleep(500);
}
System.out.println("Processed finished with exit code " + exit);
} catch (final Exception e) {
e.printStackTrace();
}
}
}
private class StreamReader implements Runnable {
private final InputStream is;
private final GUI.InfoGUI gui;
private boolean complete = false;
public StreamReader(InputStream is, GUI.InfoGUI gui) {
this.is = is;
this.gui = gui;
}
#Override
public void run() {
BufferedReader in = new BufferedReader(new InputStreamReader(is));
try {
while (!complete || in.ready()) {
while (in.ready()) {
gui.setTextAreaText(in.readLine() + "\n");
}
sleep(250);
}
} catch (final Exception e) {
e.printStackTrace();
} finally {
try {
in.close();
} catch (final Exception e) {
e.printStackTrace();
}
}
public void setComplete(final boolean complete) {
this.complete = complete;
}
}
public void sleep(final long ms) {
try {
Thread.sleep(ms);
} catch (final InterruptedException ie) {
}
}
I know my code is pretty messy, and I'm sure it contains grammatical errors.
Thanks for anything you can do to help!
You're creating a Process but you're not reading from its standard error stream. The process might be writing messages to its standard error to tell you that there's a problem, but if you're not reading its standard error, you won't be able to read these messages.
You have two options here:
Since you already have a class that reads from a stream (StreamReader), wire up another one of these to the process's standard error stream (p.getErrorStream()) and run it in another Thread. You'll also need to call setComplete on the error StreamReader when the call to p.waitFor() returns, and wait for the Thread running it to die.
Replace your use of Runtime.getRuntime().exec() with a ProcessBuilder. This class is new in Java 5 and provides an alternative way to run external processes. In my opinion its most significant improvement over Runtime.getRuntime().exec() is the ability to redirect the process's standard error into its standard output, so you only have one stream to read from.
I would strongly recommend going for the second option and choosing to redirect the process's standard error into its standard output.
I took your code and replaced the line
final Process p = Runtime.getRuntime().exec(script);
with
final ProcessBuilder pb = new ProcessBuilder(script);
pb.redirectErrorStream(true);
final Process p = pb.start();
Also, I don't have your GUI code to hand, so I wrote the output of the process to System.out instead.
When I ran your code, I got the following output:
hello. waiting 5 seconds.
ERROR: Input redirection is not supported, exiting the process immediately.
finished. goodbye.
Processed finished with exit code 0
Had you seen that error message, you might have twigged that something was up with the timeout command.
Incidentally, I noticed in one of your comments that none of the commands suggested by ughzan worked. I replaced the timeout line with ping -n 5 127.0.0.1 > NUL and the script ran as expected. I couldn't reproduce a problem with this.
The problem is definitely in timeout.exe. If you add echo %errorlevel% after line with timeout, you will see that it returns 1 if running from java. And 0 if running in usual way. Probably, it requires some specific console functionality (i.e. cursor positioning) that is suppressed when running from java process.
Is there anything I can do to get this to work while running from Java
If you don't need ability to run any batch file then consider to replace timeout with ping. Otherwise... I've tried to run batch file with JNA trough Kernel32.CreateProcess and timeout runs fine. But then you need to implement reading of process output trough native calls also.
I hope someone will suggest better way.
The ready method only tells if the stream can guarantee that something can be read immediately, without blocking. You can't really trust it because always returning false is a valid implementation. Streams with buffers may return true only when they have something buffered. So I suspect your problem is here:
while (!complete || in.ready()) {
while (in.ready()) {
gui.setTextAreaText(in.readLine() + "\n");
}
sleep(250);
}
It should rather read something like this:
String line;
while (!complete || (line=in.readLine()) != null) {
gui.setTextAreaText(line + "\n");
}
It's probably because your "timeout ..." command returned with an error.
Three ways to test it:
Check if the "timeout ..." command works in the Windows command prompt.
Replace "timeout ..." in the script with "ping -n 5 127.0.0.1 > NUL" (it essentially does the same thing)
Remove everything but "timeout /t 5 /nobreak > NUL" from your script. The process should return with an error (1) if the timeout failed because it is the last command executed.
I have a console Java program that executes sh -i in a separate process and copies the data between the processes' input/output stream and corresponding System streams:
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
class StreamCopier implements Runnable {
private InputStream in;
private OutputStream out;
public StreamCopier(InputStream in, OutputStream out) {
this.in = in;
this.out = out;
}
public void run() {
try {
int n = 0;
byte[] buffer = new byte[4096];
while (-1 != (n = in.read(buffer))) {
out.write(buffer, 0, n);
out.flush();
}
} catch (IOException e) {
System.out.println(e);
}
}
}
public class Test {
public static void main(String[] args)
throws IOException, InterruptedException {
Process process = Runtime.getRuntime().exec("sh -i");
new Thread(new StreamCopier(
process.getInputStream(), System.out)).start();
new Thread(new StreamCopier(
process.getErrorStream(), System.err)).start();
new Thread(new StreamCopier(
System.in, process.getOutputStream())).start();
process.waitFor();
}
}
Running it under Linux results in the following:
$
[1]+ Stopped java -cp . Test
Could anyone clarify why is the application stopped and how to avoid it?
This is related to my question on copying streams, but I think this particular issue deserves separate attention.
You can turn off job control by invoking the shell like sh -i +m, which will stop it taking over the tty. This means that the fg and bg commands will not work and a Ctrl+Z will suspend your Java application, the shell and all programs started from it.
If you still want job control, you should use a pseudo terminal to communicate with the shell, which creates a new tty for the shell to use, but I don't think Java supports that.
You are being stopped by SIGTTIN or SIGTTOU. These signals are sent to a background process when they attempt to do IO to a TTY. In this case "background" means "not the controlling process group of the terminal". I suspect the subshell you're forking off is creating a new pgrp and taking over your tty. Then the parent program (java) does IO (in your case probably reading from the TTY) and gets SIGTTIN.
An easy way to confirm this theory would be to replace sh with something simpler (not a shell) which will not try to take over the tty.
I have some issues regarding ProcessBuilder.
The program is basically a simple wrapper invoking a command line script.
When running the script on its own via the terminal, the memory consumption stays below 2G.
When running the script via the java wrapper, the memory consumption explodes and even 8G is quickly filled up, resulting in out-of-memory errors.
The code to launch the process is simply:
public static int execute(String command) throws IOException
{
System.out.println("Executing: " + command);
ProcessBuilder pb = new ProcessBuilder(command.split(" +"));
Process p = pb.start();
// display any output in stderr or stdout
StreamConsumer stderr = new StreamConsumer(p.getErrorStream(), "stderr");
StreamConsumer stdout = new StreamConsumer(p.getInputStream(), "stdout");
new Thread(stderr).start();
new Thread(stdout).start();
try {
return p.waitFor();
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
}
The StreamConsumer class is simply a class which consumes the stdout/stderr streams and display them on the console.
...the question is: why on earth does the memory consumption explode?
Regards,
Arnaud
Edit:
Whether I use ProcessBuilder or
Runtime.getRuntime.exec(...), the
result is the same.
The memory bursts tend to appear during unix 'sort' invoked by the
shell script called:
sort big-text-file > big-text-file.sorted
Edit 2 on request of Jim Garrison:
Ok, here is the StreamConsumer class which I omitted because it is rather simple:
class StreamConsumer implements Runnable
{
InputStream stream;
String descr;
StreamConsumer(InputStream stream, String descr) {
this.stream = stream;
this.descr = descr;
}
#Override
public void run()
{
String line;
BufferedReader brCleanUp =
new BufferedReader (new InputStreamReader (stream));
try {
while ((line = brCleanUp.readLine ()) != null)
System.out.println ("[" + descr + "] " + line);
brCleanUp.close();
} catch (IOException e) {
// TODO: handle exception
}
}
}
if you change your command like this :
sort -o big-text-file.sorted big-text-file
is it always the same ?
Maybe its because those StreamConsumer threads are not daemons so they don't die and get garbage collected when your processes return? You could try:
//...
final StreamConsumer stderr = new StreamConsumer(p.getErrorStream(), "stderr");
final StreamConsumer stdout = new StreamConsumer(p.getInputStream(), "stdout");
final Thread stderrThread = new Thread(stderr);
final Thread stdoutThread = new Thread(stdout);
stderrThread.setDaemon(true);
stdoutThread.setDaemon(true);
stderrThread.start();
stdoutThread.start();
//...
Is this behavior happening for single invocation or after doing this many times?
So I'm trying to execute a shell script which produces a lot of output(in 100s of MBs) from a Java file.
This hangs the process and never completes.
However, within the shell script, if I redirect the output of the script to some log file or /dev/null Java file executes and completes in a jiffy.
Is it because of amount of data that the Java program never completes?
If so, is there any documentation as such? or is there any limit on the amount of data(documented)?
Here's how you can simulate this scenario.
Java file will look like:
import java.io.InputStream;
public class LotOfOutput {
public static void main(String[] args) {
String cmd = "sh a-script-which-outputs-huuggee-data.sh";
try {
ProcessBuilder pb = new ProcessBuilder("bash", "-c", cmd);
pb.redirectErrorStream(true);
Process shell = pb.start();
InputStream shellIn = shell.getInputStream();
int shellExitStatus = shell.waitFor();
System.out.println(shellExitStatus);
shellIn.close();
} catch (Exception ignoreMe) {
}
}
}
The script 'a-script-which-outputs-huuggee-data.sh' may look like:
#!/bin/sh
# Toggle the line below
exec 3>&1 > /dev/null 2>&1
count=1
while [ $count -le 1000 ]
do
cat some-big-file
((count++))
done
echo
echo Yes I m done
Free beer for the right answer. :)
It's because you're not reading from the Process' output.
As per the class' Javadocs, if you don't do this then you may end up with a deadlock; the process fills its IO buffer and waits for the "shell" (or listening process) to read from it and empty it. Meanwhile your process, which should be doing this, is blocking waiting for the process to exit.
You'll want to call getInputStream() and read from that reliably (perhaps from another thread) to stop the process blocking.
Also take a look at Five Java Process Pitfalls and When Runtime.exec() Won't - both informative articles about common problems with Process.
You're never reading the input stream, so it's probably blocking because the input buffer is full.
The input/output buffer have a limited size (depending on the operating system). If I remember correctly this wasn't big or Windows XP at least. Try creating a thread that reads the InputStream as fast as possible.
Something along these lines:
class StdInWorker
implements Worker
{
private BufferedReader br;
private boolean run = true;
private int linesRead = 0;
private StdInWorker (Process prcs)
{
this.br = new BufferedReader(
new InputStreamReader(prcs.getInputStream()));
}
public synchronized void run ()
{
String in;
try {
while (this.run) {
while ((in = this.br.readLine()) != null) {
this.buffer.add(in);
linesRead++;
}
Thread.sleep(50);
}
}
catch (IOException ioe) {
ioe.printStackTrace();
}
catch (InterruptedException ie) {}
}
}
}