Develop Reference

Execute a command in Java without redirecting the output

How do I run an external command (via a shell) from a Java program, such that no redirection takes place, and wait for the command to end? I want the file descriptors of the external program to be the same as those of the Java program. In particular I do not want the output to be redirected to a pipe that the Java program is reading. Having the Java program relay the output is not a solution.
This means that a plain invocation of java.lang.Runtime.exec is not the solution. I presume that java.lang.ProcessBuilder is involved, but how do I specify that output and error streams must be the same as the calling Java process?
class A {
public static void main(String[] args) {
try {
ProcessBuilder pb = new ProcessBuilder("echo", "foo");
/*TODO: pb.out = System.out; pb.err = System.err;*/
Process p = pb.start();
p.waitFor();
} catch (Exception e) {
System.err.println(e);
System.exit(1);
}
}
}
(This may or may not be the right approach.)
In other words, I'm looking for Java's system, but all I can find is (roughly) popen.
Here's an example of a situation where relaying cannot work: if the subprocess writes to both stdout and stderr and the Java program is relaying, then the Java program has no way to know the order of the write calls in the subprocess. So the order of the output on stdout and stderr from the Java program will be observably different if the two streams end up in the same file. Mixing stdout and stderr is of course not a solution because the caller may want to keep them separate.
While I think this question is of general interest, a Linux-specific solution would solve my immediate problem.

This is the intent of ProcessBuilder.redirectError/redirectOutput which were introduced in Java 7. Using Redirect.INHERIT will make the child process share stderr/stdout with the Java process:
class A {
public static void main(String[] args) {
try {
ProcessBuilder builder = new ProcessBuilder("echo", "foo");
builder.redirectError(ProcessBuilder.Redirect.INHERIT);
builder.redirectOutput(ProcessBuilder.Redirect.INHERIT);
Process p = builder.start();
p.waitFor();
} catch (Exception e) {
System.err.println(e);
System.exit(1);
}
}
}

You might take a look at the NuProcess project. Disclaimer: I wrote it. It provides non-blocking I/O from spawned processes. You still have to relay in Java (you receive callbacks), but because it is using epoll() in the case of Linux, I would expect it to preserve the order of the underlying program. Only a single thread is epoll()'ing the pipes so you won't get any thread scheduling order issues.
I'm 100% order would be preserved on MacOS X, or any BSD variant, because it uses a kqueue which is definitely ordered. Anyway, you might want to give it a shot, it's trivial to code and test.

You can't. By default all standard I/O of the child process are redirected to the parent process (the jvm running your java program).
from the javadoc of the Process class:
By default, the created subprocess does not have its own terminal or
console. All its standard I/O (i.e. stdin, stdout, stderr)
operations will be redirected to the parent process, where they can be
accessed via the streams obtained using the methods getOutputStream(),
getInputStream(), and getErrorStream(). The parent process uses these
streams to feed input to and get output from the subprocess. Because
some native platforms only provide limited buffer size for standard
input and output streams, failure to promptly write the input stream
or read the output stream of the subprocess may cause the subprocess
to block, or even deadlock.

Invoking python from Java

I'm building a front-end for my company's internal tool kit. Half the tools are written in python and then the other half are written in several other scripting languages. So I'm building a front-end in java using swing. So far I can invoke python scripts through the following code:
public class Foo
{
public static void main(String[] args)
{
try
{
Runtime r = Runtime.getRuntime();
Process p = r.exec("python foo.py");
BufferedReader br = new BufferedReader(new InputStreamReader(p.getInputStream()));
p.waitFor();
String line = "";
while (br.ready())
System.out.println(br.readLine());
}
catch (Exception e)
{
String cause = e.getMessage();
if (cause.equals("python: not found"))
System.out.println("No python interpreter found.");
}
}
}
Which works beautifully but if the python script encounters any errors it doesn't print them out. How can I ensure that it also prints out any errors that it has?

The simple answer is to also read Process.getErrorStream.
The more complicated answer is that what you call Python likely refers to CPython which is just one implementation of the language. There is another implementation, Jython, which basically compiles Python into Java bytecode to be run on a JVM. This would allow tighter integration than simply invoking CPython via Java's Runtime.exec
P.S. Runtime.exec is sort of the old way of doing things. ProcessBuilder is often a much cleaner and more intuitive way of starting a sub-process in Java.

The prevois answer is Ok,here is a suggestion that you shoud release any resources of Process,like:
Process process = null;
try{
//xxxx
}catch(xxx){
}finally{
if(process!=null){
process.destroy();
}
}
The reason is that if you forgot to destroy process,the file handler involved would leak,
you will got an IOException show too many open files finally.

Java process invoked by ProcessBuilder sleeps forever

I'm working on a big system, which I have written entirely in Java. But I'm also at some point communicating with a C program by running it as a process, communicating some input to it through the file system and then waiting for it to finish to read the results from it. For now, I can't do anything but this. There is no time for me to link them via an API or anything like that.
This worked well until I needed to invoke this process twice to get the results. One I make the first invocation, it works just fine. However, for the second invocation the process just hangs! I don't if it is sleeping and waiting for a signal but I don't see why it should do so.
Here is the method to do the invocation:
public synchronized boolean processCommand(List command) {
try {
ProcessBuilder pb = new ProcessBuilder(command);
Process p = pb.start();
p.waitFor();
p.destroy();
} catch(Exception ex) { return false; }
return true;
}
I really do not need to communicate to the stdout or stdin. I just need the process to run and finish its job. But it just hangs when the calling process is waiting for it ONLY the second time I call it!
My calling code is just simply making the command list and calling this method from another java object.
Both calls to the method processCommand(List command) work fine when the input to the C program is smaller. Would that be some issue with the stdin or stdout?
It's just driving me crazy! Anybody has an insight into this? I appreciate your commend :)
Update:
Here is the solution based on what #Gray mentioned:
I just need to drain the InputStream and possibly the ErrorStream:
public synchronized boolean processCommand(List command) {
try {
ProcessBuilder pb = new ProcessBuilder(command);
Process p = pb.start();
handleStream(p.getInputStream);
handleStream(p.getErrorStream);
p.waitFor();
p.destroy();
} catch(Exception ex) { return false; }
return true;
}
public void handleStream(InputStream input) {
try {
int c;
while( (c=input.read())!= -1) { //Anything }
} catch(Exception ex) { }
}

Try following the solution suggested here (The original link seems to be unavailable; an archived version can be found here, although the referenced link to the example code still points to the original site...)
The best strategy for handling this situation is to kick off a thread just prior to calling waitFor that will interrupt the current thread after a suitable time period. TimerTask was designed for precisely this sort of situation, and waitFor is very responsive to interrupts in most cases. [Follow link in original for an example].
[...]
Java 6 API clearly states that failure to promptly “read the output stream of the subprocess may cause the subprocess to block, and even deadlock.”
[...]
The safe bet for handling this problem today is to explicitly clean up every instance of Process by calling close on each stream made available through Process.getOutputSteam, Process.getInputStream, and Process.getErrorStream, and then call Process.destroy even if the process is already terminated.

Help running Java runtime.exec() on multiple threads

In my program, I need to run a external command in a Ubuntu environment (ntpdate) using java. Currently my code looks like this:
Runtime rt = Runtime.getRuntime();
byte[] readBuffer = new byte[131072];
// Exec a process to do the query
Process p = null;
try {
p = rt.exec("ntpdate -q " + ip);
} catch (Exception ex) {
ex.printStackTrace();
}
if(p!= null){
try {
Thread.sleep(1000);
} catch (Exception e) {
}
// Read the input stream, copy it to the file
InputStream in = p.getInputStream();
try {
int count = 0, rc;
while ((rc = in.read(readBuffer, count, readBuffer.length - count)) != -1) {
count += rc;
if (count >= readBuffer.length) {
p.destroy();
break;
}
}
p.destroy();
result = processOutput(readBuffer, count);
} catch (IOException ex) {
ex.printStackTrace();
}
p.destroy();
This code need to be ran simultaneously on multiple threads in order to maximize performance (I need to test a list of 1.000.000 addresses using ntpdate). However, it runs very slowly, barely consuming machine processing. What am I doing wrong? How could I make this more efficient?
The same problem arises when trying to execute "dig" using .exec(), so I doubt it is because of the specific program being called. Is there some restriction in using Runtime.exec() in a multi Threaded environment?

Is Java the most appropriate approach here? Perhaps this would be better in a shell script, which calls ntpdate in the background multiple times? I'm not sure what benefit you're getting from this code snippet by doing this in Java.
What are you doing with the InputStream from the process?
A bash script could do this like:
for ip in #...IP list
do
ntpdate -q $ip > $ip.txt &
done

Why are you waiting for 1 second at each time ?
try {
Thread.sleep(1000);
} catch (Exception e) {
}
This will do nothing but slowing the execution of your application.

Not sure why it's slow but you need to do a lot more to close your resources. Runtime.exec() needs quite a bit of care and attention to avoid hang-ups and leaking of file descriptors.
http://www.javaworld.com/javaworld/jw-12-2000/jw-1229-traps.html

Are you sure the issue isn't ntpdate? If ntpdate is just sitting there waiting for a server response and has a large timeout value, then your application is going to sit there too.
Try calling ntpdate with a timeout of 0.2 and see if it makes a difference.
Also, as you're opening streams in your code, you definitely want to explicitly .close() them when you're done. Otherwise it might not happen until a GC which could be a very long time away.

I think I found the solution, and that is that there is no solution using java's Runtime.exec(). The problem seems to be that all calls to start a process are synchronized. Indeed, if you start each process alone (via synchronization) you get the exact same result of starting all processes together.
Are there any alternatives to exec? Otherwise, I will need to get some solution without linux's ntpdate...

I notice that both of the commands you tried involve network round-trips. How is the speed if you call something like echo or cat instead?

Java, system command fails to run

I'm trying to run a system command to extract a tar.bz2 file to a specified directory. This is the code:
ProcessBuilder myProc = new ProcessBuilder("tar", "-xjf", "/path/to/MyTarFile.tar.bz2");
myProc.directory(new File("/directory/i/want/results/in/"));
myProc.start();
System.out.println(myProc.command());
It runs without error, however the file is deleted and not extracted anywhere.
Any help would be greatly appreciated.

I know Runtime.exec() has a really nasty feature where if you don't manually drain STDOUT/STDERR, it effectively appears to hang. I would hope that ProcessBuilder corrected that deficiency, but this page includes this tidbit:
A word of caution about the examples in this tip. It is possible that the examples will deadlock if the subprocess generates enough output to overflow the system. A more robust solution requires draining the process stdout and stderr in separate threads.
So, make sure you're handling Process.getInputStream() and Process.getErrorStream(), as I recommended in the comments; it could solve the problem outright!

Change the myProc.start(); line to
Process p = myProc.start();
p.waitFor();
That will make sure your program doesn't exit until the tar is finished.

Run this to see errors. Perhaps one of your paths is incorrect.
import java.io.File;
import java.io.InputStream;
public class Untar {
public static void main(String[] args) throws Exception {
ProcessBuilder myProc = new ProcessBuilder("tar", "-xjf", "foo.tar.bz2");
myProc.directory(new File("newdir"));
Process p = myProc.start();
InputStream is = p.getErrorStream();
int c;
while( (c = is.read()) != -1 ){
System.out.print((char)c);
}
p.waitFor();
System.out.println(myProc.command());
}
}

Just tried this code. It works.
Check your paths.