Develop Reference

How to make Runtime.getRuntime().exec execute command one by one and get output rather than executing all commands

I am trying to run a *.bat file (which is capable of running several commands and retrieve the output one by one) from my java application. My intention is to send one command, read output use this output for second command and again retrieve the output.
To achieve this, through Runtime.getRuntime().exec I am passing more than one command as an input to PrintWriter. Issue is that after completing all the steps only I can read the output from *.bat through buffer ,but my intention is to run one command get the output and manipulate this output to send second command.
Unfortunately is not working. Any resolution for this?..
I got the idea to send more than one command to Runtime.getRuntime().exec from this link (How to execute cmd commands via Java)
The following is the same code which I got from above link
String[] command =
{
"cmd",
};
Process p = Runtime.getRuntime().exec(command);
new Thread(new SyncPipe(p.getErrorStream(), System.err)).start();
new Thread(new SyncPipe(p.getInputStream(), System.out)).start();
PrintWriter stdin = new PrintWriter(p.getOutputStream());
stdin.println("dir c:\\ /A /Q");
// write any other commands you want here
stdin.close();
int returnCode = p.waitFor();
System.out.println("Return code = " + returnCode);
class SyncPipe implements Runnable
{
public SyncPipe(InputStream istrm, OutputStream ostrm) {
istrm_ = istrm;
ostrm_ = ostrm;
}
public void run() {
try
{
final byte[] buffer = new byte[1024];
for (int length = 0; (length = istrm_.read(buffer)) != -1; )
{
ostrm_.write(buffer, 0, length);
}
}
catch (Exception e)
{
e.printStackTrace();
}
}
private final OutputStream ostrm_;
private final InputStream istrm_;
}

In your case I would not use Threads, you want a sequential execution path.

Actually, instead of trying to reinvent the wheel, I strongly suggest you to use an expect-like java library to do that kind of thing.
Just because there are several things that you'll have to deal with, such as timeout between requests, waiting for the output to return, etc.
Take a look at these libraries
http://expectj.sourceforge.net/
https://code.google.com/p/expect4j/
https://github.com/ronniedong/Expect-for-Java
http://code.google.com/p/enchanter/
In particular, I use expectj in my project and it works pretty well (although I think expect4j is more popular)
With expectj, your code will look like this (from http://expectj.sourceforge.net/)
// Create a new ExpectJ object with a timeout of 5s
ExpectJ expectinator = new ExpectJ(5);
// Fork the process
Spawn shell = expectinator.spawn("/bin/sh");
// Talk to it
shell.send("echo Chunder\n");
shell.expect("Chunder");
shell.send("exit\n");
shell.expectClose();

You can do the redirection of output of one command to other in the bat file itself using pipe.
I am sorry, i hadn't noticed that you want to manipulate the output first.
So instead of using bat file, you can run the commands that are in bat file from java using exec , get the out put, and use the out put to execute the next command.

How to execute a interactive shell script using java Runtime?

I am wondering is there any way to execute following shell script, which waits for user input using java's Runtime class?
#!/bin/bash
echo "Please enter your name:"
read name
echo "Welcome $name"
I am using following java code to do this task but it just shows blank console.
public class TestShellScript {
public static void main(String[] args) {
File wd = new File("/mnt/client/");
System.out.println("Working Directory: " +wd);
Process proc = null;
try {
proc = Runtime.getRuntime().exec("sudo ./test.sh", null, wd);
} catch (Exception e) {
e.printStackTrace();
}
}
}
Thing is when I execute above program, I believed it will execute a shell script and that shell script will wait for user input, but it just prints current directory and then exits. Is there any way to do this or it is not possible at all in java?
Thanks in advance

The reason it prints the current dir and exits is because your java app exits. You need to add a (threaded) listener to the input and error streams of your created process, and you'll probably want to add a printStream to the process's output stream
example:
proc = Runtime.getRuntime().exec(cmds);
PrintStream pw = new PrintStream(proc.getOutputStream());
FetcherListener fl = new FetcherListener() {
#Override
public void fetchedMore(byte[] buf, int start, int end) {
textOut.println(new String(buf, start, end - start));
}
#Override
public void fetchedAll(byte[] buf) {
}
};
IOUtils.loadDataASync(proc.getInputStream(), fl);
IOUtils.loadDataASync(proc.getErrorStream(), fl);
String home = System.getProperty("user.home");
//System.out.println("home: " + home);
String profile = IOUtils.loadTextFile(new File(home + "/.profile"));
pw.println(profile);
pw.flush();
To run this, you will need to download my sourceforge project: http://tus.sourceforge.net/ but hopefully the code snippet is instructive enough that you can just adapt to J2SE and whatever else you are using.

If you use a Java ProcessBuilder you should be able to get the Input, Error and Output streams of the Process you create.
These streams can be used to get information coming out of the process (like prompts for input) but they can also be written to to put information into the process directly too. For instance:
InputStream stdout = process.getInputStream ();
BufferedReader reader = new BufferedReader (new InputStreamReader(stdout));
String line;
while(true){
line = reader.readLine();
//...
That'll get you the output from the process directly. I've not done it myself, but I'm pretty sure that process.getOutputStream() gives you something that can be written to directly to send input to the process.

The problem with running interactive programs, such as sudo, from Runtime.exec is that it attaches their stdin and stdout to pipes rather than the console device they need. You can make it work by redirecting the input and output to /dev/tty.
You can achieve the same behaviour using the new ProcessBuilder class, setting up the redirection using ProcessBuilder.Redirect.INHERIT.

Note sure at all you can send input to your script from Java. However I very strongly recommend to have a look at Commons Exec if you are to execute external scripts from Java:
Commons Exec homepage
Commons Exec API

How can I launch VI from within Java under commons-exec?

I would like to be able to launch VI from within my Java program and wait for the user to quit VI before proceeding. Here's the code snippet that I have currently:
...
String previewFileName="test.txt"; // the file to edit
CommandLine cmdLine = new CommandLine("/usr/bin/vi");
cmdLine.addArgument(previewFileName);
cmdLine.addArgument(">/dev/tty");
cmdLine.addArgument("</dev/tty");
Executor executor = new DefaultExecutor();
try
{
DefaultExecuteResultHandler resultHandler = new ResetProcessResultHandler(cmdLine);
executor.execute(cmdLine, resultHandler);
} catch (IOException e)
{
throw new Error("Cannot execute command: /usr/bin/vi " + previewFileName, e);
}
log.info("waiting...");
cmdLine.wait();
log.info("...done");
...
private class ResetProcessResultHandler extends DefaultExecuteResultHandler
{
private final CommandLine mCommandLine;
public ResetProcessResultHandler(CommandLine pCommandLine)
{
mCommandLine = pCommandLine;
}
public void onProcessComplete(int exitValue)
{
log.info("Command complete rc(" + exitValue + ")");
if (exitValue != 0)
{
throw new RuntimeException("VI command error [rc=" + exitValue + "] " );
}
mCommandLine.notify();
}
public void onProcessFailed(ExecuteException e)
{
if (e.getExitValue() != 0)
{
log.error("launch VI error " + e.toString());
throw new RuntimeException("VI command failed [" + e.getCause() + "] ");
}
else
{
log.info("VI complete rc(" + e.getExitValue() + ")");
}
mCommandLine.notify();
}
}
I receive output:
Vim: output is not to a terminal
Vim: input is not from a terminal
But then I see the screen painted as if VI had started; and VI doesn't read characters I type.
So ... redirecting from /dev/tty isn't doing the trick.
Someone must have done this before - help!
Thanks,
Mark

However since Java 1.7 you can use the next example to transparently redirect and have full console functionality
System.out.println("STARTING VI");
ProcessBuilder processBuilder = new ProcessBuilder("/usr/bin/vi");
processBuilder.redirectOutput(ProcessBuilder.Redirect.INHERIT);
processBuilder.redirectError(ProcessBuilder.Redirect.INHERIT);
processBuilder.redirectInput(ProcessBuilder.Redirect.INHERIT);
Process p = processBuilder.start();
// wait for termination.
p.waitFor();
System.out.println("Exiting VI");
This will allow you to open VI transparently for JVM 1.7+.

When Java runs a program via Runtime.exec() (and this is what commons-exec does in the end), it connects the program's input, output and error streams to your Java app as input/output streams. Such a stream is certainly not a terminal, you can't for example move the text cursor in it (since it doesn't have any), change text colors, or detect if Shift key is pressed (since it's just a stream of bytes and not a physical keyborad). So, an interactive app like vi can't really function under such conditions like in a terminal.
By the way, I'm not sure if the command line args you supply are parsed by the shell or passed directly to the program. In the latter case your redirection to /dev/tty couldn't possibly work even if there was a way for Java to somehow allow the program to replace Java's connected streams with something else.
As an aside, it seems a bit strange why you would like to run vi from inside a Java program.
So I guess the best solution is to execute a terminal emulator like konsole or gnome-terminal or xterm and let it run vi by passing corresponding argument on its command line (e.g. konsole -e vi). In this case the terminal's window should pop up and vi could function inside it. Of course, it won't work if you're on a headless server, but then running vi can't be useful anyway.

I'm not sure how to do it with commons-exec,
But standard Java should be something along the lines of...
String[] command = {"/usr/bin/vi", "test.txt"};
Process vimProcess = Runtime.getRuntime().exec(command);
vimProcess.waitFor();
This will cause the current thread to wait for the process to complete. You can also use
vimProcess.getInputStream(), getOutputStream() and getErrorStream() to redirect those to log files or wherever you want it to go.
See here for more details.
http://docs.oracle.com/javase/6/docs/api/java/lang/Runtime.html
Hopefully this helps.

Calling a shell script from java hangs

So I'm trying to execute a shell script which produces a lot of output(in 100s of MBs) from a Java file.
This hangs the process and never completes.
However, within the shell script, if I redirect the output of the script to some log file or /dev/null Java file executes and completes in a jiffy.
Is it because of amount of data that the Java program never completes?
If so, is there any documentation as such? or is there any limit on the amount of data(documented)?
Here's how you can simulate this scenario.
Java file will look like:
import java.io.InputStream;
public class LotOfOutput {
public static void main(String[] args) {
String cmd = "sh a-script-which-outputs-huuggee-data.sh";
try {
ProcessBuilder pb = new ProcessBuilder("bash", "-c", cmd);
pb.redirectErrorStream(true);
Process shell = pb.start();
InputStream shellIn = shell.getInputStream();
int shellExitStatus = shell.waitFor();
System.out.println(shellExitStatus);
shellIn.close();
} catch (Exception ignoreMe) {
}
}
}
The script 'a-script-which-outputs-huuggee-data.sh' may look like:
#!/bin/sh
# Toggle the line below
exec 3>&1 > /dev/null 2>&1
count=1
while [ $count -le 1000 ]
do
cat some-big-file
((count++))
done
echo
echo Yes I m done
Free beer for the right answer. :)

It's because you're not reading from the Process' output.
As per the class' Javadocs, if you don't do this then you may end up with a deadlock; the process fills its IO buffer and waits for the "shell" (or listening process) to read from it and empty it. Meanwhile your process, which should be doing this, is blocking waiting for the process to exit.
You'll want to call getInputStream() and read from that reliably (perhaps from another thread) to stop the process blocking.
Also take a look at Five Java Process Pitfalls and When Runtime.exec() Won't - both informative articles about common problems with Process.

You're never reading the input stream, so it's probably blocking because the input buffer is full.

The input/output buffer have a limited size (depending on the operating system). If I remember correctly this wasn't big or Windows XP at least. Try creating a thread that reads the InputStream as fast as possible.
Something along these lines:
class StdInWorker
implements Worker
{
private BufferedReader br;
private boolean run = true;
private int linesRead = 0;
private StdInWorker (Process prcs)
{
this.br = new BufferedReader(
new InputStreamReader(prcs.getInputStream()));
}
public synchronized void run ()
{
String in;
try {
while (this.run) {
while ((in = this.br.readLine()) != null) {
this.buffer.add(in);
linesRead++;
}
Thread.sleep(50);
}
}
catch (IOException ioe) {
ioe.printStackTrace();
}
catch (InterruptedException ie) {}
}
}
}

Java Process "The pipe has been ended" problem

I am using Java Process API to write a class that receives binary input from the network (say via TCP port A), processes it and writes binary output to the network (say via TCP port B). I am using Windows XP. The code looks like this. There are two functions called run() and receive(): run is called once at the start, while receive is called whenever there is a new input received via the network. Run and receive are called from different threads.
The run process starts an exe and receives the input and output stream of the exe. Run also starts a new thread to write output from the exe on to the port B.
public void run() {
try {
Process prc = // some exe is `start`ed using ProcessBuilder
OutputStream procStdIn = new BufferedOutputStream(prc.getOutputStream());
InputStream procStdOut = new BufferedInputStream(prc.getInputStream());
Thread t = new Thread(new ProcStdOutputToPort(procStdOut));
t.start();
prc.waitFor();
t.join();
procStdIn.close();
procStdOut.close();
} catch (Exception e) {
e.printStackTrace();
printError("Error : " + e.getMessage());
}
}
The receive forwards the received input from the port A to the exe.
public void receive(byte[] b) throws Exception {
procStdIn.write(b);
}
class ProcStdOutputToPort implements Runnable {
private BufferedInputStream bis;
public ProcStdOutputToPort(BufferedInputStream bis) {
this.bis = bis;
}
public void run() {
try {
int bytesRead;
int bufLen = 1024;
byte[] buffer = new byte[bufLen];
while ((bytesRead = bis.read(buffer)) != -1) {
// write output to the network
}
} catch (IOException ex) {
Logger.getLogger().log(Level.SEVERE, null, ex);
}
}
}
The problem is that I am getting the following stack inside receive() and the prc.waitfor() returns immediately afterwards. The line number shows that the stack is while writing to the exe.
The pipe has been ended
java.io.IOException: The pipe has been ended
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:260)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
at java.io.FilterOutputStream.write(FilterOutputStream.java:80)
at xxx.receive(xxx.java:86)
Any advice about this will be appreciated.

This means you are writing to the pipe after the other end has already closed it.
That indicates a major error in your application protocol.

I have had the same problem recently and I have found a solution.
First of all, "The pipe has been ended" error is not a Java error - it comes from Windows system. According to MSDN:
The using process has closed the pipe or, if you are trying to write
to the pipe, there are no available readers.
Not very informative. However, if process has closed the pipe itself, it may mean that some errors occurred in process.
To check this, redirect errors coming from process, for instance, to a file:
File f = new File("errors.txt");
pb.redirectError(f);
In my case (I've been trying to execute SrcML parser) file contained this:
.\libs\srcML-Win\src2srcml.exe: unrecognised option `--language Java'
Try 'src2srcml --help' for more information.
Fixing this solved the problem.