Shell command not executed - java

I have a weird problem when trying to execute a shell command from within a java program. Since there exist thousands of websites that explain how to do it I used the following recommended code:
public String executeShellCommand (String command)
{
try
{
StringBuffer sb = new StringBuffer();
String line = "";
Process p = Runtime.getRuntime().exec(command);
BufferedReader reader = new BufferedReader(new InputStreamReader(
p.getInputStream()));
while ((line = reader.readLine()) != null)
sb.append(line + "\n");
p.waitFor();
return sb.toString();
}
catch (Exception e)
{
e.printStackTrace();
}
return null;
}
Acutally, when I try to execute for instance ls -aF is works fine and I get some output as a result. Therefore I'm pretty sure that the above code is, in principal, correct. However, I got another program I'd like to run and that produces a file as an output. I would like to execute it the above way but it never is executed and no output file is generated. Also I do not get any error, warnings or whatsoever in java. When copy and pasting the actual command argument string into the console the execution of the programm/command directly in the shell works fine and the output file is generated. So the command I pass to the method is also correct.
Are there additional things I need to pay attention to when trying to execute a shell command from within java?
UPDATE: I modified my code according to the suggestions. However, it is still hanging:
public String executeShellCommand(List<String> command, String logfile, boolean waitForProcess) { try {
ProcessBuilder pb = new ProcessBuilder(command);
System.out.println("pb.toString() = " + pb.toString());
Process p = pb.start();
System.out.println("2");
BufferedReader err = new BufferedReader(new InputStreamReader(p.getErrorStream()));
BufferedReader out = new BufferedReader(new InputStreamReader(p.getInputStream()));
System.out.println("3");
StringBuilder errSb = new StringBuilder();
StringBuilder outSb = new StringBuilder();
String line;
System.out.println("4");
while ((line = err.readLine()) != null) { // <--- code hangs here
errSb.append(line + "\n");
System.out.println("errSb = " + errSb.toString());
}
System.out.println("4a");
while ((line = out.readLine()) != null) {
outSb.append(line + "\n");
System.out.println("outSb = " + outSb.toString());
}
System.out.println("5");
if(waitForProcess) {
System.out.println("Wait for process");
p.waitFor();
} else {
System.out.println("Sleep 5000");
Thread.sleep(5000);
}
System.out.println("6");
//Log result to file
if(logfile != null) {
OutputStreamWriter outWriter = new OutputStreamWriter(new FileOutputStream(logfile));
outWriter.write(errSb.toString());
outWriter.close();
}
return errSb.toString();
} catch(Exception e) { e.printStackTrace(); } return null; }


This will block if your command writes too many characters to stderr. Like for sdtout, Java redirect stderr through a pipe, and if you do not read the pipe, it can fill up and block (size of the pipe is probably less than 256 bytes). To avoid that, you need to read from the Process.getErrorStream(), preferable from another thread as the main thread is busy reading from the Process.getInputStream().
A simpler way to avoid that is to use the ProcessBuilder class instead of Runtime.exec() and ProcessBuilder.redirectErrorStream(true) so that both stdout and stderr are merged into the Process.getInputStream()


As per Process javadoc :
Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block, or even deadlock.
You are calling p.waitFor(). If we carefully read the waitFor() documentation:
Causes the current thread to wait, if necessary, until the process represented by this Process object has terminated.
You are waiting for a process which hangs, because its error stream and output stream are never read.
What you should do, is to read these streams:
p.start();
BufferedReader err= new BufferedReader(new InputStreamReader(p.getErrorStream()));
BufferedReader out = new BufferedReader(new InputStreamReader(p.getOutputStream()));
StringBuilder errSb = new StringBuilder();
StringBuilder outSb = new Stringbuilder();
String line;
while ((line = err.readLine()) != null) {
errSb.append(line);
}
while ((line = out.readLine()) != null) {
outSB.append(line);
}
int retCode = p.waitFor(); //0 for success
System.out.println(retCode);
System.err.println(errSB.toString());
You should always read the error stream when calling external programs via the Process class, else you may find yourself in this odd situation where a process hangs forever. (well until someone else -the operating system, another application, etc- kills it, more exactly).
I've also noticed that you use the Runtime.getRuntime() which is not the recommended way to run external programs, starting with java 1.5, as per javadoc:
As of 1.5, ProcessBuilder.start() is the preferred way to create a Process.
ProcessBuilder pb = new ProcessBuilder("ls" , "-aF");
Process p = pb.start();

Related

Java process stuck at getRuntime().exec() [duplicate]

Process process = Runtime.getRuntime().exec("tasklist");
BufferedReader reader =
new BufferedReader(new InputStreamReader(process.getInputStream()));
process.waitFor();

There are many reasons that waitFor() doesn't return.
But it usually boils down to the fact that the executed command doesn't quit.
This, again, can have many reasons.
One common reason is that the process produces some output and you don't read from the appropriate streams. This means that the process is blocked as soon as the buffer is full and waits for your process to continue reading. Your process in turn waits for the other process to finish (which it won't because it waits for your process, ...). This is a classical deadlock situation.
You need to continually read from the processes input stream to ensure that it doesn't block.
There's a nice article that explains all the pitfalls of Runtime.exec() and shows ways around them called "When Runtime.exec() won't" (yes, the article is from 2000, but the content still applies!)

It appears you are not reading the output before waiting for it to finish. This is fine only if the output doesn't fill the buffer. If it does, it will wait until you read the output, catch-22.
Perhaps you have some errors which you are not reading. This would case the application to stop and waitFor to wait forever. A simple way around this is to re-direct the errors to the regular output.
ProcessBuilder pb = new ProcessBuilder("tasklist");
pb.redirectErrorStream(true);
Process process = pb.start();
BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
String line;
while ((line = reader.readLine()) != null)
System.out.println("tasklist: " + line);
process.waitFor();

Also from Java doc:
java.lang
Class Process
Because some native platforms only provide limited buffer size for standard input and
output streams, failure to promptly write the input stream or read the output stream of
the subprocess may cause the subprocess to block, and even deadlock.
Fail to clear the buffer of input stream (which pipes to the output stream of subprocess)
from Process may lead to a subprocess blocking.
Try this:
Process process = Runtime.getRuntime().exec("tasklist");
BufferedReader reader =
new BufferedReader(new InputStreamReader(process.getInputStream()));
while ((reader.readLine()) != null) {}
process.waitFor();

I would like to add something to the previous answers but since I don't have the rep to comment, I will just add an answer. This is directed towards android users which are programming in Java.
Per the post from RollingBoy, this code almost worked for me:
Process process = Runtime.getRuntime().exec("tasklist");
BufferedReader reader =
new BufferedReader(new InputStreamReader(process.getInputStream()));
while ((reader.readLine()) != null) {}
process.waitFor();
In my case, the waitFor() was not releasing because I was executing a statement with no return ("ip adddr flush eth0"). An easy way to fix this is to simply ensure you always return something in your statement. For me, that meant executing the following: "ip adddr flush eth0 && echo done". You can read the buffer all day, but if there is nothing ever returned, your thread will never release its wait.
Hope that helps someone!

There are several possibilities:
You haven't consumed all the output on the process's stdout.
You haven't consumed all the output on the process's stderr.
The process is waiting for input from you and you haven't provided it, or you haven't closed the process's stdin.
The process is spinning in a hard loop.

As others have mentioned you have to consume stderr and stdout.
Compared to the other answers, since Java 1.7 it is even more easy. You do not have to create threads yourself anymore to read stderr and stdout.
Just use the ProcessBuilder and use the methods redirectOutput in combination with either redirectError or redirectErrorStream.
String directory = "/working/dir";
File out = new File(...); // File to write stdout to
File err = new File(...); // File to write stderr to
ProcessBuilder builder = new ProcessBuilder();
builder.directory(new File(directory));
builder.command(command);
builder.redirectOutput(out); // Redirect stdout to file
if(out == err) {
builder.redirectErrorStream(true); // Combine stderr into stdout
} else {
builder.redirectError(err); // Redirect stderr to file
}
Process process = builder.start();

For the same reason you can also use inheritIO() to map Java console with external app console like:
ProcessBuilder pb = new ProcessBuilder(appPath, arguments);
pb.directory(new File(appFile.getParent()));
pb.inheritIO();
Process process = pb.start();
int success = process.waitFor();

You should try consume output and error in the same while
private void runCMD(String CMD) throws IOException, InterruptedException {
System.out.println("Standard output: " + CMD);
Process process = Runtime.getRuntime().exec(CMD);
// Get input streams
BufferedReader stdInput = new BufferedReader(new InputStreamReader(process.getInputStream()));
BufferedReader stdError = new BufferedReader(new InputStreamReader(process.getErrorStream()));
String line = "";
String newLineCharacter = System.getProperty("line.separator");
boolean isOutReady = false;
boolean isErrorReady = false;
boolean isProcessAlive = false;
boolean isErrorOut = true;
boolean isErrorError = true;
System.out.println("Read command ");
while (process.isAlive()) {
//Read the stdOut
do {
isOutReady = stdInput.ready();
//System.out.println("OUT READY " + isOutReady);
isErrorOut = true;
isErrorError = true;
if (isOutReady) {
line = stdInput.readLine();
isErrorOut = false;
System.out.println("=====================================================================================" + line + newLineCharacter);
}
isErrorReady = stdError.ready();
//System.out.println("ERROR READY " + isErrorReady);
if (isErrorReady) {
line = stdError.readLine();
isErrorError = false;
System.out.println("ERROR::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::" + line + newLineCharacter);
}
isProcessAlive = process.isAlive();
//System.out.println("Process Alive " + isProcessAlive);
if (!isProcessAlive) {
System.out.println(":::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: Process DIE " + line + newLineCharacter);
line = null;
isErrorError = false;
process.waitFor(1000, TimeUnit.MILLISECONDS);
}
} while (line != null);
//Nothing else to read, lets pause for a bit before trying again
System.out.println("PROCESS WAIT FOR");
process.waitFor(100, TimeUnit.MILLISECONDS);
}
System.out.println("Command finished");
}

I think I observed a similar problem: some processes started, seemed to run successfully but never completed. The function waitFor() was waiting forever except if I killed the process in Task Manager.
However, everything worked well in cases the length of the command line was 127 characters or shorter. If long file names are inevitable you may want to use environmental variables, which may allow you keeping the command line string short. You can generate a batch file (using FileWriter) in which you set your environmental variables before calling the program you actually want to run.
The content of such a batch could look like:
set INPUTFILE="C:\Directory 0\Subdirectory 1\AnyFileName"
set OUTPUTFILE="C:\Directory 2\Subdirectory 3\AnotherFileName"
set MYPROG="C:\Directory 4\Subdirectory 5\ExecutableFileName.exe"
%MYPROG% %INPUTFILE% %OUTPUTFILE%
Last step is running this batch file using Runtime.

Here is a method that works for me.
NOTE: There is some code within this method that may not apply to you, so try and ignore it. For example "logStandardOut(...), git-bash, etc".
private String exeShellCommand(String doCommand, String inDir, boolean ignoreErrors) {
logStandardOut("> %s", doCommand);
ProcessBuilder builder = new ProcessBuilder();
StringBuilder stdOut = new StringBuilder();
StringBuilder stdErr = new StringBuilder();
boolean isWindows = System.getProperty("os.name").toLowerCase().startsWith("windows");
if (isWindows) {
String gitBashPathForWindows = "C:\\Program Files\\Git\\bin\\bash";
builder.command(gitBashPathForWindows, "-c", doCommand);
} else {
builder.command("bash", "-c", doCommand);
}
//Do we need to change dirs?
if (inDir != null) {
builder.directory(new File(inDir));
}
//Execute it
Process process = null;
BufferedReader brStdOut;
BufferedReader brStdErr;
try {
//Start the command line process
process = builder.start();
//This hangs on a large file
// https://stackoverflow.com/questions/5483830/process-waitfor-never-returns
//exitCode = process.waitFor();
//This will have both StdIn and StdErr
brStdOut = new BufferedReader(new InputStreamReader(process.getInputStream()));
brStdErr = new BufferedReader(new InputStreamReader(process.getErrorStream()));
//Get the process output
String line = null;
String newLineCharacter = System.getProperty("line.separator");
while (process.isAlive()) {
//Read the stdOut
while ((line = brStdOut.readLine()) != null) {
stdOut.append(line + newLineCharacter);
}
//Read the stdErr
while ((line = brStdErr.readLine()) != null) {
stdErr.append(line + newLineCharacter);
}
//Nothing else to read, lets pause for a bit before trying again
process.waitFor(100, TimeUnit.MILLISECONDS);
}
//Read anything left, after the process exited
while ((line = brStdOut.readLine()) != null) {
stdOut.append(line + newLineCharacter);
}
//Read anything left, after the process exited
while ((line = brStdErr.readLine()) != null) {
stdErr.append(line + newLineCharacter);
}
//cleanup
if (brStdOut != null) {
brStdOut.close();
}
if (brStdErr != null) {
brStdOut.close();
}
//Log non-zero exit values
if (!ignoreErrors && process.exitValue() != 0) {
String exMsg = String.format("%s%nprocess.exitValue=%s", stdErr, process.exitValue());
throw new ExecuteCommandException(exMsg);
}
} catch (ExecuteCommandException e) {
throw e;
} catch (Exception e) {
throw new ExecuteCommandException(stdErr.toString(), e);
} finally {
//Log the results
logStandardOut(stdOut.toString());
logStandardError(stdErr.toString());
}
return stdOut.toString();
}

Asynchronous reading of stream combined with avoiding Wait with a timeout will solve the problem.
You can find a page explaining this here http://simplebasics.net/.net/process-waitforexit-with-a-timeout-will-not-be-able-to-collect-the-output-message/

public static void main(String[] args) throws PyException, IOException, InterruptedException
these should be the exceptions thrown

Control a command line program session via Java

I have a computer algebra program (called Reduce) that works in the shell in an interactive manner: launch Reduce in the shell, then you can define variables, compute this and that, and what not. Reduce prints the output into the shell. My idea is that I want to build a frontend for this text-based program that evaluates its output and converts it into a nice LaTeX style formula. For this I want to use Java.
I can start Reduce via exec(). But how can I emulate text input to the opened shell, and how can I read back what Reduce writes into the shell?
Thanks
Jens
Edit 1: Current Code
// get the shell
Runtime rt = Runtime.getRuntime();
// execute reduce
String[] commands = {"D:/Programme/Reduce/reduce.com", "", ""};
Process proc = null;
try {
proc = rt.exec(commands);
} catch (Exception e) {
System.out.println("Error!\n");
}
// get the associated input / output / error streams
BufferedReader stdInput = new BufferedReader(new InputStreamReader(proc.getInputStream()));
BufferedWriter stdOutput = new BufferedWriter(new OutputStreamWriter(proc.getOutputStream()));
BufferedReader stdError = new BufferedReader(new InputStreamReader(proc.getErrorStream()));
// read the output from the command
System.out.println("Here is the standard output of the command:\n");
String s = null;
try {
while ((s = stdInput.readLine()) != null) {
System.out.println(s);
}
} catch (Exception e) {
}
// read any errors from the attempted command
System.out.println("Here is the standard error of the command (if any):\n");
try {
while ((s = stdError.readLine()) != null) {
System.out.println(s);
}
} catch (Exception e) {
}

You need to get the streams associated with the process including the InputStream, OutputStream, and ErrorStream. You then can send messages to the process via the OutputStream and then read info from the process via the InputStream and the ErrorStream.
From some code of mine:
final ProcessBuilder pBuilder = new ProcessBuilder(TEST_PROCESS_ARRAY);
final Process proc = pBuilder.start();
procInputStream = proc.getInputStream();
errorStream = proc.getErrorStream();
errorSBuffer = new StringBuffer();
streamGobblerSb = new StreamGobblerSb(errorStream, "Autoit Error", errorSBuffer);
new Thread(streamGobblerSb).start();
final Scanner scan = new Scanner(procInputStream);

You may want to look into using the Process class.
http://docs.oracle.com/javase/7/docs/api/java/lang/Process.html
I believe you may be able to start the process, and then use getOutputStream() to feed commands into the process.

While this is not strictly an answer, I discovered that it is more convenient for me to stick with PHP's function proc_open(). That way I can include the output directly in the frontend and do not need to worry about the communication between my Java program and the html frontend.
For everybody who wants to stick to the Java method: the article http://www.javaworld.com/javaworld/jw-12-2000/jw-1229-traps.html is a good reference.

BufferedReader.readline() hangs

I'm trying to run /usr/bin/perl -e 'for(my $i=0;$i<1000;$i++){print 1x1000;print STDERR 2x1000;}' (which works in terminal) with my program.
ProcessBuilder pb = new ProcessBuilder(go); //go is the command
process = pb.start();
BufferedReader incommandbuf = new BufferedReader(new InputStreamReader(process.getInputStream()),1024*1000);
BufferedReader errcommandbuf = new BufferedReader(new InputStreamReader(process.getErrorStream()),1024*1000);
stdString = "";
while ((line = incommandbuf.readLine()) != null)
{
stdString += line + "\n";
}
String errorstrtemp = "";
while ((line = errcommandbuf.readLine()) != null)
{
errorstrtemp += line + "\n";
}
If I try to run this it hangs on while ((line = incommandbuf.readLine()) != null). The program runs if I change the command to /usr/bin/perl -e 'for(my $i=0;$i<64;$i++){print 1x1000;print STDERR 2x1000;}'. If it goes up to 65 and higher it doesn't work. At first I thought I just have to change the size of the my BufferedReaders but it didn't help. Any clue on what is causing this? I will provide any additional info if needed.
Thanks.

You are reading one stream at a time. When the other stream fills up the buffer, your Process will stop waiting for you to read it. The solution is to either read the streams in different threads or use ProcessBuilder.redirectErrorStream

I want realtime output of my Runtime.getRuntime().exec()

public static void executeCommand(String cmd) {
try {
Process process = Runtime.getRuntime().exec(cmd, null,
new File("/usr/hadoop-0.20.2/"));
InputStream stdin = process.getInputStream();
InputStreamReader isr = new InputStreamReader(stdin);
BufferedReader br = new BufferedReader(isr);
String line;
System.out.println("<output></output>");
while ((line = br.readLine()) != null)
System.out.println(line);
InputStreamReader esr = new InputStreamReader(
process.getErrorStream());
BufferedReader errorReader = new BufferedReader(esr);
String lineError;
while ((lineError = errorReader.readLine()) != null)
System.out.println(lineError);
process.waitFor();
System.out.println("");
} catch (Exception e) {
e.printStackTrace();
}
}
Here's my code for executing a command named 'cmd'. But I cannot get realtime output through this code. The output comes out when the command finishes. I want realtime output. Is there a way to do this?

The issue you describe is most likely caused by the application you called: many applications use unbuffered I/O when connected to a terminal, but bufferen I/O when connected to a pipe. So your cmd may simply decide not to write its output in small bits, but instead in huge chunks. The proper fix is to adjust the command, to flush its output at the appropriate times. There is little you can do about this on the Java side. See also this answer.

I think you need to have a thread for handling the output.
You should try first with the cmd which run for a while
Last time, when I try with wvdial command (this wvdial will not finish until we stop it), I need a thread to read the output of wvdial

Actually, the problem is that Process.getInputStream() returns a BufferedReader.
So, even if the called subprocess flushes all its output, a read in the calling Java program will only get it if the buffer is full.

Communicating with C++ process from Java

First, I saw a few Q's about this issue in the site, but didn't see any answer that solve my problem.
I have a program written in Java and it calls a cmd program written in C++. (this is an assumption since I don't have the actual source) I know the expected I/O of the C++ program, in the cmd it is two lines of output and then it waits for string input.
I know that the first output line of the program is through error stream, and I receive it properly (this is expected), but I don't get the second line in error or input stream.
I tried to write to the program right after the first line ( the error line) and didn't got stuck, but there was no response.
I tried using 3 different threads, for each stream, but again, nothing was received in input/error stream after the first line, and the program didn't respond to writing through output stream.
My initializers are:
Process p = Runtime.getRuntime().exec("c:\\my_prog.exe");
BufferedReader err = new BufferedReader(new InputStreamReader(p.getErrorStream()));
BufferedReader input = new BufferedReader(new InputStreamReader(p.getInputStream()));
BufferedWriter output = new BufferedWriter(new OutputStreamWriter(p.getOutputStream()));
Is it possible at all or maybe it depends on the C++ program?
Thanks,
Binyamin

If you want to call native applications like C and C++ from Java, you need to use JNI.

I would suggest to put the input in the program when it has started, it will propably use that as input when it wants it.

Here is how I execute any command line in Java. This command line may execute any program:
private String executionCommandLine(final String cmd) {
StringBuilder returnContent = new StringBuilder();
Process pr;
try {
Runtime rt = Runtime.getRuntime();
pr = rt.exec(cmd);
BufferedReader input = new BufferedReader(new InputStreamReader(pr.getInputStream()));
String line = null;
while ((line = input.readLine()) != null) {
returnContent.append(line);
}
input.close();
LOG.debug(returnContent.toString());
// return the exit code
pr.waitFor();
} catch (IOException e) {
LOG.error(e.getMessage());
returnContent = new StringBuilder();
} catch (InterruptedException e) {
LOG.error(e.getMessage());
returnContent = new StringBuilder();
}
return returnContent.toString();
}

Categories

Resources