Develop Reference

BufferedReader.reader.readLine() returns nothing in Java

I have the following code:
Process pr = run.exec(sonarScannerCommand);
BufferedReader reader = new BufferedReader(new InputStreamReader(pr.getInputStream()));
BufferedReader eReader = new BufferedReader(new InputStreamReader(pr.getErrorStream()));
StringBuilder output = new StringBuilder();
String line;
boolean error = false;
while ((line = reader.readLine()) != null) {
output.append(line + "\n");
if (line.contains("EXECUTION FAILURE")) {
error = true;
}
log.info(line);
}
String errorString = "";
while ((line = eReader.readLine()) != null) {
if (errorString.equals("") && error) {
errorString = line;
}
output.append(line + "\n");
log.error(line);
}
pr.waitFor(5, TimeUnit.MINUTES);
At some point in the process, it freezes and doesn't move forward.
It returns nothing to the readLine () method and gets stuck waiting for something that doesn't return.
How can I manage this wait?
Thank you

You must consume stdout and stderr on different threads, because if one of these output streams fills its buffer while you are only reading from the other stream, then the process blocks indefinitely.
Alternatively switch to using ProcessBuilder and map stderr to stdout, then you just read from the merged stream not both streams:
ProcessBuilder pb = new ProcessBuilder(command);
pb.redirectErrorStream(true);
Process pr = pb.start();
... Your reading code for reader, eReader is not required
int rc = exec.waitFor();

Java process stuck at getRuntime().exec() [duplicate]

Process process = Runtime.getRuntime().exec("tasklist");
BufferedReader reader =
new BufferedReader(new InputStreamReader(process.getInputStream()));
process.waitFor();

There are many reasons that waitFor() doesn't return.
But it usually boils down to the fact that the executed command doesn't quit.
This, again, can have many reasons.
One common reason is that the process produces some output and you don't read from the appropriate streams. This means that the process is blocked as soon as the buffer is full and waits for your process to continue reading. Your process in turn waits for the other process to finish (which it won't because it waits for your process, ...). This is a classical deadlock situation.
You need to continually read from the processes input stream to ensure that it doesn't block.
There's a nice article that explains all the pitfalls of Runtime.exec() and shows ways around them called "When Runtime.exec() won't" (yes, the article is from 2000, but the content still applies!)

It appears you are not reading the output before waiting for it to finish. This is fine only if the output doesn't fill the buffer. If it does, it will wait until you read the output, catch-22.
Perhaps you have some errors which you are not reading. This would case the application to stop and waitFor to wait forever. A simple way around this is to re-direct the errors to the regular output.
ProcessBuilder pb = new ProcessBuilder("tasklist");
pb.redirectErrorStream(true);
Process process = pb.start();
BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
String line;
while ((line = reader.readLine()) != null)
System.out.println("tasklist: " + line);
process.waitFor();

Also from Java doc:
java.lang
Class Process
Because some native platforms only provide limited buffer size for standard input and
output streams, failure to promptly write the input stream or read the output stream of
the subprocess may cause the subprocess to block, and even deadlock.
Fail to clear the buffer of input stream (which pipes to the output stream of subprocess)
from Process may lead to a subprocess blocking.
Try this:
Process process = Runtime.getRuntime().exec("tasklist");
BufferedReader reader =
new BufferedReader(new InputStreamReader(process.getInputStream()));
while ((reader.readLine()) != null) {}
process.waitFor();

I would like to add something to the previous answers but since I don't have the rep to comment, I will just add an answer. This is directed towards android users which are programming in Java.
Per the post from RollingBoy, this code almost worked for me:
Process process = Runtime.getRuntime().exec("tasklist");
BufferedReader reader =
new BufferedReader(new InputStreamReader(process.getInputStream()));
while ((reader.readLine()) != null) {}
process.waitFor();
In my case, the waitFor() was not releasing because I was executing a statement with no return ("ip adddr flush eth0"). An easy way to fix this is to simply ensure you always return something in your statement. For me, that meant executing the following: "ip adddr flush eth0 && echo done". You can read the buffer all day, but if there is nothing ever returned, your thread will never release its wait.
Hope that helps someone!

There are several possibilities:
You haven't consumed all the output on the process's stdout.
You haven't consumed all the output on the process's stderr.
The process is waiting for input from you and you haven't provided it, or you haven't closed the process's stdin.
The process is spinning in a hard loop.

As others have mentioned you have to consume stderr and stdout.
Compared to the other answers, since Java 1.7 it is even more easy. You do not have to create threads yourself anymore to read stderr and stdout.
Just use the ProcessBuilder and use the methods redirectOutput in combination with either redirectError or redirectErrorStream.
String directory = "/working/dir";
File out = new File(...); // File to write stdout to
File err = new File(...); // File to write stderr to
ProcessBuilder builder = new ProcessBuilder();
builder.directory(new File(directory));
builder.command(command);
builder.redirectOutput(out); // Redirect stdout to file
if(out == err) {
builder.redirectErrorStream(true); // Combine stderr into stdout
} else {
builder.redirectError(err); // Redirect stderr to file
}
Process process = builder.start();

For the same reason you can also use inheritIO() to map Java console with external app console like:
ProcessBuilder pb = new ProcessBuilder(appPath, arguments);
pb.directory(new File(appFile.getParent()));
pb.inheritIO();
Process process = pb.start();
int success = process.waitFor();

You should try consume output and error in the same while
private void runCMD(String CMD) throws IOException, InterruptedException {
System.out.println("Standard output: " + CMD);
Process process = Runtime.getRuntime().exec(CMD);
// Get input streams
BufferedReader stdInput = new BufferedReader(new InputStreamReader(process.getInputStream()));
BufferedReader stdError = new BufferedReader(new InputStreamReader(process.getErrorStream()));
String line = "";
String newLineCharacter = System.getProperty("line.separator");
boolean isOutReady = false;
boolean isErrorReady = false;
boolean isProcessAlive = false;
boolean isErrorOut = true;
boolean isErrorError = true;
System.out.println("Read command ");
while (process.isAlive()) {
//Read the stdOut
do {
isOutReady = stdInput.ready();
//System.out.println("OUT READY " + isOutReady);
isErrorOut = true;
isErrorError = true;
if (isOutReady) {
line = stdInput.readLine();
isErrorOut = false;
System.out.println("=====================================================================================" + line + newLineCharacter);
}
isErrorReady = stdError.ready();
//System.out.println("ERROR READY " + isErrorReady);
if (isErrorReady) {
line = stdError.readLine();
isErrorError = false;
System.out.println("ERROR::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::" + line + newLineCharacter);
}
isProcessAlive = process.isAlive();
//System.out.println("Process Alive " + isProcessAlive);
if (!isProcessAlive) {
System.out.println(":::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: Process DIE " + line + newLineCharacter);
line = null;
isErrorError = false;
process.waitFor(1000, TimeUnit.MILLISECONDS);
}
} while (line != null);
//Nothing else to read, lets pause for a bit before trying again
System.out.println("PROCESS WAIT FOR");
process.waitFor(100, TimeUnit.MILLISECONDS);
}
System.out.println("Command finished");
}

I think I observed a similar problem: some processes started, seemed to run successfully but never completed. The function waitFor() was waiting forever except if I killed the process in Task Manager.
However, everything worked well in cases the length of the command line was 127 characters or shorter. If long file names are inevitable you may want to use environmental variables, which may allow you keeping the command line string short. You can generate a batch file (using FileWriter) in which you set your environmental variables before calling the program you actually want to run.
The content of such a batch could look like:
set INPUTFILE="C:\Directory 0\Subdirectory 1\AnyFileName"
set OUTPUTFILE="C:\Directory 2\Subdirectory 3\AnotherFileName"
set MYPROG="C:\Directory 4\Subdirectory 5\ExecutableFileName.exe"
%MYPROG% %INPUTFILE% %OUTPUTFILE%
Last step is running this batch file using Runtime.

Here is a method that works for me.
NOTE: There is some code within this method that may not apply to you, so try and ignore it. For example "logStandardOut(...), git-bash, etc".
private String exeShellCommand(String doCommand, String inDir, boolean ignoreErrors) {
logStandardOut("> %s", doCommand);
ProcessBuilder builder = new ProcessBuilder();
StringBuilder stdOut = new StringBuilder();
StringBuilder stdErr = new StringBuilder();
boolean isWindows = System.getProperty("os.name").toLowerCase().startsWith("windows");
if (isWindows) {
String gitBashPathForWindows = "C:\\Program Files\\Git\\bin\\bash";
builder.command(gitBashPathForWindows, "-c", doCommand);
} else {
builder.command("bash", "-c", doCommand);
}
//Do we need to change dirs?
if (inDir != null) {
builder.directory(new File(inDir));
}
//Execute it
Process process = null;
BufferedReader brStdOut;
BufferedReader brStdErr;
try {
//Start the command line process
process = builder.start();
//This hangs on a large file
// https://stackoverflow.com/questions/5483830/process-waitfor-never-returns
//exitCode = process.waitFor();
//This will have both StdIn and StdErr
brStdOut = new BufferedReader(new InputStreamReader(process.getInputStream()));
brStdErr = new BufferedReader(new InputStreamReader(process.getErrorStream()));
//Get the process output
String line = null;
String newLineCharacter = System.getProperty("line.separator");
while (process.isAlive()) {
//Read the stdOut
while ((line = brStdOut.readLine()) != null) {
stdOut.append(line + newLineCharacter);
}
//Read the stdErr
while ((line = brStdErr.readLine()) != null) {
stdErr.append(line + newLineCharacter);
}
//Nothing else to read, lets pause for a bit before trying again
process.waitFor(100, TimeUnit.MILLISECONDS);
}
//Read anything left, after the process exited
while ((line = brStdOut.readLine()) != null) {
stdOut.append(line + newLineCharacter);
}
//Read anything left, after the process exited
while ((line = brStdErr.readLine()) != null) {
stdErr.append(line + newLineCharacter);
}
//cleanup
if (brStdOut != null) {
brStdOut.close();
}
if (brStdErr != null) {
brStdOut.close();
}
//Log non-zero exit values
if (!ignoreErrors && process.exitValue() != 0) {
String exMsg = String.format("%s%nprocess.exitValue=%s", stdErr, process.exitValue());
throw new ExecuteCommandException(exMsg);
}
} catch (ExecuteCommandException e) {
throw e;
} catch (Exception e) {
throw new ExecuteCommandException(stdErr.toString(), e);
} finally {
//Log the results
logStandardOut(stdOut.toString());
logStandardError(stdErr.toString());
}
return stdOut.toString();
}

Asynchronous reading of stream combined with avoiding Wait with a timeout will solve the problem.
You can find a page explaining this here http://simplebasics.net/.net/process-waitforexit-with-a-timeout-will-not-be-able-to-collect-the-output-message/

public static void main(String[] args) throws PyException, IOException, InterruptedException
these should be the exceptions thrown

Trying to run an exe file created from a cpp file in java

I'm trying to run an executable file created from a cpp program in java. If I double-click the exe file, it works just fine, but if I run the file using ProcessBuilder, it gets stuck for some reason, it prints most of the expected output and doesn't continue, also making the entire Java program not responding.
here's my code:
String filePath = FirstScreenController.getFile().getPath();
ProcessBuilder launcher = new ProcessBuilder("ClusteringProgram\\Release\\main.exe",filePath);
launcher.redirectErrorStream(true);
try {
/*File file = FirstScreenController.getFile();
Path newPath = Paths.get(System.getProperty("user.dir")+"\\ClusteringProgram").resolve("K12.fasta");//Moving the file to the
Files.copy(Paths.get(file.getPath()), newPath, StandardCopyOption.REPLACE_EXISTING);*/
System.out.println("Execution started");
p = launcher.start();
InputStream stderr = p.getInputStream();
InputStreamReader isr = new InputStreamReader(stderr);
BufferedReader br = new BufferedReader(isr);
String line = null;
while ((line = br.readLine()) != null) {
System.out.println(line);
}
p.waitFor();//Waiting for the process to finish running
System.out.println("Execution completed");
} catch (IOException | InterruptedException e) {e.printStackTrace();}

Close your stream. That's what's causing you to hang. I write code like this quite a bit.
while ((line = br.readLine()) != null) {
System.out.println(line);
}
br.close(); // You need this or p can hang
p.waitFor();
In addition, you called launcher.redirectStandardError(true); so you actually need all this to gather both stdout and stderr together: The whole rest of this answer is wrong. I don't know what is causing the deadlock. I'm leaving the large code fragment here in case it's some strange library bug and it turns out that the dual-thread reading technique is required to work around it.
final object lock = new object();
InputStream stdout = p.getInputStream();
InputStreamReader isr = new InputStreamReader(stdout);
BufferedReader br = new BufferedReader(isr);
final InputStream stderr = p.getErrorStream();
one = new Thread() {
public void run() {
InputStreamReader isr2 = new InputStreamReader(stderr);
BufferedReader br2 = new BufferedReader(isr2);
while ((line2 = br2.readLine()) != null) {
synchronized(lock) {
System.out.println(line2);
}
}
br2.close(); // you need this or p can hang
}
};
one.start();
while ((line = br.readLine()) != null) {
synchronized(lock) {
System.out.println(line);
}
}
br.close(); // You need this or p can hang
for (;;) {
try {
one.join();
break;
} catch (InterruptedException v) {
/* if there's something that might want the main thread's attention handle it here */
}
}
p.waitFor();

Run system commands in java

I am a little confused about why I can't run the following command ls -l
If I run ls or pwd it works fine.
Am I missing something?
ProcessBuilder pb = new ProcessBuilder("ls -l");
pb.redirectErrorStream(true);
Process process = pb.start();
InputStream is = process.getInputStream();
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
String line;
while ( (line = br.readLine()) != null) {
System.out.println(line);
}
br.close();
One more question: How can I run multiple system commands concurrently? Using while loop or for loops will run the command one by one. Any advice?
Thanks in advance.

Change:
new ProcessBuilder("ls -l");
To:
new ProcessBuilder("ls", "-l");

String[] st = {"ls", "bin"};
ProcessBuilder pb = new ProcessBuilder(st);
pb.redirectErrorStream(true);
Process process = pb.start();
InputStream is = process.getInputStream();
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
String line;
while ( (line = br.readLine()) != null) {
System.out.println(line);
}
br.close();

Using while loop or for loops will run the command one by one.
Only when you are doing the whole start-then-read-stdout business for each one of them, one-by-one. The processes are indeed run in parallel, it's just the reading part that's stopping you from running them concurrently. All you need to do is breaking the start and read into two parts:
Stream.of(Arrays.asList("ls", "-l"),
Arrays.asList("python", "-h"),
Arrays.asList("df"))
.map(cmd->{
// Create a process for each command, but don't read the output
try {
return new AbstractMap.SimpleImmutableEntry<>(cmd,
new ProcessBuilder(cmd)
.redirectErrorStream(true)
.start().getInputStream());
} catch (IOException e) {
e.printStackTrace();
return null;
}
})
.filter(p->p!=null)
.parallel()
.forEach(in->{
// Read and print STDOUT for each process.
try (BufferedReader reader = new BufferedReader(new InputStreamReader(in.getValue()))){
String line;
while ((line = reader.readLine()) != null) {
System.out.printf("%20s: %s\n", in.getKey(), line);
}
} catch (IOException e) {
e.printStackTrace();
}
});
The parallel() call is making the output really hard to read, it's there only to demonstrate that the processes are really running concurrently.

Shell command not executed

I have a weird problem when trying to execute a shell command from within a java program. Since there exist thousands of websites that explain how to do it I used the following recommended code:
public String executeShellCommand (String command)
{
try
{
StringBuffer sb = new StringBuffer();
String line = "";
Process p = Runtime.getRuntime().exec(command);
BufferedReader reader = new BufferedReader(new InputStreamReader(
p.getInputStream()));
while ((line = reader.readLine()) != null)
sb.append(line + "\n");
p.waitFor();
return sb.toString();
}
catch (Exception e)
{
e.printStackTrace();
}
return null;
}
Acutally, when I try to execute for instance ls -aF is works fine and I get some output as a result. Therefore I'm pretty sure that the above code is, in principal, correct. However, I got another program I'd like to run and that produces a file as an output. I would like to execute it the above way but it never is executed and no output file is generated. Also I do not get any error, warnings or whatsoever in java. When copy and pasting the actual command argument string into the console the execution of the programm/command directly in the shell works fine and the output file is generated. So the command I pass to the method is also correct.
Are there additional things I need to pay attention to when trying to execute a shell command from within java?
UPDATE: I modified my code according to the suggestions. However, it is still hanging:
public String executeShellCommand(List<String> command, String logfile, boolean waitForProcess) { try {
ProcessBuilder pb = new ProcessBuilder(command);
System.out.println("pb.toString() = " + pb.toString());
Process p = pb.start();
System.out.println("2");
BufferedReader err = new BufferedReader(new InputStreamReader(p.getErrorStream()));
BufferedReader out = new BufferedReader(new InputStreamReader(p.getInputStream()));
System.out.println("3");
StringBuilder errSb = new StringBuilder();
StringBuilder outSb = new StringBuilder();
String line;
System.out.println("4");
while ((line = err.readLine()) != null) { // <--- code hangs here
errSb.append(line + "\n");
System.out.println("errSb = " + errSb.toString());
}
System.out.println("4a");
while ((line = out.readLine()) != null) {
outSb.append(line + "\n");
System.out.println("outSb = " + outSb.toString());
}
System.out.println("5");
if(waitForProcess) {
System.out.println("Wait for process");
p.waitFor();
} else {
System.out.println("Sleep 5000");
Thread.sleep(5000);
}
System.out.println("6");
//Log result to file
if(logfile != null) {
OutputStreamWriter outWriter = new OutputStreamWriter(new FileOutputStream(logfile));
outWriter.write(errSb.toString());
outWriter.close();
}
return errSb.toString();
} catch(Exception e) { e.printStackTrace(); } return null; }

This will block if your command writes too many characters to stderr. Like for sdtout, Java redirect stderr through a pipe, and if you do not read the pipe, it can fill up and block (size of the pipe is probably less than 256 bytes). To avoid that, you need to read from the Process.getErrorStream(), preferable from another thread as the main thread is busy reading from the Process.getInputStream().
A simpler way to avoid that is to use the ProcessBuilder class instead of Runtime.exec() and ProcessBuilder.redirectErrorStream(true) so that both stdout and stderr are merged into the Process.getInputStream()

As per Process javadoc :
Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block, or even deadlock.
You are calling p.waitFor(). If we carefully read the waitFor() documentation:
Causes the current thread to wait, if necessary, until the process represented by this Process object has terminated.
You are waiting for a process which hangs, because its error stream and output stream are never read.
What you should do, is to read these streams:
p.start();
BufferedReader err= new BufferedReader(new InputStreamReader(p.getErrorStream()));
BufferedReader out = new BufferedReader(new InputStreamReader(p.getOutputStream()));
StringBuilder errSb = new StringBuilder();
StringBuilder outSb = new Stringbuilder();
String line;
while ((line = err.readLine()) != null) {
errSb.append(line);
}
while ((line = out.readLine()) != null) {
outSB.append(line);
}
int retCode = p.waitFor(); //0 for success
System.out.println(retCode);
System.err.println(errSB.toString());
You should always read the error stream when calling external programs via the Process class, else you may find yourself in this odd situation where a process hangs forever. (well until someone else -the operating system, another application, etc- kills it, more exactly).
I've also noticed that you use the Runtime.getRuntime() which is not the recommended way to run external programs, starting with java 1.5, as per javadoc:
As of 1.5, ProcessBuilder.start() is the preferred way to create a Process.
ProcessBuilder pb = new ProcessBuilder("ls" , "-aF");
Process p = pb.start();