Develop Reference

ProcessBuilder BufferedReader read() blocking

I'm using ProcessBuilder to run a command line tool we are using. During its run the tool asks 2 yes/no questions, so usually I answer 'y' twice and then press enter after each time. My problem is that the tool always finishes its run when running from cmd, but when I run it through my java code, it sometimes work and sometimes gets stuck on while ((n = op.read(buffer)) != -1) (with the same input).
Here is my code. Am I doing something wrong? What am I missing? Thanks.
List<String> processArgs = new ArrayList<>();
processArgs.add(0, "java");
processArgs.add(1, "-jar");
processArgs.add(2, JAR_PATH);
processArgs.add(3, "-put");
processArgs.addAll(args);
try
{
// run tool with put
ProcessBuilder pb = new ProcessBuilder(processArgs);
pb.directory(new File("src\\temp"));
pb.redirectErrorStream(true);
Process p = pb.start();
// write 'y' to the tool's stdin.
String answer = "y" + System.getProperty("line.separator");
// yes to first question
p.getOutputStream().write(answer.getBytes());
p.getOutputStream().flush();
// read tool's process stdout
this.op = new BufferedReader(new InputStreamReader(p.getInputStream()));
StringWriter sw = new StringWriter();
int n = 0;
boolean answered = false;
char[] buffer = new char[BUFFER_SIZE];
while ((n = op.read(buffer)) != -1)
{
sw.write(buffer, 0, n);
if (sw.toString().contains("second question") && !answered)
{
// yes to second question
p.getOutputStream().write(answer.getBytes());
p.getOutputStream().flush();
answered = true;
}
}
stdout = sw.toString();
exitCode = p.waitFor();
}
catch (IOException | InterruptedException e)
{
throw new ToolException("process had an exception:\n" + e.getMessage());
}
UPDATE:
I changed my code and added pb.redirectErrorStream(true), but now the process still gets blocked on op.read(buffer). When I debug it seems that it's stuck on the second question, even though I wrote 'y' twice to the output stream. Am I using getOutputStream() incorrectly?
SECOND UPDATE:
The second question didn't get the second 'y' as an answer, and it caused the process to wait for input. I changed the code so i will show the proper way to insert input to the subprocess outputstream.

The output and error stream are buffered. When the buffer fills up the program stops waiting for you to read it. However you only read the output first so if the error stream fills you have a deadlock.
A simple solution is to redirect the error to the output so you have only one stream to read. i.e.
pb.redirectErrorStream(true);
As per the documentation https://docs.oracle.com/javase/8/docs/api/java/lang/ProcessBuilder.html
Alternatives include; writing error to a file, or reading it in another thread.

Shell command not executed

I have a weird problem when trying to execute a shell command from within a java program. Since there exist thousands of websites that explain how to do it I used the following recommended code:
public String executeShellCommand (String command)
{
try
{
StringBuffer sb = new StringBuffer();
String line = "";
Process p = Runtime.getRuntime().exec(command);
BufferedReader reader = new BufferedReader(new InputStreamReader(
p.getInputStream()));
while ((line = reader.readLine()) != null)
sb.append(line + "\n");
p.waitFor();
return sb.toString();
}
catch (Exception e)
{
e.printStackTrace();
}
return null;
}
Acutally, when I try to execute for instance ls -aF is works fine and I get some output as a result. Therefore I'm pretty sure that the above code is, in principal, correct. However, I got another program I'd like to run and that produces a file as an output. I would like to execute it the above way but it never is executed and no output file is generated. Also I do not get any error, warnings or whatsoever in java. When copy and pasting the actual command argument string into the console the execution of the programm/command directly in the shell works fine and the output file is generated. So the command I pass to the method is also correct.
Are there additional things I need to pay attention to when trying to execute a shell command from within java?
UPDATE: I modified my code according to the suggestions. However, it is still hanging:
public String executeShellCommand(List<String> command, String logfile, boolean waitForProcess) { try {
ProcessBuilder pb = new ProcessBuilder(command);
System.out.println("pb.toString() = " + pb.toString());
Process p = pb.start();
System.out.println("2");
BufferedReader err = new BufferedReader(new InputStreamReader(p.getErrorStream()));
BufferedReader out = new BufferedReader(new InputStreamReader(p.getInputStream()));
System.out.println("3");
StringBuilder errSb = new StringBuilder();
StringBuilder outSb = new StringBuilder();
String line;
System.out.println("4");
while ((line = err.readLine()) != null) { // <--- code hangs here
errSb.append(line + "\n");
System.out.println("errSb = " + errSb.toString());
}
System.out.println("4a");
while ((line = out.readLine()) != null) {
outSb.append(line + "\n");
System.out.println("outSb = " + outSb.toString());
}
System.out.println("5");
if(waitForProcess) {
System.out.println("Wait for process");
p.waitFor();
} else {
System.out.println("Sleep 5000");
Thread.sleep(5000);
}
System.out.println("6");
//Log result to file
if(logfile != null) {
OutputStreamWriter outWriter = new OutputStreamWriter(new FileOutputStream(logfile));
outWriter.write(errSb.toString());
outWriter.close();
}
return errSb.toString();
} catch(Exception e) { e.printStackTrace(); } return null; }

This will block if your command writes too many characters to stderr. Like for sdtout, Java redirect stderr through a pipe, and if you do not read the pipe, it can fill up and block (size of the pipe is probably less than 256 bytes). To avoid that, you need to read from the Process.getErrorStream(), preferable from another thread as the main thread is busy reading from the Process.getInputStream().
A simpler way to avoid that is to use the ProcessBuilder class instead of Runtime.exec() and ProcessBuilder.redirectErrorStream(true) so that both stdout and stderr are merged into the Process.getInputStream()

As per Process javadoc :
Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block, or even deadlock.
You are calling p.waitFor(). If we carefully read the waitFor() documentation:
Causes the current thread to wait, if necessary, until the process represented by this Process object has terminated.
You are waiting for a process which hangs, because its error stream and output stream are never read.
What you should do, is to read these streams:
p.start();
BufferedReader err= new BufferedReader(new InputStreamReader(p.getErrorStream()));
BufferedReader out = new BufferedReader(new InputStreamReader(p.getOutputStream()));
StringBuilder errSb = new StringBuilder();
StringBuilder outSb = new Stringbuilder();
String line;
while ((line = err.readLine()) != null) {
errSb.append(line);
}
while ((line = out.readLine()) != null) {
outSB.append(line);
}
int retCode = p.waitFor(); //0 for success
System.out.println(retCode);
System.err.println(errSB.toString());
You should always read the error stream when calling external programs via the Process class, else you may find yourself in this odd situation where a process hangs forever. (well until someone else -the operating system, another application, etc- kills it, more exactly).
I've also noticed that you use the Runtime.getRuntime() which is not the recommended way to run external programs, starting with java 1.5, as per javadoc:
As of 1.5, ProcessBuilder.start() is the preferred way to create a Process.
ProcessBuilder pb = new ProcessBuilder("ls" , "-aF");
Process p = pb.start();

Java: is there a way to run a system command and print the output during execution?

I have a python script and it takes a long time to finish. I would like to run it from Java, but also output the script's output while it is executing, so that I can tell if it is properly running.
I've searched and only found examples where we output the output after the system command has finished, rather than during its execution.
Any way to do it while the script is running?
Here's what I have
public void doSomething() throws IOException {
String[] callAndArgs = {"python", "/hi.py"};
Process p = Runtime.getRuntime().exec(callAndArgs);
BufferedReader stdInput = new BufferedReader(new InputStreamReader(p.getInputStream()));
BufferedReader stdError = new BufferedReader(new InputStreamReader(p.getErrorStream()));
String s;
while ((s = stdInput.readLine()) != null) {
System.out.println(s);
}
while ((s = stdError.readLine()) != null) {
System.out.println(s);
}
}

i managed to get it working like this (Note it requires java7):
package test;
import java.lang.ProcessBuilder.Redirect;
public class Test {
public static void main(String... args) throws Exception {
ProcessBuilder pb = new ProcessBuilder("python","/home/foobar/Programming/test/src/test/test.py");
pb.redirectOutput(Redirect.INHERIT);
Process p = pb.start();
p.waitFor();
}
}
python (note i flush on python to make it work using sys.stdout.flush())
import time,sys
c =0
while c<=50:
time.sleep(1)
print("----")
c = c +1
sys.stdout.flush()
Note if you don't want to flush in a loop you can use this:
ProcessBuilder pb = new ProcessBuilder("python","-u","/home/foobar/Programming/NetBeansProjects/test/src/test/test.py");
Redirect.INHERIT
Indicates that subprocess I/O source or destination will be the same as those of the current process. This is the normal behavior of most operating system command interpreters (shells).

I've searched and only found examples where we output the output after
the system command has finished, rather than during its execution.
That's weird, because your example should be dumping the output as the command is executing.
Instead of using BufferedReader, you could try reading directly from the InputStream instead as the required conditions for readLine might not be being met until after the process exits.
I'd also recommend that you use a ProcessBuilder over Process directly, as, apart from anything else, it allows you to redirect the output from the error stream into the input stream, allowing you to read just one stream instead of two...
This might also be an issue with Python and how it flushes it output buffers...
For example, rather then waiting for the BufferedReader to decide when to return, try printing each character from the stream as it occurs/is reported
ProcessBuilder pb = new ProcessBuilder("test.py");
pb.redirectError();
Process p = pb.start();
InputStream is = null;
try {
is = p.getInputStream();
int in = -1;
while ((in = is.read()) != -1) {
System.out.print((char)in);
}
} finally {
try {
is.close();
} catch (Exception e) {
}
}
Update
Doing a little reading, Python seems to be buffering its out out before sending it to the stdout. I don't think you can fix this on the a Java side, but need to alter either the way Python is run or the script works.
See How to flush output of Python print? for more details

I'm suspecting that you are writing to stderr, which you can't see because you are blocking on stdin. Use a ProcessBuilder instead of doing exec. This way, you can redirect stderr and stdin into a single stream.
Here is an example:
import java.io.*;
public class Test {
public static void main(String... args) throws IOException {
ProcessBuilder pb =
new ProcessBuilder("test.py");
pb.redirectErrorStream(true);
Process proc = pb.start();
Reader reader = new InputStreamReader(proc.getInputStream());
BufferedReader bf = new BufferedReader(reader);
String s;
while ((s = bf.readLine()) != null) {
System.out.println(s);
}
}
}
Alternatively you can spawn threads to read from stdin/stderr respectively.
Another thing to look for is output buffering by python. You can see if this is the cause by doing:
import sys
sys.stdout.flush()
after you write to stdout

Don't use #readLine as the conditional in your while loop. Instead wrap your inputStream in a scanner and use #hasNextLine()
Scanner in = new Scanner(p.getInputStream());
while (in.hasNextLine()) {
System.out.println(in.nextLine());
}

BufferedReader.readline() hangs

I'm trying to run /usr/bin/perl -e 'for(my $i=0;$i<1000;$i++){print 1x1000;print STDERR 2x1000;}' (which works in terminal) with my program.
ProcessBuilder pb = new ProcessBuilder(go); //go is the command
process = pb.start();
BufferedReader incommandbuf = new BufferedReader(new InputStreamReader(process.getInputStream()),1024*1000);
BufferedReader errcommandbuf = new BufferedReader(new InputStreamReader(process.getErrorStream()),1024*1000);
stdString = "";
while ((line = incommandbuf.readLine()) != null)
{
stdString += line + "\n";
}
String errorstrtemp = "";
while ((line = errcommandbuf.readLine()) != null)
{
errorstrtemp += line + "\n";
}
If I try to run this it hangs on while ((line = incommandbuf.readLine()) != null). The program runs if I change the command to /usr/bin/perl -e 'for(my $i=0;$i<64;$i++){print 1x1000;print STDERR 2x1000;}'. If it goes up to 65 and higher it doesn't work. At first I thought I just have to change the size of the my BufferedReaders but it didn't help. Any clue on what is causing this? I will provide any additional info if needed.
Thanks.

You are reading one stream at a time. When the other stream fills up the buffer, your Process will stop waiting for you to read it. The solution is to either read the streams in different threads or use ProcessBuilder.redirectErrorStream

How to get output from a .bat process to display in a JTextPane?

My program requires that I run a .bat file which will compile java source. This is running fine, however I am looking for a solution which will get the output (and possible errors) of compile.bat and add it to a text pane on a GUI. I have the following code, however when executed the process happens without printing anything to the pane and without any errors.
GenerationDebugWindow.main(null);
Process process = rut.exec(new String[] {file.getAbsolutePath() + "\\compile.bat"});
Scanner input = new Scanner(process.getInputStream());
InputStream is = process.getInputStream();
InputStreamReader isr = new InputStreamReader(is);
BufferedReader reader = new BufferedReader(isr);
String line;
int exit = -1;
while ((line = reader.readLine()) != null) {
// Outputs your process execution
try {
exit = process.exitValue();
GenerationDebugWindow.writeToPane(line);
System.out.println(line);
if (exit == 0) {
GenerationDebugWindow.writeToPane("Compilation Finished!");
if(new File(file + "/mod_" + WindowMain.modName.getText()).exists()){
GenerationDebugWindow.writeToPane("Compilation May Have Experienced Errors.");
}
}
} catch (IllegalThreadStateException t) {
}
}
GenerationDebugWindow
private static JTextPane outputPane;
public static void writeToPane(String i){
outputPane.setText(outputPane.getText() + i + "\r\n");
}

Use:
Runtime.getRuntime().exec( "cmd.exe /C " + file.getAbsolutePath() + "\\compile.bat" );

Reference this question: Java Process with Input/Output Stream
It's likely that the output of the process is going to the error stream. However, ProcessBuilder is a more useful class than directly using System.getRuntime().exec()
In the example below, we're telling the ProcessBuilder to redirect the error stream to the same stream as the output goes to, which simplifies the code.
ProcessBuilder builder = new ProcessBuilder("cmd.exe /C " + file.getAbsolutePath() + "\\compile.bat");
builder.redirectErrorStream(true);
builder.directory(executionDirectory); // if you want to run from a specific directory
Process process = builder.start();
Reader reader = ...;
String line = null;
while ((line = reader.readLine ()) != null) {
System.out.println ("Stdout: " + line);
}
int exitValue = process.exitValue();

My program requires that I run a .bat file which will compile java source.
Your users on *nix and OS X require that source be compiled using the JavaCompiler.
The STBC is an example of using the JavaCompiler. It is open source. It uses a JTextArea as opposed to a JTextPane to hold the source and errors, but should be trivial to adapt.