Develop Reference

process builder with redirecterrorstream hangs in readline

I would like to run a hidden script file that resides in the current location using process builder. with the following code
// System.out.println("line"+reader.readLine());
ProcessBuilder builder = new ProcessBuilder(shfile.getAbsolutePath());
builder.redirectErrorStream(true);
Process process = builder.start();
BufferedReader br = new BufferedReader(new InputStreamReader(process.getInputStream()));
String output = null;
System.out.println("out"); //===printing this
while (null != (output = br.readLine()))
{
System.out.println("in"); //not printing this
System.out.println(">>"+output);
}
int rs = process.waitFor();
but it hangs in the br.readline()..
but when I run the same script file using the following command in terminal
sh .script.sh
it executes and gives me the expected results
I looked into all the loops in the forum everyone asks to handle input stream and error stream in threads or do a redirect error stream. I have added a redirect error stream but still it hangs.
when i press ctrl+c it prints the initial lines of the output and exits.
Content of my script file
#!/bin/sh
cd /home/ats/cloudripper/lke_factory_asb_v2/lk_assets_factory_release/
sh ./LKE_run_Diablo.sh 0a0e0c3dc893
So how to handle this situation.

Process builder have special API to redirect child process input, output and error streams. See documentation
If you need both child and parent process to use same console you should use INHERIT mode redirection. An example:
public class ChildProcessOutputProxy {
public static void main(String[] args) {
ProcessBuilder builder = new ProcessBuilder("whoami");
builder.redirectOutput(Redirect.INHERIT);
builder.redirectErrorStream(true);
try {
var child = builder.start();
child.waitFor();
} catch (IOException e) {
System.err.println(e.getMessage());
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
}

Shell script exit too early due to docker pull, while running through ProcessBuilder

I have the following problem which seems to be caused by the "docker pull" in my shell script, as the pull works concurrently
#!/bin/bash
#VARIABLES
NAME="my-app"
IMAGE="my-image:latest"
#DOCKER
docker stop $NAME
docker rm $NAME
docker pull -q $IMAGE
docker run --name $NAME -d -p 1234:8080 --log-opt fluentd-address=localhost:2233 $IMAGE
Running the script through the terminal works just fine everything works as expected. But when I run it with the Java's ProcessBuilder the script exits much quicker and it seems that it skips the "docker pull" step. As i am not a Java developer and I am not very well familiar with the Language I have the feeling that is something related to the multi-concurrent nature of the docker pull command and the way how the Java Process Builder executes the shell script
The Java class that runs the shell script is this
try {
Collection<Task> tasks = taskService.getProjectTasksByProjectKey(projectId);
Task findTask = findTaskByTaskId(tasks, taskId);
if (findTask.getTaskId() != null) {
ProcessBuilder pb = new ProcessBuilder(findTask.getCmdPath());
Process process = pb.start();
String output;
try (InputStream in = process.getInputStream();
InputStream err = process.getErrorStream();
OutputStream closeOnly = process.getOutputStream()) {
while (process.isAlive()) {
long skipped = in.skip(in.available())
+ err.skip(err.available());
if(skipped == 0L) {
process.waitFor(5L, TimeUnit.MILLISECONDS);
}
}
output = loadStream(in);
} finally {
process.destroy();
}
// String error = loadStream(process.getErrorStream());
// int rc = process.waitFor();
// log.debug("exit code ->>> " + rc);
// StringBuilder output = new StringBuilder();
// BufferedReader reader = new BufferedReader(
// new InputStreamReader(process.getInputStream()));
//
// String line;
//
// while ((line = reader.readLine()) != null) {
// output.append(line + "\n");
// }
//
// int exitVal = process.waitFor();
// if (exitVal == 0) {
// System.out.println(output);
//
// return output.toString();
// } else {
// //abnormal...
// }
return output;
}
else {
throw new InvalidTaskModelException(taskId);
}
} catch (InvalidModelException e) {
throw new InvalidModelException(projectId);
} catch (IOException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
private static String loadStream(InputStream s) throws Exception
{
BufferedReader br = new BufferedReader(new InputStreamReader(s));
StringBuilder sb = new StringBuilder();
String line;
while((line=br.readLine()) != null)
sb.append(line).append("\n");
return sb.toString();
}
The commented lines are different ways I tried to do it.
If anyone encountered a similar problem any help would be much appreciated!

It is good that you already take care for the processes' STDOUT and STDIN. But rather than skipping copy them to System.out so you can see what is going on. I suspect something is not going as per your expectations.
Looking at the bash script you posted and the fact you are trying to run several processes: Is it possible your java code is running the bash script line by line? Be aware your java program it is not a BASH interpreter, so e.g. variable substitution should not work.

why you can not run each command in thread and joins them so that unless therad 1 is not completed . next thread can not start .
also please add command to verify image is downloaded successfully
docker pull -q $IMAGE
docker images | grep $IMAGE
docker run --name $NAME -d -p 1234:8080 --log-opt fluentd-address=localhost:2233 $IMAGE
i am guessing there are 2 possiblities are here
Check local directory persmission if possible give 755 permission to
it .
Java process itself is not able to execute docker command due
to permission issue, run process as sudo user.

How to run .sh file, using process builder?

I already create the .sh file, and the inside is:
sudo iptables --flush
sudo iptables -A INPUT -m mac --mac-source 00:00:00:00:00:00 -j DROP
It works normally when I run it on the terminal, but when I use processbuilder, it didn't do anything. No error, but didn't happen anything, this is the code on my java:
Process pb = new ProcessBuilder("/bin/bash","/my/file.sh").start();
I already looking for the answer, but I still failed to run the .sh file, even I do the same thing with people that already done it.
Sorry if this is a bad question, thank you.

Are you sure that the bash is not run? Do you checked the Process object returned by the startmethod? You can get the output value, the output stream, etc. from this objects.

Check your streams and exitvalue for errors... sudo is probably the problem here.
Not necessarily the best code but it gets the job done. Executes a process, takes the process.streams and prints them to System.out. Might helpt to find out what the issue actually is atlest.
ProcessBuilder pb = new ProcessBuilder(args);
pb.redirectErrorStream(true);
final Process proc = pb.start();
final StringBuilder builder = new StringBuilder("Process output");
final Thread logThread = new Thread() {
#Override
public void run() {
InputStream is = proc.getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(is));
try {
String line;
do {
line = reader.readLine();
builder.append("");
builder.append(line == null ? "" : line);
builder.append("<br/>");
} while(line != null);
} catch (IOException e) {
builder.append("Exception! ").append(e.getMessage());
} finally {
try {
reader.close();
} catch (IOException e) {
builder.append("Exception! ").append(e.getMessage());
}
}
}
};
logThread.start();
int retVal = proc.waitFor();
System.out.println(builder.toString());

From Java API Runtime : http://docs.oracle.com/javase/7/docs/api/java/lang/Runtime.html
// Java runtime
Runtime runtime = Runtime.getRuntime();
// Command
String[] command = {"/bin/bash", "/my/file.sh"};
// Process
Process process = runtime.exec(command);
Also you should be careful with sudo commands that may ask for root password.

Unable to perform any action before Process.Runtime.exec statement line

my very first post here after long time of searching but yet get an answer regarding this issue, please help me in this issue.
I am using Netbean 6.9.1 to build a Java application which massive call to few different external program, therefore I used process and runtime function to call for external program.
The whole application process is separated into few stages and I wish to inform the user till which stage the application is currently running by updating the GUI textarea, the code is showed as below:
public void executeCommand (String cmd, File path)
{
try
{
****areaOutput.setText("Executing audio decoding, please wait till process is done\n");****
btnTranscribe.setEnabled(false);
areaOutput.setEditable(false);
areaOutput.setEnabled(false);
Runtime rt = Runtime.getRuntime();
Process proc = rt.exec(cmd , null, path);
InputStream stderr = proc.getErrorStream();
InputStreamReader isr = new InputStreamReader(stderr);
BufferedReader br = new BufferedReader(isr);
String line = null;
System.out.println("<ERROR>");
while ( (line = br.readLine()) != null)
System.out.println(line);
System.out.println("</ERROR>");
int exitVal = proc.waitFor();
System.out.println("Process exitValue: " + exitVal);
areaOutput.append("\n\nConversion is done, processing with features extraction....");
} catch (Throwable t)
{
t.printStackTrace();
}
}
As showed in the code above, I wish to set the Textarea and disable some button before executing the command, but when the application ran, all of these line seems unable to work and nothing is changed at the application itself till the command is finish executed, any solution to run the pre-command code execute first before the .exec() start to run?
I appreciate your great help and advice regarding this issue.
Best regards,
Striky
P/S:
hi there, I have make a Thread class for this CmdExec in order to execute cmd in different thread:
public class CmdExec extends Thread
{
private String cmd;
private File path;
public CmdExec() {
}
public CmdExec(String cmd, File path) {
this.cmd = cmd;
this.path = path;
}
public void run(){
try
{
Runtime rt = Runtime.getRuntime();
Process proc = rt.exec(cmd , null, path);
InputStream stderr = proc.getErrorStream();
InputStreamReader isr = new InputStreamReader(stderr);
BufferedReader br = new BufferedReader(isr);
String line = null;
System.out.println("<ERROR>");
while ( (line = br.readLine()) != null)
System.out.println(line);
System.out.println("</ERROR>");
int exitVal = proc.waitFor();
System.out.println("Process exitValue: " + exitVal);
} catch (Throwable t)
{
t.printStackTrace();
}
}
}
and in order to call to this class,
CmdExec tryDemo = new CmdExec();
tryDemo = new CmdExec(strSegment, fSegment);
tryDemo.run();
is used to start the thread, but I failed to put SwingUtilities.invokeLater in any part of these process, it simply won't run the tryDemo.run() because it is void...
Also, may I know so far am I doing right?? Very thank you for your kind help regarding this issue
P/S 2: I have just added another runnable code (so threads for process executing, runnable to GUI update) for GUI update command as below:
Runnable doWorkRunnable = new Runnable() {
public void run() {
System.out.println("hello world");
btnTranscribe.setEnabled(false);
areaOutput.setEditable(false);
areaOutput.setEnabled(false);
areaOutput.setText("Performing segmentation, please wait till process is done\n"); }
};
and I used SwingUtilies.invokeLater before the execution of process as below:
SwingUtilities.invokeLater(doWorkRunnable);
Runtime rt = Runtime.getRuntime();
Process proc = rt.exec(cmd , null, path);
But all these failed, am I get the wrong sequence for the GUI and process thread coordination?

you are executing this work on the EDT (the thread which updates the gui). so, the gui cannot update until all this work finishes. what you want to do is run a separate thread which does all the work and periodically calls SwingUtilities.invokeLater with a status update.

Try to put sleep before execute method. To verify what is happening.

Calling a shell script from java hangs

So I'm trying to execute a shell script which produces a lot of output(in 100s of MBs) from a Java file.
This hangs the process and never completes.
However, within the shell script, if I redirect the output of the script to some log file or /dev/null Java file executes and completes in a jiffy.
Is it because of amount of data that the Java program never completes?
If so, is there any documentation as such? or is there any limit on the amount of data(documented)?
Here's how you can simulate this scenario.
Java file will look like:
import java.io.InputStream;
public class LotOfOutput {
public static void main(String[] args) {
String cmd = "sh a-script-which-outputs-huuggee-data.sh";
try {
ProcessBuilder pb = new ProcessBuilder("bash", "-c", cmd);
pb.redirectErrorStream(true);
Process shell = pb.start();
InputStream shellIn = shell.getInputStream();
int shellExitStatus = shell.waitFor();
System.out.println(shellExitStatus);
shellIn.close();
} catch (Exception ignoreMe) {
}
}
}
The script 'a-script-which-outputs-huuggee-data.sh' may look like:
#!/bin/sh
# Toggle the line below
exec 3>&1 > /dev/null 2>&1
count=1
while [ $count -le 1000 ]
do
cat some-big-file
((count++))
done
echo
echo Yes I m done
Free beer for the right answer. :)

It's because you're not reading from the Process' output.
As per the class' Javadocs, if you don't do this then you may end up with a deadlock; the process fills its IO buffer and waits for the "shell" (or listening process) to read from it and empty it. Meanwhile your process, which should be doing this, is blocking waiting for the process to exit.
You'll want to call getInputStream() and read from that reliably (perhaps from another thread) to stop the process blocking.
Also take a look at Five Java Process Pitfalls and When Runtime.exec() Won't - both informative articles about common problems with Process.

You're never reading the input stream, so it's probably blocking because the input buffer is full.

The input/output buffer have a limited size (depending on the operating system). If I remember correctly this wasn't big or Windows XP at least. Try creating a thread that reads the InputStream as fast as possible.
Something along these lines:
class StdInWorker
implements Worker
{
private BufferedReader br;
private boolean run = true;
private int linesRead = 0;
private StdInWorker (Process prcs)
{
this.br = new BufferedReader(
new InputStreamReader(prcs.getInputStream()));
}
public synchronized void run ()
{
String in;
try {
while (this.run) {
while ((in = this.br.readLine()) != null) {
this.buffer.add(in);
linesRead++;
}
Thread.sleep(50);
}
}
catch (IOException ioe) {
ioe.printStackTrace();
}
catch (InterruptedException ie) {}
}
}
}