Develop Reference

Shell script exit too early due to docker pull, while running through ProcessBuilder

I have the following problem which seems to be caused by the "docker pull" in my shell script, as the pull works concurrently
#!/bin/bash
#VARIABLES
NAME="my-app"
IMAGE="my-image:latest"
#DOCKER
docker stop $NAME
docker rm $NAME
docker pull -q $IMAGE
docker run --name $NAME -d -p 1234:8080 --log-opt fluentd-address=localhost:2233 $IMAGE
Running the script through the terminal works just fine everything works as expected. But when I run it with the Java's ProcessBuilder the script exits much quicker and it seems that it skips the "docker pull" step. As i am not a Java developer and I am not very well familiar with the Language I have the feeling that is something related to the multi-concurrent nature of the docker pull command and the way how the Java Process Builder executes the shell script
The Java class that runs the shell script is this
try {
Collection<Task> tasks = taskService.getProjectTasksByProjectKey(projectId);
Task findTask = findTaskByTaskId(tasks, taskId);
if (findTask.getTaskId() != null) {
ProcessBuilder pb = new ProcessBuilder(findTask.getCmdPath());
Process process = pb.start();
String output;
try (InputStream in = process.getInputStream();
InputStream err = process.getErrorStream();
OutputStream closeOnly = process.getOutputStream()) {
while (process.isAlive()) {
long skipped = in.skip(in.available())
+ err.skip(err.available());
if(skipped == 0L) {
process.waitFor(5L, TimeUnit.MILLISECONDS);
}
}
output = loadStream(in);
} finally {
process.destroy();
}
// String error = loadStream(process.getErrorStream());
// int rc = process.waitFor();
// log.debug("exit code ->>> " + rc);
// StringBuilder output = new StringBuilder();
// BufferedReader reader = new BufferedReader(
// new InputStreamReader(process.getInputStream()));
//
// String line;
//
// while ((line = reader.readLine()) != null) {
// output.append(line + "\n");
// }
//
// int exitVal = process.waitFor();
// if (exitVal == 0) {
// System.out.println(output);
//
// return output.toString();
// } else {
// //abnormal...
// }
return output;
}
else {
throw new InvalidTaskModelException(taskId);
}
} catch (InvalidModelException e) {
throw new InvalidModelException(projectId);
} catch (IOException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
private static String loadStream(InputStream s) throws Exception
{
BufferedReader br = new BufferedReader(new InputStreamReader(s));
StringBuilder sb = new StringBuilder();
String line;
while((line=br.readLine()) != null)
sb.append(line).append("\n");
return sb.toString();
}
The commented lines are different ways I tried to do it.
If anyone encountered a similar problem any help would be much appreciated!

It is good that you already take care for the processes' STDOUT and STDIN. But rather than skipping copy them to System.out so you can see what is going on. I suspect something is not going as per your expectations.
Looking at the bash script you posted and the fact you are trying to run several processes: Is it possible your java code is running the bash script line by line? Be aware your java program it is not a BASH interpreter, so e.g. variable substitution should not work.

why you can not run each command in thread and joins them so that unless therad 1 is not completed . next thread can not start .
also please add command to verify image is downloaded successfully
docker pull -q $IMAGE
docker images | grep $IMAGE
docker run --name $NAME -d -p 1234:8080 --log-opt fluentd-address=localhost:2233 $IMAGE
i am guessing there are 2 possiblities are here
Check local directory persmission if possible give 755 permission to
it .
Java process itself is not able to execute docker command due
to permission issue, run process as sudo user.

How to access file path with spaces from command prompt in Java

I am trying to run a batch file from a Java program.
For instance: I have a batch "abc.bat" in a folder in "Program Files".
I want to execute this batch from my Java Program. I am using CommandLine class, Commons-exec jar.
CommandLine cmdLine = CommandLine.parse("cmd");
cmdLine.addArgument("/c start \"\" \"C:\\Program Files\\abc.bat\"");
DefaultExecutor exec = new DefaultExecutor();
Process p = Runtime.getRuntime().exec(cmdLine.toString());
exec.execute(cmdLine);
The above code throws an error saying "Windows cant find the file. Make sure you typed the name correctly, and try again". And, that is because of the spaces in the path.
So, I tried the answer provided here by #brso05 and that works. But I want it to be in a Future Class. Please find my code below and help me fix it.
final CommandLine cmdLine = CommandLine.parse("cmd.exe");
cmdLine.addArgument("/c");
cmdLine.addArgument("start");
cmdLine.addArgument("\""+ batchFileExecute.getParent().toString() + "\"");
ExecutorService es = Executors.newFixedThreadPool(1);
Future<?> future = es.submit(new Runnable() {
public void run() {
DefaultExecutor exec = new DefaultExecutor();
try {
Process p = Runtime.getRuntime().exec(cmdLine.toString());
exec.execute(cmdLine);
System.out.println(p.waitFor());
}
catch (IOException e)
{
e.printStackTrace();
}
catch (InterruptedException e)
{
e.printStackTrace();
}
}
});
String thread_status = null;
try
{
thread_status = future.get().toString();
System.out.println(thread_status+" completed the execution");
}
catch (NullPointerException e)
{
System.out.println("The execution of the received project is complete.");
// In here I want to do some processing again.
}
The code I mentioned works but it doesnt work if my batch file has a spaces in the path. Can you help me fix this?
Becuase the snippet you've given works but then I cant put it into Future. It doesnt work in the desired manner.
Thanks in advance!

This is an alternative way:
Runtime rt = Runtime.getRuntime();
rt.exec("cmd.exe /c start \"\" \"C:\\Program Files\\abc.bat\"");

Have you tried with single quotes? According to this, it should work.

I had the same filenames with spaces issue while using ImageMagick. Here is the code to solve the issue:
String imageOutput = null;
ByteArrayOutputStream identifyStdout = new ByteArrayOutputStream();
ByteArrayOutputStream identifyStderr = new ByteArrayOutputStream();
try
{
DefaultExecutor identifyExecutor = new DefaultExecutor();
// End the process if it exceeds 60 seconds
ExecuteWatchdog identifyWatchdog = new ExecuteWatchdog(60000);
identifyExecutor.setWatchdog(identifyWatchdog);
PumpStreamHandler identifyPsh = new PumpStreamHandler(identifyStdout, identifyStderr);
identifyExecutor.setStreamHandler(identifyPsh);
identifyExecutor.setExitValue(0); //0 is success
CommandLine identifyCommandline = new CommandLine("identify");
identifyCommandline.addArgument(destFile.getAbsolutePath(), false);
DefaultExecuteResultHandler identifyResultHandler = new DefaultExecuteResultHandler();
identifyExecutor.execute(identifyCommandline, identifyResultHandler);
identifyResultHandler.waitFor();
if (identifyResultHandler.getExitValue() != 0)
{
String output = identifyStdout.toString();
_logger.debug("Standard Out = " + output);
_logger.debug("Standard Err = " + identifyStderr.toString());
String msg = "ImageMagick overlay encountered an error. ImageMagick returned a value of " + identifyResultHandler.getExitValue();
throw new Exception(msg);
}
else
{
imageOutput = identifyStdout.toString();
_logger.debug("Standard Out = " + imageOutput);
identifyStdout.close();
identifyStderr.close();
}
}
catch(Exception e)
{
_logger.debug("Error: " + e.getLocalizedMessage(), e);
}
finally
{
identifyStdout.close();
identifyStderr.close();
}
The important part here is:
identifyCommandline.addArgument(destFile.getAbsolutePath(), false);
This line allows a filepath with spaces to process correctly.

When using the CommandLine class addArgument method without defining the boolean handleQuoting, it will set handleQuoting to true for you, basically adding quotes to the argument. This is the method invoking:
public CommandLine addArgument(String argument) {
return this.addArgument(argument, true);
}
public CommandLine addArgument(String argument, boolean handleQuoting) {
if (argument == null) {
return this;
} else {
if (handleQuoting) {
StringUtils.quoteArgument(argument);
}
this.arguments.add(new Argument(argument, handleQuoting));
return this;
}
}
Changing my method from:
CommandLine cmd = new CommandLine("pdfinfo");
cmd.addArgument("-box");
cmd.addArgument(pdfFile.getAbsolutePath());
To:
CommandLine cmd = new CommandLine("pdfinfo");
cmd.addArgument("-box");
cmd.addArgument(pdfFile.getAbsolutePath(), false); <-- change here
Solved the issue for me. No quotes were added and the CommandLine was able to find the file.

Java Powershell CreateProcess error=2, The system cannot find the file specified

I am executing powershell commands in java and I have written two programs, however the strange part is one works fine and the other throws the error. The code that throws the error is as shown
I have tried the following
1) Spcifying the fully specified path of powershell
2) My path variable has the following - "C:\WINDOWS\system32\WindowsPowerShell\v1.0"
I know I might be doing something trivial but its been a day and I am unable to figure out what the issue might be
import java.io.IOException;
public class FileCount {
public static void main(String[] args) {
Process flCntProcess = null;
try {
String test = "C:\\WINDOWS\\system32\\windowspowershell\\v1.0\\powershell.exe -Command \"& { Get-ChildItem C:\\test -Recurse -force | Measure-Object }\"";
System.out.println("Powershell command : " + test);
ProcessBuilder builder = new ProcessBuilder(test);
builder.redirectErrorStream(true);
flCntProcess = builder.start();
// FILE COUNT OUTPUT STREAM PROCESSING
NotifyThreadComplete outputThread = new ProcessHandler(flCntProcess.getInputStream(),"OUTPUT");
outputThread.addListener(new ThreadCompleteListener() {
#Override
public void notifyCompletion(Thread t, long startTm, boolean didErrorOut, String noOfLines) {
System.out.println("Completed Output Stream Processing");
System.out.println("Printing values");
System.out.println("No of Lines : " + noOfLines);
System.out.println("Did Error out : " + didErrorOut);
if(didErrorOut) {
System.out.println("Do not continue with processing");
} else {
System.out.println("Continue with processing");
}
}
});
System.out.println("Starting output thread ");
outputThread.start();
} catch (Exception e) {
System.err.println("Exception while counting files using Powershell Command" + e.getMessage());
} finally {
if(flCntProcess != null && flCntProcess.getOutputStream() != null) {
try {
flCntProcess.getOutputStream().close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
}

Error code indicates the file to execute can't be found. Try splitting up the program from its arguments:
String ps = "C:\\WINDOWS\\system32\\windowspowershell\\v1.0\\powershell.exe";
String args = "-Command \"& { Get-ChildItem C:\\test -Recurse -force | Measure-Object}\"";
ProcessBuilder builder = new ProcessBuilder(ps, args);

The constructor of ProcessBuilder does not accept a single String containing a cli invocation, but an array of Strings containing in order :
the program to be executed
its arguments
See the javadoc
So it interprets your whole String test as the program name, splitting it up should work :
final String psh = "C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe";
final String args = "-Command & { Get-ChildItem C:\\temp -Recurse -force | Measure-Object }";
final ProcessBuilder builder = new ProcessBuilder(psh, args);

trying to execute mysqldump from java app, multiple problems

I am writing a java app that needs to perform mysql dump, and I am using the runtime.exec, based in the when runtime.exec won't article. The code is below:
public int exectuteCommand(){
Runtime rt = Runtime.getRuntime();
logger.debug("exexuting cmd: " + showCommand());
int exit = -1;
try {
Process proc = rt.exec(cmd);
ExtProcessStreamHandler errorHandler = new ExtProcessStreamHandler(proc.getErrorStream(), "ERROR");
ExtProcessStreamHandler outHandler = new ExtProcessStreamHandler(proc.getInputStream(), "OUTPUT");
// kick it off
errorHandler.start();
outHandler.start();
exit = proc.waitFor();
} catch (IOException e) {
logger.error("ERROR!! ~~ executing command " + showCommand(), e);
e.printStackTrace();
} catch (InterruptedException e) {
logger.error("ERROR!! ~~ unexpected return for " + showCommand() + " , returned " + exit, e);
e.printStackTrace();
}
return exit;
}
1) The command that the process returns works in the shell (I'm running this on a mac). The first error I had was an inability to find the mysqldump command. That results in this error:
java.io.IOException: Cannot run program "mysqldump": error=2, No such file or directory
I resolved that by adding the complete path of the file to the command. The $PATH var shows
/usr/local/mysql/bin/mysqldump
as the complete path. How can I make sure my java app has that info?
2) when adding the complete path to the command, I get this error msg:
INFO [Thread-1] (ExtProcessStreamHandler.java:28) - external process ERROR : mysqldump: Couldn't find table: ">"
Here is the code that builds the command array:
return new String[] {MYSQLDUMP_CMD, "-u", USER_DEFAULT, "-p"+ PW_DEFAULT, TEST_DB_NAME,
">", DUMP_LOC};
again, when I copy the command passed to the java app into the shell on my mac, it works. Not sure what I'm doing wrong.
thanks in advance!

It thinks ">" is an argument intended for mysqldump. You are invoking an executable, not evaluating a shell expression. If you want to pipe your output, do it with the outHandler and errorHandler in your code.
An alternative is to invoke a shell and pass the expression you want to evaluate as an argument:
expr = new StringBuilder()
.append(MYSQLDUMP_CMD).append(' ')
.append("-u").append(USER_DEFAULT).append(' ')
.append("-p").append(PW_DEFAULT).append(' ')
.append(TEST_DB_NAME).append(' ')
.append(">").append(' ')
.append(DUMP_LOC)
.toString();
return new String[] {"/bin/bash", "-c", expr};
If your code to build the command array doesn't wrap spaced arguments in single quotes (or if the JDK doesn't do this for you), then modify the StringBuilder statement to create the wrapped quotes for you.

Below Code is worked for me
public static void backup() {
String currentDate = LocalDate.now().format(DateTimeFormatter.ofPattern("yyyy_MM_dd"));
String backupPath = String.format("%s/%s.%s", Helper.BACKUP_PATH, currentDate, "sql");
File backupFile = new File(backupPath);
if (!backupFile.exists()) {
try {
backupFile.createNewFile();
String mysqlCom=String.format("mysqldump -u%s -p%s %s",USER_NAME,PASSWORD,DB);
String[] command = new String[] { "/bin/bash", "-c",mysqlCom};
ProcessBuilder processBuilder = new ProcessBuilder(Arrays.asList(command));
processBuilder.redirectError(Redirect.INHERIT);
processBuilder.redirectOutput(Redirect.to(backupFile));
Process process = processBuilder.start();
process.waitFor();
LOGGER.info("Backup done");
} catch (IOException e1) {
e1.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
}
} else {
LOGGER.info("Database already backuped today");
}
}

Java blocks after calling runtime.exec()

First some code:
Runtime runtime = Runtime.getRuntime();
String args[] = new String[2];
// args[0] = "/bin/bash";
// args[1] = "-c";
// args[2] = "/usr/bin/rpm2cpio "+archiveFile.getCanonicalPath()+" | /bin/cpio -idmv";
args[0] = "/usr/bin/rpm2cpio";
args[1] = archiveFile.getCanonicalPath();
Process rpm2cpioProcess = runtime.exec(args, null, dir);
// System.out.println("started rpm2cpio");
String args2[] = new String[3];
args2[0] = "/bin/cpio";
args2[1] = "-idmu";
args2[2] = "--quiet";
Process cpioProcess = runtime.exec(args2, null, dir);
// System.out.println("started cpio");
InputStream fromRpm2cpio = rpm2cpioProcess.getInputStream();
new ProcessInputStreamer(rpm2cpioProcess.getErrorStream());
OutputStream fromCpio = cpioProcess.getOutputStream();
new PipedStreamer(fromRpm2cpio, fromCpio);
new ProcessInputStreamer(cpioProcess.getErrorStream());
// System.out.println("pipe created");
while(cpioProcess!=null && fromRpm2cpio!=null) {
boolean doSleep = true;
// System.out.println("waking up");
if (cpioProcess!=null) {
try {
if (cpioProcess.exitValue()==0) {
cpioProcess = null;
doSleep = false;
}
} catch(IllegalThreadStateException e) {
}
}
if (rpm2cpioProcess!=null) {
try {
if (rpm2cpioProcess.exitValue()==0) {
rpm2cpioProcess = null;
doSleep = false;
}
} catch(IllegalThreadStateException e) {
}
}
if (doSleep) {
Thread.sleep(30);
}
// System.out.println("still running");
}
I'm trying to extract the content of an rpm archive. This code works fine after multiple modifications. My first attempt was to execute the next code through Java:
/bin/bash -c '/usr/bin/rpm2cpio <archive-file> | /bin/cpio -idmv'
Which worked fine the first time I ran it (you can see it in the code commented above). The second time I ran the code it got blocked since the extracted files already existed. So I thought maybe it has to do with the piping and thus split the call into two separate processes. This didn't help much either. So I then modified the arguments of the /bin/cpio from '-idmv' to '-idmu --quiet' and now it works. Unfortunately the -u option overwrites existing files 'unconditionally' which is not really needed. My question is why does it block with -idmv and why doesn't it block with -idmu ?

It could be waiting on standard input for some inputs. Redirect your standard input and/or output to /dev/null

I'd guess that your ProcessInputStreamer and/or PipedStreamer implement Runnable or extent Thread and you're not running them anywhere.