I have the following problem which seems to be caused by the "docker pull" in my shell script, as the pull works concurrently
#!/bin/bash
#VARIABLES
NAME="my-app"
IMAGE="my-image:latest"
#DOCKER
docker stop $NAME
docker rm $NAME
docker pull -q $IMAGE
docker run --name $NAME -d -p 1234:8080 --log-opt fluentd-address=localhost:2233 $IMAGE
Running the script through the terminal works just fine everything works as expected. But when I run it with the Java's ProcessBuilder the script exits much quicker and it seems that it skips the "docker pull" step. As i am not a Java developer and I am not very well familiar with the Language I have the feeling that is something related to the multi-concurrent nature of the docker pull command and the way how the Java Process Builder executes the shell script
The Java class that runs the shell script is this
try {
Collection<Task> tasks = taskService.getProjectTasksByProjectKey(projectId);
Task findTask = findTaskByTaskId(tasks, taskId);
if (findTask.getTaskId() != null) {
ProcessBuilder pb = new ProcessBuilder(findTask.getCmdPath());
Process process = pb.start();
String output;
try (InputStream in = process.getInputStream();
InputStream err = process.getErrorStream();
OutputStream closeOnly = process.getOutputStream()) {
while (process.isAlive()) {
long skipped = in.skip(in.available())
+ err.skip(err.available());
if(skipped == 0L) {
process.waitFor(5L, TimeUnit.MILLISECONDS);
}
}
output = loadStream(in);
} finally {
process.destroy();
}
// String error = loadStream(process.getErrorStream());
// int rc = process.waitFor();
// log.debug("exit code ->>> " + rc);
// StringBuilder output = new StringBuilder();
// BufferedReader reader = new BufferedReader(
// new InputStreamReader(process.getInputStream()));
//
// String line;
//
// while ((line = reader.readLine()) != null) {
// output.append(line + "\n");
// }
//
// int exitVal = process.waitFor();
// if (exitVal == 0) {
// System.out.println(output);
//
// return output.toString();
// } else {
// //abnormal...
// }
return output;
}
else {
throw new InvalidTaskModelException(taskId);
}
} catch (InvalidModelException e) {
throw new InvalidModelException(projectId);
} catch (IOException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
private static String loadStream(InputStream s) throws Exception
{
BufferedReader br = new BufferedReader(new InputStreamReader(s));
StringBuilder sb = new StringBuilder();
String line;
while((line=br.readLine()) != null)
sb.append(line).append("\n");
return sb.toString();
}
The commented lines are different ways I tried to do it.
If anyone encountered a similar problem any help would be much appreciated!
It is good that you already take care for the processes' STDOUT and STDIN. But rather than skipping copy them to System.out so you can see what is going on. I suspect something is not going as per your expectations.
Looking at the bash script you posted and the fact you are trying to run several processes: Is it possible your java code is running the bash script line by line? Be aware your java program it is not a BASH interpreter, so e.g. variable substitution should not work.
why you can not run each command in thread and joins them so that unless therad 1 is not completed . next thread can not start .
also please add command to verify image is downloaded successfully
docker pull -q $IMAGE
docker images | grep $IMAGE
docker run --name $NAME -d -p 1234:8080 --log-opt fluentd-address=localhost:2233 $IMAGE
i am guessing there are 2 possiblities are here
Check local directory persmission if possible give 755 permission to
it .
Java process itself is not able to execute docker command due
to permission issue, run process as sudo user.
I am trying to execute it via:
Process process = Runtime.getRuntime().exec(spark_cmd);
with no luck. The command ran via shell starts my application which succeeds. Running it via exec start a process which dies shortly after and does nothing.
When I try
process.waitFor();
it hangs and waits forever. Real magic begins when I try to read something from the process:
InputStreamReader isr = new InputStreamReader(process.getErrorStream());
BufferedReader br = new BufferedReader(isr);
To do so I start a thread that reads from the stream in a while loop:
class ReadingThread extends Thread {
BufferedReader reader;
Wontekk(BufferedReader reader) {
this.reader = reader;
}
#Override
public void run() {
String line;
try {
while ((line = reader.readLine()) != null) {
System.out.println(line);
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
Application starts, does some stuff, and hangs. When I abort my application, spark application wakes up (??????????) and completes remaining work. Does anyone have reasonable explanation of what is happening?
thanks
You can send spark job as spark-submit with the help of java code with the help of SparkLauncher so you can go though below link and check it our
https://spark.apache.org/docs/1.4.0/api/java/org/apache/spark/launcher/SparkLauncher.html
One way is Spark launcher as told by #Sandeep Purohit
I'd offer shell script approach with nohup command to submit job like this...
This worked for me incase of mapreduce executions... same way you can
try for spark background jobs as well.
Have a look https://en.wikipedia.org/wiki/Nohup
"nohup spark-submit <parameters> 2>&1 < /dev/null &"
When ever, you get messages then you can poll that event and call this shell script. Below is the code snippet to do this...
/**
* This method is to spark submit
* <pre> You can call spark-submit or mapreduce job on the fly like this.. by calling shell script... </pre>
* #param commandToExecute String
*/
public static Boolean executeCommand(final String commandToExecute) {
try {
final Runtime rt = Runtime.getRuntime();
// LOG.info("process command -- " + commandToExecute);
final String[] arr = { "/bin/sh", "-c", commandToExecute};
final Process proc = rt.exec(arr);
// LOG.info("process started ");
final int exitVal = proc.waitFor();
LOG.trace(" commandToExecute exited with code: " + exitVal);
proc.destroy();
} catch (final Exception e) {
LOG.error("Exception occurred while Launching process : " + e.getMessage());
return Boolean.FALSE;
}
return Boolean.TRUE;
}
Moreover to debug
ps -aef | grep "your pid or process name"
Below command will list the open files opened by the process..
lsof -p <your process id >
Also, have a look at process.waitFor() never returns
May be this question is too easy but it took a lot my time but I could not figure out. I am trying to execute echo command in which I am trying to append some data in a text file. Everything works when run in terminal but when I try to run it through Runtime.getRuntime().exec("echo my text >> file.txt"), then >> does not work.
Is there any way to give code or something so that this command should work? I tried > but it did not work for me.
The problem is: In which shell should the echo command be executed? Runtime.exec can get a cmdArray. http://docs.oracle.com/javase/7/docs/api/java/lang/Runtime.html#exec%28java.lang.String[],%20java.lang.String[]%29
import java.io.*;
class RuntimeExecLinux {
public static void main(String[] args){
try {
// Execute a command with arguments
String[] cmd = new String[]{"/bin/bash", "-c", "echo 'my text from java' >> file.txt"};
Process child = Runtime.getRuntime().exec(cmd);
cmd = new String[]{"/bin/bash", "-c", "cat file.txt"};
child = Runtime.getRuntime().exec(cmd);
// Get the input stream and read from it
InputStream in = child.getInputStream();
int c;
while ((c = in.read()) != -1) {
System.out.print((char)c);
}
in.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
Greetings
Axel
what I would like to do is run a batch file multiple times from a java application. Therefore I set up a for-loop that runs this code n times:
for (int i = 0; i < n; i++) {
Runtime.getRuntime().exec("cmd /c start somefile.bat");
}
The problem is that now each time the command is run a new cmd window pops up. However, what I want is just one window that pops up at the beginning and that is used to display all data from the following command calls.
How can I do that?
With && you can execute more than one commands, one after another:
Runtime.getRuntime().exec("cmd /c \"start somefile.bat && start other.bat && cd C:\\test && test.exe\"");
Using multiple commands and conditional processing symbols
You can run multiple commands from a single command line or script using conditional processing symbols. When you run multiple commands with conditional processing symbols, the commands to the right of the conditional processing symbol act based upon the results of the command to the left of the conditional processing symbol.
For example, you might want to run a command only if the previous command fails. Or, you might want to run a command only if the previous command is successful.
You can use the special characters listed in the following table to pass multiple commands.
& [...] command1 & command2
Use to separate multiple commands on one command line. Cmd.exe runs the first command, and then the second command.
&& [...] command1 && command2
Use to run the command following && only if the command preceding the symbol is successful. Cmd.exe runs the first command, and then runs the second command only if the first command completed successfully.
|| [...] command1 || command2
Use to run the command following || only if the command preceding || fails. Cmd.exe runs the first command, and then runs the second command only if the first command did not complete successfully (receives an error code greater than zero).
( ) [...] (command1 & command2)
Use to group or nest multiple commands.
; or , command1 parameter1;parameter2
Use to separate command parameters.
I would use Java's ProcessBuilder or another class which simulates/uses a shell. The following snippet demonstrates the idea (for Linux with bash).
import java.util.Scanner;
import java.io.*;
public class MyExec {
public static void main(String[] args)
{
//init shell
ProcessBuilder builder = new ProcessBuilder( "/bin/bash" );
Process p=null;
try {
p = builder.start();
}
catch (IOException e) {
System.out.println(e);
}
//get stdin of shell
BufferedWriter p_stdin =
new BufferedWriter(new OutputStreamWriter(p.getOutputStream()));
// execute the desired command (here: ls) n times
int n=10;
for (int i=0; i<n; i++) {
try {
//single execution
p_stdin.write("ls");
p_stdin.newLine();
p_stdin.flush();
}
catch (IOException e) {
System.out.println(e);
}
}
// finally close the shell by execution exit command
try {
p_stdin.write("exit");
p_stdin.newLine();
p_stdin.flush();
}
catch (IOException e) {
System.out.println(e);
}
// write stdout of shell (=output of all commands)
Scanner s = new Scanner( p.getInputStream() );
while (s.hasNext())
{
System.out.println( s.next() );
}
s.close();
}
}
Please note that it is only a snippet, which needs to be adapted for Windows, but in general it should work with cmd.exe.
public void TestCommandRun(){
Process process = null;
String[] command_arr = new String[]{"cmd.exe","/K","start"};
ProcessBuilder pBuilder = new ProcessBuilder("C:/Windows/System32/cmd.exe");
try{
process = pBuilder.start();
}
catch(IOException e){
e.printStackTrace();
System.out.println("Process failed");
}
if(null != process){
OutputStream out = process.getOutputStream();
OutputStreamWriter outWriter = new OutputStreamWriter(out);
BufferedWriter bWriter = new BufferedWriter(outWriter);
try{
bWriter.write("dir");
bWriter.newLine();
bWriter.write("ipconfig");
bWriter.flush();
bWriter.close();
}
catch(IOException e){
e.printStackTrace();
System.out.println("bWriter Failed");
}
}
}
It is quite simple to run a Unix command from Java.
Runtime.getRuntime().exec(myCommand);
But is it possible to run a Unix shell script from Java code? If yes, would it be a good practice to run a shell script from within Java code?
You should really look at Process Builder. It is really built for this kind of thing.
ProcessBuilder pb = new ProcessBuilder("myshellScript.sh", "myArg1", "myArg2");
Map<String, String> env = pb.environment();
env.put("VAR1", "myValue");
env.remove("OTHERVAR");
env.put("VAR2", env.get("VAR1") + "suffix");
pb.directory(new File("myDir"));
Process p = pb.start();
You can use Apache Commons exec library also.
Example :
package testShellScript;
import java.io.IOException;
import org.apache.commons.exec.CommandLine;
import org.apache.commons.exec.DefaultExecutor;
import org.apache.commons.exec.ExecuteException;
public class TestScript {
int iExitValue;
String sCommandString;
public void runScript(String command){
sCommandString = command;
CommandLine oCmdLine = CommandLine.parse(sCommandString);
DefaultExecutor oDefaultExecutor = new DefaultExecutor();
oDefaultExecutor.setExitValue(0);
try {
iExitValue = oDefaultExecutor.execute(oCmdLine);
} catch (ExecuteException e) {
System.err.println("Execution failed.");
e.printStackTrace();
} catch (IOException e) {
System.err.println("permission denied.");
e.printStackTrace();
}
}
public static void main(String args[]){
TestScript testScript = new TestScript();
testScript.runScript("sh /root/Desktop/testScript.sh");
}
}
For further reference, An example is given on Apache Doc also.
I think you have answered your own question with
Runtime.getRuntime().exec(myShellScript);
As to whether it is good practice... what are you trying to do with a shell script that you cannot do with Java?
I would say that it is not in the spirit of Java to run a shell script from Java. Java is meant to be cross platform, and running a shell script would limit its use to just UNIX.
With that said, it's definitely possible to run a shell script from within Java. You'd use exactly the same syntax you listed (I haven't tried it myself, but try executing the shell script directly, and if that doesn't work, execute the shell itself, passing the script in as a command line parameter).
Yes it is possible to do so. This worked out for me.
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import org.omg.CORBA.portable.InputStream;
public static void readBashScript() {
try {
Process proc = Runtime.getRuntime().exec("/home/destino/workspace/JavaProject/listing.sh /"); //Whatever you want to execute
BufferedReader read = new BufferedReader(new InputStreamReader(
proc.getInputStream()));
try {
proc.waitFor();
} catch (InterruptedException e) {
System.out.println(e.getMessage());
}
while (read.ready()) {
System.out.println(read.readLine());
}
} catch (IOException e) {
System.out.println(e.getMessage());
}
}
Here is my example. Hope it make sense.
public static void excuteCommand(String filePath) throws IOException{
File file = new File(filePath);
if(!file.isFile()){
throw new IllegalArgumentException("The file " + filePath + " does not exist");
}
if(isLinux()){
Runtime.getRuntime().exec(new String[] {"/bin/sh", "-c", filePath}, null);
}else if(isWindows()){
Runtime.getRuntime().exec("cmd /c start " + filePath);
}
}
public static boolean isLinux(){
String os = System.getProperty("os.name");
return os.toLowerCase().indexOf("linux") >= 0;
}
public static boolean isWindows(){
String os = System.getProperty("os.name");
return os.toLowerCase().indexOf("windows") >= 0;
}
Yes, it is possible and you have answered it! About good practises, I think it is better to launch commands from files and not directly from your code. So you have to make Java execute the list of commands (or one command) in an existing .bat, .sh , .ksh ... files.
Here is an example of executing a list of commands in a file MyFile.sh:
String[] cmd = { "sh", "MyFile.sh", "\pathOfTheFile"};
Runtime.getRuntime().exec(cmd);
To avoid having to hardcode an absolute path, you can use the following method that will find and execute your script if it is in your root directory.
public static void runScript() throws IOException, InterruptedException {
ProcessBuilder processBuilder = new ProcessBuilder("./nameOfScript.sh");
//Sets the source and destination for subprocess standard I/O to be the same as those of the current Java process.
processBuilder.inheritIO();
Process process = processBuilder.start();
int exitValue = process.waitFor();
if (exitValue != 0) {
// check for errors
new BufferedInputStream(process.getErrorStream());
throw new RuntimeException("execution of script failed!");
}
}
As for me all things must be simple.
For running script just need to execute
new ProcessBuilder("pathToYourShellScript").start();
The ZT Process Executor library is an alternative to Apache Commons Exec. It has functionality to run commands, capturing their output, setting timeouts, etc.
I have not used it yet, but it looks reasonably well-documented.
An example from the documentation: Executing a command, pumping the stderr to a logger, returning the output as UTF8 string.
String output = new ProcessExecutor().command("java", "-version")
.redirectError(Slf4jStream.of(getClass()).asInfo())
.readOutput(true).execute()
.outputUTF8();
Its documentation lists the following advantages over Commons Exec:
Improved handling of streams
Reading/writing to streams
Redirecting stderr to stdout
Improved handling of timeouts
Improved checking of exit codes
Improved API
One liners for quite complex use cases
One liners to get process output into a String
Access to the Process object available
Support for async processes ( Future )
Improved logging with SLF4J API
Support for multiple processes
This is a late answer. However, I thought of putting the struggle I had to bear to get a shell script to be executed from a Spring-Boot application for future developers.
I was working in Spring-Boot and I was not able to find the file to be executed from my Java application and it was throwing FileNotFoundFoundException. I had to keep the file in the resources directory and had to set the file to be scanned in pom.xml while the application was being started like the following.
<resources>
<resource>
<directory>src/main/resources</directory>
<filtering>true</filtering>
<includes>
<include>**/*.xml</include>
<include>**/*.properties</include>
<include>**/*.sh</include>
</includes>
</resource>
</resources>
After that I was having trouble executing the file and it was returning error code = 13, Permission Denied. Then I had to make the file executable by running this command - chmod u+x myShellScript.sh
Finally, I could execute the file using the following code snippet.
public void runScript() {
ProcessBuilder pb = new ProcessBuilder("src/main/resources/myFile.sh");
try {
Process p;
p = pb.start();
} catch (IOException e) {
e.printStackTrace();
}
}
Hope that solves someone's problem.
Here is an example how to run an Unix bash or Windows bat/cmd script from Java. Arguments can be passed on the script and output received from the script. The method accepts arbitrary number of arguments.
public static void runScript(String path, String... args) {
try {
String[] cmd = new String[args.length + 1];
cmd[0] = path;
int count = 0;
for (String s : args) {
cmd[++count] = args[count - 1];
}
Process process = Runtime.getRuntime().exec(cmd);
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
try {
process.waitFor();
} catch (Exception ex) {
System.out.println(ex.getMessage());
}
while (bufferedReader.ready()) {
System.out.println("Received from script: " + bufferedReader.readLine());
}
} catch (Exception ex) {
System.out.println(ex.getMessage());
System.exit(1);
}
}
When running on Unix/Linux, the path must be Unix-like (with '/' as separator), when running on Windows - use '\'. Hier is an example of a bash script (test.sh) that receives arbitrary number of arguments and doubles every argument:
#!/bin/bash
counter=0
while [ $# -gt 0 ]
do
echo argument $((counter +=1)): $1
echo doubling argument $((counter)): $(($1+$1))
shift
done
When calling
runScript("path_to_script/test.sh", "1", "2")
on Unix/Linux, the output is:
Received from script: argument 1: 1
Received from script: doubling argument 1: 2
Received from script: argument 2: 2
Received from script: doubling argument 2: 4
Hier is a simple cmd Windows script test.cmd that counts number of input arguments:
#echo off
set a=0
for %%x in (%*) do Set /A a+=1
echo %a% arguments received
When calling the script on Windows
runScript("path_to_script\\test.cmd", "1", "2", "3")
The output is
Received from script: 3 arguments received
It is possible, just exec it as any other program. Just make sure your script has the proper #! (she-bang) line as the first line of the script, and make sure there are execute permissions on the file.
For example, if it is a bash script put #!/bin/bash at the top of the script, also chmod +x .
Also as for if it's good practice, no it's not, especially for Java, but if it saves you a lot of time porting a large script over, and you're not getting paid extra to do it ;) save your time, exec the script, and put the porting to Java on your long-term todo list.
I think with
System.getProperty("os.name");
Checking the operating system on can manage the shell/bash scrips if such are supported.
if there is need to make the code portable.
String scriptName = PATH+"/myScript.sh";
String commands[] = new String[]{scriptName,"myArg1", "myArg2"};
Runtime rt = Runtime.getRuntime();
Process process = null;
try{
process = rt.exec(commands);
process.waitFor();
}catch(Exception e){
e.printStackTrace();
}
Just the same thing that Solaris 5.10 it works like this ./batchstart.sh there is a trick I don´t know if your OS accept it use \\. batchstart.sh instead. This double slash may help.
for linux use
public static void runShell(String directory, String command, String[] args, Map<String, String> environment)
{
try
{
if(directory.trim().equals(""))
directory = "/";
String[] cmd = new String[args.length + 1];
cmd[0] = command;
int count = 1;
for(String s : args)
{
cmd[count] = s;
count++;
}
ProcessBuilder pb = new ProcessBuilder(cmd);
Map<String, String> env = pb.environment();
for(String s : environment.keySet())
env.put(s, environment.get(s));
pb.directory(new File(directory));
Process process = pb.start();
BufferedReader inputReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
BufferedWriter outputReader = new BufferedWriter(new OutputStreamWriter(process.getOutputStream()));
BufferedReader errReader = new BufferedReader(new InputStreamReader(process.getErrorStream()));
int exitValue = process.waitFor();
if(exitValue != 0) // has errors
{
while(errReader.ready())
{
LogClass.log("ErrShell: " + errReader.readLine(), LogClass.LogMode.LogAll);
}
}
else
{
while(inputReader.ready())
{
LogClass.log("Shell Result : " + inputReader.readLine(), LogClass.LogMode.LogAll);
}
}
}
catch(Exception e)
{
LogClass.log("Err: RunShell, " + e.toString(), LogClass.LogMode.LogAll);
}
}
public static void runShell(String path, String command, String[] args)
{
try
{
String[] cmd = new String[args.length + 1];
if(!path.trim().isEmpty())
cmd[0] = path + "/" + command;
else
cmd[0] = command;
int count = 1;
for(String s : args)
{
cmd[count] = s;
count++;
}
Process process = Runtime.getRuntime().exec(cmd);
BufferedReader inputReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
BufferedWriter outputReader = new BufferedWriter(new OutputStreamWriter(process.getOutputStream()));
BufferedReader errReader = new BufferedReader(new InputStreamReader(process.getErrorStream()));
int exitValue = process.waitFor();
if(exitValue != 0) // has errors
{
while(errReader.ready())
{
LogClass.log("ErrShell: " + errReader.readLine(), LogClass.LogMode.LogAll);
}
}
else
{
while(inputReader.ready())
{
LogClass.log("Shell Result: " + inputReader.readLine(), LogClass.LogMode.LogAll);
}
}
}
catch(Exception e)
{
LogClass.log("Err: RunShell, " + e.toString(), LogClass.LogMode.LogAll);
}
}
and for usage;
ShellAssistance.runShell("", "pg_dump", new String[]{"-U", "aliAdmin", "-f", "/home/Backup.sql", "StoresAssistanceDB"});
OR
ShellAssistance.runShell("", "pg_dump", new String[]{"-U", "aliAdmin", "-f", "/home/Backup.sql", "StoresAssistanceDB"}, new Hashmap<>());