Develop Reference

python - JAR works in terminal but not from python

I working on a python 3 script for doing some bench (school purpose). So I need to invoke my JAR.
I use subprocess.check_output for that.
java_out = subprocess.check_output("java -jar my_jar.jar -p input_file", shell=True)
In terminal it works fine, I get the expected output and exit code is 0.
But in python, I get this :
Syntax error. (One of my java exception, but it might not happen in this case)
Traceback (most recent call last):
File "C:/Users/Jeremy/PycharmProjects/bench_bf/bench_script.py", line 41, in <module>
main()
File "C:/Users/Jeremy/PycharmProjects/bench_bf/bench_script.py", line 32, in main
result_list.append(bench(bf_file, stats_file))
File "C:/Users/Jeremy/PycharmProjects/bench_bf/bench_script.py", line 10, in bench
java_out = subprocess.check_output("java -jar my_jar.jar -p input_file", shell=True)
File "C:\Python34\lib\subprocess.py", line 620, in check_output
raise CalledProcessError(retcode, process.args, output=output)
subprocess.CalledProcessError: Command 'java -jar my_jar.jar -p input_file' returned non-zero exit status 5
Process finished with exit code 1
That does not make any sense to me. Can anyone help me ? Thanks !
The full code is following (I've also tried with absolute path) :)
import subprocess
import os
import re
FILE_NAME = "input_file"
JAR_NAME = "my_jar.jar"
TEST_ITER = 5
def bench(bf_file, stats_file):
java_out = subprocess.check_output("java -jar "+ JAR_NAME + " -p " + FILE_NAME, shell=True)
print(java_out)
m = re.search(".*EXEC_TIME : (\d*) ms.*EXEC_MOVE : (\d*)", java_out)
return [m.group(0), m.group(1)]
def init_stats(f):
f.write("Iterations; Exec time; exec move")
def write_file(f):
f.write("+++")
def main():
bf_file = open(FILE_NAME, "w", encoding="utf-8")
stats_file = open("bench-result.csv", "w")
write_file(bf_file)
init_stats(stats_file);
result_list = []
for i in range(0,TEST_ITER):
result_list.append(bench(bf_file, stats_file))
average_time = 0;
for res in result_list:
average_time += res[0]
average_time /= TEST_ITER;
stats_file.write(average_time + ";" + result_list[0][1])
main()
EDIT: I also tried java_out = subprocess.check_output(["java", "-jar", "my_jar.jar", "-p", "input_file"], shell=True), it changes nothing.
EDIT 2: Same result using absolute path or os.system
* SOLUTION *
Since I open the file in write mode, my JAR can't open it, and consider it's empty... Thanks my mate DjNikita :)

My first thought would be that there is something in your environment that is not transferring to the subprocess. Try this and see if it outputs anything that looks relevant
import os
for key in os.environ:
if any(token in key.lower() for token in ['java', 'jre', 'jdk']):
print(key, os.environ[key])
I've had another thought too. Some programs expect their input to be a tty (ie. interactive terminal) and get angry when they're fed in a pipe. Is there anything in your Java program that might cause it to expect a certain type of input stream?

Try specifying the absolute path of the jar file, as it might be that your sub-process isn't running the directory you think it is.
Try running 'dir' and seeing where it returns, for instance. Maybe check that 'java --V' (the version flag? not in a position to check at the moment) returns something indicating that Java ran, rather than an error. Basically, try and get a simple thing running via Python, then extend it.

Redirecs with /bin/bash from Java Runtime exec()

I'm trying to use a redirect in a command executed with the exec()-method of java.lang.Runtime on Ubuntu 14.04
public static void main(String[] args) throws IOException, Exception {
Runtime runtime = Runtime.getRuntime();
String command = "echo bla > bla.txt";
System.out.println("Command : " + command);
Process process = runtime.exec(command);
printLines(" stdout", process.getInputStream());
printLines(" error", process.getErrorStream());
process.waitFor();
System.out.println("ExitValue : " + process.exitValue());
}
private static void printLines(String name, InputStream ins) throws Exception {
try(Stream<String> lines = new BufferedReader(new InputStreamReader(ins)).lines()) {
lines.forEach(line -> System.out.println(name + " : " + line));
}
}
The output is:
Command : echo bla > bla.txt
stdout : bla > bla.txt
ExitValue : 0
So bla > bla.txt is written to stdout but of course there is no redirect to bla.txt.
Perhapes shell redirects are not possible in a simple exec().
So I tried to change the command = "/bin/bash -c 'echo bla > bla.txt'" to use the whole echo and redirect as a parameter to /bin/bash.
With this I get the result:
Command : /bin/bash -c 'echo bla > bla.txt'
error : bla: -c: line 0: unexpected EOF while looking for matching `''
error : bla: -c: line 1: syntax error: unexpected end of file
ExitValue : 1
Of course /bin/bash -c 'echo bla > bla.txt'works fine on Ubuntu and creates the desired file.
I found no place where I could set the single quotes to get a satisfying result, and I also tried with all kinds of escape characters to escape the spaces or the redirect (>).
It works if I use a command array like
String cmdArray[] = {"/bin/bash", "-c", "echo bla > bla.txt"};
Process process = runtime.exec(cmdArray);
, but it has to be a single string because the entire command has to be build somewhere else.
I surely know
that there are better ways to write strings to files,
that it may be a bad idea to execute commands build somewhere else,
and so on ...
I'm just curious why this does not work.

The cause is that exec uses simple StringTokenizer with any white space as a delimiter to parse the actual command. Therefore it's portable as it does work nowhere when you pass something complex :-)
The workaround you chose is correct way, portable and most of all safest asyou ddon't need to escape if the command contained for example quotes etc.

String command = "echo bla > bla.txt";
Process process = runtime.exec(command);
The > blah.txt text in your command is shell syntax to redirect standard output. It's implemented by shells like sh or bash.
Runtime.exec() doesn't use a shell to run commands. The way it launches commands is described here and here. Basically, it uses its own logic to split the command string into arguments at spaces, then directly executes the resulting command.
If you want to invoke a command which should be interpreted as a shell command, you need to explicitly invoke a shell:
String[] command = { "/bin/bash", "-c", "echo bla > bla.txt" };
Process process = runtime.exec(command);

Execute command from java which might include spaces

I am trying to execute the following command from a java program:
java -jar /opt/plasma/fr.inria.plasmalab.plasmalab-1.3.4.jar -t -a montecarlo -A"Total samples"=1000 -m models/translated/plasma/NaCl2.rml:rml --format csv -r models/translated/plasma/NaCl2.bltl:bltl
with the following code:
String totalSample = "-A\"Total samples\"=1000";
String mcCommand = "java -jar " + MChecker.getAppPath() + " -t "
+ "-a " + "montecarlo " + totalSample
+ " -m " + mcModelRelPath + ":rml " + "--format " + "csv "
+ "-r " + customQueryRelPath + ":bltl";
Process process = Runtime.getRuntime().exec(mcCommand);
int errCode = process.waitFor();
//then get the output, and error
But it results in the following error:
Wrong parameter description : Dynamic parameter expected a value of the form a=b but got:"Total
I ran the same command in a terminal and it worked without any problem. But when I create the command in Java and try to invoke the tool it does not work.
I think it's confused because of the totalSample parameter which includes a space. What I did next was to put "\ " space escape in paramater(String totalSample = "-A\"Total\\ samples\"=1000";), but it still refused to accept it. It gave the following error:
Wrong parameter description : Dynamic parameter expected a value of the form a=b but got:"Total\
Then I run the same parameters with the ProcessBuilder object, like the following:
String[] mcCommand = {"java", "-jar", MChecker.getAppPath(), "-t",
"-a", "montecarlo",totalSample, "-m",
mcModelRelPath + ":rml", "--format", "csv", "-r",
customQueryRelPath + ":bltl" };
ProcessBuilder pb = new ProcessBuilder(mcCommand);
Process process = pb.start();
process.waitFor();
But it still did not work and threw some custom exceptions.
I am currently out of options -- do you have any idea why this command does not work with Java, when it works just fine from the terminal interface?
BTW: I ran the same code on Windows it worked perfectly, but I have to run this code on Ubuntu OS.
Many Thanks

It was weird that, Java Process and ProcessBuilder classes could not pass the parameters properly. I don't know why, but since I was able to execute the command from terminal. I decide to call the terminal first and then execute the command. Therefore, I changed my command as following
String mcCommand[] = {
"/bin/sh",
"-c",
"java -jar /opt/plasma/fr.inria.plasmalab.plasmalab-1.3.4.jar -t -a montecarlo "+totalSample+" -m models/translated/plasma/NaCl2.rml:rml --format csv -r models/translated/plasma/NaCl2.bltl:bltl" };
Despite it is platform dependant solution, currently it is ok for me.

Don't add extra quotes for Total samples when using ProcessBuilder:
ProcessBuilder pb = new ProcessBuilder("java", "-jar", MChecker.getAppPath(), "-t",
"-a", "montecarlo", "-ATotal samples=1000",
"-m", mcModelRelPath + ":rml", "--format", "csv",
"-r", customQueryRelPath + ":bltl");
//...

Can't execute Shellscript successfully using getruntime.exec()

I have written a code to execute a script from java :
String wrapper_script=homedir+"/blast_distribute.sh "+" --seqs="+seqs+" --i="+formobj.getUpFile().getFileName()+" "+formobj.getSelected_program();
script_exec=Runtime.getRuntime().exec(wrapper_script);
This works perfect for me as command is executed successfully. Now I need to run this command as other user so I need to execute a command in a format like :
su username -c 'command'
SO I have edited above script_exec string as :
String wrapper_script1="su - "+username+" -c "+"'"+wrapper_script+"'";
I have printed wrapper_script1 which diplays:
su - abhijeet -c '/home/abhijeet//blast_distribute.sh --seqs=1562
--i=mPS_0.contigs.fasta'
If I run same command directly on Linux,It works exactly as I need.But When I am running through :
script_exec=Runtime.getRuntime().exec(wrapper_script1);
It does't work properly ,In my error stream I get error as
su: unrecognized option '--seqs=1562' Try `su --help' for more
information.
I have tried a lot but could't resolve the issue.What can be the reason for this issue?

You should use multiple parameters run for this:
Runtime.getRuntime().exec(new String[] {"su", "-", username, "-c",
homedir + "/blast_distribute.sh " + " --seqs=" + seqs + " --i=" + formobj.getUpFile().getFileName() + " " + formobj.getSelected_program()
});
You're getting the error, because ' is not treated here as a enclosing characters, but as a parameters for su command.
Let's look into details. When you type into console some command like su - jsmith -c 'aba --cabga', what you're doing is: "run command su with such an arguments: -, jsmith, -c, aba --caba (one argument for multiple words)".
The same thing you should do in your Java code. There's special command in Java API for running the concrete command with concrete arguments instead of parsing line as shell do: Runtime.exec(String[]): the first argument is command, the next are the arguments. That's what we do in this code block.

Running shell script from java - doesn't finish task

I have a java program which is supposed to launch a shell script. The script contains 6 tasks which are to be executed in sequence. The java program launches the script and it starts(as I see the logs). But after 10-15 seconds, the execution stops, even before the first task in the shell script is completed. The strange thing is that the script runs fine when I launch it in terminal. To avoid risking the program to hang while the script is being executed, I launch it in a separate thread. What might be a probable reason?
Java code -
try {
log.info("run cmd - "+optionsRun);
String[] cmdLine = (String[]) optionsRun.toArray(new String[optionsRun.size()]);
Process process = Runtime.getRuntime().exec(cmdLine);
log.info("end run cmd " + this.getScriptPath());
//
// BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(process.getOutputStream()));
// writer.write("mypwd");
// writer.flush();
// writer.close();
InputStream is = process.getErrorStream();
String error = inputStreamToStringValue(is);
log.trace("Eventual error was : " + error);
InputStream os = process.getInputStream();
String output = inputStreamToStringValue(os);
log.info("Eventual output was : " + output);
if (error!=null & error.length()>0) {
throw new ActionProcessingException("An error occurred when running the script :'"+this.getScriptPath()+"' with following error message : "+error);
}else {
log.info("Script run ended successfully.");
}
And the shell script looks this way -
#!/bin/sh
# ./publish <path-to-publish-home-folder> <workspace_id> <start_date> <end_date>
# ./publish <path-to-publish-home-folder> 100011 2010-01-06-12:00:00-CET 2012-01-14-19:00:00-CET
rm -f $1/publish.log
echo 'Start publish' >> $1/publish.log
echo $0 $1 $2 $3 $4 $5 $6 $7 $8 $9 >> $1/publish.log
# lancement de l'export RDF du domaine
cd $1/1-export-domain
echo "Starting export domain with the command - ./export.sh $2" >> $1/publish.log
./export.sh $2
# lancement de l'export des translations du domaine
cd $1/2-export-trans
echo "Starting export domain(translated) with the command - ./export.sh $2" >> $1/publish.log
./export.sh $2
.....
.....
a couple of more steps like 1 and 2
....
Thanks in advance,

I'm not sure, but I'll recommend two links that might help you figure it out.
The first is a very old one about Runtime.exec():
http://www.javaworld.com/javaworld/jw-12-2000/jw-1229-traps.html
The second is about ProcessBuilder, the new class intended to replace Runtime.exec():
http://www.java-tips.org/java-se-tips/java.util/from-runtime.exec-to-processbuilder.html

I cannot be sure I my guess is that the problem is in your method inputStreamToStringValue(is). It reads STDERR and it is blocking on read. When it has nothing to read from STDERR but the process is not terminated yet you will be blocked forever.
I'd recommend you to use ProcessBuilder:
ProcessBuilder b = new ProcessBuilder();
b.redirectErrorStream(true);
Now you can read STDIN and STDERR together.
If you still want to read them separately you have 2 solutions.
First do as you are doing now but do not be block on read, i.e. call in.available() before each call of read and then read only number of bytes that were previously available.
Second way is to use the shell redirection. Run you script and redirect its STDOUT and STDERR to temporary files. Then wait until your process terminates and then read from files. I personally think that this solution is easier and more robust.
Good luck.