I working on a python 3 script for doing some bench (school purpose). So I need to invoke my JAR.
I use subprocess.check_output for that.
java_out = subprocess.check_output("java -jar my_jar.jar -p input_file", shell=True)
In terminal it works fine, I get the expected output and exit code is 0.
But in python, I get this :
Syntax error. (One of my java exception, but it might not happen in this case)
Traceback (most recent call last):
File "C:/Users/Jeremy/PycharmProjects/bench_bf/bench_script.py", line 41, in <module>
main()
File "C:/Users/Jeremy/PycharmProjects/bench_bf/bench_script.py", line 32, in main
result_list.append(bench(bf_file, stats_file))
File "C:/Users/Jeremy/PycharmProjects/bench_bf/bench_script.py", line 10, in bench
java_out = subprocess.check_output("java -jar my_jar.jar -p input_file", shell=True)
File "C:\Python34\lib\subprocess.py", line 620, in check_output
raise CalledProcessError(retcode, process.args, output=output)
subprocess.CalledProcessError: Command 'java -jar my_jar.jar -p input_file' returned non-zero exit status 5
Process finished with exit code 1
That does not make any sense to me. Can anyone help me ? Thanks !
The full code is following (I've also tried with absolute path) :)
import subprocess
import os
import re
FILE_NAME = "input_file"
JAR_NAME = "my_jar.jar"
TEST_ITER = 5
def bench(bf_file, stats_file):
java_out = subprocess.check_output("java -jar "+ JAR_NAME + " -p " + FILE_NAME, shell=True)
print(java_out)
m = re.search(".*EXEC_TIME : (\d*) ms.*EXEC_MOVE : (\d*)", java_out)
return [m.group(0), m.group(1)]
def init_stats(f):
f.write("Iterations; Exec time; exec move")
def write_file(f):
f.write("+++")
def main():
bf_file = open(FILE_NAME, "w", encoding="utf-8")
stats_file = open("bench-result.csv", "w")
write_file(bf_file)
init_stats(stats_file);
result_list = []
for i in range(0,TEST_ITER):
result_list.append(bench(bf_file, stats_file))
average_time = 0;
for res in result_list:
average_time += res[0]
average_time /= TEST_ITER;
stats_file.write(average_time + ";" + result_list[0][1])
main()
EDIT: I also tried java_out = subprocess.check_output(["java", "-jar", "my_jar.jar", "-p", "input_file"], shell=True), it changes nothing.
EDIT 2: Same result using absolute path or os.system
* SOLUTION *
Since I open the file in write mode, my JAR can't open it, and consider it's empty... Thanks my mate DjNikita :)
My first thought would be that there is something in your environment that is not transferring to the subprocess. Try this and see if it outputs anything that looks relevant
import os
for key in os.environ:
if any(token in key.lower() for token in ['java', 'jre', 'jdk']):
print(key, os.environ[key])
I've had another thought too. Some programs expect their input to be a tty (ie. interactive terminal) and get angry when they're fed in a pipe. Is there anything in your Java program that might cause it to expect a certain type of input stream?
Try specifying the absolute path of the jar file, as it might be that your sub-process isn't running the directory you think it is.
Try running 'dir' and seeing where it returns, for instance. Maybe check that 'java --V' (the version flag? not in a position to check at the moment) returns something indicating that Java ran, rather than an error. Basically, try and get a simple thing running via Python, then extend it.
Related
This question already has an answer here:
Why does Runtime.exec(String) work for some but not all commands?
(1 answer)
Closed 10 months ago.
I am doing an exercise related to Runtime.exec(), I understand that Runtime.exec is not a shell interpreter, that's why I execute "bash -c 'command'" instead, but for some reason, I can execute commands like ls bash -c 'ls' but not echo or redirection or multiple commands. These does not work:
bash -c 'echo 1234'
bash -c 'ls > abc'
bash -c 'ls;id'
bash -c 'ls -al'
Here is my java code:
import java.util.*;
import java.io.*;
public class runtime {
public static void main(String args[]) throws Exception{
String cmd = args[0];
System.out.println(cmd);
Process p = Runtime.getRuntime().exec(cmd);
OutputStream os = p.getOutputStream();
InputStream in = p.getInputStream();
DataInputStream dis = new DataInputStream(in);
String disr = dis.readLine();
while ( disr != null ) {
System.out.println("Out: " + disr);
disr = dis.readLine();
}
}
}
I run the above commands with the syntax:
java runtime "bash -c 'command'"
This works:
$ java runtime "bash -c 'ls'"
bash -c 'ls'
Out: Main.class
Out: Main.java
Out: runtime.class
Out: runtime.java
I am using openjdk 11.0.15 on Ubuntu 20.04 and zsh.
Can anyone tell me why Runtime doesn't work in this case? Thank you!
Because of shell parsing.
These are all concepts that the OS just does not have:
The concept of 'every space separates one argument from another (and the command from the list of arguments'). The concept that a single string can possibly run anything, in fact; at the OS level that's just not what the 'run this thing' API looks like. When you type commands on the command prompt, your shell app is 'interpreting' these strings into a command execution request.
The concept of figuring out that bash means /bin/bash, i.e. $PATH resolution.
The concept that *.txt is supposed to refer to all files that ends in .txt.
The concept that $FOO should be replaced with the value of the environment variable 'FOO'
The concept that ; separates 2 commands, and it's supposed to run both.
The concept that single and double quotes escape things. "Escape things" implies that things can cause interpretation to happen. The OS interprets nothing, therefore there's nothing to escape. Obvious, then, that the OS doesn't know what ' is, or ".
That >foo means: Please set the standard output of the spawned process such that it sends it all to file 'foo'.
In windows shells, that # in front of the command means 'do not echo the command itself'. This and many other things are shellisms: /bin/bash does that, maybe /bin/zsh does something else. Windows's built in shell thing definitely is quite different from bash!
Instead, an OS simply wants you to provide it a full path to an executable along with a list of strings, and pick targets for standard in, standard out, and standard err. It does no processing on any of that, just starts the process you named, and passes it the strings verbatim.
You're sort of half there, as you already figured out that e.g. ls >foo cannot work if you execute it on its own, but it can work if you tell bash to execute it. As ALL of that stuff in the above list? Your shell does that.
It gets more complicated: Turning *.txt into foo.txt bar.txt is a task of bash and friends, e.g. if you attempted to run: ls '*.txt' it does not work. But on windows, it's not the shell's job; the shell just passes it verbatim to dir, and it is the command's job to undo it. What a mess, right? Executing things is hard!
So, what's wrong here? Two things:
Space splitting isn't working out.
Quote application isn't being done.
When you write:
bash -c 'ls >foo'
in a bash shell, bash has to first split this up, into a command, and a list of arguments. Bash does so as follows:
Command: bash
arg1: -c
arg2: ls >foo
It knows that ls >foo isn't 2 arguments because, effectively, "space" is the bash operator for: "... and here is the next argument", and with quotes (either single or double), the space becomes a literal space instead of the '... move to next argument ...' operator.
In your code, you ask bash to run java, and then java to run bash. So, bash first does:
cmd: java
arg1: bash -c 'ls >foo'
With the same logic at work. Your java app then takes that entire string (that's args[0]: "bash -c 'ls >foo'"), and you then feed it to a method you should never use: Runtime.exec(). Always use ProcessBuilder, and always use the list-based form of it.
Given that you're using the bad method, you're now asking java to do this splitting thing. After all, if you just tell the OS verbatim, please run "bash -c 'ls >foo'", the OS dutifully reports: "I'm sorry, but file ./bash -c ;ls >foo' does not exist", because it does not processing at all". This is unwieldy so java's exec() method is a disaster you should never use: Because people are confused about this, it tries to do some extremely basic processing, except every OS and shell is different, java does not know this, so it does a really bad job at it.
Hence, do not use it.
In this case, java doesn't realize that those quotes mean it shouldn't split, so java tells the OS:
Please run:
cmd: /bin/bash (java DOES do path lookup; but you should avoid this, do not use relative path names, you should always write them out in full)
arg1: -c
arg2: 'ls
arg3: >foo'
and now you understand why this is just going completely wrong.
Instead, you want java to tell the OS:
cmd: /bin/bash
arg1: -c
arg2: ls >foo
Note: ls >foo needs to be one argument, and NOTHING in the argument should be containing quotes, anywhere. The reason you write:
/bin/bash -c 'ls >foo'
In bash, is because you [A] want bash not to treat that space between ls and >foo as an arg separator (you want ls >foo to travel to /bin/bash as one argument), and [B] that you want >foo to just be sent to the bash you're spawning and not to be treated as 'please redirect the output to file foo' at the current shell level.
Runtime.exec isn't a shell, so the quotes stuff? Runtime.exec has no idea.
This means more generally your plan of "I shall write an app where the entire argument you pass to it is just run" is oversimplified and can never work unless you write an entire quotes and escaper analyser for it.
An easy way out is to take the input, write it out to a shell script on disk, set the exec flag on it, and always run /bin/bash -c /path/to/script-you-just-wrote, sidestepping any need to attempt to parse anything in java: Let bash do it.
The ONE weird bizarro thing I cannot explain, is that literally passing 'ls' to /bin/bash -c, with quotes intact, DOES work and runs ls as normal, but 'ls *' does not, perhaps because now bash thinks you want executable file /bin/ls * which obviously does not exist (a star cannot be in a file name, or at least, that's not the ls executable, and it's not an alias for the ls built-in). At any rate, you want to pass ls without the quotes.
Let's try it!
import java.util.*;
import java.io.*;
public class runtime {
public static void main(String args[]) throws Exception{
ProcessBuilder pb = new ProcessBuilder();
pb.command("/bin/bash", "-c", "echo 1234");
// pb.command("/bin/bash", "-c", "'echo 1234'");
Process p = pb.start();
OutputStream os = p.getOutputStream();
InputStream in = p.getInputStream();
DataInputStream dis = new DataInputStream(in);
String disr = dis.readLine();
while ( disr != null ) {
System.out.println("Out: " + disr);
disr = dis.readLine();
}
int errCode = p.waitFor();
System.out.println("Process exit code: " + errCode);
}
}
The above works fine. Replace the .command line with the commented out variant and you'll notice it does not work at all, and you get an error. On my mac I get a '127' error; perhaps this is bash reporting back: I could not find the command you were attempting to execute. 0 is what you're looking for when you invoke waitFor: That's the code for 'no errors'.
I was developing my spring boot server on Windows. Now I have upgraded to Ubuntu 20.04.
the project executes a python script which should return a result as a txt file with this command:
python3 -c "from main import *;main(function,'/tmp/execution12480676806364930620/executionResponse.txt')"
Thanks to this code:
List<String> items = Arrays.asList(project.getExecutorType().buildAndGetExecutionCommandByProject(project));
ProcessBuilder pb = new ProcessBuilder(items);
pb.directory(new File(project.getPath()));
Process p = pb.start();
p.waitFor(5, TimeUnit.SECONDS);
when I print in the console the array passed in the Item variable:
[python3, -c, "from main import *;main(function, '/tmp/execution12480676806364930620/executionResponse.txt')"]
and the path of the array passed in pb.directory :
/tmp/execution12480676806364930620
My problem is that the project is not running and returning nothing.
when i go to the folder and run the same command from terminal everything works.
And that on windows 10 this same process worked fine.
Looking at similar issues I modified my code like this but it doesn't change anything:
List<String> items = Arrays.asList(project.getExecutorType().buildAndGetExecutionCommandByProject(project));
ProcessBuilder pb = new ProcessBuilder();
pb.command(items);
pb.redirectErrorStream(true);
pb.directory(new File(project.getPath()));
Process p = pb.start();
p.waitFor(5, TimeUnit.SECONDS);
What am I doing wrong?
Edit :
My command for read outputs :
private String inputStreamToString(InputStream inputStream){
return new BufferedReader(
new InputStreamReader(inputStream, StandardCharsets.UTF_8)).lines()
.collect(Collectors.joining("\n"));
}
And I call it like that :
System.out.println(this.inputStreamToString(p.getErrorStream()));
System.out.println(this.inputStreamToString(p.getInputStream()));
What works, when I just run "python main.py" I get the errors and print them out.
I can easily add the command at the end of the main file but I don't understand why the python -c "..." is not working? I am not receiving any errors ... I manage several languages and this could be a problem for me later
Aha! On closer inspection I think you're right it's not executing anything (and thus not producing any output either normal or error for you to see).
You don't show how the array of strings is created, but your printout suggests you have actually put quotemarks in the third string. That's wrong. When you give the shell command line python -c "import this; dothat" the shell uses the quotemarks to control parsing of this command line, but it does not pass them to the python process; the args passed to the python process (shown vertically for clarity, and omitting the argv[0]=program used in C but omitted in Java) are actually
-c
import this; dothat
If you pass an argument actually containing quotemarks like
-c
"import this; dothat"
then python doesn't execute the commands import and dothat; instead it evaluates the string literal "import this; dothat" and (since it isn't running interactively) discards the result.
Try not including, or removing, the " at the beginning and end. But leave the ' inside the string value because you do want python to receive those.
I'm trying to execute bash script using karate. I'm able to execute the script from karate-config.js and also from .feature file. I'm also able to pass the arguments to the script.
The problem is, that if the script fails (exits with something else than 0) the test execution continues and finishes as succesfull.
I found out that when the script echo-es something then i can access it as a result of the script so I could possibly echo the exit value and do assertion on it (in some re-usable feature), but this seems like a workaround rather than a valid clean solution. Is there some clean way of accessing the exit code without echo-ing it? Am I missing on something?
script
#!/bin/bash
#possible solution
#echo 3
exit 3;
karate-config.js
var result = karate.exec('script.sh arg1')
feture file
def result = karate.exec('script.sh arg1')
Great timing. We very recently did some work for CLI testing which I am sure you can use effectively. Here is a thread on Twitter: https://twitter.com/maxandersen/status/1276431309276151814
And we have just released version 0.9.6.RC4 and new we have a new karate.fork() option that returns an instance of Command on which you can call exitCode
Here's an example:
* def proc = karate.fork('script.sh arg1')
* proc.waitSync()
* match proc.exitCode == 0
You can get more ideas here: https://github.com/intuit/karate/issues/1191#issuecomment-650087023
Note that the argument to karate.fork() can take multiple forms. If you are using karate.exec() (which will block until the process completes) the same arguments work.
string - full command line as seen above
string array - e.g. ['script.sh', 'arg1']
json where the keys can be
line - string (OR)
args - string array
env - optional environment properties (as JSON)
redirectErrorStream - boolean, true by default which means Sys.err appears in Sys.out
workingDir - working directory
useShell - default false, auto-prepend cmd /c or sh -c depending on OS
And since karate.fork() is async, you need to call waitSync() if needed as in the example above.
Do provide feedback and we can tweak further if needed.
EDIT: here's a very advanced example that shows how to listen to the process output / log, collect the log, and conditionally exit: fork-listener.feature
Another answer which can be a useful reference: Conditional match based on OS
And here's how to use cURL for advanced HTTP tests ! https://stackoverflow.com/a/73230200/143475
In case you need to do a lot of local file manipulation, you can use the karate.toJavaFile() utility so you can convert a relative path or a "prefixed" path to an absolute path.
* def file = karate.toJavaFile('classpath:some/file.txt')
* def path = file.getPath()
I am having issues to use the java program https://github.com/antonydeepak/ResumeParser/ in Django project in a localserver in MAC OS.
I have installed ResumeParser in the Django Project like:
-- Django Project
-- app1
-- app2
-- ResumerParser
Here is my code but it says "Could not find or load main class".
if form.is_valid():
f = form.save(commit=False)
resume = form.cleaned_data['resume']
cmd = ['java', '-cp', 'bin/:../GATEFiles/lib/:../GATEFiles/bin/gate.jar:lib/*', 'code4goal.antony.resumeparser.ResumeParserProgram %s textOutput.json' % resume]
subprocess.Popen(cmd)
Any clues of how to solve this? I've tried every post related to this theme in StackOverflow no success.
Thanks in advance
You're mixing up well-delimiter parameters with parameters grouped with spaces.
cmd = ['java', '-cp', 'bin/:../GATEFiles/lib/:../GATEFiles/bin/gate.jar:lib/*', 'code4goal.antony.resumeparser.ResumeParserProgram %s textOutput.json' % resume]
Your last parameter is seen as a single parameter and is protected by spaces by subprocess:
"code4goal.antony.resumeparser.ResumeParserProgram resume_value textOutput.json"
=> The whole "class<space>param1<space>param2" is seen as your class: no wonder why it's not found.
Split all your parameters and it will work, subprocess won't group your parameters, no quoting (note the forced conversion of the resume object to str):
cmd = ['java', '-cp', 'bin/:../GATEFiles/lib/:../GATEFiles/bin/gate.jar:lib/*', 'code4goal.antony.resumeparser.ResumeParserProgram', str(resume),'textOutput.json']
After two weeks I got it working.
All files must be copied to the ResumeParser/ResumeTransducer dir.
Also, there's a need to inform the current dir for file parsing.
Here is the implementation:
# first save the file
if form.is_valid():
f = form.save(commit=False)
resume = form.cleaned_data['resume']
f.resume = resume
f.save()
# copy file to the CV parser dir so java can parse the file
cf = "." + f.resume.url
shutil.copy2(cf, 'ResumeParser/ResumeTransducer')
# get file to convert
fl_name = str(f.resume).split('/')[-1]
# get file name to make json output
base_name = os.path.splitext(fl_name)[0]
cmd = "java -cp 'bin/*:../GATEFiles/lib/*:../GATEFiles/bin/gate.jar:lib/*' code4goal.antony.resumeparser.ResumeParserProgram %s %s.json" % (fl_name, base_name)
# get the current working dir
os.chdir("ResumeParser/ResumeTransducer")
# call java
subprocess.Popen(cmd, shell=True)
Thanks Jean-François Fabre!
I run a bash script from my Java program which takes a chunk of data, manipulates it, and splits it up.
It's not a question of whether the bash script works -- I can see the split files in the directory.
Say the original file was "bigFile" in data/
Then
try
{
Process proc = Runtime.getRuntime().exec("bash " + SCRIPT_DIR + "/" + SPLIT_SCRIPT_NAME + " " + args[_MESSAGES_PER_UPLOAD_] + " " + args[_MAXIMUM_MESSAGES_PER_FEED_] + " " + (60000*Integer.parseInt(args[_DURATION_BEFORE_EACH_UPLOAD_IN_MINUTES_])/Integer.parseInt(args[_DURATION_OF_EACH_FEED_IN_MILLISECONDS_])));
proc.waitFor();
}
catch(IOException e) { error(e); }
String fileNames;
File folder = new File(DATA_DIR);
File[] filesToUpload = folder.listFiles();
for (int i = 0; i < filesToUpload.length; ++i)
if (filesToUpload[i].isFile())
{
fileNames = filesToUpload[i].getName();
System.out.println(fileNames);
}
Will print bigFile, not...
$ ls data/
dataChunk_00000
dataChunk_00001
dataChunk_00002
dataChunk_00003
dataChunk_00004
dataChunk_00005
dataChunk_00006
dataChunk_00007
dataChunk_00008
dataChunk_00009
dataChunk_00010
dataChunk_00011
dataChunk_00012
dataChunk_00013
dataChunk_00014
dataChunk_00015
dataChunk_00016
dataChunk_00017
dataChunk_00018
dataChunk_00019
dataChunk_00020
dataChunk_00021
dataChunk_00022
dataChunk_00023
dataChunk_00024
dataChunk_00025
dataChunk_00026
dataChunk_00027
as it should. I'm guessing this is a compiler optimization or something.
Edit: If somebody could explain to me why proc.waitFor() isn't working and/or a better way to solve this, I'd much appreciate it.
The problem with this is not compiler optimization or anything like that.
Its because you are invoking your script with a "bash" in front of it . This causes the process to fork -- so your bash command returns successfully immediately , but your script continues to run in the background and terminate.
The proc.waitFor() has nothing to wait for, the rest of the java program executes before your file has been "split".
You cannot change the directory with java.
If you want to "simulate" it, all you need to do is set the property "user.dir".
I am guessing that your bash script is performing actions asynchronously from its own process/thread. This means that the script finishes executing before the work is complete. This would still pass the waitFor() check and continue executing the code.
EDIT:
Kal's answer explains this more clearly, and it was posted first. The problem is the fact that you use the bash command to execute the script.
I suspect your arguments aren't all passed to your script.
Put all your arguments in an ArrayList instance, pass the instance to the ProcessBuilder, then call the start method on the builder instance, which returns the proc on which you call waitFor.
Here's sample Scala code to show what I mean (I can port it to Java if you're really interested ;-):
import java.lang.{ Process => JProcess, ProcessBuilder => JProcessBuilder }
import java.util.{ArrayList => JArrayList, List => JList, Map => JMap}
import java.io.{InputStreamReader, BufferedReader}
def call(args: String*) = {
val command: JList[String] = new JArrayList[String]()
args.foreach {arg =>
command.add(arg)
}
//log.debug("argument list: %s", command.toString)
val builder = new JProcessBuilder(command)
val proc: JProcess = builder.start()
proc.waitFor()
val read = new BufferedReader(new InputStreamReader(proc.getInputStream()));
val sb: StringBuffer = new StringBuffer()
while(read.ready()) {
sb.append(read.readLine)
}
// sb now has the output of the called process...
val exitValue: Int = proc.exitValue
// http://stuffthathappens.com/blog/2007/11/28/crash-boom-too-many-open-files/
read.close
proc.destroy
(exitValue, sb.toString) // return it
}
Example call in REPL:
scala> call("date")
res156: (Int, java.lang.String) = (0,Mon 18 Jul 2011 22:29:58 BST)
There are a number of wrong assumptions with this program:
Every time you do 'exec' you fork a new process, with its own environment, current directory, etc. Any change of the current directory would have been local to that process and will not affect the parent (your Java process). In other words, there is no way to change the current path of an application using a command in a sub-process, there is no Java API for that either - if you really need this, you have to use native call.
The 'cd' command on Unix is a real command, you do not need the shell in order to run it (unlike Windows).
When you fork a process, you need to make sure that you drain the stdout and stderr, or it is going to block when the OS buffer gets full (see next)
Process.waitFor() works. Always.
A better way to approach the problem is to read carefully the File API and as much as possible work with absolute paths. The 'current directory' is something very usefull when you are in shell, but for applications it ends up being more confusing, so the sooner you resolve it to absolute path - the better.