I know, that we can execute commands in a BASH using the -c parameter:
bash --login -c "command parameter"
Since we need to be able to execute any command with arbitrary parameters (that can contain spaces, quotes or apostrophes) programmatically (from Java), we would like BASH to treat all parameters as command, something like the -- usually used to separate options from file names, e.g. with Git. Is there a way to execute the above call similar to
bash --login -<some option> -- command parameter
which could be executed from Java easily without escaping command or parameter?
new ProcessBuilder(new String[] {
"bash",
"--login",
"-<some option>",
"--"
command,
parameter
}).start();
-c command must be the last option argument to bash. Any subsequent arguments, whether or not they start with a dash, are assigned to the positional parameters $0, $1, ... in the shell executing the command.
That's usually convenient, since it lets you substitute strings into the command being executed without worrying about quoting them.
Since the first argument following the command is assigned to $0, it is very common to see invocations like
bash -c 'complicated command using "$#"' _ arg1 arg2 arg3
so that arg1 will be assigned to $1 (thus allowing "$#" to work as expected, for example).
As an example:
new ProcessBuilder( new String[] {
"bash",
"--login",
"-<some option>",
"-c",
"for arg in \"$#\"; do frobnicate \"$arg\" 42; done",
"_",
"first thing to frobnicate",
"second thing", ...
}).start();
Related
This question already has an answer here:
Why does Runtime.exec(String) work for some but not all commands?
(1 answer)
Closed 10 months ago.
I am doing an exercise related to Runtime.exec(), I understand that Runtime.exec is not a shell interpreter, that's why I execute "bash -c 'command'" instead, but for some reason, I can execute commands like ls bash -c 'ls' but not echo or redirection or multiple commands. These does not work:
bash -c 'echo 1234'
bash -c 'ls > abc'
bash -c 'ls;id'
bash -c 'ls -al'
Here is my java code:
import java.util.*;
import java.io.*;
public class runtime {
public static void main(String args[]) throws Exception{
String cmd = args[0];
System.out.println(cmd);
Process p = Runtime.getRuntime().exec(cmd);
OutputStream os = p.getOutputStream();
InputStream in = p.getInputStream();
DataInputStream dis = new DataInputStream(in);
String disr = dis.readLine();
while ( disr != null ) {
System.out.println("Out: " + disr);
disr = dis.readLine();
}
}
}
I run the above commands with the syntax:
java runtime "bash -c 'command'"
This works:
$ java runtime "bash -c 'ls'"
bash -c 'ls'
Out: Main.class
Out: Main.java
Out: runtime.class
Out: runtime.java
I am using openjdk 11.0.15 on Ubuntu 20.04 and zsh.
Can anyone tell me why Runtime doesn't work in this case? Thank you!
Because of shell parsing.
These are all concepts that the OS just does not have:
The concept of 'every space separates one argument from another (and the command from the list of arguments'). The concept that a single string can possibly run anything, in fact; at the OS level that's just not what the 'run this thing' API looks like. When you type commands on the command prompt, your shell app is 'interpreting' these strings into a command execution request.
The concept of figuring out that bash means /bin/bash, i.e. $PATH resolution.
The concept that *.txt is supposed to refer to all files that ends in .txt.
The concept that $FOO should be replaced with the value of the environment variable 'FOO'
The concept that ; separates 2 commands, and it's supposed to run both.
The concept that single and double quotes escape things. "Escape things" implies that things can cause interpretation to happen. The OS interprets nothing, therefore there's nothing to escape. Obvious, then, that the OS doesn't know what ' is, or ".
That >foo means: Please set the standard output of the spawned process such that it sends it all to file 'foo'.
In windows shells, that # in front of the command means 'do not echo the command itself'. This and many other things are shellisms: /bin/bash does that, maybe /bin/zsh does something else. Windows's built in shell thing definitely is quite different from bash!
Instead, an OS simply wants you to provide it a full path to an executable along with a list of strings, and pick targets for standard in, standard out, and standard err. It does no processing on any of that, just starts the process you named, and passes it the strings verbatim.
You're sort of half there, as you already figured out that e.g. ls >foo cannot work if you execute it on its own, but it can work if you tell bash to execute it. As ALL of that stuff in the above list? Your shell does that.
It gets more complicated: Turning *.txt into foo.txt bar.txt is a task of bash and friends, e.g. if you attempted to run: ls '*.txt' it does not work. But on windows, it's not the shell's job; the shell just passes it verbatim to dir, and it is the command's job to undo it. What a mess, right? Executing things is hard!
So, what's wrong here? Two things:
Space splitting isn't working out.
Quote application isn't being done.
When you write:
bash -c 'ls >foo'
in a bash shell, bash has to first split this up, into a command, and a list of arguments. Bash does so as follows:
Command: bash
arg1: -c
arg2: ls >foo
It knows that ls >foo isn't 2 arguments because, effectively, "space" is the bash operator for: "... and here is the next argument", and with quotes (either single or double), the space becomes a literal space instead of the '... move to next argument ...' operator.
In your code, you ask bash to run java, and then java to run bash. So, bash first does:
cmd: java
arg1: bash -c 'ls >foo'
With the same logic at work. Your java app then takes that entire string (that's args[0]: "bash -c 'ls >foo'"), and you then feed it to a method you should never use: Runtime.exec(). Always use ProcessBuilder, and always use the list-based form of it.
Given that you're using the bad method, you're now asking java to do this splitting thing. After all, if you just tell the OS verbatim, please run "bash -c 'ls >foo'", the OS dutifully reports: "I'm sorry, but file ./bash -c ;ls >foo' does not exist", because it does not processing at all". This is unwieldy so java's exec() method is a disaster you should never use: Because people are confused about this, it tries to do some extremely basic processing, except every OS and shell is different, java does not know this, so it does a really bad job at it.
Hence, do not use it.
In this case, java doesn't realize that those quotes mean it shouldn't split, so java tells the OS:
Please run:
cmd: /bin/bash (java DOES do path lookup; but you should avoid this, do not use relative path names, you should always write them out in full)
arg1: -c
arg2: 'ls
arg3: >foo'
and now you understand why this is just going completely wrong.
Instead, you want java to tell the OS:
cmd: /bin/bash
arg1: -c
arg2: ls >foo
Note: ls >foo needs to be one argument, and NOTHING in the argument should be containing quotes, anywhere. The reason you write:
/bin/bash -c 'ls >foo'
In bash, is because you [A] want bash not to treat that space between ls and >foo as an arg separator (you want ls >foo to travel to /bin/bash as one argument), and [B] that you want >foo to just be sent to the bash you're spawning and not to be treated as 'please redirect the output to file foo' at the current shell level.
Runtime.exec isn't a shell, so the quotes stuff? Runtime.exec has no idea.
This means more generally your plan of "I shall write an app where the entire argument you pass to it is just run" is oversimplified and can never work unless you write an entire quotes and escaper analyser for it.
An easy way out is to take the input, write it out to a shell script on disk, set the exec flag on it, and always run /bin/bash -c /path/to/script-you-just-wrote, sidestepping any need to attempt to parse anything in java: Let bash do it.
The ONE weird bizarro thing I cannot explain, is that literally passing 'ls' to /bin/bash -c, with quotes intact, DOES work and runs ls as normal, but 'ls *' does not, perhaps because now bash thinks you want executable file /bin/ls * which obviously does not exist (a star cannot be in a file name, or at least, that's not the ls executable, and it's not an alias for the ls built-in). At any rate, you want to pass ls without the quotes.
Let's try it!
import java.util.*;
import java.io.*;
public class runtime {
public static void main(String args[]) throws Exception{
ProcessBuilder pb = new ProcessBuilder();
pb.command("/bin/bash", "-c", "echo 1234");
// pb.command("/bin/bash", "-c", "'echo 1234'");
Process p = pb.start();
OutputStream os = p.getOutputStream();
InputStream in = p.getInputStream();
DataInputStream dis = new DataInputStream(in);
String disr = dis.readLine();
while ( disr != null ) {
System.out.println("Out: " + disr);
disr = dis.readLine();
}
int errCode = p.waitFor();
System.out.println("Process exit code: " + errCode);
}
}
The above works fine. Replace the .command line with the commented out variant and you'll notice it does not work at all, and you get an error. On my mac I get a '127' error; perhaps this is bash reporting back: I could not find the command you were attempting to execute. 0 is what you're looking for when you invoke waitFor: That's the code for 'no errors'.
In both Java and C++ arguments are wrapped by ' like :
system("cmd arg1 arg2");// C++
Runtime.getRuntime().exec("cmd arg1 arg2");// Java
new ProcessBuilder("cmd","arg1","arg2").start();// Java
//=============== output
cmd 'arg1' 'arg2'
Which prevents arguments to be interpreted as wildcard like: l* will be interpreted as "lstar" not "l{everything-is-possible}"
Also if I add a ' ,it will converted to \'
How can I pass wildcard via these programming languages?
It is the shell (bash, csh) etc that handles wildcards and passes specific arguments to your process. For example you may run ls *.txt - the shell expands *.txt and calls ls a.txt b.txt c.txt (depending on what files exist in the current directory).
Calling exec doesn't invoke any shell and so there is no "wildcard handling".
Processes could do their own wildcard handling (if (arg.equals("*")) { ...) but I think it is pretty rare.
So either the calling process has to do the matching and pass the correct args to exec or you need to invoke exec with a shell like Runtime.getRuntime().exec("bash -c ls *.txt");
Hello I am trying to execute following :
Process p = null;
StringBuffer rbCmd = new StringBuffer();
rbCmd.append("rsync -e \"ssh -i /root/.ssh/key\" -va --relative /home/lego/hyb/abc/PVR2/Testdata/./R887/SCM/System root#myMachine:/xyz/data/SCMdata/");
p = Runtime.getRuntime().exec(rbCmd.toString());
But I am getting following error on command line.Command executes correctly on command line
Missing trailing-" in remote-shell command.
rsync error: syntax or usage error (code 1) at main.c(361) [sender=3.0.6]
Issue is because of double quotes inside the command where I mention ssh key.
Please help with correction.
Your approach won't work because Runtime.exec() does not realize that "ssh -i /root/.ssh/key" is a single argument to rsync. Escaping the double-quotes keeps the compiler happy, but doesn't remove the fundamental problem, which is the limits of the built-in tokenizer.
You might have more luck with something like this:
Process p = Runtime.getRuntime().exec
(new String[]{"rsync", "-e", "ssh -i /root/.ssh/key", "-va" "--relative" ... });
That is, tokenize the command line yourself, and form the individual tokens into a String[]. You're deciding in advance what the arguments to rsync are, rather than allowing exec() to figure it out (wrongly).
Don't forget that if rsync produces any output, you'll need to arrange for your application to consume its stdout and stderr, or it could stall.
I'm doing some benchmark testing of command line execution of postgres queries vs. JDBC using the Lehigh Benchmark Dataset as test data, including their test queries.
Everything else is working fine, except when I try to execute a single string query using psql.
The query is the string I'm creating and executing using
Runtime.getRuntime().exec("time ./psql -U lehigh -d lehigh -c 'SELECT u.id FROM undergraduatestudent u;';")
It works fine on the mac command line, but when run from within Java, I get the following errors:
---> psql: warning: extra command-line argument "u.id" ignored
---> psql: warning: extra command-line argument "FROM" ignored
---> psql: warning: extra command-line argument "undergraduatestudent" ignored
---> psql: warning: extra command-line argument "u;';" ignored
---> ERROR: unterminated quoted string at or near "'SELECT"
---> LINE 1: 'SELECT
---> ^
Essentially everything after "SELECT" isn't being recognized as part of the query. There are 14 queries where this is happening, but this is the shortest one, and will suffice to solve the problem.
I've tried moving the -U and -d to afterwards, as was suggested in a different post, but I got the same results.
I'm running this in Eclipse on a Mac in Capitan (though I don't believe either of these things are having an effect on things).
I'm aware of the -f option, but I have to time this per query, so I prefer the -c option.
As far as I am aware, exec(String) uses spaces as a delimiter, so your query isnt being treated as a single arguments, but rather as a set of arguments ('SELECT, u.id, FROM, undergraduatestudent, and u;').
Try using the exec(String[]) version, where the String[] is like this:
{"time", "./psql", "-U", "lehigh", "-d", "lehigh", "-c", "'SELECT", "u.id", "FROM", "undergraduatestudent u;'"}`
Or, better yet, use a ProcessBuilder:
new ProcessBuilder(new String[]{"time", "./psql", "-U", "lehigh", "-d", "lehigh", "-c", "'SELECT", "u.id", "FROM", "undergraduatestudent u;'"}).start();
I'm trying to execute the following command (which works on the terminal) within Java:
awk -F';' 'NR>1{gsub(/; +/,";",$0);printf("{msisdn:\"%s\",imei:\"%s\",brand:\"%s\",model:\"%s\",sap:\"%s\",sap_cod:\"%s\",file_name:\"teste\",company:{\"$ref\":\"company\",\"$id\":ObjectId\"456\")}}\n",$2,$15,$16,$17,$18,$20)}' /Users/milena/Desktop/giant.csv
The Java code I am using is this:
String fileName = "test";
String company = "456";
String awk = "awk -F';' 'NR>1";
String gsub = "{gsub(/; +/,\";\",$0);";
String printf = "printf(\"{msisdn:\\\"%s\\\",imei:\\\"%s\\\",brand:\\\"%s\\\","
+ "model:\\\"%s\\\",sap:\\\"%s\\\",sap_cod:\\\"%s\\\",file_name:\\\""+fileName+"\\\",company:"
+ "{\\\"$ref\\\":\\\"company\\\",\\\"$id\\\":ObjectId\\\""+company+"\\\")}}\\n\",$2,$15,$16,$17,$18,$20)}\' ";
String path = "/Users/milena/Desktop/giant.csv";
String command = awk + gsub + printf + path;
Process p
p = Runtime.getRuntime().exec(command);
The error I am getting is:
awk: syntax error at source line 1
context is
>>> ' <<<
missing }
missing )
awk: bailing out at source line 1
Any ideas of what am I doing wrong?
When you use Runtime.getRuntime().exec(command), it takes the string and breaks it into a command and arguments based on spaces.
This simple parsing is not the same as the parsing done by the shell when you invoke the command. The shell, for example, takes quotes into consideration. This means that if you have a command line like:
cmd 'abc' 'def'
The arguments that the shell will send to the cmd command are going to be abc and def. But if you give the same command to Runtime.getRuntime().exec(command), it will send 'abc' and 'def' to cmd as arguments. Yes, including the quotes!
The situation gets worse if you have spaces in any of the arguments. If the shell gets
cmd 'my single argument'
It will invoke the command with a single argument my single argument. But Runtime.getRuntime().exec(command) will invoke cmd with three arguments: 'my, single and argument'!
So it's not recommended to use this particular overload for anything but very simple commands. Instead, you should use the overload that accepts an array of strings. The first element should be the command name, and each argument should be in a separate element:
String[] command = { "awk",
"-F;",
"NR>1{gsub(/; +/,\";\",$0);printf(\"{msisdn:\\\"%s\\\",...",
"/Users/milena/Desktop/giant.csv"
};
Process p = Runtime.getRuntime().exec(command);
Note: It is recommended to use the ProcessBuilder class to build a Process rather than Runtime.getRuntime.exec - it gives you better control of the command and you can pass the separate arguments right to the constructor.