What is the accepted, portable way to include interpreter options in the shebang line, ie. how can I do something like
#!/usr/bin/env python -c
or (more importantly) something like
#!/usr/bin/env java -cp "./jars/*:./src" -Xmn1G -Xms1G -server
and get it to be parsed correctly? Right now ubuntu seems to just glom the whole thing together, although other systems will parse this with no problem.
http://en.wikipedia.org/wiki/Shebang_%28Unix%29
describes the problem but offers no solution.
There's no good solution, as different unices treat multi-word #! lines differently. Portable #! use limits you to at most one argument to the interpreter on the #! line, and no whitespace in the interpreter or argument.
If the language allows it, you can make the script a shell script which takes care of loading the interpreter with whatever command line it likes. For example, in Perl, from the perl manual:
#!/bin/sh -- # -*- perl -*- -p
eval 'exec perl -wS "$0" ${1+"$#"}'
if $running_under_some_shell;
The shell stops processing after the second line, and Perl sees lines 2–3 as an instruction that does nothing. Some lisp/scheme dialects make #!...!# a comment, allowing you to write
#!/bin/sh
exec guile -s "$0" "$#"
!# ;; scheme code starts here
In general, the only solutions involve two files. You can write #!/usr/bin/env mywrapper where mywrapper is a program (it can be a script) that calls the actual interpreter with whatever argument it wants. Or you can make the executable itself the wrapper script and keep the interpreted file separate. The second solution has the advantage of working even if the interpreter doesn't accept a leading #! line.
Related
Is this a simple and good way to execute a Shell command via Java?
Runtime.getRuntime().exec( some command );
Or is this bad practice?
It depends.
The original purpose and basic functionality of a Unix shell is to let you run programs, optionally passing them arguments. For example the command ls runs the ls program, and the command grep foo bar runs the grep program with the arguments foo and bar. If your command only runs a (fixed) program with fixed if any arguments, Runtime.exec can do it. There are two subcases:
the overloads taking a String parse the line into 'words' (program name and arguments) using any whitespace; this is essentially the same as the default parsing (with no quoting) done by standard shells.
if you need any different parsing, for example if your command would use any quoting in shell, you must do that parsing yourself and pass the results to one of the overloads taking a String[].
But note that when you run a program from an interactive shell -- one using a terminal or equivalent (sometimes called a console) for input and output -- the program's input and output default to that terminal. The I/O for a program run by Runtime.exec is always pipes from and to the Java process, and some programs behave differently when their input and/or output is/are pipe(s) -- or file(s) -- instead of a terminal. Plus you must write code to send (write) any desired input and receive (read) any output. Of course, shells can be and sometimes are run without a terminal too.
However, shells can be and routinely are used to do much much more than the basics:
shell can execute commands with contents different from the input by variable (formally parameter) substitution (possibly with modification/editing), command substitution, process substitution, special notations like squiggle and bang, and filename expansion aka 'globbing' (so called because in the early versions of Unix it was done by a separate program named glob). Runtime.exec doesn't do these, although you can write Java code to produce the same resulting command execution by very different means.
shell executes some commands directly in the shell rather than by running a program, because these commands affect the shell process itself,
like cd umask ulimit exec source/. eval exit alias/unalias, or variables in the shell like set shift unset export local readonly declare typeset let read readarray/mapfile,
or child process like jobs fg bg, or special parsing like [[ ]] and (( )) (in some shells). These are called 'builtin' and Runtime.exec can't do them,
with two partial exceptions: it can run a program with a different working directory and/or env var settings, equivalent to having previously executed cd or export or equivalent.
Shell also often has builtins that duplicate, or modify, a 'normal' program; these commonly include test/[ echo printf kill time. Runtime.exec can only do the program version, not the builtin version.
shell has control structures (compound commands) like if/then/else/elif/fi and while/for/do/done and trap && || ( ) { }. Runtime.exec can't do these, although in some cases you could use Java logic to produce the same results.
shell can also have user-defined functions and aliases that can be used as commands; Runtime.exec does not.
shell can redirect the I/O of programs it runs, including forming pipes. Runtime.exec can't do these, but see below.
Since 1.5, Java also has ProcessBuilder, which provides the same functionality and more, in a more flexible and arguably clearer API, and thus is generally recommended instead. ProcessBuilder does support redirecting I/O for the program it runs, including using the terminal/console if the JVM was run on/from one (which is not always the case), and since 9 it can build a pipeline. It does not have the word-splitting functionality of Runtime.exec(String) but you can easily get the same result with string.split("[ \t]+") or in most cases just " +".
Note shell is itself a program, so you can use either Runtime.exec or ProcessBuilder to run a shell and pass it a command, either as an argument using option -c (on standard shells at least) or as input, and unsurprisingly this shell command can do anything a shell command can do.
But this can be a portability issue because different systems may have different shells, although any system claiming Unix certification or POSIX conformance must have a shell named sh that meets certain minimum requirements.
The actual shell used on different systems might be any of bash dash ksh ash or even more. OTOH this is true for other programs as well; some programs that typically differ significantly on different systems are awk sed grep and anything to do with administration like netstat.
A few of the existing Qs that show shell commands that don't work in Runtime.exec at least as-is:
a command for sherlock.py is interpreted differently from linux command line and java process api
Execute shell script multiple commands in one line using Process Builder in Java (Unix)
Check in Java if a certain application is in focus
Problem in executing command on AIX through Java
ProcessBuilder doesn't recognise embedded command
File not Found when executing a python scipt from java
Java system command to load sqlite3 db from file fails
Curl To Download Image In JAVA
Keytool command does not work when invoked with Java
use javap from within a java program on all the files
Using SSMTP and ProcessBuilder
Process Builder Arguments
Whitespace in bash path with java
java.lang.Runtime exception "Cannot run program"
Why does Runtime.exec(String) work for some but not all commands?
How to save Top command output in a text or csv file in java?
Execute bash-command in Java won't give a return
Using Java's Runtime.getRuntime().exec I get error with some commands, success with others -- how can this be determined?
Java and exec command - pipe multiple commands
Java exec() does not return expected result of pipes' connected commands
How to make pipes work with Runtime.exec()?
How to use Pipe Symbol through exec in Java
In Runtime.getRuntime().exec() getting error: /bin/bash: No such file or directory
Java exec linux command
How to use pipes in a java Runtime.exec
Java Runtime.getRuntime().exec and the vertical bar
Whenever I execute terminal command from code it gives "cannot run program" error=2 No such file or directory
Command line proccess read linux in java
Java Command line system call does not work properly
I have a powershell script that is working and runs a java executable. Before I was generating a bunch of powershell script files that were run through the command prompt. Now I want to make it so there does not need to be file creation.
Here is what the line looks like from the working (.ps1) file:
java <mem opts here> "-Doption1=3" "-Doption2=`` ` ``"true`` ` ``" jar.exe
I want to be able to do something like this in command prompt:
Powershell -Command "java <mem opts here> "-Doption1=3" "-Doption2=`` ` ``"true`` ` ``" jar.exe"
Even just asking this question I am having problems with the escape characters. What is the proper way to handle escape characters when you have quotes in quotes in quotes when calling java through powershell through command prompt? (I understand it is a bit messy)
You can lessen the quoting headaches if you focus just on the parts that require quoting (assuming that option value true truly needs quoting):
REM From cmd.exe
C:\> powershell -Command java -Doption1=3 -Doption2="'\"true\"'" jar.exe
The above will make java.exe see:
java -Doption1=3 -Doption2="true" jar.exe
As you can see, even this simple case of the desired resultant quoting is quite obscure, because you're dealing with 3 layers of interpretation:
cmd.exe's own interpretation of the initial command line
PowerShell's interpretation of the arguments it receives.
How PowerShell's translates the arguments into a call to an external program (java.exe).
In any event, the final layer of interpretation is how the target program (java.exe) parses the command line.
That said, in order to call java.exe, i.e. an external program, you don't need PowerShell at all; invoke it directly:
REM From cmd.exe
C:\> java -Doption1=3 -Doption2="true" jar.exe
I'm trying to restart process when OOME happens. Java binary is launched using two shell scripts, one of them imports other. I don't have any control of the first one but can modify the second one as I want.
This is a prototype what I'm trying to do:
First shell script test.sh:
#!/bin/sh
JAVA_OPTS="$JAVA_OPTS -Xmx10m"
. test1.sh
echo $JAVA_OPTS
java $JAVA_OPTS $es_params TestMemory
Second shell script test1.sh:
#!/bin/sh
pidfile="test.pid"
touch $pidfile
params="$parms -Dpidfile=$pidfile"
kill_command="kill -9 \$(cat $pidfile)"
dir=$( cd $(dirname $0) ; pwd -P )
path="$dir/$(basename $0)"
start_command="$path $#"
restart_command="$kill_command;sleep 2;$start_command"
JAVA_OPTS="$JAVA_OPTS -XX:OnOutOfMemoryError=\"$restart_command\""
Generally what it does is JAVA_OPTS is constructed inside test1.sh and then used to run Java binary, which just writes PID in pidfile and then creates OOME.
Problem happens during execution, java can't understand what is a parameter and what is a class to run. I think it might be a problem of quoting, I tried different ways to escape JAVA_OPTS, but without any result. I'm either getting:
Unrecognized option: -9
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Or
Error: Could not find or load main class "-XX:OnOutOfMemoryError=kill
If I just take a value of JAVA_OPTS and put it manually in test.sh it runs perfectly.
Any ideas how can I change test1.sh to make it work? I think I tried almost every possible way of putting double and single quotes, but without any success. Also if I put restart_command in restart.sh file and use it instead of the variable, it works fine.
After running set -x I saw that shell modifies every single space character to ' ' - adds ' on both sides. Escaping doesn't gives any result. Any idea how to avoid this? So final commend is:
+ java -Xmx10m '"-XX:OnOutOfMemoryError=kill' '$(cat' 'test.pid);sleep' '2;/Users/davidt/test/TestMemory/bin/test.sh' '")' -Des.pidfile=test.pid TestMemory
Update
I can run simplified command successfully
java "-XX:OnOutOfMemoryError=echo 'Ups'" $es_params TestMemory
But it seems a general problem, shell just hates spaces into variables I guess:
JAVA_OPTS="\"-XX:OnOutOfMemoryError=echo 'Ups'\""
set -x
java $JAVA_OPTS TestMemory
This script fails and the last line is interpreted as:
java '"-XX:OnOutOfMemoryError=echo' ''\''Ups'\''"' TestMemory
I tried different options to escape
This is a shell problem. Based on the evidence, I'd say that one of the ; characters ... and possibly some why space ... is being interpretted by the shell when you don't want / need this to happen.
If you run set -x in the shell before running the command that is trying to start the JVM, you will see the actual command that is being used.
It seems shell translates every single space to ' ',
Not exactly. The single quotes are inserted by the shell into the output you are getting from set -x. They simply indicating where the argument boundaries are. They are not really there ... and they are certainly NOT being passed to the java command.
Any idea how to [a]void it?
What you need to do is start from the (final) command that you are trying execute ...
java -Xmx10m -XX:OnOutOfMemoryError="kill NNNN;sleep 2;/Users/davidt/test/TestMemory/bin/test.sh" -Des.pidfile=test.pid TestMemory
... and work backwards, so that the shell variables, expansions and escaping give you what you need.
The other thing to note is that this:
java -Xmx10m -XX:OnOutOfMemoryError="kill $(cat test.pid); ..."
probably won't work. The kill $(cat test.pid) command is using shell syntax and requires shell functionality to interpolate the contents of the PID file. I doubt that the JVM is going to know what to do with that. (Or more accurately. It will do what you have literally told it to do, but that will not be what you want ...)
If you really need to interpolate the pid file content when the restart command is run as you appear to be trying to do, then suggest that turn the restart command into a free-standing shell script, and set the file mode so that it is executable. It will be simpler and a lot easier to get working.
As a general piece of advice, is is a bad idea to be too clever with shell scripts. The exact semantics of variable expansion and command parsing are rather tricky, and it is easy to get yourself really confused ... if you are trying to do this at multiple levels.
I ended up put the script I wanted to execute in a separate file and gave it as a parameter to JVM to execute when OOME happens.
echo "echo 'UPS'" >> oome_happened.sh
JAVA_OPTS="\"-XX:OnOutOfMemoryError='oome_happened.sh'\""
set -x
java $JAVA_OPTS TestMemory
Like #DaTval said, you should put the command in a script. The script should be someting like.
#!/bin/bash
kill -9 $PPID
Kill the caller of scripts.
I just want to be able to do ./whatever.jar instead of java -jar whatever.jar.
I've found a way:
#!/bin/bash
java -jar $0 $*
exit
# jar goes here...
but it doesn't work. Java just complains that it's an invalid/corrupt jarfile.
I also tried piping:
#!/bin/bash
tail -n +4 $0 | java -jar
exit
# jar goes here...
but this doesn't work.
One way to do it is to somehow split the file into two separate parts (the script part and the jar part), and then execute the jar, but that'd be redundant. You'd might as well make a script that executes the jar and execute that script.
So I need to figure out how to somehow tail it and fake the file.
I thought I could do it using /dev/stdout:
#!/bin/bash
java -jar /dev/stdout
tail -n +5 $0
exit
# jar goes here...
That doesn't work either. It just prints the contents of the jar and java complains that it's invalid. (I figured out later that there's nothing to read in /dev/stdout)
So I need to read from stdout some other way. I really wish I could pipe it though. It would make things SO much easier :)
You need a service called jexec some linux distros come with this installed check for /etc/init.d/jexec. My CentOS 5.5 definitely does.
What it does is register the jexec interpreter with the binfmt system.
For more information you might what to have a quick read of binfmt_misc.
Assuming you have the kernel source code installed, check out /usr/src/linux/Documentation/java.txt for a way to run Java code directly using the kernel's BINFMT_MISC support (assuming it's compiled into the version of the kernel you're running, but I think it is on most major distros). If you don't have the source installed, you should be able to find it online easy enough (here's one example).
FYI, if you wanted to do it your original way it would go like this:
$ cat jar.sh
#!/usr/bin/env bash
java -jar <(tail -n +4 "$0")
exit
$ cat jar.sh runme.jar > works.jar
$ chmod a+x works.jar
$ ./works.jar
Presuming a recent bash with support for <()
java -jar does not work with stdin, apparently it does some seeks rather than straight reads.
On a system you can't mod, you have to use a tmp. for example.
#!/bin/bash
JF=/tmp/junk$$.jar
(uudecode -o /dev/stdout >$JF;java -jar $JF;unlink $JF) <<JAR
begin-base64 644 junk.jar
UEsDBBQACAAIAEaBz0AAAAAAAAAAAAAAAAAJAAQATUVUQS1JTkYv/soAAAMA
UEsHCAAAAAACAAAAAAAAAFBLAwQUAAgACABGgc9AAAAAAAAAAAAAAAAAFAAA
AE1FVEEtSU5GL01BTklGRVNULk1G803My0xLLS7RDUstKs7Mz7NSMNQz4OVy
LkpNLElN0XWqBAmY6RnEG5koaASX5in4ZiYX5RdXFpek5hYreOYl62nycvkm
ZubpOuckFhdbKWSV5mXzcvFyAQBQSwcIBHn3CVkAAABZAAAAUEsDBBQACAAI
AEd+z0AAAAAAAAAAAAAAAAAKAAAAanVuay5jbGFzc21QTUvDQBB926RJE6Ot
ramfhXoQogcDXiteBPFQVIjowdOmXcrGZCMxEfxZelDw4A/wR4mzUShCF3Zn
9s2bt2/26/vjE8ARBi4stB10sNpC10UPazZ8G30G61gqWZ4wGMH+DYN5mk8F
Q3sslbioslgU1zxOCTEzLhVDP7gbJ/yJhylXszAqC6lmI93oRnlVTMSZ1GQn
qdT9oeZ5sNGyse5hA5sM3rlI03x4mxfpdNfGlodt7JC45jN05sqXcSIm5T8o
en4sRUZG84oK/q8NmYdX5KEkJ4JnI4beApjBftC3lAbwg0X+MUSTvkivBm3y
DJqCsgFFRrF58A72QglNSqdVg5qyBO+PuketGnVe0egabzDndLdWNUjVJGS5
fmXlB1BLBwjDUWL/IAEAAJ4BAABQSwECFAAUAAgACABGgc9AAAAAAAIAAAAA
AAAACQAEAAAAAAAAAAAAAAAAAAAATUVUQS1JTkYv/soAAFBLAQIUABQACAAI
AEaBz0AEefcJWQAAAFkAAAAUAAAAAAAAAAAAAAAAAD0AAABNRVRBLUlORi9N
QU5JRkVTVC5NRlBLAQIUABQACAAIAEd+z0DDUWL/IAEAAJ4BAAAKAAAAAAAA
AAAAAAAAANgAAABqdW5rLmNsYXNzUEsFBgAAAAADAAMAtQAAADACAAAAAA==
====
JAR
Or I could just install the jarwrapper (Ubuntu) package.
Write a separate shell script:
whatever.sh
#!/bin/bash
java -jar whatever.jar $*
You can't make the JAR file directly executable because it's not an executable file. It's Java bytecode which can't be read directly by the machine nor any standard shell interpreter that I know of.
First, I already googled but only found examples where a compressed file (say a .tar.gz) is embedded into a shell script.
Basically if I have a C program (hello.c) that prints a string, say Hello World!.
I compile it to get an executable binary
gcc hello.c -o hello
Now I have a shell script testEmbed.sh
What I am asking is if it is possible to embed the binary (hello) inside the shell script so that when I run
./testEmbed.sh
it executes the binary to print Hello World!.
Clarification:
One alternative is that I compress the executable into an archive and then extract it when the script runs. What I am asking is if it is possible to run the program without that.
Up until now, I was trying the method here. But it does not work for me. I guess the author was using some other distribution on another architecture. So, basically this did not work for me. :P
Also, if the workflow for a C program differs from a Java jar, I would like to know that too!
Yes, this can be done. It's actually quite similar in concept to your linked article. The trick is to use uuencode to encode the binary into text format then tack it on to the end of your script.
Your script is then written in such a way that it runs uudecode on itself to create a binary file, change the permissions then execute it.
uuencode and uudecode were originally created for shifting binary content around on the precursor to the internet, which didn't handles binary information that well. The conversion into text means that it can be shipped as a shell script as well. If, for some reason your distribution complains when you try to run uuencode, it probably means you have to install it. For example, on Debian Squeeze:
sudo aptitude install sharutils
will get the relevant executables for you. Here's the process I went through. First create and compile your C program hello.c:
pax> cat hello.c
#include <stdio.h>
int main (void) {
printf ("Hello\n");
return 0;
}
pax> gcc -o hello hello.c
Then create a shell script testEmbed.sh, which will decode itself:
pax> cat testEmbed.sh
#!/bin/bash
rm -f hello
uudecode $0
./hello
rm -f hello
exit
The first rm statement demonstrates that the hello executable is being created anew by this script, not left hanging around from your compilation. Since you need the payload in the file as well, attach the encoded executable to the end of it:
pax> uuencode hello hello >>testEmbed.sh
Afterwards, when you execute the script testEmbed.sh, it extracts the executable and runs it.
The reason this works is because uudecode looks for certain marker lines in its input (begin and end) which are put there by uuencode, so it only tries to decode the encoded program, not the entire script:
pax> cat testEmbed.sh
#!/bin/bash
rm -f hello
uudecode $0
./hello
rm -f hello
exit
begin 755 hello
M?T5,1#$!`0````````````(``P`!````$(,$"#0```#`!#```````#0`(``'
M`"#`'#`;``8````T````-(`$"#2`!`C#````X`````4````$`````P```!0!
: : :
M:&%N9&QE`%]?1%1/4E]%3D1?7P!?7VQI8F-?8W-U7VEN:70`7U]B<W-?<W1A
M<G0`7V5N9`!P=71S0$!'3$E"0U\R+C``7V5D871A`%]?:38X-BYG971?<&-?
4=&AU;FLN8G#`;6%I;#!?:6YI=```
`
end
There are other things you should probably worry about, such as the possibility that your program may require shared libraries that don't exist on the target system, but the process above is basically what you need.
The process for a JAR file is very similar, except that the way you run it is different. It's still a single file but you need to replace the line:
./hello
with something capable of running JAR files, such as:
java -jar hello.jar
I think makeself is what you're describing.
The portable way to do this is with the printf command and octal escapes:
printf '\001\002\003'
to print bytes 1, 2, and 3. Since you probably don't want to write that all by hand, the od -b command can be used to generate an octal dump of the file, then you can use a sed script to strip off the junk and put the right backslashes in place.