I am trying to make a subprocess call from my python script to a java class built inside a jar. I am running the python code on a docker container in AWS Batch. I set the CLASSPATH environment variable in the Dockerfile to include the directory containing the jar file.
ENV CLASSPATH /path/to/dir/containing/jar/file
When I pass the entire command with arguments as a string to subprocess, it works fine.
runnable_command = "java $JAVA_OPTS " \
"RunCommand " \
"-b arg_b"
sp = subprocess.Popen(runnable_command,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
close_fds=True,
shell=True,
universal_newlines=True,
env=os.environ)
result, stderr_data = sp.communicate()
print(result)
But for that I had to make the variable "shell=True" which has a security risk. So I modified the variable 'shell=False" and I pass in the command and arguments as a list to the subprocess. This also works fine.
runnable_command = ["/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.x86_64/jre/bin/java", "$JAVA_OPTS", "<ClassName>"]
However, I am setting JAVA_OPTS environment variable in the Dockerfile to pass the log4j configuration file to JVM.
ENV JAVA_OPTS="-Dlog4j.configurationFile=/opt/amazon/lib/log4j2.xml"
This is important because I want to pipe the logs from this java script to my python script.
When I add JAVA_OPTS to the command, it fails with the following error:
runnable_command = ["/usr/lib/jvm/java-1.8.0-jdk-1.8.0.252.b09-2.x86_64/jre/bin/java", "$JAVA_OPTS", "<ClassName>"]
I am not able to pass JAVA_OPTS to the list of args in the subprocess command. It fails to find the log4j.xml file. I followed this question from stackoverflow but it fails with the same error, even after adding the JAVA_OPTS to the "env" argument.
'Error: Could not find or load main class $JAVA_OPTS\n'
Also, when I pass the arguments as a list, I am not able to run 'java' but I am forced to pass the absolute path of java executable.
Can someone help me with the following questions?
How can i pass the log4j configuration to this java command?
Why am I having to pass the absolute path to java command when running subprocess with a list and not a string?
Related
Normally if I were to run .jar file, then I can pass properties like
$ java -Dfoo=bar -jar my_application.jar
And my_application will be able to access the value of foo.
But while trying to run the file generated by distTar it is not able to access the value of foo.
I generated tar file using
$ ./gradlew distTar
$ tar -xvf build/distributions/my_application-1.0.tar
$ my_application-1.0/bin/my_application -Dfoo=bar
My application is not able to get the value of foo.
So turns out we have to pass it as an environment variable. If I open my_application-1.0/bin/my_application, there is comment something like
# Add default JVM options here. You can also use JAVA_OPTS and MY_APPLICATION_OPTS to pass JVM options to this script.
So all I needed to do was to set my arguments in MY_APPLICATION_OPTS in environment variable.
I am trying to integrate a javaagent for application monitoring. I'm using docker and I've setup an OpenJDK base image which is inherited by other application images.
Since javaagent requires a path to the jar file, for maintenance purposes I've defined the path in the base image as another env variable(AGENT_PATH) and I want to reuse the same env variable across all my app images. For some reason the environment variable isn't picked and the application container exits with error.
Base Image's Dockerfile
AGENT_PATH=/agent/agent.jar
This is how I've configured JAVA_TOOL_OPTIONS in application's Dockerfile.
JAVA_TOOL_OPTIONS="-javaagent:$AGENT_PATH + other JVM options"
This is the error message
Picked up JAVA_TOOL_OPTIONS: -javaagent:$AGENT_PATH
Error opening zip file or JAR manifest missing : $AGENT_PATH
Error occurred during initialization of VM
agent library failed to init: instrument
Why is AGENT_PATH not getting substituted properly ?
I've grepped through Hotspot implementation for understanding. I've found this.
This will depend on the Docker step. If you use RUN step the variables processing is not supported:
Unlike the shell form, the exec form does not invoke a command shell. This means that normal shell processing does not happen. For example, RUN [ "echo", "$HOME" ] will not do variable substitution on $HOME. If you want shell processing then either use the shell form or execute a shell directly, for example: RUN [ "sh", "-c", "echo $HOME" ]. When using the exec form and executing a shell directly, as in the case for the shell form, it is the shell that is doing the environment variable expansion, not docker.
In dockerfiles, all directives require a certain codeword. Assigning an environment variable is no different from this and needs the ENV codeword (read here).
Using this, the excerpt of your dockerfile should read
ENV AGENT_PATH=/agent/agent.jar
I was facing the same issue where the environment variables set in the JAVA_TOOL_OPTIONS were not getting substituted. So I had to mention those variables as -D parameters along with java command.
java -Djavax.net.ssl.keyStorePassword=$KeyStore_Secret .....
You can check if that is the case by hardcoding the values in the JAVA_TOOL_OPTIONS and it should work. Meanwhile, I am still looking at the reason for this.
I have the following spring reference jars in a file springClasspath
CLASSPATH=/apps/cab/spring/spring-core-3.1.1.RELEASE.jar:/apps/cab/spring/commons-logging-1.2.jar:/apps/cab/spring/spring-aop-4.2.2.RELEASE.jar:/apps/cab/spring/spring-beans-4.2.2.RELEASE.jar
export CLASSPATH
I want to reference the springClasspath file to my shell script file start_order.sh for my java program to reference the required spring dependencies.How can I do so?
Since you have multiple declarations in the springClasspath file which you are planning to use in the shell script. You need it to source the file in the script to use the variables further.
By sourcing the file in the script, you are making the variables defined in it to be available in the sub-shell in which the script is being run from. For example, in the start_order.sh line after setting the interpreter to bash source the script as below
#!/usr/bin/env bash
# Give the full path to the file if it is present in another location
. ./springClasspath
# with the above source done, the variables can be used as below in your
# script
echo "$CLASSPATH"
Remember export-ing the file won't work in this case, unless your source the file and also the script so that they run in the same parent shell with the variables set.
I have set environment variable by executing the command
export test=abcd
I can see test=abcd when I run printenvcommand
I have deployed a springboot.jar application and I am passing the JAVA_OPTS from the springboot.conf file.
JAVA_OPTS='-Dspring.profiles.active=aaa -Denv=$test'
I started the app by service springboot start . When I check the process, env variable doesn't have the value of $test environment variable.
/usr/bin/java -Dsun.misc.URLClassPath.disableJarChecking=true -Dspring.profiles.active=aaa -Denv=.
How can I access the environment variable in the conf file? I read somewhere the environment variables will be stripped off when run as service. Basically I want to run as service springboot start which internally executes the below command
java -Dspring.profiles.active=aws -Denv=${whatever is set for env environment variable} -jar springboot.jar
I have tried the below configurations but nothing worked
JAVA_OPTS='-Dspring.profiles.active=aaa -Denv='$test
JAVA_OPTS='-Dspring.profiles.active=aaa -Denv='${test}
JAVA_OPTS='-Dspring.profiles.active=aaa -Denv=${test}'
JAVA_OPTS="-Dspring.profiles.active=aaa -Denv=$test"
Be careful about your quotes. Assuming that you use a "normal" shell, variables won't be substituted in single quotes.
java -Dspring.profiles.active=aws -Denv="$myvariable" -jar springboot.jar should lead to env being available in the JVM, no matter if you run it as a service or not.
If you can't get it to work, try to specify a hard coded value like this java -Dspring.profiles.active=aws -Denv=foo -jar springboot.jar. If env is now available in the JVM, your problem is with your shell or run mechanism. Verify that the user who runs the command (i.e. do you use sudo?) has the variable set.
I had the same problem where my .conf was referencing an environment variable which was in the .bashrc.
What I found out is:
The problem is service strips all environment variables but TERM, PATH and LANG which is a good thing. If you are executing the script directly nothing removes the environment variables so everything works.
https://unix.stackexchange.com/questions/44370/how-to-make-unix-service-see-environment-variables
One solution would be to install your app as a systemd service:
https://docs.spring.io/spring-boot/docs/1.3.x-SNAPSHOT/reference/html/deployment-install.html
Or another way is to use docker and you can specify extra configuration in the docker file, like loading a file which contains your environment variables.
As those solutions where not available in my case I ended up with having the value in the .conf file, like: -Denv=prod
I am able to get localhost:16010 running. But, somehow the Hbase shell is not launching when I use :
01HW993798:bin tcssig$ cd /Users/tcssig/Downloads/hbase-1.0.3/bin
01HW993798:bin tcssig$ hbase shell
-bash: hbase: command not found
When I directly launch Hbase Unix executable, it generates the below error log.
Error: JAVA_HOME is not set
Although I have set it. After this only, the localhost:16010 is running.
NOTE : I know there is one similar question, but no relevant answers are present there.
Using this I am able to invoke the command, but now it gives the error :
./hbase: line 403: /Users/tcssig/Downloads/hbase-
1.0.3/bin/JAVA_HOME:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Cont``ents/Home/bin/java: No such file or directory
Although I have java file there.
Your hbase invocation should be like this:
cd /Users/tcssig/Downloads/hbase-1.0.3/bin
./hbase shell [Note the ./]
When you just type hbase shell linux searches for hbase executable in all directories included in PATH environment variable. Since above bin directory is not included it errors out.
Alternatively you can also update your path variable, based on linux distribution, the command to do that may vary. It should be something like:
export PATH=/Users/tcssig/Downloads/hbase-1.0.3/bin:$PATH
Put this command in your .bashrc or .bash_profile and then source this file. That way the bin directory is now included in PATH and hbase command is available.
Go into $HBASE_HOME/bin path, and try:
./hbase shell