I'm running Apache Airflow using Docker. I have installed OpenJDK12 and defined its variables in both of:
~/.bachrc
/usr/local/airflow/.bachrc
export JAVA_HOME=/usr/local/src/jdk-12
export PATH=$PATH:$JAVA_HOME/bin
and I can see Java is running normally especially when I run the Task from the command line :
airflow test ip-importer Download 2020-2-19
where it's working properly without errors , where the task is for :
Executing java command to run a Jar file
But now when I schedule the task on the webserver UI and when I run it, It doesn't run properly and gives me this error:
[2020-02-18 00:46:39,941] {{bash_operator.py:126}} INFO -
/tmp/airflowtmpdzijcupu/Download886rmbol: line 1: java: command not
found
so seems like It's unable to see the defined paths for Java.
Any hints / ideas for how to solve this issue ?
Another question , In case if the java application has thrown an exception/error , Airflow doesn't assume it as an error , it assumes it as a succeeded task , Could you advise how to enforce airflow to assume the exception as a real interrupter for the task ?
Check if /usr/bin folder from Docker container has the executable file pointing out to Java cmd. If not, create one with the following cmd:
Example:
ln -s /usr/local/airflow/java/jdk1.8.0_161-x64/bin/java /usr/bin/java
Related
I have a Java Code where I am able to run it on Intellij using custom configuration. The configuration has following attributes :-
module : java 8 (Oracle OpenJDK 1.8.0_321)
classpath : -cp XYZ.main()
main : com.ABC.XYZ.ManageTraffic
CLI arguements : server XYZ.yml
But when I try to run the jar that was build using gradle from terminal , it gives me Error , could not find or load main class com.ABC.XYZ.ManageTraffic
So far I have tried the following things looking at other solutions at Stackoverflow , still getting the same error
java -jar ques.jar
java -jar ques.jar com.ABC.XYZ.ManageTraffic
java -cp /build/libs/ques.jar com.ABC.XYZ.ManageTraffic
Just to cross check , I unzipped the creataed jar and found that com.ABC.XYZ.ManageTraffic class file is available there , still getting error. What could be the issue?
Run it from the IDE, and while it is running, try to get the command used by the IDE from the process list.
not sure what OS you are using but something like this should work on linux/mac:
ps -ef | grep java
After you have the command you can try to understand why its not working for you, or just use that command
Just want to add how I managed to run it. I created a new shaded jar file of the same application. refreshed its dependencies and now it works. I am yet to find out how creating a shaded jar instead of normal jar helped. Right now the only reason I could figure out is there may be version clashes with some dependencies but I wonder how it could throw could not found main class error.
Anyways , then I ran the file with the following command from terminal:
java -jar ./build/libs/ques-shaded.jar server XYZ.yml
I'm learning to deploy Spring Boot apps on AWS EC2. And I know how to automate app launch, when I start the EC2 instance, I don't need to manually use the command java -jar java-service.jar, I just add this command in the /etc/rc.local file and that is all. But I have 2 microservice, and I want to start both of them automatically, but if I try to add both commands in the /etc/rc.local it's not working, only the first service will start, the second service will not start.
So I have the commands added like this:
And after I start the EC2 instance only the first service is started:
Thank you!
I am not a unix expert, but I see the only issue in running 2 java commands from terminal is that unless the first command returns, the next command is not executed. So, I think the solution would be run the 1st command in some interactive mode so that the other commands can be executed simultaneously.
There are ways in unix shell to run a command in background. I found this useful link - https://www.maketecheasier.com/run-bash-commands-background-linux/
In bash terminal, a command can be made to run in background by appending it with &. So, I think you should be able to start both jars if you do something like -
java -jar /home/ec2-user/first.jar &
java -jar /home/ec2-user/second.jar
I recommend to use Systemd.
Create a Systemd unit file for every microservice, save it in /etc/systemd/system/my-app.service. Something like that:
[Unit]
Description=My Java app
After=syslog.target network.target
[Service]
EnvironmentFile=/etc/sysconfig/my-app-env
WorkingDirectory=/my/app/home
ExecStart=/usr/bin/java $JAVA_OPTS -jar my-app.jar
KillMode=process
User=my-app-user
Restart=on-failure
[Install]
WantedBy=multi-user.target
Then, run:
systemctl daemon-reload
systemctl enable --now my-app
After that, you can use:
systemctl status my-app
systemctl stop my-app
systemctl start my-app
Another solution is to bundle your jars into Docker images. This of course requires Docker runtime and adds an overhead, but it also has some benefits:
Complete separation of jar files. Easily use different java versions.
No need to worry about differences of local and ec2 environment.
Easily scale to 3 or more jars.
Use Docker Cli to build and start containers. Works great in a Devops Pipeline.
You can read here to learn how to create Spring Boot Docker images. After you build an image. You start it like this.:
docker run -p 8080:8080 springio/gs-spring-boot-docker
You can run as many docker run commands you need, one after another.
I am not sure which system you are using in starting application:
For linux base system, you can use crontab to schedule the task when the server reboot.
Follow this steps:
Download crontab
#apt-get install cron
Edit the file file to enable the task
crontab -e
(Choose Vim or nano to edit the task)
Add this code to your server
#reboot /usr/bin/java -jar XXXXX.jar
Save your file
Check the result
crontab -l
#systemctl status cron
This method works in my Debian system. For more details, you can refer to
How to automatically run program on Linux startup
If you are running from bash, then join two jar commands with "&" like below.
java -jar /home/ec2-user/first.jar&java -jar /home/ec2-user/second.jar
coupon service
Run the command 'java -jar /home/ec2-user/coupon-service-0.0.1-SNAPSHOT.JAR'
Press CTRL+Z, type bg, press Enter, type disown, press Enter.
product service
Run the command 'java -jar /home/ec2-user/product-service-0.0.1-SNAPSHOT.JAR'
Press CTRL+Z, type bg, press Enter, type disown, press Enter.
NOTE: Both services should have different ports.
I can run kubectl get secret/rabbitmq-expressmode -n expressmode in shell successfully.
However when I try to run it in java with either ProcessBuilder or Runtime.getRuntime().exec, the error:
Error from server (BadRequest): the server rejected our request for an unknown reason
is thrown out.
What's the possible reason?
Best approach would be to use official java client library.
Installation is pretty easy, and code examples shows you how to use it.
If the official one does not meet your requirements, there are also community-maintained client libraries:
Java (OSGi)
Java (Fabric8, OSGi)
Java
I had same issue but resolved by programmatically creating bash file containing kubectl... command I was trying to execute directly.
For example, create bash file and put your kubectl command there (i.e.: some-file.sh). Then try then executing it in bash process running within your java code (bash some-file.sh).
I am trying to run the following command:
java -jar jar-from-application.jar --file myown.properties
Couple of problems I faced earlier while trying to run this on target machine. Tomcat needs to be running and a war file needs to be deployed completely before I run the above command. This is what I did to insure that pre conditions are met. For Tomcat:
nohup startup.sh
and then check for deployed file bu using curl to check for status 200:
- name: Check if URL is available
shell: curl --head --silent http://target-mchine:8080/restoftheurl
register: result
until: result.stdout.find("200 OK") != -1
retries: 12
delay: 10
It does above two steps correctly but when the next task is to run the
java -jar jar-from-application.jar --file myown.properties
it for some reason says that it cannot read the properties file. However, if I run the same jar command on the remote machine locally, then it works perfectly. I even added some wait time after successful execution of curl command, just in case, but it doesn't make any difference. I tried doing the following too:
nohup java -jar jar-from-application.jar --file myown.properties
as it was suggested online but that didn't make any difference either. Any suggestions will be appreciated. Thank You.
Make sure that you set the correct path to the properties file in the command by using the chdir argument of the command module or set the full path in the command.
I have a runnable jar file produced from clojure code that I am running on an embedded system (udoo - http://www.udoo.org/), which is running a version of Ubuntu for its OS (udoobuntu - http://www.udoo.org/udoobuntu-the-official-udoo-linux-operating-system/).
The jar file contains a clojure library I wrote, that includes a some Java swing stuff.
Running the jar manually via the command line using:
sudo java -jar myjar.jar
works fine. The sudo is needed for certain usb device permissions.
My problem arises when I try to start the jar using an upstart script called testjob. The relevant part of /etc/init/testjob.conf looks like:
start on (desktop-session-start)
expect fork
script
exec ./home/ubuntu/start > /home/ubuntu/boot-jar.log 2>&1 &
end script
Where /home/ubuntu/start is the following shell script:
#!/bin/sh
sudo java -jar /home/ubuntu/myjar.jar
exit 0
When this runs, either manually via:
sudo start testjob
or automatically by rebooting the system, I get the following output to the log file:
Exception in thread "main" java.lang.ExceptionInInitializerError
... <bunch of meaningless classloading stuff>
Caused by: java.awt.HeadlessException:
No X11 DISPLAY variable was set, but this program performed an operation which requires it.
at sun.awt.HeadlessToolkit.getMenuShortcutKeyMask(HeadlessToolkit.java:236)
at seesaw.keystroke$preprocess_descriptor.invoke(keystroke.clj:25)
at seesaw.keystroke$keystroke.invoke(keystroke.clj:50)
at seesaw.keystroke$keystroke.invoke(keystroke.clj:49)
at seesaw.keymap$map_key.doInvoke(keymap.clj:107)
at clojure.lang.RestFn.invoke(RestFn.java:445)
at seesaw.widgets.log_window$log_window.doInvoke(log_window.clj:88)
at clojure.lang.RestFn.invoke(RestFn.java:457)
at physicloud.utils__init.load(Unknown Source)
at physicloud.utils__init.<clinit>(Unknown Source)
... 52 more
I don't actually make any UI calls, but the run fails on classloading of the library. I know that a simple fix would be to remove the Java Swing code from the library, but the library is generalized to allow ui output to capable machines, and therefore I am looking for a workaround. It doesn't make sense to me why the jar will run via java -jar but not in the script.
I tried different cases for the upstart script's "start on" condition, all of which produced the same result.
I tried setting the $DISPLAY environment variable in the upstart script before executing the jar, but to no avail.
I also tried scheduling the start script using crontab #reboot, but the error was the same.
Anyone have suggestions?
You can pass -Djava.awt.headless=true to java to allow running non-GUI applications that happen to depend on some UI libraries.
For example,
java -Djava.awt.headless=true -jar something.jar