I am facing a problem which I could not figure out:
Scenario:
When a new instance powers up on the morning (Autoscaling), it starts with tomcat on a failed state, to solve this issue I just connect to the instancia via SSH and execute:
systemctl restart tomcat This solves the issue but I would like to find the root cause on why Tomcat9 is failing to start when the instance is powered on by the first time
Looking thru the logs, I find that when Tomcat9 is restarted, It appears as this:
[root#ip-172-31-177-124 bin]# systemctl status tomcat
● tomcat.service - Apache Tomcat
Loaded: loaded (/etc/systemd/system/tomcat.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2021-10-05 09:06:09 CEST; 2h 36min ago
Process: 3626 ExecStop=/usr/local/apache-tomcat/bin/shutdown.sh (code=exited, status=1/FAILURE)
Process: 3735 ExecStart=/usr/local/apache-tomcat/bin/startup.sh (code=exited, status=0/SUCCESS)
Main PID: 3742 (java)
CGroup: /system.slice/tomcat.service
└─3742 /usr/lib/jvm/java-1.8.0-openjdk/bin/java -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties -Djava.util.log...
Oct 05 09:06:09 ip-172-31-177-124.eu-west-1.compute.internal systemd[1]: Starting Apache Tomcat...
Oct 05 09:06:09 ip-172-31-177-124.eu-west-1.compute.internal startup.sh[3735]: /usr/local/apache-tomcat/bin/catalina.sh: line 514: /run/tomcat9.pid: P...enied
Oct 05 09:06:09 ip-172-31-177-124.eu-west-1.compute.internal startup.sh[3735]: Tomcat started.
Oct 05 09:06:09 ip-172-31-177-124.eu-west-1.compute.internal systemd[1]: Started Apache Tomcat.
Here are the journalctl -xe logs when Tomcat is restarted manually:
-- Unit tomcat.service has begun shutting down.
sep 17 20:00:37 ip-172-31-177-144.eu-west-1.compute.internal shutdown.sh[5311]: $CATALINA_PID was set but the specified file does not exist. Is Tomcat running?
sep 17 20:00:37 ip-172-31-177-144.eu-west-1.compute.internal systemd[1]: tomcat.service: control process exited, code=exited status=1
sep 17 20:00:37 ip-172-31-177-144.eu-west-1.compute.internal systemd[1]: tomcat.service: main process exited, code=exited, status=143/n/a
sep 17 20:00:37 ip-172-31-177-144.eu-west-1.compute.internal systemd[1]: Unit tomcat.service entered failed state.
sep 17 20:00:37 ip-172-31-177-144.eu-west-1.compute.internal systemd[1]: tomcat.service failed.
sep 17 20:00:37 ip-172-31-177-144.eu-west-1.compute.internal systemd[1]: Starting Apache Tomcat...
-- Subject: Unit tomcat.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit tomcat.service has begun starting up.
sep 17 20:00:37 ip-172-31-177-144.eu-west-1.compute.internal startup.sh[5322]: /usr/local/apache-tomcat/bin/catalina.sh: line 514: /run/tomcat9.pid: Permission
sep 17 20:00:37 ip-172-31-177-144.eu-west-1.compute.internal startup.sh[5322]: Tomcat started.
sep 17 20:00:37 ip-172-31-177-144.eu-west-1.compute.internal systemd[1]: Started Apache Tomcat.
Does anybody know what is going on here?
EDIT: Catalina/Java Logs when crash occours:
2021-10-01 08:03:03,628 ERROR org.springframework.scheduling.support.TaskUtils$LoggingErrorHandler [scheduling-1]Unexpected error occurred in scheduled task.java.lang.ExceptionInInitializerError: null at javax.crypto.JceSecurityManager.<clinit>(JceSecurityManager.java:65) at javax.crypto.Cipher.getConfiguredPermission(Cipher.java:2590) at javax.crypto.Cipher.getMaxAllowedKeyLength(Cipher.java:2614) at sun.security.ssl.CipherSuite$BulkCipher.isUnlimited(CipherSuite.java:535) at sun.security.ssl.CipherSuite$BulkCipher.<init>(CipherSuite.java:507) at sun.security.ssl.CipherSuite.<clinit>(CipherSuite.java:614) at sun.security.ssl.SSLContextImpl.getApplicableSupportedCipherSuiteList(SSLContextImpl.java:294) at sun.security.ssl.SSLContextImpl.access$100(SSLContextImpl.java:42) at sun.security.ssl.SSLContextImpl$AbstractTLSContext.<clinit>(SSLContextImpl.java:529) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at java.security.Provider$Service.getImplClass(Provider.java:1634) at java.security.Provider$Service.newInstance(Provider.java:1592) at sun.security.jca.GetInstance.getInstance(GetInstance.java:236) at sun.security.jca.GetInstance.getInstance(GetInstance.java:164) at javax.net.ssl.SSLContext.getInstance(SSLContext.java:156) at javax.net.ssl.SSLContext.getDefault(SSLContext.java:96) at javax.net.ssl.SSLSocketFactory.getDefault(SSLSocketFactory.java:122) at
Java version:
# java -version
openjdk version "1.8.0_302"
OpenJDK Runtime Environment (build 1.8.0_302-b08)
OpenJDK 64-Bit Server VM (build 25.302-b08, mixed mode)
Related
I have installed and configured the sonarqube on the server and created a symlink with systemd/systemctl, it properly running in terminal without any error but not running on browser even with default port. The version i have tried is 8.71 and 9, but in both it doesn't work, i have configured the postgres credentials and all in "sonar.properties"
but even after anything it doesn't work,
sonarqube#sonar2020:~/sonarqube-8.3.1.34397$
bin/linux-x86-64/sonar.sh start\
; Starting SonarQube... Started SonarQube. sonarqube#sonar2020:~/sonarqube-8.3.1.34397$ bin/linux-x86-64/sonar.sh
status SonarQube is running (12957).
sonarqube#sonar2020:~/sonarqube-8.3.1.34397$ sudo nano
/etc/systemd/system/sonar.service
sonarqube#sonar2020:~/sonarqube-8.3.1.34397$ sudo systemctl enable
sonar Created symlink
/etc/systemd/system/multi-user.target.wants/sonar.service →
/etc/systemd/system/sonar.service.
sonarqube#sonar2020:~/sonarqube-8.3.1.34397$ sudo systemctl start
sonar.service sonarqube#sonar2020:~/sonarqube-8.3.1.34397$ sudo
systemctl status sonar.service ● sonar.service - SonarQube service
Loaded: loaded (/etc/systemd/system/sonar.service; enabled; vendor
preset: enabled) Active: active (running) since Tue 2022-11-22
09:12:15 UTC; 1s ago Process: 13574
ExecStop=/home/sonarqube/sonarqube-8.3.1.34397/bin/linux-x86-64/sonar.sh
stop (code=exited, status=0/SUCCESS) Process: 13634
ExecStart=/home/sonarqube/sonarqube-8.3.1.34397/bin/linux-x86-64/sonar.sh
start (code=exited, status=0/SUCCESS) Main PID: 13697 (wrapper)
Tasks: 19 (limit: 4915) CGroup: /system.slice/sonar.service
├─13697 /home/sonarqube/sonarqube-8.3.1.34397/bin/linux-x86-64/./wrapper
/home/sonarqube/sonarqube-8.3.1.34397/bin/linux-x86-64/../../conf/wrapper.conf
wrapper.syslog.ident=SonarQube
wrapper.pidfile=/home/sonarqube/sonarqube-8.3.1.34397/bin/linux-x86-
└─13702 java -Dsonar.wrapped=true -Djava.awt.headless=true -Xms8m -Xmx32m -Djava.library.path=./lib -classpath ../../lib/jsw/wrapper-3.2.3.jar:../../lib/common/picocontainer-2.15.jar:../../lib/common/lucene-join-7.7.2.jar:../../lib/common/sonar-db-mig
Nov 22 09:12:14 sonar2020 systemd[1]: sonar.service: Scheduled restart
job, restart counter is at 3. Nov 22 09:12:14 sonar2020 systemd[1]:
Stopped SonarQube service. Nov 22 09:12:14 sonar2020 systemd[1]:
Starting SonarQube service... Nov 22 09:12:14 sonar2020
sonar.sh[13634]: Starting SonarQube... Nov 22 09:12:15 sonar2020
sonar.sh[13634]: Started SonarQube. Nov 22 09:12:15 sonar2020
systemd[1]: Started SonarQube service.
Now whenever i try to open it on browser, it shows connection refused. How can it be refused when all ports are open in local??
I attempted to add the nvm-wrapper plugin (https://plugins.jenkins.io/nvm-wrapper/) to jenkins and reboot and am no longer able to reach the site (This site can’t be reached). It is deployed as an azure VM. The VM is running, I can ssh in and systemctl status jenkins.service is showing:
jenkins.service - LSB: Start Jenkins at boot time
Loaded: loaded (/etc/init.d/jenkins; generated)
Active: failed (Result: exit-code) since Mon 2022-04-04 21:04:43 UTC; 14h ago
Docs: man:systemd-sysv-generator(8)
Process: 2530 ExecStart=/etc/init.d/jenkins start (code=exited, status=7)
Apr 04 21:04:42 pipeline jenkins[2530]: Correct java version found
Apr 04 21:04:42 pipeline jenkins[2530]: * Starting Jenkins Automation Server jenkins
Apr 04 21:04:42 pipeline su[2587]: Successful su for jenkins by root
Apr 04 21:04:42 pipeline su[2587]: + ??? root:jenkins
Apr 04 21:04:42 pipeline su[2587]: pam_unix(su:session): session opened for user jenkins by (uid=0)
Apr 04 21:04:42 pipeline su[2587]: pam_unix(su:session): session closed for user jenkins
Apr 04 21:04:43 pipeline jenkins[2530]: ...fail!
Apr 04 21:04:43 pipeline systemd[1]: jenkins.service: Control process exited, code=exited status=7
Apr 04 21:04:43 pipeline systemd[1]: jenkins.service: Failed with result 'exit-code'.
Apr 04 21:04:43 pipeline systemd[1]: Failed to start LSB: Start Jenkins at boot time.
I had previously been running into Incorrect Java version due to having installed Java 12, so I modified the /etc/init.d/jenkins like so:
# Which Java versions can be used to run Jenkins
JAVA_ALLOWED_VERSIONS=( "18" "110" "120" )
# Work out the JAVA version we are working with:
JAVA_VERSION=$($JAVA -version 2>&1 | sed -n ';s/.* version "\([0-9]*\)\.\([0-9]*\)\..*".*/\1\2/p;')
to allow for Java 12. Any thoughts on how/why either the plugin addition or Java version and/or /etc/init.d/jenkins edit could be impacting things? My sense is that the initial reboot failed due to the Incorrect Java version issue, but not sure how I can resolve things and get it back up and running. It should, by default, be available at 8080 and that is where I am seeing This site can’t be reached.
I also have the networking set up like so:
and port 8080 should allow traffic. I have attempted to restart, hard start and stop/start to no avail as well. "Resource health" says the VM is available which should be obvious since it is running and I can ssh in. Do I need to redeploy perhaps?
First, please delegate the service to a non root user.
Second, azure, means in the net, please use SSL, the times without SSL are gone.
JAVA_ALLOWED_VERSIONS=( "18" "110" "120" )
you are trying java 12, but its configured java 120, i think you should fix that to
JAVA_ALLOWED_VERSIONS=( "18" "11" "12" )
I have this java file, it has to be executed using the following command otherwise it defaults to ipv6 and effectively useless for my use-case
-jar Djava.net.preferIPv4Stack=true EdOwl-Springboot.jar
I tried following this guide: https://computingforgeeks.com/how-to-run-java-jar-application-with-systemd-on-linux/
However using this shows:
root#edowl:~# systemctl status EdOwl-Springboot.service
● EdOwl-Springboot.service - EdowlSpringboot
Loaded: loaded (/etc/systemd/system/EdOwl-Springboot.service; disabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Tue 2022-02-22 11:28:26 UTC; 10s ago
Main PID: 186854 (code=exited, status=203/EXEC)
Tasks: 0 (limit: 614)
Memory: 0B
CGroup: /system.slice/EdOwl-Springboot.service
Feb 22 11:28:36 edowl.online systemd[1]: EdOwl-Springboot.service: Scheduled restart job, restart counter is at 125919.
Feb 22 11:28:36 edowl.online systemd[1]: Stopped EdowlSpringboot.
Feb 22 11:28:36 edowl.online systemd[1]: Started EdowlSpringboot.
Feb 22 11:28:36 edowl.online systemd[186888]: EdOwl-Springboot.service: Failed to execute command: No such file or directory
Feb 22 11:28:36 edowl.online systemd[186888]: EdOwl-Springboot.service: Failed at step EXEC spawning /bin/java: No such file or directory
Feb 22 11:28:36 edowl.online systemd[1]: EdOwl-Springboot.service: Main process exited, code=exited, status=203/EXEC
Feb 22 11:28:36 edowl.online systemd[1]: EdOwl-Springboot.service: Failed with result 'exit-code'.
the service file is as follows:
[Unit]
Description=EdowlSpringboot
[Service]
WorkingDirectory=/home
ExecStart=/bin/java -Xms128m -Xmx256m -jar Djava.net.preferIPv4Stack=true EdOwl-Springboot.jar
Type=simple
Restart=on-failure
RestartSec=10
[Install]
WantedBy=multi-user.target
The file is located in the /home dir
I am certain it is the exec start line that is messing this up, is it necessary to state: /bin/java -Xms128m -Xmx256mor am I messing up something else here?
Suggestions are welcome cheers
The JVM argument java.net.preferIPv4Stack should be prefixed with -D instead you have D
Also, as others pointed out in comments check your java path it might be /usr/bin/java.
So, change the command to /usr/bin/java -Xms128m -Xmx256m -jar -Djava.net.preferIPv4Stack=true EdOwl-Springboot.jar
I have created a GUI Javafx app and exported it to a jar file. The GUI app is working fine. After creating this app i have moved it to linux system and it is working fine here. Now i want to run this GUI app in linux using services so that i can make this service run at start of linux system.
For that I have created a pointless.sh file which on run runs the jar file. It is working perfectly the code for this pointless.sh file are
java -jar GUI1.jar
on Runinng this script i.e. "./pointless" the GUI is shown on the screen.
Problem starts when i try to make a service for it.
I made a service for it in /etc/systemd/system pointless.service
[Service]
ExecStart=/home/mandeep/Documents/pointless.sh
on running sudo systemctl start pointless.service nothing happens and the status shows following errors.
● pointless.service
Loaded: loaded (/etc/systemd/system/pointless.service; static; vendor preset: enabled)
Active: failed (Result: exit-code) since Sat 2019-07-06 13:44:00 EDT; 25min ago
Process: 1449 ExecStart=/home/mandeep/Documents/pointless.sh (code=exited, status=203/EXEC)
Main PID: 1449 (code=exited, status=203/EXEC)
Jul 06 13:44:00 debian systemd[1]: Started pointless.service.
Jul 06 13:44:00 debian systemd[1449]: pointless.service: Failed at step EXEC spawning /home/mandeep/Documents/pointless.sh: Exec format error
Jul 06 13:44:00 debian systemd[1]: pointless.service: Main process exited, code=exited, status=203/EXEC
Jul 06 13:44:00 debian systemd[1]: pointless.service: Unit entered failed state.
Jul 06 13:44:00 debian systemd[1]: pointless.service: Failed with result 'exit-code'.
Update !!!
Step 1
As expected I created a simple Java app helloworld and exported it to jar file. I ran this file as java -jar helloworld and It worked.
Step 2
I created a simple service for it and service also worked as expected.
the service codes are
[Service]
ExecStart=/usr/bin/java -jar /home/mandeep/Documents/helloworld.jar
Output
Jul 07 17:08:13 debian systemd[1]: Started pointless.service.
Jul 07 17:08:13 debian java[2765]: Hello World!
Step 3
I Updated service back to GUI1.jar to check original problem posted.
code
[Service]
ExecStart=/usr/bin/java -jar /home/mandeep/Documents/GUI1.jar
and again it shows the error
Output
● pointless.service
Loaded: loaded (/etc/systemd/system/pointless.service; static; vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2019-07-07 17:14:37 EDT; 4s ago
Process: 2816 ExecStart=/usr/bin/java -jar /home/mandeep/Documents/GUI1.jar (code=exited, status=1/FAILURE)
Main PID: 2816 (code=exited, status=1/FAILURE)
Jul 07 17:14:37 debian java[2816]: at com.sun.glass.ui.Application.run(Application.java:146)
Jul 07 17:14:37 debian java[2816]: at com.sun.javafx.tk.quantum.QuantumToolkit.startup(QuantumToolkit.java:257)
Jul 07 17:14:37 debian java[2816]: at com.sun.javafx.application.PlatformImpl.startup(PlatformImpl.java:211)
Jul 07 17:14:37 debian java[2816]: at com.sun.javafx.application.LauncherImpl.startToolkit(LauncherImpl.java:675)
Jul 07 17:14:37 debian java[2816]: at com.sun.javafx.application.LauncherImpl.launchApplicationWithArgs(LauncherImpl.java:337)
Jul 07 17:14:37 debian java[2816]: at com.sun.javafx.application.LauncherImpl.launchApplication(LauncherImpl.java:328)
Jul 07 17:14:37 debian java[2816]: ... 5 more
Jul 07 17:14:37 debian systemd[1]: pointless.service: Main process exited, code=exited, status=1/FAILURE
Jul 07 17:14:37 debian systemd[1]: pointless.service: Unit entered failed state.
Jul 07 17:14:37 debian systemd[1]: pointless.service: Failed with result 'exit-code'.
Found the answer
Made following changes under [Service] in pointless.service
a) use "Environment=DISPLAY:=0"
b) declare "User=<user>"
and under [Install] optional
c) declare "WantedBy=<user>#.service"
One of my server is running with tomcat for a java application. The application is working fine with tomcat but in syslog we are getting the following error continuously in Ubuntu:
Oct 15 16:00:18 HiT-APP-Primary shutdown.sh[115760]: PID file found but either no matching process was found or the current user does not have permission to stop the process. Stop aborted.
Oct 15 16:00:18 HiT-APP-Primary systemd[1]: tomcat.service: Control process exited, code=exited status=1
Oct 15 16:00:18 HiT-APP-Primary systemd[1]: tomcat.service: Unit entered failed state.
Oct 15 16:00:18 HiT-APP-Primary systemd[1]: tomcat.service: Failed with result 'exit-code'.
Oct 15 16:00:28 HiT-APP-Primary systemd[1]: tomcat.service: Service hold-off time over, scheduling restart.
Oct 15 16:00:28 HiT-APP-Primary systemd[1]: Stopped Apache Tomcat Web Application Container.
Oct 15 16:00:28 HiT-APP-Primary systemd[1]: Starting Apache Tomcat Web Application Container...
Oct 15 16:00:28 HiT-APP-Primary startup.sh[115772]: Existing PID file found during start.
Oct 15 16:00:28 HiT-APP-Primary startup.sh[115772]: Removing/clearing stale PID file.
Oct 15 16:00:28 HiT-APP-Primary startup.sh[115772]: Tomcat started.
Oct 15 16:00:28 HiT-APP-Primary systemd[1]: Started Apache Tomcat Web Application Container.
Oct 15 16:00:35 HiT-APP-Primary shutdown.sh[115867]: PID file found but either no matching process was found or the current user does not have permission to stop the process. Stop aborted.
Oct 15 16:00:35 HiT-APP-Primary systemd[1]: tomcat.service: Control process exited, code=exited status=1
Oct 15 16:00:35 HiT-APP-Primary systemd[1]: tomcat.service: Unit entered failed state.
Oct 15 16:00:35 HiT-APP-Primary systemd[1]: tomcat.service: Failed with result 'exit-code'.
Can anyone solve it please?
in my case java home was not properly setup in tomcat service file to make this fixed open tomcat service file.
vim /etc/systemd/system/tomcat.service
and verify JAVA_HOME is pointing to the correct JRE version if not then make it correct
Environment="JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre"
After this reload daemon and start service again
Oct 15 16:00:18 HiT-APP-Primary shutdown.sh[115760]: PID file found but either no matching process was found or the current user does not have permission to stop the process. Stop aborted.
Looks like a permission issue. Check the permissions.
Otherwise
set CATALINA_PID=/tmp/tomcat.pid, or to a directory where tomcat has access. Then restart tomcat.