Tomcat 9 no longer starting using systemctl but will start manually - java

Been digging on this for a while. I reviewed multiple articles on this issue. This one was the closest:
Tomcat 8 on CentOS 7 does not start as service (but it starts manually ....)
The difference being that I am running Tomcat 9.0.33. Here are the particulars:
java version "1.8.0_121"\
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)\
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)\
Tomcat 9.0.33
NAME="CentOS Linux"\
VERSION="7 (Core)"\
ID="centos"\
ID_LIKE="rhel fedora"\
VERSION_ID="7"\
PRETTY_NAME="CentOS Linux 7 (Core)"\
ANSI_COLOR="0;31"\
CPE_NAME="cpe:/o:centos:centos:7"\
HOME_URL="https://www.centos.org/"\
BUG_REPORT_URL="https://bugs.centos.org/"\
CENTOS_MANTISBT_PROJECT="CentOS-7"\
CENTOS_MANTISBT_PROJECT_VERSION="7"\
REDHAT_SUPPORT_PRODUCT="centos"\
REDHAT_SUPPORT_PRODUCT_VERSION="7"\
As a side note, everything was starting normally with no issues until recently. As far as I know there haven't been any major changes to the environment. But, when I ran the "systemctl restart" command recently, the startup began to fail. There are 5 instances of Tomcat 9.0.33 running at different ports and paths and those have not changed. I have not restarted two of the instance (afraid they won't start) the other three flat out won't start. Details below:
Systemd unit file for tomcat\
[Unit]\
Description=Apache Tomcat Web Application Container in Liferay 7.32 TEST for UAT\
After=syslog.target network.target
[Service]\
Type=forking
Environment=JAVA_HOME=/opt/jdk1.8.0_121/jre\
Environment=CATALINA_PID=/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33/temp/tomcat.pid\
Environment=CATALINA_HOME=/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33\
Environment=CATALINA_BASE=/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33\
Environment='CATALINA_OPTS=-Xms1024m -Xmx2048m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=20 -XX:ParallelGCThreads=8 -server -Xdebug -Xrunjdwp:transport=dt_socket,address=5000,server=y,suspend=n'\
Environment='JAVA_OPTS=-Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom -Duser.timezone=GMT -Dfile.encoding=UTF-8'
ExecStart=/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33/bin/startup.sh\
ExecStop=/bin/kill -15 $MAINPID
User=tomcat\
Group=tomcat\
UMask=0007
[Install]\
WantedBy=multi-user.target\
Results when running systemctl start liferayuat
● liferayuat.service - Apache Tomcat Web Application Container in Liferay 7.32 TEST for UAT\
Loaded: loaded (/etc/systemd/system/liferayuat.service; enabled; vendor preset: disabled)\
Active: failed (Result: exit-code) since Sat 2020-12-05 08:44:08 CST; 3s ago\
Process: 10891 ExecStop=/bin/kill -15 $MAINPID (code=exited, status=1/FAILURE)\
Process: 10851 ExecStart=/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33/bin/startup.sh \(code=exited, status=0/SUCCESS)\
Main PID: 10861 (code=exited, status=0/SUCCESS)
Dec 05 08:44:08 systemd[1]: Starting Apache Tomcat Web Application Container in Liferay 7.32 TEST for UAT...\
Dec 05 08:44:08 startup.sh[10851]: Existing PID file found during start.\
Dec 05 08:44:08 startup.sh[10851]: Removing/clearing stale PID file.\
Dec 05 08:44:08 startup.sh[10851]: Tomcat started.\
Dec 05 08:44:08 systemd[1]: Started Apache Tomcat Web Application Container in Liferay 7.32 TEST for UAT.\
Dec 05 08:44:08 systemd[1]: liferayuat.service: control process exited, code=exited status=1\
Dec 05 08:44:08 systemd[1]: Unit liferayuat.service entered failed state.\
Dec 05 08:44:08 systemd[1]: liferayuat.service failed.
Then the ONLY thing in catalina.out:
Listening for transport dt_socket at address: 5000\
java.lang.ClassNotFoundException: org.apache.catalina.startup.Catalina\
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)\
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)\
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)\
at org.apache.catalina.startup.Bootstrap.init(Bootstrap.java:261)\
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:443)\
So, when I start the instance with systemctl start it will fail. But if I run this command (as root...) then it will start:
/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33/bin/startup.sh
If I run that full commmand AS tomcat it doesn't start with the same error. So, it appears that the issue is permissions. The tomcat user and group are owners of all files and folders. But, somehow, the tomcat user either doesn't have permissions or the path gets jacked up so that the class files can't be found. I followed the suggestions in the article I referenced above but the changes had no impact.
I tripped across one article on SELINX that seemed to point to an issue there. This are the SELINUX settings:
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: permissive
Mode from config file: permissive
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 31\
The workaround to keep the instances running is just to manually start them but what is causing systemctl start NOT to work? I suspect permissions but I sure as heck can't see why since everything is owned by tomcat:tomcat

So, this is self-inflicted as most "mysteries" are. I still cannot account for some of the differences I see when looking into SELinux contexts between the instances but the REAL cause was subtle (to me). Permissions on the {tomcat root}/lib and {tomcat root}/lib/ext had no execute permissions. That may have been due to a jar that was added recently and then needed to be updated by owner and permissions. In any case, the original issue resulted in many trial and error attempts to fix it which complicated matters further.
I discovered the solution by doing a folder by folder, file by file comparison between working and non-working instances. Apparently the new jar and the owner/permission changes were applied to all but the production version.
Thanks for the suggestions.

Related

Jenkins in Azure VM "This site can’t be reached" at 8080

I attempted to add the nvm-wrapper plugin (https://plugins.jenkins.io/nvm-wrapper/) to jenkins and reboot and am no longer able to reach the site (This site can’t be reached). It is deployed as an azure VM. The VM is running, I can ssh in and systemctl status jenkins.service is showing:
jenkins.service - LSB: Start Jenkins at boot time
Loaded: loaded (/etc/init.d/jenkins; generated)
Active: failed (Result: exit-code) since Mon 2022-04-04 21:04:43 UTC; 14h ago
Docs: man:systemd-sysv-generator(8)
Process: 2530 ExecStart=/etc/init.d/jenkins start (code=exited, status=7)
Apr 04 21:04:42 pipeline jenkins[2530]: Correct java version found
Apr 04 21:04:42 pipeline jenkins[2530]: * Starting Jenkins Automation Server jenkins
Apr 04 21:04:42 pipeline su[2587]: Successful su for jenkins by root
Apr 04 21:04:42 pipeline su[2587]: + ??? root:jenkins
Apr 04 21:04:42 pipeline su[2587]: pam_unix(su:session): session opened for user jenkins by (uid=0)
Apr 04 21:04:42 pipeline su[2587]: pam_unix(su:session): session closed for user jenkins
Apr 04 21:04:43 pipeline jenkins[2530]: ...fail!
Apr 04 21:04:43 pipeline systemd[1]: jenkins.service: Control process exited, code=exited status=7
Apr 04 21:04:43 pipeline systemd[1]: jenkins.service: Failed with result 'exit-code'.
Apr 04 21:04:43 pipeline systemd[1]: Failed to start LSB: Start Jenkins at boot time.
I had previously been running into Incorrect Java version due to having installed Java 12, so I modified the /etc/init.d/jenkins like so:
# Which Java versions can be used to run Jenkins
JAVA_ALLOWED_VERSIONS=( "18" "110" "120" )
# Work out the JAVA version we are working with:
JAVA_VERSION=$($JAVA -version 2>&1 | sed -n ';s/.* version "\([0-9]*\)\.\([0-9]*\)\..*".*/\1\2/p;')
to allow for Java 12. Any thoughts on how/why either the plugin addition or Java version and/or /etc/init.d/jenkins edit could be impacting things? My sense is that the initial reboot failed due to the Incorrect Java version issue, but not sure how I can resolve things and get it back up and running. It should, by default, be available at 8080 and that is where I am seeing This site can’t be reached.
I also have the networking set up like so:
and port 8080 should allow traffic. I have attempted to restart, hard start and stop/start to no avail as well. "Resource health" says the VM is available which should be obvious since it is running and I can ssh in. Do I need to redeploy perhaps?
First, please delegate the service to a non root user.
Second, azure, means in the net, please use SSL, the times without SSL are gone.
JAVA_ALLOWED_VERSIONS=( "18" "110" "120" )
you are trying java 12, but its configured java 120, i think you should fix that to
JAVA_ALLOWED_VERSIONS=( "18" "11" "12" )

ubuntu server 16.04 - tomcat7 failed to start

For the past 2 months I run tomcat7 perfectly on my ubuntu server. Today tomcat7 failed to start.
I started tomcat 7
sudo systemctl start tomcat 7
Job for tomcat7.service failed because the control process exited with error code.
See "systemctl status tomcat7.service" and "journalctl -xe" for details.
Then I checked for the status
sudo systemctl status tomcat7
● tomcat7.service - LSB: Start Tomcat.
Loaded: loaded (/etc/init.d/tomcat7; generated)
Active: failed (Result: exit-code) since Sat 2020-05-16 15:47:00 HKT; 8min ago
Docs: man:systemd-sysv-generator(8)
Process: 6043 ExecStart=/etc/init.d/tomcat7 start (code=exited, status=1/FAILURE
May 16 15:46:55 www.example.com systemd[1]: Starting LSB: Start Tomcat....
May 16 15:46:55 www.example.com tomcat7[6043]: * Starting Tomcat servlet engine
May 16 15:47:00 www.example.com tomcat7[6043]: ...fail!
May 16 15:47:00 www.example.com systemd[1]: tomcat7.service: Control process exit
May 16 15:47:00 www.example.com systemd[1]: tomcat7.service: Failed with result '
May 16 15:47:00 www.example.com systemd[1]: Failed to start LSB: Start Tomcat..
Then, I checked for the log error and I have no idea how to fix this
sudo tail /etc/var/log/tomcat7/catalina.out
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
-Djava.endorsed.dirs=/usr/share/tomcat7/endorsed is not supported. Endorsed standards and standalone APIs
in modular form will be supported via the concept of upgradeable modules.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
-Djava.endorsed.dirs=/usr/share/tomcat7/endorsed is not supported. Endorsed standards and standalone APIs
in modular form will be supported via the concept of upgradeable modules.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Somebody please help ....

Unable to install Jetty9 as a service in Ubuntu

I've followed the docs in order to install Jetty9 as a service but whenever I run
service jetty start
It would fail with no messages, my JETTY_HOME is /opt/jetty9, contains the home distribution for version 9.4.14. I've also created my JETTY_BASE at /usr/share/jetty9 with my webapp and modules.
Both Jetty Home and Base are owned by the user jetty. I've then symlinked to my init.d folder as:
ln -s /opt/jetty9/bin/jetty.sh /etc/init.d/jetty
Then I created a /etc/default/jetty file with the following content:
# change to 1 to prevent Jetty from starting
NO_START=0
# change to 'no' or uncomment to use the default setting in /etc/default/rcS
VERBOSE=yes
# Run Jetty as this user ID (default: jetty)
# Set this to an empty string to prevent Jetty from starting automatically
JETTY_USER=jetty
# The home directory of the Java Runtime Environment (JRE). You need at least
# Java 6. If JAVA_HOME is not set, some common directories for OpenJDK and
# the Oracle JDK are tried.
#JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
# Extra options to pass to the JVM
#JAVA_OPTIONS="-Xmx256m -Djava.awt.headless=true"
# Timeout in seconds for the shutdown of all webapps
#JETTY_SHUTDOWN=30
# Additional arguments to pass to Jetty
#JETTY_ARGS=
# Jetty uses a directory to store temporary files like unpacked webapps
TMPDIR=/opt/jetty9/tmp
JETTY_HOME=/opt/jetty9
JETTY_BASE=/usr/share/jetty9
# Default for number of days to keep old log files in /var/log/jetty9/
#LOGFILE_DAYS=14
# If you run Jetty on port numbers that are all higher than 1023, then you # do not need authbind. It is used for binding Jetty to lower port numbers.
# (yes/no, default: no)
#AUTHBIND=yes
JETTY_HOST=0.0.0.0
If I start Jetty using java -jar $JETTY_HOME/start.jar in my base folder it would work with no problem. Also, if I run
service jetty supervise
It would also run with no issues, but when I call start it fails with:
root#app:/usr/share/jetty9# service jetty start
Job for jetty.service failed because the control process exited with error code.
See "systemctl status jetty.service" and "journalctl -xe" for details.
root#app:/usr/share/jetty9# service jetty status
● jetty.service - LSB: Jetty start script.
Loaded: loaded (/etc/init.d/jetty; generated)
Active: failed (Result: exit-code) since Mon 2018-12-03 15:05:26 UTC; 14s ago
Docs: man:systemd-sysv-generator(8)
Process: 21162 ExecStop=/etc/init.d/jetty stop (code=exited, status=0/SUCCESS)
Process: 21202 ExecStart=/etc/init.d/jetty start (code=exited, status=1/FAILURE)
Dec 03 15:05:22 app systemd[1]: Stopped LSB: Jetty start script..
Dec 03 15:05:22 app systemd[1]: Starting LSB: Jetty start script....
Dec 03 15:05:26 app jetty[21202]: Starting Jetty: FAILED Mon Dec 3 15:05:26 UTC 2018
Dec 03 15:05:26 app systemd[1]: jetty.service: Control process exited, code=exited status=1
Dec 03 15:05:26 app systemd[1]: jetty.service: Failed with result 'exit-code'.
Dec 03 15:05:26 app systemd[1]: Failed to start LSB: Jetty start script..
This is the output of service jetty check:
root#app:/usr/share/jetty9# service jetty check
Jetty NOT running
JAVA = /usr/bin/java
JAVA_OPTIONS = -Djetty.home=/opt/jetty9 -Djetty.base=/usr/share/jetty9 -Djava.io.tmpdir=/opt/jetty9/tmp
JETTY_HOME = /opt/jetty9
JETTY_BASE = /usr/share/jetty9
START_D = /usr/share/jetty9/start.d
START_INI = /usr/share/jetty9/start.ini
JETTY_START = /opt/jetty9/start.jar
JETTY_CONF = /opt/jetty9/etc/jetty.conf
JETTY_ARGS = jetty.state=/usr/share/jetty9/jetty.state jetty-started.xml
JETTY_RUN = /var/run/jetty
JETTY_PID = /var/run/jetty/jetty.pid
JETTY_START_LOG = /var/run/jetty/jetty-start.log
JETTY_STATE = /usr/share/jetty9/jetty.state
JETTY_START_TIMEOUT = 60
RUN_CMD = /usr/bin/java -Djetty.home=/opt/jetty9 -Djetty.base=/usr/share/jetty9 -Djava.io.tmpdir=/opt/jetty9/tmp -jar /opt/jetty9/start.jar jetty.state=/usr/share/jetty9/jetty.state jetty-started.xml
Any ideas?
UPDATE
Changing the user in /etc/default/jetty to root would solve the issue, but this is not a solution, isn't it?
# Run Jetty as this user ID (default: jetty)
# Set this to an empty string to prevent Jetty from starting automatically
JETTY_USER=root
I finally got this working, the jetty user should have permissions to the following folders and /usr/sbin/nologin as shell as described here.
JETTY_HOME
JETTY_BASE
/var/run/jetty <-- couldn't find a reference to this folder in the docs
And add the following to your /etc/default/jetty:
JETTY_SHELL=/bin/sh
JETTY_LOGS=/usr/share/jetty9/logs
JETTY_START_LOG=/usr/share/jetty9/logs/jetty-start-log.log
Also you should double check that there are no remaining log files owned by other user than jetty in your folders.

Unsupported major.minor version 52.0 with web app built with eclipse, server is tomcat 7 [duplicate]

This question already has answers here:
How to fix java.lang.UnsupportedClassVersionError: Unsupported major.minor version
(51 answers)
Closed 4 years ago.
I have recently upgraded my development machine to Ubuntu 18.04 and have been revisiting and checking some old projects, one of which is built under Eclipse oxygen and deployed using Tomcat 7.
To the best of my memory this project worked when deploying the .war file to the private instance of tomcat (running on port 10080) and the main, autostarting instance on 8080.
It still works on 10080 but when I deploy to 8080 and open through a browser, I see
javax.servlet.ServletException: java.lang.UnsupportedClassVersionError: org/eclipse/jdt/internal/compiler/env/INameEnvironment : Unsupported major.minor version 52.0
I have $JAVA_HOME as JAVA_HOME="/usr/lib/jvm/java-8-oracle"
The main tomcat manager page shows :
Apache Tomcat/7.0.68 (Ubuntu) JVM version 1.7.0_80-b15
I have set the the eclipse compiler compliance level to 1.7.
I am completely baffled as to why one instance is OK and the other not. I don't want to change the JAVA_HOME, so I think it must be something to do with the tomcat configuration or the internal compiler but can't seem to sort it out.
All sensible suggestions will be gratefully received.
Edit:
This is not a duplicate of a question, as I have built one war file and exported it to ~/junk. After that I have copied and pasted it to the separate webapps folders for the two Tomcat instances. I think it's a tomact configuration issue but I can't see what. Please don't mark it as duplicate
Edit 2:
Looking at how the tomcat was started as a service with systemctl I see:
nick#nick-X555LAB:~$ systemctl status tomcat7.service
tomcat7.service - LSB: Start Tomcat.
Loaded: loaded (/etc/init.d/tomcat7; generated)
Active: active (running) since Mon 2018-08-20 12:04:29 BST; 55s ago
Docs: man:systemd-sysv-generator(8)
Process: 10050 ExecStart=/etc/init.d/tomcat7 start (code=exited, status=0/SUCC
Tasks: 20 (limit: 4915)
CGroup: /system.slice/tomcat7.service
└─10109 /usr/lib/jvm/java-7-oracle/bin/java -Djava.util.logging.confi
Aug 20 12:04:24 nick-X555LAB systemd[1]: Starting LSB: Start Tomcat....
Aug 20 12:04:24 nick-X555LAB tomcat7[10050]: * Starting Tomcat servlet engine t
Aug 20 12:04:29 nick-X555LAB tomcat7[10050]: ...done.
Aug 20 12:04:29 nick-X555LAB systemd[1]: Started LSB: Start Tomcat..
Note the reference to java7 - I'd like to switch this to java 8
Solved
Solved it. Found a reference to java 7 in /etc/default/tomcat7, changed it to java 8 and all is well.
Use update-alternatives to change the default JDK to java8
as
sudo update-alternatives --config java
see https://linux.die.net/man/8/update-alternatives

neo4j cannot find JAVA_HOME when checking service status

I installed neo4j 1.8.2 on opensuse 12.2/64. To do so I had to add the JAVA_HOME path in the /etc/profile file as:
export JAVA_HOME=/opt/java/64/jdk1.7.0_21/jre/:
export PATH=$PATH:/opt/java/64/jdk1.7.0_21/jre/bin/;
Now when I try to check the server status I get the following error
>service neo4j-service status
neo4j-service.service - LSB: The Neo4J graph database server. See http://neo4j.org
Loaded: loaded (/etc/init.d/neo4j-service)
Active: failed (Result: exit-code) since Fri, 26 Apr 2013 17:13:56 +0200; 10s ago
Process: 7234 ExecStart=/etc/init.d/neo4j-service start (code=exited, status=1/FAILURE)
CGroup: name=systemd:/system/neo4j-service.service
Apr 26 17:13:56 linux-wwcz neo4j-service[7234]: which: no java in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/...bin)
Apr 26 17:13:56 linux-wwcz neo4j-service[7234]: Error: JAVA_HOME is not defined correctly.
Apr 26 17:13:56 linux-wwcz neo4j-service[7234]: We cannot execute
It's quite puzzling considering neo4j-service links to ./bin/neo4j, namely the file used at installation time with
./bin/neo4j install
Some ideas on what is going on here?
Thanks
SOLVED
Actually I was using jdk 7 instead of jdk 6
EDIT 2
According to the official neo4j page one runs the server using neo4j start. But I got into troubles when trying to run service neo4j start/status/stop as suggested in the Installing Neo4j in Linux how-to.
Try changing them to this:
export JAVA_HOME=/opt/java/64/jdk1.7.0_21/:
export PATH=$PATH:/opt/java/64/jdk1.7.0_21/bin/;
Not exactly for the question, but I reached here because I also though Neo4j could not find the environment variable JAVA_HOME when
sudo neo4j start
My problem was that OS reset environment variables when using sudo.
sudo -E neo4j start
solved my problem.

Categories

Resources