I attempted to add the nvm-wrapper plugin (https://plugins.jenkins.io/nvm-wrapper/) to jenkins and reboot and am no longer able to reach the site (This site can’t be reached). It is deployed as an azure VM. The VM is running, I can ssh in and systemctl status jenkins.service is showing:
jenkins.service - LSB: Start Jenkins at boot time
Loaded: loaded (/etc/init.d/jenkins; generated)
Active: failed (Result: exit-code) since Mon 2022-04-04 21:04:43 UTC; 14h ago
Docs: man:systemd-sysv-generator(8)
Process: 2530 ExecStart=/etc/init.d/jenkins start (code=exited, status=7)
Apr 04 21:04:42 pipeline jenkins[2530]: Correct java version found
Apr 04 21:04:42 pipeline jenkins[2530]: * Starting Jenkins Automation Server jenkins
Apr 04 21:04:42 pipeline su[2587]: Successful su for jenkins by root
Apr 04 21:04:42 pipeline su[2587]: + ??? root:jenkins
Apr 04 21:04:42 pipeline su[2587]: pam_unix(su:session): session opened for user jenkins by (uid=0)
Apr 04 21:04:42 pipeline su[2587]: pam_unix(su:session): session closed for user jenkins
Apr 04 21:04:43 pipeline jenkins[2530]: ...fail!
Apr 04 21:04:43 pipeline systemd[1]: jenkins.service: Control process exited, code=exited status=7
Apr 04 21:04:43 pipeline systemd[1]: jenkins.service: Failed with result 'exit-code'.
Apr 04 21:04:43 pipeline systemd[1]: Failed to start LSB: Start Jenkins at boot time.
I had previously been running into Incorrect Java version due to having installed Java 12, so I modified the /etc/init.d/jenkins like so:
# Which Java versions can be used to run Jenkins
JAVA_ALLOWED_VERSIONS=( "18" "110" "120" )
# Work out the JAVA version we are working with:
JAVA_VERSION=$($JAVA -version 2>&1 | sed -n ';s/.* version "\([0-9]*\)\.\([0-9]*\)\..*".*/\1\2/p;')
to allow for Java 12. Any thoughts on how/why either the plugin addition or Java version and/or /etc/init.d/jenkins edit could be impacting things? My sense is that the initial reboot failed due to the Incorrect Java version issue, but not sure how I can resolve things and get it back up and running. It should, by default, be available at 8080 and that is where I am seeing This site can’t be reached.
I also have the networking set up like so:
and port 8080 should allow traffic. I have attempted to restart, hard start and stop/start to no avail as well. "Resource health" says the VM is available which should be obvious since it is running and I can ssh in. Do I need to redeploy perhaps?
First, please delegate the service to a non root user.
Second, azure, means in the net, please use SSL, the times without SSL are gone.
JAVA_ALLOWED_VERSIONS=( "18" "110" "120" )
you are trying java 12, but its configured java 120, i think you should fix that to
JAVA_ALLOWED_VERSIONS=( "18" "11" "12" )
Related
I am facing a problem which I could not figure out:
Scenario:
When a new instance powers up on the morning (Autoscaling), it starts with tomcat on a failed state, to solve this issue I just connect to the instancia via SSH and execute:
systemctl restart tomcat This solves the issue but I would like to find the root cause on why Tomcat9 is failing to start when the instance is powered on by the first time
Looking thru the logs, I find that when Tomcat9 is restarted, It appears as this:
[root#ip-172-31-177-124 bin]# systemctl status tomcat
● tomcat.service - Apache Tomcat
Loaded: loaded (/etc/systemd/system/tomcat.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2021-10-05 09:06:09 CEST; 2h 36min ago
Process: 3626 ExecStop=/usr/local/apache-tomcat/bin/shutdown.sh (code=exited, status=1/FAILURE)
Process: 3735 ExecStart=/usr/local/apache-tomcat/bin/startup.sh (code=exited, status=0/SUCCESS)
Main PID: 3742 (java)
CGroup: /system.slice/tomcat.service
└─3742 /usr/lib/jvm/java-1.8.0-openjdk/bin/java -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties -Djava.util.log...
Oct 05 09:06:09 ip-172-31-177-124.eu-west-1.compute.internal systemd[1]: Starting Apache Tomcat...
Oct 05 09:06:09 ip-172-31-177-124.eu-west-1.compute.internal startup.sh[3735]: /usr/local/apache-tomcat/bin/catalina.sh: line 514: /run/tomcat9.pid: P...enied
Oct 05 09:06:09 ip-172-31-177-124.eu-west-1.compute.internal startup.sh[3735]: Tomcat started.
Oct 05 09:06:09 ip-172-31-177-124.eu-west-1.compute.internal systemd[1]: Started Apache Tomcat.
Here are the journalctl -xe logs when Tomcat is restarted manually:
-- Unit tomcat.service has begun shutting down.
sep 17 20:00:37 ip-172-31-177-144.eu-west-1.compute.internal shutdown.sh[5311]: $CATALINA_PID was set but the specified file does not exist. Is Tomcat running?
sep 17 20:00:37 ip-172-31-177-144.eu-west-1.compute.internal systemd[1]: tomcat.service: control process exited, code=exited status=1
sep 17 20:00:37 ip-172-31-177-144.eu-west-1.compute.internal systemd[1]: tomcat.service: main process exited, code=exited, status=143/n/a
sep 17 20:00:37 ip-172-31-177-144.eu-west-1.compute.internal systemd[1]: Unit tomcat.service entered failed state.
sep 17 20:00:37 ip-172-31-177-144.eu-west-1.compute.internal systemd[1]: tomcat.service failed.
sep 17 20:00:37 ip-172-31-177-144.eu-west-1.compute.internal systemd[1]: Starting Apache Tomcat...
-- Subject: Unit tomcat.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit tomcat.service has begun starting up.
sep 17 20:00:37 ip-172-31-177-144.eu-west-1.compute.internal startup.sh[5322]: /usr/local/apache-tomcat/bin/catalina.sh: line 514: /run/tomcat9.pid: Permission
sep 17 20:00:37 ip-172-31-177-144.eu-west-1.compute.internal startup.sh[5322]: Tomcat started.
sep 17 20:00:37 ip-172-31-177-144.eu-west-1.compute.internal systemd[1]: Started Apache Tomcat.
Does anybody know what is going on here?
EDIT: Catalina/Java Logs when crash occours:
2021-10-01 08:03:03,628 ERROR org.springframework.scheduling.support.TaskUtils$LoggingErrorHandler [scheduling-1]Unexpected error occurred in scheduled task.java.lang.ExceptionInInitializerError: null at javax.crypto.JceSecurityManager.<clinit>(JceSecurityManager.java:65) at javax.crypto.Cipher.getConfiguredPermission(Cipher.java:2590) at javax.crypto.Cipher.getMaxAllowedKeyLength(Cipher.java:2614) at sun.security.ssl.CipherSuite$BulkCipher.isUnlimited(CipherSuite.java:535) at sun.security.ssl.CipherSuite$BulkCipher.<init>(CipherSuite.java:507) at sun.security.ssl.CipherSuite.<clinit>(CipherSuite.java:614) at sun.security.ssl.SSLContextImpl.getApplicableSupportedCipherSuiteList(SSLContextImpl.java:294) at sun.security.ssl.SSLContextImpl.access$100(SSLContextImpl.java:42) at sun.security.ssl.SSLContextImpl$AbstractTLSContext.<clinit>(SSLContextImpl.java:529) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at java.security.Provider$Service.getImplClass(Provider.java:1634) at java.security.Provider$Service.newInstance(Provider.java:1592) at sun.security.jca.GetInstance.getInstance(GetInstance.java:236) at sun.security.jca.GetInstance.getInstance(GetInstance.java:164) at javax.net.ssl.SSLContext.getInstance(SSLContext.java:156) at javax.net.ssl.SSLContext.getDefault(SSLContext.java:96) at javax.net.ssl.SSLSocketFactory.getDefault(SSLSocketFactory.java:122) at
Java version:
# java -version
openjdk version "1.8.0_302"
OpenJDK Runtime Environment (build 1.8.0_302-b08)
OpenJDK 64-Bit Server VM (build 25.302-b08, mixed mode)
Been digging on this for a while. I reviewed multiple articles on this issue. This one was the closest:
Tomcat 8 on CentOS 7 does not start as service (but it starts manually ....)
The difference being that I am running Tomcat 9.0.33. Here are the particulars:
java version "1.8.0_121"\
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)\
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)\
Tomcat 9.0.33
NAME="CentOS Linux"\
VERSION="7 (Core)"\
ID="centos"\
ID_LIKE="rhel fedora"\
VERSION_ID="7"\
PRETTY_NAME="CentOS Linux 7 (Core)"\
ANSI_COLOR="0;31"\
CPE_NAME="cpe:/o:centos:centos:7"\
HOME_URL="https://www.centos.org/"\
BUG_REPORT_URL="https://bugs.centos.org/"\
CENTOS_MANTISBT_PROJECT="CentOS-7"\
CENTOS_MANTISBT_PROJECT_VERSION="7"\
REDHAT_SUPPORT_PRODUCT="centos"\
REDHAT_SUPPORT_PRODUCT_VERSION="7"\
As a side note, everything was starting normally with no issues until recently. As far as I know there haven't been any major changes to the environment. But, when I ran the "systemctl restart" command recently, the startup began to fail. There are 5 instances of Tomcat 9.0.33 running at different ports and paths and those have not changed. I have not restarted two of the instance (afraid they won't start) the other three flat out won't start. Details below:
Systemd unit file for tomcat\
[Unit]\
Description=Apache Tomcat Web Application Container in Liferay 7.32 TEST for UAT\
After=syslog.target network.target
[Service]\
Type=forking
Environment=JAVA_HOME=/opt/jdk1.8.0_121/jre\
Environment=CATALINA_PID=/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33/temp/tomcat.pid\
Environment=CATALINA_HOME=/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33\
Environment=CATALINA_BASE=/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33\
Environment='CATALINA_OPTS=-Xms1024m -Xmx2048m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=20 -XX:ParallelGCThreads=8 -server -Xdebug -Xrunjdwp:transport=dt_socket,address=5000,server=y,suspend=n'\
Environment='JAVA_OPTS=-Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom -Duser.timezone=GMT -Dfile.encoding=UTF-8'
ExecStart=/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33/bin/startup.sh\
ExecStop=/bin/kill -15 $MAINPID
User=tomcat\
Group=tomcat\
UMask=0007
[Install]\
WantedBy=multi-user.target\
Results when running systemctl start liferayuat
● liferayuat.service - Apache Tomcat Web Application Container in Liferay 7.32 TEST for UAT\
Loaded: loaded (/etc/systemd/system/liferayuat.service; enabled; vendor preset: disabled)\
Active: failed (Result: exit-code) since Sat 2020-12-05 08:44:08 CST; 3s ago\
Process: 10891 ExecStop=/bin/kill -15 $MAINPID (code=exited, status=1/FAILURE)\
Process: 10851 ExecStart=/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33/bin/startup.sh \(code=exited, status=0/SUCCESS)\
Main PID: 10861 (code=exited, status=0/SUCCESS)
Dec 05 08:44:08 systemd[1]: Starting Apache Tomcat Web Application Container in Liferay 7.32 TEST for UAT...\
Dec 05 08:44:08 startup.sh[10851]: Existing PID file found during start.\
Dec 05 08:44:08 startup.sh[10851]: Removing/clearing stale PID file.\
Dec 05 08:44:08 startup.sh[10851]: Tomcat started.\
Dec 05 08:44:08 systemd[1]: Started Apache Tomcat Web Application Container in Liferay 7.32 TEST for UAT.\
Dec 05 08:44:08 systemd[1]: liferayuat.service: control process exited, code=exited status=1\
Dec 05 08:44:08 systemd[1]: Unit liferayuat.service entered failed state.\
Dec 05 08:44:08 systemd[1]: liferayuat.service failed.
Then the ONLY thing in catalina.out:
Listening for transport dt_socket at address: 5000\
java.lang.ClassNotFoundException: org.apache.catalina.startup.Catalina\
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)\
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)\
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)\
at org.apache.catalina.startup.Bootstrap.init(Bootstrap.java:261)\
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:443)\
So, when I start the instance with systemctl start it will fail. But if I run this command (as root...) then it will start:
/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33/bin/startup.sh
If I run that full commmand AS tomcat it doesn't start with the same error. So, it appears that the issue is permissions. The tomcat user and group are owners of all files and folders. But, somehow, the tomcat user either doesn't have permissions or the path gets jacked up so that the class files can't be found. I followed the suggestions in the article I referenced above but the changes had no impact.
I tripped across one article on SELINX that seemed to point to an issue there. This are the SELINUX settings:
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: permissive
Mode from config file: permissive
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 31\
The workaround to keep the instances running is just to manually start them but what is causing systemctl start NOT to work? I suspect permissions but I sure as heck can't see why since everything is owned by tomcat:tomcat
So, this is self-inflicted as most "mysteries" are. I still cannot account for some of the differences I see when looking into SELinux contexts between the instances but the REAL cause was subtle (to me). Permissions on the {tomcat root}/lib and {tomcat root}/lib/ext had no execute permissions. That may have been due to a jar that was added recently and then needed to be updated by owner and permissions. In any case, the original issue resulted in many trial and error attempts to fix it which complicated matters further.
I discovered the solution by doing a folder by folder, file by file comparison between working and non-working instances. Apparently the new jar and the owner/permission changes were applied to all but the production version.
Thanks for the suggestions.
For the past 2 months I run tomcat7 perfectly on my ubuntu server. Today tomcat7 failed to start.
I started tomcat 7
sudo systemctl start tomcat 7
Job for tomcat7.service failed because the control process exited with error code.
See "systemctl status tomcat7.service" and "journalctl -xe" for details.
Then I checked for the status
sudo systemctl status tomcat7
● tomcat7.service - LSB: Start Tomcat.
Loaded: loaded (/etc/init.d/tomcat7; generated)
Active: failed (Result: exit-code) since Sat 2020-05-16 15:47:00 HKT; 8min ago
Docs: man:systemd-sysv-generator(8)
Process: 6043 ExecStart=/etc/init.d/tomcat7 start (code=exited, status=1/FAILURE
May 16 15:46:55 www.example.com systemd[1]: Starting LSB: Start Tomcat....
May 16 15:46:55 www.example.com tomcat7[6043]: * Starting Tomcat servlet engine
May 16 15:47:00 www.example.com tomcat7[6043]: ...fail!
May 16 15:47:00 www.example.com systemd[1]: tomcat7.service: Control process exit
May 16 15:47:00 www.example.com systemd[1]: tomcat7.service: Failed with result '
May 16 15:47:00 www.example.com systemd[1]: Failed to start LSB: Start Tomcat..
Then, I checked for the log error and I have no idea how to fix this
sudo tail /etc/var/log/tomcat7/catalina.out
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
-Djava.endorsed.dirs=/usr/share/tomcat7/endorsed is not supported. Endorsed standards and standalone APIs
in modular form will be supported via the concept of upgradeable modules.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
-Djava.endorsed.dirs=/usr/share/tomcat7/endorsed is not supported. Endorsed standards and standalone APIs
in modular form will be supported via the concept of upgradeable modules.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Somebody please help ....
I've followed the docs in order to install Jetty9 as a service but whenever I run
service jetty start
It would fail with no messages, my JETTY_HOME is /opt/jetty9, contains the home distribution for version 9.4.14. I've also created my JETTY_BASE at /usr/share/jetty9 with my webapp and modules.
Both Jetty Home and Base are owned by the user jetty. I've then symlinked to my init.d folder as:
ln -s /opt/jetty9/bin/jetty.sh /etc/init.d/jetty
Then I created a /etc/default/jetty file with the following content:
# change to 1 to prevent Jetty from starting
NO_START=0
# change to 'no' or uncomment to use the default setting in /etc/default/rcS
VERBOSE=yes
# Run Jetty as this user ID (default: jetty)
# Set this to an empty string to prevent Jetty from starting automatically
JETTY_USER=jetty
# The home directory of the Java Runtime Environment (JRE). You need at least
# Java 6. If JAVA_HOME is not set, some common directories for OpenJDK and
# the Oracle JDK are tried.
#JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
# Extra options to pass to the JVM
#JAVA_OPTIONS="-Xmx256m -Djava.awt.headless=true"
# Timeout in seconds for the shutdown of all webapps
#JETTY_SHUTDOWN=30
# Additional arguments to pass to Jetty
#JETTY_ARGS=
# Jetty uses a directory to store temporary files like unpacked webapps
TMPDIR=/opt/jetty9/tmp
JETTY_HOME=/opt/jetty9
JETTY_BASE=/usr/share/jetty9
# Default for number of days to keep old log files in /var/log/jetty9/
#LOGFILE_DAYS=14
# If you run Jetty on port numbers that are all higher than 1023, then you # do not need authbind. It is used for binding Jetty to lower port numbers.
# (yes/no, default: no)
#AUTHBIND=yes
JETTY_HOST=0.0.0.0
If I start Jetty using java -jar $JETTY_HOME/start.jar in my base folder it would work with no problem. Also, if I run
service jetty supervise
It would also run with no issues, but when I call start it fails with:
root#app:/usr/share/jetty9# service jetty start
Job for jetty.service failed because the control process exited with error code.
See "systemctl status jetty.service" and "journalctl -xe" for details.
root#app:/usr/share/jetty9# service jetty status
● jetty.service - LSB: Jetty start script.
Loaded: loaded (/etc/init.d/jetty; generated)
Active: failed (Result: exit-code) since Mon 2018-12-03 15:05:26 UTC; 14s ago
Docs: man:systemd-sysv-generator(8)
Process: 21162 ExecStop=/etc/init.d/jetty stop (code=exited, status=0/SUCCESS)
Process: 21202 ExecStart=/etc/init.d/jetty start (code=exited, status=1/FAILURE)
Dec 03 15:05:22 app systemd[1]: Stopped LSB: Jetty start script..
Dec 03 15:05:22 app systemd[1]: Starting LSB: Jetty start script....
Dec 03 15:05:26 app jetty[21202]: Starting Jetty: FAILED Mon Dec 3 15:05:26 UTC 2018
Dec 03 15:05:26 app systemd[1]: jetty.service: Control process exited, code=exited status=1
Dec 03 15:05:26 app systemd[1]: jetty.service: Failed with result 'exit-code'.
Dec 03 15:05:26 app systemd[1]: Failed to start LSB: Jetty start script..
This is the output of service jetty check:
root#app:/usr/share/jetty9# service jetty check
Jetty NOT running
JAVA = /usr/bin/java
JAVA_OPTIONS = -Djetty.home=/opt/jetty9 -Djetty.base=/usr/share/jetty9 -Djava.io.tmpdir=/opt/jetty9/tmp
JETTY_HOME = /opt/jetty9
JETTY_BASE = /usr/share/jetty9
START_D = /usr/share/jetty9/start.d
START_INI = /usr/share/jetty9/start.ini
JETTY_START = /opt/jetty9/start.jar
JETTY_CONF = /opt/jetty9/etc/jetty.conf
JETTY_ARGS = jetty.state=/usr/share/jetty9/jetty.state jetty-started.xml
JETTY_RUN = /var/run/jetty
JETTY_PID = /var/run/jetty/jetty.pid
JETTY_START_LOG = /var/run/jetty/jetty-start.log
JETTY_STATE = /usr/share/jetty9/jetty.state
JETTY_START_TIMEOUT = 60
RUN_CMD = /usr/bin/java -Djetty.home=/opt/jetty9 -Djetty.base=/usr/share/jetty9 -Djava.io.tmpdir=/opt/jetty9/tmp -jar /opt/jetty9/start.jar jetty.state=/usr/share/jetty9/jetty.state jetty-started.xml
Any ideas?
UPDATE
Changing the user in /etc/default/jetty to root would solve the issue, but this is not a solution, isn't it?
# Run Jetty as this user ID (default: jetty)
# Set this to an empty string to prevent Jetty from starting automatically
JETTY_USER=root
I finally got this working, the jetty user should have permissions to the following folders and /usr/sbin/nologin as shell as described here.
JETTY_HOME
JETTY_BASE
/var/run/jetty <-- couldn't find a reference to this folder in the docs
And add the following to your /etc/default/jetty:
JETTY_SHELL=/bin/sh
JETTY_LOGS=/usr/share/jetty9/logs
JETTY_START_LOG=/usr/share/jetty9/logs/jetty-start-log.log
Also you should double check that there are no remaining log files owned by other user than jetty in your folders.
This is the status of tomcat8 in ubuntu server. but i am not able to view it in browser.
tomcat8.service - LSB: Start Tomcat.
Loaded: loaded (/etc/init.d/tomcat8; bad; vendor preset: enabled)
Active: active (running) since Sat 2017-12-16 15:48:34 IST; 1 day 19h ago
Docs: man:systemd-sysv-generator(8)
Tasks: 51 (limit: 512)
CGroup: /system.slice/tomcat8.service
└─28131 /usr/lib/jvm/default-java/bin/java -Djava.util.logging.config
Dec 16 15:48:29 dellgpu1-C6100 systemd[1]: Starting LSB: Start Tomcat....
Dec 16 15:48:29 dellgpu1-C6100 tomcat8[28104]: * Starting Tomcat servlet engine
Dec 16 15:48:34 dellgpu1-C6100 tomcat8[28104]: ...done.
Dec 16 15:48:34 dellgpu1-C6100 systemd[1]: Started LSB: Start Tomcat..
lines 1-12/12 (END)
You can check the port number in server.xml file
Go to tomcat folder>conf
open server.xml file and check for connector port number.
In browser open the tomcat home page like
http://SystemIP:connectorPORT
I all tried all possible ways. I couldn't do anything, then i contacted network administrator of my campus he solved the issue. It was a firewall issue. Thank you