Develop Reference

Tomcat 9 no longer starting using systemctl but will start manually

Been digging on this for a while. I reviewed multiple articles on this issue. This one was the closest:
Tomcat 8 on CentOS 7 does not start as service (but it starts manually ....)
The difference being that I am running Tomcat 9.0.33. Here are the particulars:
java version "1.8.0_121"\
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)\
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)\
Tomcat 9.0.33
NAME="CentOS Linux"\
VERSION="7 (Core)"\
ID="centos"\
ID_LIKE="rhel fedora"\
VERSION_ID="7"\
PRETTY_NAME="CentOS Linux 7 (Core)"\
ANSI_COLOR="0;31"\
CPE_NAME="cpe:/o:centos:centos:7"\
HOME_URL="https://www.centos.org/"\
BUG_REPORT_URL="https://bugs.centos.org/"\
CENTOS_MANTISBT_PROJECT="CentOS-7"\
CENTOS_MANTISBT_PROJECT_VERSION="7"\
REDHAT_SUPPORT_PRODUCT="centos"\
REDHAT_SUPPORT_PRODUCT_VERSION="7"\
As a side note, everything was starting normally with no issues until recently. As far as I know there haven't been any major changes to the environment. But, when I ran the "systemctl restart" command recently, the startup began to fail. There are 5 instances of Tomcat 9.0.33 running at different ports and paths and those have not changed. I have not restarted two of the instance (afraid they won't start) the other three flat out won't start. Details below:
Systemd unit file for tomcat\
[Unit]\
Description=Apache Tomcat Web Application Container in Liferay 7.32 TEST for UAT\
After=syslog.target network.target
[Service]\
Type=forking
Environment=JAVA_HOME=/opt/jdk1.8.0_121/jre\
Environment=CATALINA_PID=/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33/temp/tomcat.pid\
Environment=CATALINA_HOME=/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33\
Environment=CATALINA_BASE=/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33\
Environment='CATALINA_OPTS=-Xms1024m -Xmx2048m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=20 -XX:ParallelGCThreads=8 -server -Xdebug -Xrunjdwp:transport=dt_socket,address=5000,server=y,suspend=n'\
Environment='JAVA_OPTS=-Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom -Duser.timezone=GMT -Dfile.encoding=UTF-8'
ExecStart=/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33/bin/startup.sh\
ExecStop=/bin/kill -15 $MAINPID
User=tomcat\
Group=tomcat\
UMask=0007
[Install]\
WantedBy=multi-user.target\
Results when running systemctl start liferayuat
● liferayuat.service - Apache Tomcat Web Application Container in Liferay 7.32 TEST for UAT\
Loaded: loaded (/etc/systemd/system/liferayuat.service; enabled; vendor preset: disabled)\
Active: failed (Result: exit-code) since Sat 2020-12-05 08:44:08 CST; 3s ago\
Process: 10891 ExecStop=/bin/kill -15 $MAINPID (code=exited, status=1/FAILURE)\
Process: 10851 ExecStart=/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33/bin/startup.sh \(code=exited, status=0/SUCCESS)\
Main PID: 10861 (code=exited, status=0/SUCCESS)
Dec 05 08:44:08 systemd[1]: Starting Apache Tomcat Web Application Container in Liferay 7.32 TEST for UAT...\
Dec 05 08:44:08 startup.sh[10851]: Existing PID file found during start.\
Dec 05 08:44:08 startup.sh[10851]: Removing/clearing stale PID file.\
Dec 05 08:44:08 startup.sh[10851]: Tomcat started.\
Dec 05 08:44:08 systemd[1]: Started Apache Tomcat Web Application Container in Liferay 7.32 TEST for UAT.\
Dec 05 08:44:08 systemd[1]: liferayuat.service: control process exited, code=exited status=1\
Dec 05 08:44:08 systemd[1]: Unit liferayuat.service entered failed state.\
Dec 05 08:44:08 systemd[1]: liferayuat.service failed.
Then the ONLY thing in catalina.out:
Listening for transport dt_socket at address: 5000\
java.lang.ClassNotFoundException: org.apache.catalina.startup.Catalina\
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)\
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)\
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)\
at org.apache.catalina.startup.Bootstrap.init(Bootstrap.java:261)\
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:443)\
So, when I start the instance with systemctl start it will fail. But if I run this command (as root...) then it will start:
/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33/bin/startup.sh
If I run that full commmand AS tomcat it doesn't start with the same error. So, it appears that the issue is permissions. The tomcat user and group are owners of all files and folders. But, somehow, the tomcat user either doesn't have permissions or the path gets jacked up so that the class files can't be found. I followed the suggestions in the article I referenced above but the changes had no impact.
I tripped across one article on SELINX that seemed to point to an issue there. This are the SELINUX settings:
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: permissive
Mode from config file: permissive
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 31\
The workaround to keep the instances running is just to manually start them but what is causing systemctl start NOT to work? I suspect permissions but I sure as heck can't see why since everything is owned by tomcat:tomcat

So, this is self-inflicted as most "mysteries" are. I still cannot account for some of the differences I see when looking into SELinux contexts between the instances but the REAL cause was subtle (to me). Permissions on the {tomcat root}/lib and {tomcat root}/lib/ext had no execute permissions. That may have been due to a jar that was added recently and then needed to be updated by owner and permissions. In any case, the original issue resulted in many trial and error attempts to fix it which complicated matters further.
I discovered the solution by doing a folder by folder, file by file comparison between working and non-working instances. Apparently the new jar and the owner/permission changes were applied to all but the production version.
Thanks for the suggestions.

Current Language: ~~ERROR~~ NullPointerException: null (minecraft modding eclipse)

I've been trying to make a minecraft mod in eclipse but whenever I start it the minecraft window it flashes open and then closes and I get a crash report.
---- Minecraft Crash Report ----
Time: 6/2/20 9:43 PM
Description: Initializing game
java.lang.ExceptionInInitializerError
at net.minecraftforge.fml.client.FMLClientHandler.beginMinecraftLoading(FMLClientHandler.java:217)
at net.minecraft.client.Minecraft.init(Minecraft.java:467)
at net.minecraft.client.Minecraft.run(Minecraft.java:378)
at net.minecraft.client.main.Main.main(Main.java:118)
Caused by: java.lang.NullPointerException
at net.minecraftforge.fml.client.SplashProgress.createResourcePack(SplashProgress.java:731)
at net.minecraftforge.fml.client.SplashProgress.<clinit>(SplashProgress.java:99)
... 4 more
A detailed walkthrough of the error, its code path and all known details is as follows:
-- Head --
Thread: Client thread
Stacktrace:
at net.minecraftforge.fml.client.FMLClientHandler.beginMinecraftLoading(FMLClientHandler.java:217)
at net.minecraft.client.Minecraft.init(Minecraft.java:467)
-- Initialization --
Details:
Stacktrace:
at net.minecraft.client.Minecraft.run(Minecraft.java:378)
at net.minecraft.client.main.Main.main(Main.java:118)
-- System Details --
Details:
Minecraft Version: 1.12.2
Operating System: Windows 10 (amd64) version 10.0
Java Version: 1.8.0_251, Oracle Corporation
Java VM Version: Java HotSpot(TM) 64-Bit Server VM (mixed mode), Oracle Corporation
Memory: 197682624 bytes (188 MB) / 253755392 bytes (242 MB) up to 3782737920 bytes (3607 MB)
JVM Flags: 0 total;
IntCache: cache: 0, tcache: 0, allocated: 0, tallocated: 0
FML:
Loaded coremods (and transformers):
Launched Version: 1.12.2
LWJGL: 2.9.4
OpenGL: Radeon (TM) RX 480 Graphics GL version 4.6.13587 Compatibility Profile Context 20.4.2 26.20.15029.27017, ATI Technologies Inc.
GL Caps: Using GL 1.3 multitexturing.
Using GL 1.3 texture combiners.
Using framebuffer objects because OpenGL 3.0 is supported and separate blending is supported.
Shaders are available because OpenGL 2.1 is supported.
VBOs are available because OpenGL 1.5 is supported.
Using VBOs: Yes
Is Modded: Definitely; Client brand changed to 'fml,forge'
Type: Client (map_client.txt)
Resource Packs:
Current Language: ~~ERROR~~ NullPointerException: null
Profiler Position: N/A (disabled)
CPU: 8x Intel(R) Core(TM) i7-4770K CPU # 3.50GHz
I've tried everything I can find on how to fix this (drivers are up to date, I have java 8, minecraft 1.12.2).
Normal minecraft works completely fine and my modpacks using twitch launcher work fine, it's only in eclipse that minecraft doesn't start.

ubuntu server 16.04 - tomcat7 failed to start

For the past 2 months I run tomcat7 perfectly on my ubuntu server. Today tomcat7 failed to start.
I started tomcat 7
sudo systemctl start tomcat 7
Job for tomcat7.service failed because the control process exited with error code.
See "systemctl status tomcat7.service" and "journalctl -xe" for details.
Then I checked for the status
sudo systemctl status tomcat7
● tomcat7.service - LSB: Start Tomcat.
Loaded: loaded (/etc/init.d/tomcat7; generated)
Active: failed (Result: exit-code) since Sat 2020-05-16 15:47:00 HKT; 8min ago
Docs: man:systemd-sysv-generator(8)
Process: 6043 ExecStart=/etc/init.d/tomcat7 start (code=exited, status=1/FAILURE
May 16 15:46:55 www.example.com systemd[1]: Starting LSB: Start Tomcat....
May 16 15:46:55 www.example.com tomcat7[6043]: * Starting Tomcat servlet engine
May 16 15:47:00 www.example.com tomcat7[6043]: ...fail!
May 16 15:47:00 www.example.com systemd[1]: tomcat7.service: Control process exit
May 16 15:47:00 www.example.com systemd[1]: tomcat7.service: Failed with result '
May 16 15:47:00 www.example.com systemd[1]: Failed to start LSB: Start Tomcat..
Then, I checked for the log error and I have no idea how to fix this
sudo tail /etc/var/log/tomcat7/catalina.out
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
-Djava.endorsed.dirs=/usr/share/tomcat7/endorsed is not supported. Endorsed standards and standalone APIs
in modular form will be supported via the concept of upgradeable modules.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
-Djava.endorsed.dirs=/usr/share/tomcat7/endorsed is not supported. Endorsed standards and standalone APIs
in modular form will be supported via the concept of upgradeable modules.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Somebody please help ....

Jelastic GC agent does not work with Tomcat 8.5.x

I use Jelastic platform from different providers: dogado.de (Jelastic version is 4.6.2) and mirhosting.com (Jelastic version is 4.6.1). I have some environments on both platforms. These environments have next configuration:
Java 8
Apache Tomcat 8.5.3
MySQL 5.7.10
The Tomcat prints next info to the log file:
Server version: Apache Tomcat/8.5.3
Server number: 8.5.3.0
OS Name: Linux
OS Version: 2.6.32-042stab113.21
Architecture: amd64
Java Home: /usr/java/jdk1.8.0_72/jre
JVM Version: 1.8.0_72-b15
JVM Vendor: Oracle Corporation
CATALINA_BASE: /opt/repo/versions/8.5.3
CATALINA_HOME: /opt/repo/versions/8.5.3
I'm trying to enable Jelastic GC agent. So I changed the conf/variables.conf file, so now it contains next lines:
-javaagent:/opt/repo/versions/8.5.3/lib/jelastic-gc-agent.jar=debug=true,period=60
It means that the debug mode should be enabled now and the agent must print every 60 seconds the info about memory releasing. For previous Tomcat version (7.0.39; on the same platform, but another environment) it looks like this:
Jul 14, 2016 6:08:30 PM com.jelastic.java.gc.JelasticGCAgent$1 run
INFO: JelasticGCAgent - Start Full GC : [free memory] : 181834896 bytes
Jul 14, 2016 6:08:30 PM com.jelastic.java.gc.JelasticGCAgent$1 run
INFO: JelasticGCAgent - Finish Full GC : [free memory] : 74885120 bytes
But it does not work for Tomcat 8: memory usage is not changing at all time, there are no any new messages in the log file. I asked supports of these providers how to fix this issue, but the issue still there. And even more it looks like a bug in Jelastic agent or in the whole platform.
Did anybody face with the same issue already? Any known ways to fix it? Maybe I need to use another jar files?
Any ideas are welcome and of course I want to ask Jelastic team about this problem.

It seems to be a problem with configuration file for variables parsing.
As a workaround:
log into your Tomcat node via SSH
navigate to /opt/repo/versions/8.5.3/bin/
edit variablesparser.sh file and change the third line from
CONFFILE='/opt/repo/versions/${Version}/conf/variables.conf'
to
CONFFILE="/opt/repo/versions/${Version}/conf/variables.conf"
restart Tomcat

JBoss crashes on libclient64.dylib access

I´m trying to get Oracle instantclient up and running on OS X with Java 1.6.0_65. I did all steps described in the Oracle documentation: https://docs.oracle.com/cd/E11882_01/install.112/e38228/toc.htm
but the problem keeps the same:
Invalid memory access of location 0x0 rip=0x106369f87
The stack trace is:
Process: java [6234]
Path: /Library/Java/JavaVirtualMachines/jdk1.6.0_65.jdk/Contents/Home/bin/java
Identifier: com.apple.javajdk16.cmd
Version: 1.0 (1.0)
Code Type: X86-64 (Native)
Parent Process: sh [6222]
Responsible: Terminal [1179]
User ID: 33291
PlugIn Path: /Library/Java/JavaVirtualMachines/jdk1.6.0_65.jdk/Contents/Home/bundle/Libraries/libclient64.dylib
PlugIn Identifier: libclient64.dylib
PlugIn Version: ??? (1)
Date/Time: 2015-07-28 11:22:49.211 +0200
OS Version: Mac OS X 10.10.4 (14E46)
Report Version: 11
Anonymous UUID: 54BA4C92-323A-644A-55CF-CDBEDA054F4E
Time Awake Since Boot: 5500 seconds
Crashed Thread: 27 Java: main
Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x0000000000000000
VM Regions Near 0:
-->
__TEXT 0000000106233000-000000010623b000 [ 32K] r-x/rwx SM=COW /Library/Java/JavaVirtualMachines/jdk1.6.0_65.jdk/Contents/Home/bin/java
Application Specific Information:
Java information:
Exception type: Bus Error (0xa) at pc=106369f87
Java VM: Java HotSpot(TM) 64-Bit Server VM (20.65-b04-462 mixed mode macosx-amd64)
Current thread (7fddcf86d800): JavaThread "main" [_thread_in_vm, id=309616640, stack(112646000,112746000)]
Stack: [112646000,112746000]
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j oracle.jdbc.driver.T2CConnection.t2cSetSessionTimeZone(JLjava/lang/String;)I+0
j oracle.jdbc.driver.T2CConnection.logon()V+825
j oracle.jdbc.driver.PhysicalConnection.<init>(Ljava/lang/String;Ljava/util/Properties;Loracle/jdbc/driver/OracleDriverExtension;)V+323
....
I tried with both 32 and 64 versions. I did it using the appropriate client version together with either activating or deactivating the -d32 flag on the application start.
Both seam to have the same problem.
Does anyone has an idea what can be wrong here?
Thanks
UPDATE:
I replaced OCI by THIN and moved the crash ahead.
jdbc:oracle:thin:#${dbserver:our.domain.de}:${dbport:1234}:${dbsid:OURSID}
jdbc:oracle:oci:#(description=(ADDRESS=(COMMUNITY=tcp.world)(PROTOCOL=TCP)(Host=our.domain.de)(Port=1521))(connect_data=(sid=OURSID)))
It crashes now after reading of the data sources:
12:53:52,490 INFO [ConnectionProviderFactory] Initializing connection provider: org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider
12:53:52,490 INFO [InjectedDataSourceConnectionProvider] Using provided datasource
Invalid memory access of location 0x0 rip=0x10ff50f87

AFAIK client libs were not well maintained by Oracle and they simply did not support the newest OS X version - for years. They crashed on Mac
Try to use 12c drivers. But when even thin drivers crash your JVM, then there must be something wrong with your Java installation. Maybe you have multiple JDBC drivers in your classpath?