Develop Reference

Tomcat 9 no longer starting using systemctl but will start manually

Been digging on this for a while. I reviewed multiple articles on this issue. This one was the closest:
Tomcat 8 on CentOS 7 does not start as service (but it starts manually ....)
The difference being that I am running Tomcat 9.0.33. Here are the particulars:
java version "1.8.0_121"\
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)\
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)\
Tomcat 9.0.33
NAME="CentOS Linux"\
VERSION="7 (Core)"\
ID="centos"\
ID_LIKE="rhel fedora"\
VERSION_ID="7"\
PRETTY_NAME="CentOS Linux 7 (Core)"\
ANSI_COLOR="0;31"\
CPE_NAME="cpe:/o:centos:centos:7"\
HOME_URL="https://www.centos.org/"\
BUG_REPORT_URL="https://bugs.centos.org/"\
CENTOS_MANTISBT_PROJECT="CentOS-7"\
CENTOS_MANTISBT_PROJECT_VERSION="7"\
REDHAT_SUPPORT_PRODUCT="centos"\
REDHAT_SUPPORT_PRODUCT_VERSION="7"\
As a side note, everything was starting normally with no issues until recently. As far as I know there haven't been any major changes to the environment. But, when I ran the "systemctl restart" command recently, the startup began to fail. There are 5 instances of Tomcat 9.0.33 running at different ports and paths and those have not changed. I have not restarted two of the instance (afraid they won't start) the other three flat out won't start. Details below:
Systemd unit file for tomcat\
[Unit]\
Description=Apache Tomcat Web Application Container in Liferay 7.32 TEST for UAT\
After=syslog.target network.target
[Service]\
Type=forking
Environment=JAVA_HOME=/opt/jdk1.8.0_121/jre\
Environment=CATALINA_PID=/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33/temp/tomcat.pid\
Environment=CATALINA_HOME=/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33\
Environment=CATALINA_BASE=/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33\
Environment='CATALINA_OPTS=-Xms1024m -Xmx2048m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=20 -XX:ParallelGCThreads=8 -server -Xdebug -Xrunjdwp:transport=dt_socket,address=5000,server=y,suspend=n'\
Environment='JAVA_OPTS=-Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom -Duser.timezone=GMT -Dfile.encoding=UTF-8'
ExecStart=/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33/bin/startup.sh\
ExecStop=/bin/kill -15 $MAINPID
User=tomcat\
Group=tomcat\
UMask=0007
[Install]\
WantedBy=multi-user.target\
Results when running systemctl start liferayuat
● liferayuat.service - Apache Tomcat Web Application Container in Liferay 7.32 TEST for UAT\
Loaded: loaded (/etc/systemd/system/liferayuat.service; enabled; vendor preset: disabled)\
Active: failed (Result: exit-code) since Sat 2020-12-05 08:44:08 CST; 3s ago\
Process: 10891 ExecStop=/bin/kill -15 $MAINPID (code=exited, status=1/FAILURE)\
Process: 10851 ExecStart=/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33/bin/startup.sh \(code=exited, status=0/SUCCESS)\
Main PID: 10861 (code=exited, status=0/SUCCESS)
Dec 05 08:44:08 systemd[1]: Starting Apache Tomcat Web Application Container in Liferay 7.32 TEST for UAT...\
Dec 05 08:44:08 startup.sh[10851]: Existing PID file found during start.\
Dec 05 08:44:08 startup.sh[10851]: Removing/clearing stale PID file.\
Dec 05 08:44:08 startup.sh[10851]: Tomcat started.\
Dec 05 08:44:08 systemd[1]: Started Apache Tomcat Web Application Container in Liferay 7.32 TEST for UAT.\
Dec 05 08:44:08 systemd[1]: liferayuat.service: control process exited, code=exited status=1\
Dec 05 08:44:08 systemd[1]: Unit liferayuat.service entered failed state.\
Dec 05 08:44:08 systemd[1]: liferayuat.service failed.
Then the ONLY thing in catalina.out:
Listening for transport dt_socket at address: 5000\
java.lang.ClassNotFoundException: org.apache.catalina.startup.Catalina\
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)\
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)\
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)\
at org.apache.catalina.startup.Bootstrap.init(Bootstrap.java:261)\
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:443)\
So, when I start the instance with systemctl start it will fail. But if I run this command (as root...) then it will start:
/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33/bin/startup.sh
If I run that full commmand AS tomcat it doesn't start with the same error. So, it appears that the issue is permissions. The tomcat user and group are owners of all files and folders. But, somehow, the tomcat user either doesn't have permissions or the path gets jacked up so that the class files can't be found. I followed the suggestions in the article I referenced above but the changes had no impact.
I tripped across one article on SELINX that seemed to point to an issue there. This are the SELINUX settings:
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: permissive
Mode from config file: permissive
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 31\
The workaround to keep the instances running is just to manually start them but what is causing systemctl start NOT to work? I suspect permissions but I sure as heck can't see why since everything is owned by tomcat:tomcat

So, this is self-inflicted as most "mysteries" are. I still cannot account for some of the differences I see when looking into SELinux contexts between the instances but the REAL cause was subtle (to me). Permissions on the {tomcat root}/lib and {tomcat root}/lib/ext had no execute permissions. That may have been due to a jar that was added recently and then needed to be updated by owner and permissions. In any case, the original issue resulted in many trial and error attempts to fix it which complicated matters further.
I discovered the solution by doing a folder by folder, file by file comparison between working and non-working instances. Apparently the new jar and the owner/permission changes were applied to all but the production version.
Thanks for the suggestions.

getting errors in oracle nodemanager - command 'SYNC_CHANGELIST' failed

after installing Oracle Weblogic 12.2.1.2 with Form and Reports 12.2.1.2 I see a lot of errors like this:
Cannot load the plugin factory oracle.forms.plugin.cam.FormsPluginFactory from the classloader. java.lang.ExceptionInInitializerError
Dec 17, 2016 11:28:15 AM oracle.dms.cam.metrics.LocalDmsChannel oracle.dms.cam.metrics.LocalDmsChannel
INFO: DMS startup class is not started. DMS runs with reduced functionality.
<Dec 17, 2016 11:28:16 AM EET> <WARNING> <Processing for FORMS server 'forms1' failed due to: [Plugin not found for system component type 'FORMS', plugin type 'PROCESS']. Please check Node Manager log and/or server 'forms1' log for detailed information.>
<Dec 17, 2016 1:04:20 PM EET> <WARNING> <Node Manager command 'SYNC_CHANGELIST' failed unexpectedly due to: [Plugin not found for system component type 'FORMS', plugin type 'REPLICATION']. Please check Node Manager log and/or server log for detailed information.>
in nodemanager log.
And I can't start ohs, getting error:
Received error message from Node Manager Server: [Node Manager command 'SYNC_CHANGELIST' failed unexpectedly due to: [Plugin not found for system component type 'FORMS', plugin type 'REPLICATION']. Please check Node Manager log and/or server log for detailed information.]. Please check Node Manager log for details.
Wasn't able to find any information in google. Maybe someone know how to fix it?

java permission error on first run of newly installed Neo4J server

I'm setting up a new server for an application (node.js with Neo4j 1.8.3) that hasn't been used for a while. I installed Oracle JDK 7 (latest), which might explain this issue, as it's not recommended
for 1.8.3, will retest with JDK 6 tomorrow.
I installed Neo4J 1.8.3 on Thursday, on Ubuntu 14.04 running on Linode. Today I reviewed the log file (neo4j.0.0.log) and saw the following error:
Oct 02, 2014 5:02:48 PM java.util.prefs.FileSystemPreferences$1 run
WARNING: Couldn't create user preferences directory. User preferences are unusable.
After that there are numerous errors as follows:
WARNING: java.io.IOException: No such file or directory
Oct 02, 2014 12:00:46 PM java.util.prefs.FileSystemPreferences checkLockFile0ErrorCode
This discussion seems relevant,
java system preferences under different users in linux
But before trying this I wanted to ask here. Is this a known problem with a standard solution?

Tomcat7 starts too late on Ubuntu 14.04 x64 [Digitalocean]

i am using digitalocean and trying to install and start tomcat on ubuntu but unfortunately i can not do it. (created new droplets and tried 10 times)
1GB Ram 30GB SSD Disk Amsterdam 2 Ubuntu 14.04 x64
When i start tomcat, it says "Tomcat started". But i can not access page from browser. and ./shutdown.sh returns error.
What can be the problem ?
I noticed something now. While i am writing this question, tomcat page is displayed. it took 28 minutes to display the page
catalina.out says: INFO: Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [1,718,769] milliseconds.
Here are my installation steps (These steps works on different vps but doesn't work on digitalocean droplets):
Install oracle jdk
sudo apt-get install python-software-properties
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java7-installer
sudo apt-get install oracle-java7-set-default
java -version
java version "1.7.0_72"
Java(TM) SE Runtime Environment (build 1.7.0_72-b14)
Java HotSpot(TM) 64-Bit Server VM (build 24.72-b04, mixed mode)
Set java path
sudo nano /etc/environment
JAVA_HOME="/usr/lib/jvm/java-7-oracle"
source /etc/environment
wget http://ftp.itu.edu.tr/Mirror/Apache/tomcat/tomcat-7/v7.0.56/bin/apache-tomcat-7.0.56.tar.gz
tar xvzf apache-tomcat-7.0.56.tar.gz
mv apache-tomcat-7.0.56/ apache-tomcat-7.0.56-server-1/
Start Tomcat
./startup.sh
Using CATALINA_BASE: /usr/local/apache-tomcat-7.0.56-server-1
Using CATALINA_HOME: /usr/local/apache-tomcat-7.0.56-server-1
Using CATALINA_TMPDIR: /usr/local/apache-tomcat-7.0.56-server-1/temp
Using JRE_HOME: /usr/lib/jvm/java-7-oracle/jre
Using CLASSPATH: /usr/local/apache-tomcat-7.0.56-server-1/bin/bootstrap.jar:/usr/local/apache-tomcat-7.0.56-server-1/bin/tomcat-juli.jar
Tomcat started.
Checkout Port 8080
netstat -ln
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp6 0 0 :::8009 :::* LISTEN
tcp6 0 0 :::8080 :::* LISTEN
tcp6 0 0 :::22 :::* LISTEN
Checkout process
ps -ef | grep tomcat
root 2825 1 1 14:23 pts/0 00:00:03 /usr/lib/jvm/java-7-oracle/jre/bin/java -Djava.util.logging.config.file=/usr/local/apache-tomcat-7.0.56-server-1/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/usr/local/apache-tomcat-7.0.56-server-1/endorsed -classpath /usr/local/apache-tomcat-7.0.56-server-1/bin/bootstrap.jar:/usr/local/apache-tomcat-7.0.56-server-1/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat-7.0.56-server-1 -Dcatalina.home=/usr/local/apache-tomcat-7.0.56-server-1 -Djava.io.tmpdir=/usr/local/apache-tomcat-7.0.56-server-1/temp org.apache.catalina.startup.Bootstrap start
Open web site at port 8080 http://5.101.107.56:8080/ Page is waiting... [content is displayed after 28 minute or more]
Try to shutdown tomcat if content is not displayed yet (before tomcat starts properly).
./shutdown.sh
SEVERE: Could not contact localhost:8005. Tomcat may not be running.
Oct 17, 2014 2:40:29 PM org.apache.catalina.startup.Catalina stopServer
SEVERE: Catalina.stop:
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSoc
Checkout logs
catalina.out
Oct 17, 2014 2:31:47 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["ajp-bio-8009"]
Oct 17, 2014 2:31:47 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 1492 ms
Oct 17, 2014 2:31:47 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Catalina
Oct 17, 2014 2:31:47 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.56
Oct 17, 2014 2:31:47 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /usr/local/apache-tomcat-7.0.56-server-1/webapps/host-manager
I also installed nginx and navigate to http://5.XXX.XXX.XX/ nginx welcome page is opened immediately
I checked catalina.out when i see the page in browser, it says:
Oct 17, 2014 2:31:47 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /usr/local/apache-tomcat-7.0.56-server-1/webapps/host-manager
Oct 17, 2014 3:00:27 PM org.apache.catalina.util.SessionIdGenerator createSecureRandom
INFO: Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took **[1,718,769] milliseconds.**
Memory:
total used free shared buffers cached
Mem: 1017912 849512 168400 332 18780 688468

Replacing securerandom.source=file:/dev/urandom with securerandom.source=file:/dev/./urandom in $JAVA_PATH/jre/lib/security/java.security has solved my problem.
Even when file:/dev/urandom is specified, JRE will still use /dev/random for SHA1PRNG (see bug JDK-4705093):
In SHA1PRNG, there is a SeedGenerator which does various things
depending on the configuration.
If java.security.egd or securerandom.source point to "file:/dev/random" or "file:/dev/urandom", we will use
NativeSeedGenerator, which calls super() which calls
SeedGenerator.URLSeedGenerator(/dev/random). (A nested class within
SeedGenerator.) The only things that changed in this bug was that
urandom will also trigger use of this code path.
If those properties point to another URL that exists, we'll initialize SeedGenerator.URLSeedGenerator(url). This is why
"file:///dev/urandom", "file:/./dev/random", etc. will work.
From Wikipedia on /dev/random:
In this implementation, the generator keeps an estimate of the number
of bits of noise in the entropy pool. From this entropy pool random
numbers are created. When read, the /dev/random device will only
return random bytes within the estimated number of bits of noise in
the entropy pool. /dev/random should be suitable for uses that need
very high quality randomness such as one-time pad or key generation.
When the entropy pool is empty, reads from /dev/random will block
until additional environmental noise is gathered. The intent is to
serve as a cryptographically secure pseudorandom number generator,
delivering output with entropy as large as possible. This is suggested
for use in generating cryptographic keys for high-value or long-term
protection.
Environmental noise?
The random number generator gathers environmental noise from device
drivers and other sources into an entropy pool. The generator also
keeps an estimate of the number of bits of noise in the entropy pool.
From this entropy pool random numbers are created.
That means in practice, it’s possible to block tomcat for an unknown amount of time.

This also works:
Actually, by setting the following in /etc/default/tomcat7, I was fine:
JAVA_OPTS="-Djava.security.egd=file:/dev/./urandom -Djava.awt.headless=true -Xmx1024m -XX:MaxPermSize=512m -XX:+UseConcMarkSweepGC"
Comment from :
https://www.digitalocean.com/community/tutorials/how-to-install-apache-tomcat-7-on-ubuntu-14-04-via-apt-get

While using /dev/urandom as the source for entropy is a workaround that reduces the startup time for Tomcat, it is not a good idea because it can have unintended side effects.
Other components running in the Tomcat server (e.g. web applications) might depend on a securely initialized SecureRandom instance and there might be security issues when the entropy for the random numbers is not sufficient.
Actually, this is one of the reasons why using /dev/urandom does not work, but /dev/./urandom does. The SHA1PRNG heavily relies on a good seed. If the seed is not good, the random numbers are predictable. Therefore, the developer ensured that for this purpose /dev/random is used as the source of entropy, even if the JVM is configured to use /dev/urandom. There are two bug reports about this (bug 1, bug 2).
So instead of changing the entropy source to /dev/urandom, one should rather make sure that /dev/random has enough entropy. If the system has a hardware RNG, installing rng-tools should do the trick. Otherwise, installing haveged provides a very good source of entropy that does not rely on a special hardware RNG to be present. In a virtual machine, rng-tools can use entropy from the host through a virtual hardware RNG. As an alternative to this, EGD could be used, but at the moment this software is not included in the Ubuntu repositories, so that it is bothersome to use it.

Google App Engine - How to get demos to work?

I'm embarrassed to have to ask this, but I can't for the life of me figure out what I'm doing wrong.
I've downloaded the appengine-java-sdk-1.4.2 via the eclipse 3.6 plugin as per the instructions on Installing the Java SDK page. I am running Mac OSX 10.5.8 with JRE version 1.5.0_26. I have made the dev_appserver.sh file executable (chmod u+x dev_appserver.sh) and when I attempt to start the app server with one of the demos via the command
bin/dev_appserver.sh demos/guestbook/war/
I get the following output, but the server does not start up:
2011-03-13 17:52:43.404 java[839:80f] [Java CocoaComponent compatibility mode]: Enabled
2011-03-13 17:52:43.405 java[839:80f] [Java CocoaComponent compatibility mode]: Setting timeout for SWT to 0.100000
********************************************************
Warning: Future versions of the Dev App Server will require Java 1.6 or later. Please upgrade your JRE.
********************************************************
Mar 13, 2011 9:52:44 PM com.google.apphosting.utils.jetty.JettyLogger info
INFO: Logging to JettyLogger(null) via com.google.apphosting.utils.jetty.JettyLogger
Mar 13, 2011 9:52:44 PM com.google.apphosting.utils.config.AppEngineWebXmlReader readAppEngineWebXml
INFO: Successfully processed /Applications/eclipse_3.6/plugins/com.google.appengine.eclipse.sdkbundle.1.4.2_1.4.2.v201102111811/appengine-java-sdk-1.4.2/demos/guestbook/war/WEB-INF/appengine-web.xml
Mar 13, 2011 9:52:44 PM com.google.apphosting.utils.config.AbstractConfigXmlReader readConfigXml
INFO: Successfully processed /Applications/eclipse_3.6/plugins/com.google.appengine.eclipse.sdkbundle.1.4.2_1.4.2.v201102111811/appengine-java-sdk-1.4.2/demos/guestbook/war/WEB-INF/web.xml
I see the warning about upgrading the JRE, but I don't think this is the problem, as there are no exceptions reported. It just won't start up. I've tried creating a hello world app from within Eclipse using the plugin and running as Web server, but that also spits out the same text but does not start up the web server.
Any idea what I'm doing wrong and how to remedy the problem?

By changing the line in dev_appserver.sh to
/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Commands/java -ea -cp "$JAR_FILE" \
com.google.appengine.tools.KickStart \
com.google.appengine.tools.development.DevAppServerMain $*
instead of using the default java (which is set to 1.5.0_26 as I mentioned earlier) the web server starts up.

I had the same problem and simply tried running Eclipse as Administrator and then debugging worked fine.