Develop Reference

Tomcat 9 no longer starting using systemctl but will start manually

Been digging on this for a while. I reviewed multiple articles on this issue. This one was the closest:
Tomcat 8 on CentOS 7 does not start as service (but it starts manually ....)
The difference being that I am running Tomcat 9.0.33. Here are the particulars:
java version "1.8.0_121"\
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)\
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)\
Tomcat 9.0.33
NAME="CentOS Linux"\
VERSION="7 (Core)"\
ID="centos"\
ID_LIKE="rhel fedora"\
VERSION_ID="7"\
PRETTY_NAME="CentOS Linux 7 (Core)"\
ANSI_COLOR="0;31"\
CPE_NAME="cpe:/o:centos:centos:7"\
HOME_URL="https://www.centos.org/"\
BUG_REPORT_URL="https://bugs.centos.org/"\
CENTOS_MANTISBT_PROJECT="CentOS-7"\
CENTOS_MANTISBT_PROJECT_VERSION="7"\
REDHAT_SUPPORT_PRODUCT="centos"\
REDHAT_SUPPORT_PRODUCT_VERSION="7"\
As a side note, everything was starting normally with no issues until recently. As far as I know there haven't been any major changes to the environment. But, when I ran the "systemctl restart" command recently, the startup began to fail. There are 5 instances of Tomcat 9.0.33 running at different ports and paths and those have not changed. I have not restarted two of the instance (afraid they won't start) the other three flat out won't start. Details below:
Systemd unit file for tomcat\
[Unit]\
Description=Apache Tomcat Web Application Container in Liferay 7.32 TEST for UAT\
After=syslog.target network.target
[Service]\
Type=forking
Environment=JAVA_HOME=/opt/jdk1.8.0_121/jre\
Environment=CATALINA_PID=/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33/temp/tomcat.pid\
Environment=CATALINA_HOME=/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33\
Environment=CATALINA_BASE=/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33\
Environment='CATALINA_OPTS=-Xms1024m -Xmx2048m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=20 -XX:ParallelGCThreads=8 -server -Xdebug -Xrunjdwp:transport=dt_socket,address=5000,server=y,suspend=n'\
Environment='JAVA_OPTS=-Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom -Duser.timezone=GMT -Dfile.encoding=UTF-8'
ExecStart=/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33/bin/startup.sh\
ExecStop=/bin/kill -15 $MAINPID
User=tomcat\
Group=tomcat\
UMask=0007
[Install]\
WantedBy=multi-user.target\
Results when running systemctl start liferayuat
● liferayuat.service - Apache Tomcat Web Application Container in Liferay 7.32 TEST for UAT\
Loaded: loaded (/etc/systemd/system/liferayuat.service; enabled; vendor preset: disabled)\
Active: failed (Result: exit-code) since Sat 2020-12-05 08:44:08 CST; 3s ago\
Process: 10891 ExecStop=/bin/kill -15 $MAINPID (code=exited, status=1/FAILURE)\
Process: 10851 ExecStart=/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33/bin/startup.sh \(code=exited, status=0/SUCCESS)\
Main PID: 10861 (code=exited, status=0/SUCCESS)
Dec 05 08:44:08 systemd[1]: Starting Apache Tomcat Web Application Container in Liferay 7.32 TEST for UAT...\
Dec 05 08:44:08 startup.sh[10851]: Existing PID file found during start.\
Dec 05 08:44:08 startup.sh[10851]: Removing/clearing stale PID file.\
Dec 05 08:44:08 startup.sh[10851]: Tomcat started.\
Dec 05 08:44:08 systemd[1]: Started Apache Tomcat Web Application Container in Liferay 7.32 TEST for UAT.\
Dec 05 08:44:08 systemd[1]: liferayuat.service: control process exited, code=exited status=1\
Dec 05 08:44:08 systemd[1]: Unit liferayuat.service entered failed state.\
Dec 05 08:44:08 systemd[1]: liferayuat.service failed.
Then the ONLY thing in catalina.out:
Listening for transport dt_socket at address: 5000\
java.lang.ClassNotFoundException: org.apache.catalina.startup.Catalina\
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)\
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)\
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)\
at org.apache.catalina.startup.Bootstrap.init(Bootstrap.java:261)\
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:443)\
So, when I start the instance with systemctl start it will fail. But if I run this command (as root...) then it will start:
/opt/liferay/uatapi/liferay-ce-portal-7.3.2-ga3/tomcat-9.0.33/bin/startup.sh
If I run that full commmand AS tomcat it doesn't start with the same error. So, it appears that the issue is permissions. The tomcat user and group are owners of all files and folders. But, somehow, the tomcat user either doesn't have permissions or the path gets jacked up so that the class files can't be found. I followed the suggestions in the article I referenced above but the changes had no impact.
I tripped across one article on SELINX that seemed to point to an issue there. This are the SELINUX settings:
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: permissive
Mode from config file: permissive
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 31\
The workaround to keep the instances running is just to manually start them but what is causing systemctl start NOT to work? I suspect permissions but I sure as heck can't see why since everything is owned by tomcat:tomcat

So, this is self-inflicted as most "mysteries" are. I still cannot account for some of the differences I see when looking into SELinux contexts between the instances but the REAL cause was subtle (to me). Permissions on the {tomcat root}/lib and {tomcat root}/lib/ext had no execute permissions. That may have been due to a jar that was added recently and then needed to be updated by owner and permissions. In any case, the original issue resulted in many trial and error attempts to fix it which complicated matters further.
I discovered the solution by doing a folder by folder, file by file comparison between working and non-working instances. Apparently the new jar and the owner/permission changes were applied to all but the production version.
Thanks for the suggestions.

Issue in Jenkins installation plugins

I am facing issue while installing any of the Jenkins plugins suggested.
Actually after downloading,Jenkins.war file(which is latest 2.141) when i tried to execute the jar with
java -jar jenkins.war so it gave me an error of Jenkins require java 8 but you are using 10. Also,it says that java class version 54.0 is running,but it requires java 52.0.
But I was able to resolve this issue by setting --enable-future-java flag.
java -jar jenkins.war --enable-future-java flag
Now,after writing this command,jenkins is up and running but i am unable to install the plugins.
Also,im cmd prompt after the Jenkins is upa d running.There is one error also.
PFB :-
Sep 17, 2018 4:38:49 PM hudson.WebAppMain$3 run
INFO: Jenkins is fully up and running
[31mSep 17, 2018 4:39:02 PM hudson.model.UpdateSite updateData
SEVERE: ERROR: SHA-512 based signature in the update center doesn't match with the certificate in 'update site 'default''
[0mSep 17, 2018 4:39:02 PM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished Download metadata. 15,407 ms

You need to add a flag which allows starting Jenkins with unsupported Java versions. You can do some google research on this.

If we change the Jenkins war file version from 2.141 to 2.814 then it is suitable with Java 10 and works fine for plugins installation of Jenkins

Issue with Redis setup cluster mode in docker(Windows 7)

I am trying to set up Redis in cluster mode and when I try to connect to Redis using Jedis API, I am seeing below exception.
Exception in thread "main" redis.clients.jedis.exceptions.JedisNoReachableClusterNodeException: No reachable node in cluster
at redis.clients.jedis.JedisSlotBasedConnectionHandler.getConnection(JedisSlotBasedConnectionHandler.java:57)
at redis.clients.jedis.JedisSlotBasedConnectionHandler.getConnectionFromSlot(JedisSlotBasedConnectionHandler.java:74)
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:116)
at redis.clients.jedis.JedisClusterCommand.run(JedisClusterCommand.java:31)
at redis.clients.jedis.JedisCluster.set(JedisCluster.java:103)
at com.redis.main.Main.main(Main.java:18)
I am using below command to start the Redis
$ docker run -v /d/redis.conf:/usr/bin/redis.conf --name myredis redis redis-server /usr/bin/redis.conf
And my simple redis.conf looks like below.
port 6379
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
And below are redis start up logs.
$ docker run -v /d/redis.conf:/usr/bin/redis.conf --name myredis redis redis-se
rver /usr/bin/redis.conf
1:C 11 Oct 18:06:01.657 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 11 Oct 18:06:01.663 # Redis version=4.0.2, bits=64, commit=00000000, modifi
d=0, pid=1, just started
1:C 11 Oct 18:06:01.664 # Configuration loaded
1:M 11 Oct 18:06:01.685 * Running mode=standalone, port=6379.
1:M 11 Oct 18:06:01.690 # WARNING: The TCP backlog setting of 511 cannot be enf
rced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 11 Oct 18:06:01.692 # Server initialized
1:M 11 Oct 18:06:01.696 # WARNING overcommit_memory is set to 0! Background sav
may fail under low memory condition. To fix this issue add 'vm.overcommit_memo
y = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overco
mit_memory=1' for this to take effect.
1:M 11 Oct 18:06:01.697 # WARNING you have Transparent Huge Pages (THP) support
enabled in your kernel. This will create latency and memory usage issues with R
dis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent
hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain
he setting after a reboot. Redis must be restarted after THP is disabled.
1:M 11 Oct 18:06:01.700 * Ready to accept connections
And below is the simple java program.
public class Main {
public static void main(String[] args) {
Set<HostAndPort> jedisClusterNodes = new HashSet<HostAndPort>();
jedisClusterNodes.add(new HostAndPort("127.0.0.1", 6379));
JedisCluster jc = new JedisCluster(jedisClusterNodes);
//Jedis jc = new Jedis("192.168.99.100");
jc.set("prime", "1 is prime");
String keyVal = jc.get("prime");
System.out.println(keyVal);
}
}
Not really sure what is going wrong here and will appreciate any help on this.

You need to expose the port when starting the redis container
docker run -v /d/redis.conf:/usr/bin/redis.conf -p 6379:6379 --name myredis redis redis-server /usr/bin/redis.conf

neo4j failing to start after disk full

i had the /var disk full on my debian server and neo4j stopped as expected. I freed up the space on the disk but the neo4j server does not start throwing the error given below in the logs. I don't have any java or neo4j process running on the server that i may kill.
Things were stable with my neo4j setup for the past 6 months with around 1000 nodes. I am little novice on the java side so please let me know if i have missed out on anything basic.
Output on command line: service neo4j-service restart
Restarting Neo4j Graph Database: neo4jWARNING: Max 1024 open files
allowed, minimum of 40 000 recommended. See the Neo4j manual. WARNING!
You are using an unsupported Java runtime.
* Please use Oracle(R) Java(TM) 7 to run Neo4j Server. Download "Java Platform (JDK) 7" from:
http://www.oracle.com/technetwork/java/javase/downloads/index.html
* Please see http://docs.neo4j.org/ for Neo4j Server installation instructions. Using additional JVM arguments: -server
-XX:+DisableExplicitGC -Dorg.neo4j.server.properties=conf/neo4j-server.properties -Djava.util.logging.config.file=conf/logging.properties -Dlog4j.configuration=file:conf/log4j.properties -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled Starting Neo4j Server...WARNING: not changing user process [4733]... waiting for
server to be ready..... Failed to start within 120 seconds. Neo4j
Server may have failed to start, please check the logs. failed!
Logfile:
Sep 26, 2014 3:31:38 PM org.neo4j.server.logging.Logger log
SEVERE: Failed to start Neo Server on port [7474]
Sep 26, 2014 3:33:00 PM org.neo4j.server.logging.Logger log
WARNING: You are using an unsupported Java runtime. Please use Oracle(R) Java(TM) Runtime Environment 7.
Sep 26, 2014 3:33:00 PM org.neo4j.server.logging.Logger log
INFO: Setting startup timeout to: 120000ms based on -1
Sep 26, 2014 3:33:03 PM org.neo4j.server.logging.Logger log
SEVERE:
org.neo4j.server.ServerStartupException: Starting Neo4j Server failed: Error starting org.neo4j.kernel.EmbeddedGraphDatabase, /var/lib/neo4j/data/graph.db
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:218)
at org.neo4j.server.Bootstrapper.start(Bootstrapper.java:87)
at org.neo4j.server.Bootstrapper.main(Bootstrapper.java:50)
Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.EmbeddedGraphDatabase, /var/lib/neo4j/data/graph.db
at org.neo4j.kernel.InternalAbstractGraphDatabase.run(InternalAbstractGraphDatabase.java:330)
at org.neo4j.kernel.EmbeddedGraphDatabase.<init>(EmbeddedGraphDatabase.java:63)
at org.neo4j.graphdb.factory.GraphDatabaseFactory$1.newDatabase(GraphDatabaseFactory.java:92)
at org.neo4j.graphdb.factory.GraphDatabaseBuilder.newGraphDatabase(GraphDatabaseBuilder.java:198)
at org.neo4j.kernel.impl.recovery.StoreRecoverer.recover(StoreRecoverer.java:115)
at org.neo4j.server.preflight.PerformRecoveryIfNecessary.run(PerformRecoveryIfNecessary.java:59)
at org.neo4j.server.preflight.PreFlightTasks.run(PreFlightTasks.java:70)
at org.neo4j.server.AbstractNeoServer.runPreflightTasks(AbstractNeoServer.java:333)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:152)
... 2 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.impl.transaction.TxManager#61615142' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:509)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:115)
at org.neo4j.kernel.InternalAbstractGraphDatabase.run(InternalAbstractGraphDatabase.java:307)
... 10 more
Caused by: org.neo4j.graphdb.TransactionFailureException: Unable to start TM
at org.neo4j.kernel.impl.transaction.TxManager.openLog(TxManager.java:824)
at org.neo4j.kernel.impl.transaction.TxManager.start(TxManager.java:198)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:503)
... 12 more
Caused by: java.io.IOException: Branch[����] found for [GlobalId[NENEOK|5930761323375553953|5569093129887285248|13248], BranchId[ ]] but no record list found in map
at org.neo4j.kernel.impl.transaction.TxLog.readBranchAddRecordInto(TxLog.java:520)
at org.neo4j.kernel.impl.transaction.TxLog.getDanglingRecords(TxLog.java:440)
at org.neo4j.kernel.impl.transaction.TxLog.recreateActiveTransactionState(TxLog.java:133)
at org.neo4j.kernel.impl.transaction.TxLog.<init>(TxLog.java:128)
at org.neo4j.kernel.impl.transaction.TxManager.openLog(TxManager.java:796)
... 14 more
Sep 26, 2014 3:33:03 PM org.neo4j.server.logging.Logger log
SEVERE: Failed to start Neo Server on port [7474]
I am using an openjdk 1.7 runtime environment which i think i should upgrade but i don't understand what is the cause of the error since everything was working fine before. Thanks for any help!

You should never ever manually modify anything inside the graph.db directory unless you're 100% sure what you're doing.
To prevent the datastore directory from continuously growing check your setting for keep_logical_logs in neo4j.properties, see http://docs.neo4j.org/chunked/stable/configuration-logical-logs.html.
Unless you require logical logs for online backup or cluster synchronization, you might be save deleting nioneo_logical.log.v* files. Make sure to have a backup first!
Also check your settings for open file limits, http://docs.neo4j.org/chunked/stable/linux-performance-guide.html#_setting_the_number_of_open_files.
There used to be a outdated version of Neo4j having a bug that might corrupt your datastore when running out of disc space. If that has happened to you either need to manually fix it on a binary level (which requires a lot of knowledge on Neo4j internals) or restore a previous backup from a time before you run out of disc.

neo4j cannot find JAVA_HOME when checking service status

I installed neo4j 1.8.2 on opensuse 12.2/64. To do so I had to add the JAVA_HOME path in the /etc/profile file as:
export JAVA_HOME=/opt/java/64/jdk1.7.0_21/jre/:
export PATH=$PATH:/opt/java/64/jdk1.7.0_21/jre/bin/;
Now when I try to check the server status I get the following error
>service neo4j-service status
neo4j-service.service - LSB: The Neo4J graph database server. See http://neo4j.org
Loaded: loaded (/etc/init.d/neo4j-service)
Active: failed (Result: exit-code) since Fri, 26 Apr 2013 17:13:56 +0200; 10s ago
Process: 7234 ExecStart=/etc/init.d/neo4j-service start (code=exited, status=1/FAILURE)
CGroup: name=systemd:/system/neo4j-service.service
Apr 26 17:13:56 linux-wwcz neo4j-service[7234]: which: no java in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/...bin)
Apr 26 17:13:56 linux-wwcz neo4j-service[7234]: Error: JAVA_HOME is not defined correctly.
Apr 26 17:13:56 linux-wwcz neo4j-service[7234]: We cannot execute
It's quite puzzling considering neo4j-service links to ./bin/neo4j, namely the file used at installation time with
./bin/neo4j install
Some ideas on what is going on here?
Thanks
SOLVED
Actually I was using jdk 7 instead of jdk 6
EDIT 2
According to the official neo4j page one runs the server using neo4j start. But I got into troubles when trying to run service neo4j start/status/stop as suggested in the Installing Neo4j in Linux how-to.

Try changing them to this:
export JAVA_HOME=/opt/java/64/jdk1.7.0_21/:
export PATH=$PATH:/opt/java/64/jdk1.7.0_21/bin/;

Not exactly for the question, but I reached here because I also though Neo4j could not find the environment variable JAVA_HOME when
sudo neo4j start
My problem was that OS reset environment variables when using sudo.
sudo -E neo4j start
solved my problem.